Long story short: both my 2.5" Caviar blue disks failed some days apart (GAH!). So I had to send them both back to WD for an RMA. But since all my stuff in there was accessible as-is, and also since I didn't have any other device to wipe out the disks (actually one had failed beyond the point where I could wipe out the disk even if I wanted to), I had to consider every ssh key, puppet SSL cert, passwords, etc, as compromised (double-GAH!!). Who knows what happens to the data after your disk is refurbished... I don't trust WD to wipe them out before shipping them to new clients.

Luckily, I had a recent backup of the entire /etc/puppet directory from the puppet master, and I was able with this to rebuild all VMs. Friends don't let friends run computers without a proper, regularly updated backup!

So! I decided I would add an encryption layer to the new disks, and since I don't have a screen plugged into the mini-ITX, I needed a way to delay the system from asking the password for decrypting the disk.

The disk layout looks something like the following:

(/dev/sda1)   (/dev/sdb1)    (/dev/sda2)   (/dev/sdb2)
        \        /                   \        /
         \      /                     \      /
    (RAID1, /dev/md0)            (RAID1, /dev/md1)
            |                            |
           \|/                          \|/
  (ext4, mounted as / )    (LUKS encryption, mapped to /dev/dm-0 when decrypted)
                                         |
                                        \|/
                                (LVM physical disk)
                                         |
                                        \|/
                                  (LVM group "vm")
                                         |
                          --------------------------------- - - -
                          |                            |
           (ext4, /dev/mapper/vm--vm1-disk)   (swap, /dev/mapper/vm--vm1-swap)

I won't cover how to create the above layout on the disk, but if you'd like me to explain some parts of it, or to point you to documentation about creating some parts of it, don't be shy to ask it out in the comments.

I don't really care about encrypting the dom0 because I don't store anything in there. If I ever loose another disk, I'll revoke the dom0's SSL certificate for puppet and create and sign a new one for it. Annoying, yes, but less so than setting up a fully encrypted host.

The above layout garantees that all data in VMs get encrypted. Even their swap is.

Since the dom0 is not encrypted we can skip the painful steps of having to to customize the initrd to include a special SSH blend that lets you enter the encryption password, and just ask the system to not decrypt md1 at boot. To do this, we can simply modify /etc/crypttab to add a "noauto" option to the cryptdisk:

md1_crypt UUID=4a0dc9d8-b48f-4d6a-8a75-c4c3c6065493 none luks noauto

Now the system boots completely up to the prompt. No more "plugging-in-a-screen-and-a-keyboard" time lost, and you can now SSH in the machine. You'll see that Xen domUs fail to start since the LVs are not accessible.

To finish the boot sequence manually, and start your Xen domUs, you'll have to enter the following commands as root (change the UUID of the disk to the one that really corresponds to your disk):

# cryptsetup luksOpen /dev/disk/by-uuid/4a0dc9d8-b48f-4d6a-8a75-c4c3c6065493 md1_crypt
# vgscan
# vgchange -a y vm
# service xendomains restart

For a really quick explanation of the above commands, they're doing the following, in order:

  1. decrypting the disk and mapping the decrypted contents of the disk to the device /dev/dm-0 and creating an alias to it in /dev/mapper/md1_crypt
  2. scanning devices in /dev for LVM physical volumes and finding out volume groups in them. This'll make your LVs visible with lvs
  3. The LVs are set to "inactive" after the previous step, so you then activate all LVs in the "vm" group
  4. Once all this poutine is delt with, you can start your VMs. the /etc/init.d/xendomains script conveniently starts all VMs in the auto-start directory (by default, /etc/xen/auto)

Done. Your setup is fully booted up.