posts | Debugging crashes on debian - divide and conquer →

I finally got the hang of vagrant base box creation. I used to download images from ppl that I trusted, but then I was always caught with having to either wait for new releases to get captured into a box or just not using a certain OS at all in vagrant.

But it's not complex at all to create your own, as the vagrant site documents:

https://www.vagrantup.com/docs/boxes/base.html

This documentation page got me started. So here are lists of commands and instructions for how I create boxes (unfortunately I couldn't get instructions for CentOS finished since I'm still experiencing a bug about ip not showing up the network interface while starting up intances once the box is imported).

Creating a base box

The idea here is really simple. You need to manually create a VM and install the OS of your choice in it. Make sure it's using DHCP, has OpenSSH installed, the configuration manager of your choice, that the insecure vagrant public key lets you login to the "vagrant" user inside the VM and finally that you can sudo to root from the "vagrant" user without a password.

We also include perform some other tricks and install other stuff that might be interesting for our use cases. My base boxes are used for testing puppet modules so I want them to be as untouched as possible, but you can install whatever you need to make them smell exactly like your production setups.

Debian box (currently jessie)

Start by downloading a netinst iso from the debian web site. If you want to have a box using the testing branch of packages you'll need to install a stable release first and then upgrade to testing right before cleaning up and packaging up into a box. The OS upgrade is out of the scope of this document.

In virt-manager, I usually create a VM with 512Mb of RAM and 20Gb of disk (with qcow2 since I'm using vagrant-libvirt). Then I just follow instructions from the installer. In tasksel, uncheck all the options except for "SSH server" and "Common system utils". Place all files in one partition (not crypted otherwise it's impractical to update packages in the base box by merging a snapshot). Set the root password to "vagrant", then choose "vagrant" as a user name and set its password to "vagrant".

On the host:

# You'll have to know which IP the VM configured once booted up after install
ssh-copy-id -o UserKnownHostsFile=/dev/null -i ~/.vagrant.d/insecure_private_key.pub vagrant@192.168.122.56

Inside the VM:

su -
sed -i 's/^\(GRUB_TIMEOUT\)=.*$/\1=1/' /etc/default/grub
update-grub2
echo UseDNS no >> /etc/ssh/sshd_config
apt install -y sudo
echo "vagrant ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/vagrant
logout; sudo -i  # test that sudo is working OK and without password

Here you can optionally upgrade the OS if you want to use testing or sid instead.

Still in the VM:

apt install -y puppet rsync
systemctl disable puppet.service
# Now you can install whatever else that you need. I usually install vim-nox here.
apt-get clean
dd if=/dev/zero of=/EMPTY #reclaim emtpy space. this operation needs 20Gb on host
rm /EMPTY
history -c; history -w
logout # go back to the vagrant user
history -c; history -w
sudo -i
shutdown -h now

The main part of the work is done. Now follow instructions in the section below about packaging and importing the base box.

FreeBSD box

This procedure was put together using FreeBSD 11.

Notice: I use the ports system to install software which is insanely long and needs constant attention since some software needs you to choose compilation options. There is probably a better way, but I'm now steering away from FreeBSD packages because of their nasty choices in default compilation options.

Start by downloading an image that ends with -bootonly.iso. In the installer, choose the keyboard layout of your preference. Choose guided ZFS partitioning (or if you don't want ZFS, you can choose guided normal). Don't set any crypto since this'll make upgrading software in the base box by merging a snapshot impractical later. Set network to DHCP and type in a hostname that'll be somwhat valid (e.g. a real FQDN hostname even if the hostname won't resolve). Don't activate IPv6 (that choice might be reviewed in the future.. depending on whether the local network on laptop is IPv6). Don't activate any hardening features. Choose sshd in the list of software to install to the system. Set the root password to "vagrant". Choose to create a user and name it "vagrant" with a password of "vagrant".

On the host:

# You'll have to know which IP the VM configured once booted up after install
ssh-copy-id -o UserKnownHostsFile=/dev/null -i ~/.vagrant.d/insecure_private_key.pub vagrant@192.168.122.56

Inside the VM:

su -
echo 'autoboot_delay="1"' >> /boot/loader.conf
echo UseDNS no >> /etc/ssh/sshd_config
cd /usr/ports
# Installing bash is optional and might be avoided to have a system that's more
# "pure" or "vanilla". But I personally hate csh
(cd shells/bash; make install clean)
# Following line needed for bash
echo "fdesc /dev/fd fdescfs rw 0 0" >> /etc/fstab; mount /dev/fd
chsh -s bash; chsh -s bash vagrant
(cd security/sudo; make install clean)
echo "vagrant ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/vagrant
logout; sudo -i  # test that sudo is working OK and without password
# This is SO annoying. why is that enabled by default?
sed -i -e '/freebsd-tips/d' ~vagrant/.profile
(cd net/rsync; make install clean)
# Here you can choose other available versions of puppet. currently 3.7, 3.8 or 4
(cd sysutils/puppet38; make install clean)
# Now you can install whatever you want. I usually do: (cd editors/vim-lite; make install clean)
history -c; history -w
shutdown -p now

note: since we're using ZFS, cleaning up disk space by writing a huge file and deleting it doesn't work since data compression is enabled by default.

You're done with installing your box. Now follow the instructions in the section below about Packaging and importing the base box.

Packaging and importing the base box

Once the VM is installed we need to package it into a box and then import that as a base box.

These instructions are made for my setup that uses vagrant-libvirt. You'll have to find out how to perform disk space reclaiming and box export with virtualbox or other providers. I believe these instructions are very easy to find at least for virtualbox.

Obviously the image name and path where you store the final copy of the image must be changed to fit your current setup.

On the host:

sudo -i
cd /var/lib/libvirt/images
qemu-img convert -O qcow2 stretch.qcow2 ~myusername/dev/vm/stretch.qcow2
chown myusername:mygroup ~myusername/dev/vm/stretch.qcow2
logout
cd ~/dev/vm
~/.vagrant.d/gems/gems/vagrant-libvirt-*/tools/create_box.sh stretch.qcow2
vagrant box add stretch.box --name stretch

Now you can create a vagrant project with the new base box. Test that it works correctly, and then you can remove the qcow2 image in your home dir. You can also remove the base box, or you can store it for future uses or even publish it so that others can use it!

You can also remove the VM that was manually created.

Upgrade packages inside of the box

From time to time it's useful to upgrade packages/software inside of the base boxes to avoid downloading too much during your tests or even hitting package not found if the version you're requesting doesn't exist anymore.

Again, these instructions are meant for vagrant-libvirt users. However, if I remember correctly it's even easier to do with virtualbox and snapshots.

I've scripted the following procedure since it's very mechanical and has no real variability. Checkout box update.sh.

To perform upgrades, make sure you're using a vagrant project that:

doesn't install stuff inside the VM with a configuration manager
doesn't use any additional network interface
doesn't have any files in the vagrant project other than the Vagrantfile

Backup

Yes, you are possibly going to break the base box. So it's a great idea to start by taking a backup of the base box disk image before starting:

sudo cp /var/lib/libvirt/images/jessie_vagrant_box_image_0.img .

Warning: Don't place the image backup inside the directory of the vagrant project you're using for running the upgrades. This directory gets rsync'ed to the VM when starting up.

In case of a total meltdown of the base box, run "vagrant destroy" on all VMs that use this base box, then squash the file inside /var/lib/libvirt/images/ with the backup you've taken.

Perform the upgrade

Those instructions are fit for Debian, but the upgrade commands run inside the VM can easily be changed to perform upgrades on any system.

On the host:

vagrant up
vagrant ssh

Inside the VM:

sudo sh -c "apt update && apt -y upgrade && apt -y dist-upgrade && apt-get clean"
sudo bash -c "history -c; history -w"; history -c; history -w

On the host:

cat ~/.vagrant.d/insecure_private_key.pub | vagrant ssh -c "cat > ~/.ssh/authorized_keys"
vagrant halt
# The image file name must be the snapshot that corresponds to the vagrant
# instance you've spun up. This should be shown in the output of vagrant when
# running vagrant up at the beginning of this procedure.
sudo qemu-img commit /var/lib/libvirt/images/jessiepuppet_jessiepuppet.img
vagrant destroy

Done! now you can start any VM using that base box and the upgrades should be available to the new instances.