pages tagged vagrantlelutin.cahttps://lelutin.ca//tags/vagrant/lelutin.caikiwiki2018-01-02T23:41:19ZCreating and maintaining vagrant base boxeshttps://lelutin.ca//posts/Creating_and_maintaining_vagrant_base_boxes/2017-12-14T21:51:16Z2016-12-13T08:09:19Z
<p>I finally got the hang of vagrant base box creation. I used to download images
from ppl that I trusted, but then I was always caught with having to either
wait for new releases to get captured into a box or just not using a certain OS
at all in vagrant.</p>
<p>But it's not complex at all to create your own, as the vagrant site documents:</p>
<p><a href="https://www.vagrantup.com/docs/boxes/base.html">https://www.vagrantup.com/docs/boxes/base.html</a></p>
<p>This documentation page got me started. So here are lists of commands and
instructions for how I create boxes (unfortunately I couldn't get instructions
for CentOS finished since I'm still experiencing a bug about <code>ip</code> not showing
up the network interface while starting up intances once the box is imported).</p>
<h1>Creating a base box</h1>
<p>The idea here is really simple. You need to manually create a VM and install
the OS of your choice in it. Make sure it's using DHCP, has OpenSSH installed,
the configuration manager of your choice, that the insecure vagrant public key
lets you login to the "vagrant" user inside the VM and finally that you can
sudo to root from the "vagrant" user without a password.</p>
<p>We also include perform some other tricks and install other stuff that might be
interesting for our use cases. My base boxes are used for testing puppet
modules so I want them to be as untouched as possible, but you can install
whatever you need to make them smell exactly like your production setups.</p>
<h2>Debian box (currently jessie)</h2>
<p>Start by downloading a <code>netinst</code> iso from the debian web site. If you want to
have a box using the <code>testing</code> branch of packages you'll need to install a
stable release first and then upgrade to testing right before cleaning up and
packaging up into a box. The OS upgrade is out of the scope of this document.</p>
<p>In virt-manager, I usually create a VM with 512Mb of RAM and 20Gb of disk (with
qcow2 since I'm using vagrant-libvirt). Then I just follow instructions from
the installer. In tasksel, uncheck all the options except for "SSH server" and
"Common system utils". Place all files in one partition (not crypted otherwise
it's impractical to update packages in the base box by merging a snapshot). Set
the root password to "vagrant", then choose "vagrant" as a user name and set
its password to "vagrant".</p>
<p>On the host:</p>
<pre><code># You'll have to know which IP the VM configured once booted up after install
ssh-copy-id -o UserKnownHostsFile=/dev/null -i ~/.vagrant.d/insecure_private_key.pub vagrant@192.168.122.56
</code></pre>
<p>Inside the VM:</p>
<pre><code>su -
sed -i 's/^\(GRUB_TIMEOUT\)=.*$/\1=1/' /etc/default/grub
update-grub2
echo UseDNS no >> /etc/ssh/sshd_config
apt install -y sudo
echo "vagrant ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/vagrant
logout; sudo -i # test that sudo is working OK and without password
</code></pre>
<p>Here you can optionally upgrade the OS if you want to use testing or sid
instead.</p>
<p>Still in the VM:</p>
<pre><code>apt install -y puppet rsync
systemctl disable puppet.service
# Now you can install whatever else that you need. I usually install vim-nox here.
apt-get clean
dd if=/dev/zero of=/EMPTY #reclaim emtpy space. this operation needs 20Gb on host
rm /EMPTY
history -c; history -w
logout # go back to the vagrant user
history -c; history -w
sudo -i
shutdown -h now
</code></pre>
<p>The main part of the work is done. Now follow instructions in the section below
about packaging and importing the base box.</p>
<h2>FreeBSD box</h2>
<p>This procedure was put together using FreeBSD 11.</p>
<p>Notice: I use the ports system to install software which is insanely long and
needs constant attention since some software needs you to choose compilation
options. There is probably a better way, but I'm now steering away from FreeBSD
packages because of their nasty choices in default compilation options.</p>
<p>Start by downloading an image that ends with <code>-bootonly.iso</code>. In the installer,
choose the keyboard layout of your preference. Choose guided ZFS partitioning
(or if you don't want ZFS, you can choose guided normal). Don't set any crypto
since this'll make upgrading software in the base box by merging a snapshot
impractical later. Set network to DHCP and type in a hostname that'll be
somwhat valid (e.g. a real FQDN hostname even if the hostname won't resolve).
Don't activate IPv6 (that choice might be reviewed in the future.. depending on
whether the local network on laptop is IPv6). Don't activate any hardening
features. Choose sshd in the list of software to install to the system. Set the
root password to "vagrant". Choose to create a user and name it "vagrant" with
a password of "vagrant".</p>
<p>On the host:</p>
<pre><code># You'll have to know which IP the VM configured once booted up after install
ssh-copy-id -o UserKnownHostsFile=/dev/null -i ~/.vagrant.d/insecure_private_key.pub vagrant@192.168.122.56
</code></pre>
<p>Inside the VM:</p>
<pre><code>su -
echo 'autoboot_delay="1"' >> /boot/loader.conf
echo UseDNS no >> /etc/ssh/sshd_config
cd /usr/ports
# Installing bash is optional and might be avoided to have a system that's more
# "pure" or "vanilla". But I personally hate csh
(cd shells/bash; make install clean)
# Following line needed for bash
echo "fdesc /dev/fd fdescfs rw 0 0" >> /etc/fstab; mount /dev/fd
chsh -s bash; chsh -s bash vagrant
(cd security/sudo; make install clean)
echo "vagrant ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/vagrant
logout; sudo -i # test that sudo is working OK and without password
# This is SO annoying. why is that enabled by default?
sed -i -e '/freebsd-tips/d' ~vagrant/.profile
(cd net/rsync; make install clean)
# Here you can choose other available versions of puppet. currently 3.7, 3.8 or 4
(cd sysutils/puppet38; make install clean)
# Now you can install whatever you want. I usually do: (cd editors/vim-lite; make install clean)
history -c; history -w
shutdown -p now
</code></pre>
<p>note: since we're using ZFS, cleaning up disk space by writing a huge file and
deleting it doesn't work since data compression is enabled by default.</p>
<p>You're done with installing your box. Now follow the instructions in the
section below about Packaging and importing the base box.</p>
<h2>Packaging and importing the base box</h2>
<p>Once the VM is installed we need to package it into a box and then import that
as a base box.</p>
<p>These instructions are made for my setup that uses vagrant-libvirt. You'll have
to find out how to perform disk space reclaiming and box export with virtualbox
or other providers. I believe these instructions are very easy to find at least
for virtualbox.</p>
<p>Obviously the image name and path where you store the final copy of the image
must be changed to fit your current setup.</p>
<p>On the host:</p>
<pre><code>sudo -i
cd /var/lib/libvirt/images
qemu-img convert -O qcow2 stretch.qcow2 ~myusername/dev/vm/stretch.qcow2
chown myusername:mygroup ~myusername/dev/vm/stretch.qcow2
logout
cd ~/dev/vm
~/.vagrant.d/gems/gems/vagrant-libvirt-*/tools/create_box.sh stretch.qcow2
vagrant box add stretch.box --name stretch
</code></pre>
<p>Now you can create a vagrant project with the new base box. Test that it works
correctly, and then you can remove the qcow2 image in your home dir. You can
also remove the base box, or you can store it for future uses or even publish
it so that others can use it!</p>
<p>You can also remove the VM that was manually created.</p>
<h1>Upgrade packages inside of the box</h1>
<p>From time to time it's useful to upgrade packages/software inside of the base
boxes to avoid downloading too much during your tests or even hitting package
not found if the version you're requesting doesn't exist anymore.</p>
<p>Again, these instructions are meant for vagrant-libvirt users. However, if I
remember correctly it's even easier to do with virtualbox and snapshots.</p>
<p>I've scripted the following procedure since it's very mechanical and has no
real variability. Checkout <a href="https://lelutin.ca//posts/files/box_update.sh">box update.sh</a>.</p>
<p>To perform upgrades, make sure you're using a vagrant project that:</p>
<ul>
<li>doesn't install stuff inside the VM with a configuration manager</li>
<li>doesn't use any additional network interface</li>
<li>doesn't have any files in the vagrant project other than the Vagrantfile</li>
</ul>
<h2>Backup</h2>
<p>Yes, you are possibly going to break the base box. So it's a great idea to
start by taking a backup of the base box disk image before starting:</p>
<pre><code>sudo cp /var/lib/libvirt/images/jessie_vagrant_box_image_0.img .
</code></pre>
<p>Warning: Don't place the image backup inside the directory of the vagrant
project you're using for running the upgrades. This directory gets rsync'ed to
the VM when starting up.</p>
<p>In case of a total meltdown of the base box, run "vagrant destroy" on all VMs
that use this base box, then squash the file inside <code>/var/lib/libvirt/images/</code>
with the backup you've taken.</p>
<h2>Perform the upgrade</h2>
<p>Those instructions are fit for Debian, but the upgrade commands run inside the
VM can easily be changed to perform upgrades on any system.</p>
<p>On the host:</p>
<pre><code>vagrant up
vagrant ssh
</code></pre>
<p>Inside the VM:</p>
<pre><code>sudo sh -c "apt update && apt -y upgrade && apt -y dist-upgrade && apt-get clean"
sudo bash -c "history -c; history -w"; history -c; history -w
</code></pre>
<p>On the host:</p>
<pre><code>cat ~/.vagrant.d/insecure_private_key.pub | vagrant ssh -c "cat > ~/.ssh/authorized_keys"
vagrant halt
# The image file name must be the snapshot that corresponds to the vagrant
# instance you've spun up. This should be shown in the output of vagrant when
# running vagrant up at the beginning of this procedure.
sudo qemu-img commit /var/lib/libvirt/images/jessiepuppet_jessiepuppet.img
vagrant destroy
</code></pre>
<p>Done! now you can start any VM using that base box and the upgrades should be
available to the new instances.</p>
Vagrant puppeteer searches new virtual machinehttps://lelutin.ca//posts/Vagrant_puppeteer_searches_new_virtual_machine/2018-01-02T23:41:19Z2013-09-15T03:18:00Z
<p>Well... enough with the sedentary life, I'm going Vagrant.</p>
<p>.. oh and by Vagrant I mean <a href="http://vagrantup.com/" title="Vagrant">this project</a> <img src="https://lelutin.ca//smileys/smile4.png" alt=";)" /></p>
<p>I've heard a lot of people talk about this project lately, and I read the feature list and found pretty it interesting. A very quick summary of the features would be that it can automatically create customized VirtualBox VMs and provision them with Chef/Puppet.</p>
<p>Today I'm leaving my home behind.</p>
<p>The cool thing about Vagrant is that you can create a full dev environment, with every tool/library required for development already installed and ready to use, then package this up (and the VM's configuration) into a single file that you can use as a template. So, you can have one single file that you distribute to your dev team and everyone can then use the same environment to build, abuse and break ... but that environment is on their own computer! (with the hope that the developers can minimally manage the VM) That means that a heavy SQL request won't be killing the single development server, and also that a devloper can bring a dev server along on the bus, plane or train to continue work while offline.</p>
<p>It also means you can build a VM that's all ready for a live presentation in a conference and remove the really annoying dependency on Internet (which, we all know <em>never</em> works as expected at conferences).</p>
<p>I'm using Ubuntu 10.10 (Maverick) as the host OS. So all commands found below will deal with that OS. I'll be building a cardboard box for my puppet manifests to live in.</p>
<h1>Getting started</h1>
<p>So, where do we start? Got your cardboard, your scissors and some glue? ... OK, now for the basics: the Vagrant web site has a <a href="http://vagrantup.com/docs/getting-started/index.html">pretty good starter guide</a> that I suggest you read diagonally before installing the thing. This way you'll be more familiar about Vagrant lingo. You can refer to the guide later to really try the suggested commands.</p>
<p>The project's web site claims that you simply blink your eyes twice and Vagrant is installed on your computer. However, I'm using Ubuntu Maverick here, and things didn't go as smoothly for me as "gem install"-and-we're-ready! I found out this <a href="http://www.theodo.fr/blog/2011/03/start-using-vagrant/">VirtualBox + Vagrant install howto for Maverick</a> that filled in [almost] all the gaps.</p>
<p>Since the writing of the above howto, Vagrant and VirtualBox have progressed, and so have the requirements. So you must change the VirtualBox package name from <em>virtualbox-4.0</em> to <em>virtualbox-4.1</em>.</p>
<p>I also added a line to the end of my <em>~/.bashrc</em> script so that the <em>vagrant</em> binary is always available without its full path:</p>
<pre><code>PATH=$PATH:/var/lib/gems/1.8/bin
</code></pre>
<p>A step that is missing from the howto is to add yourself to the <em>vboxusers</em> group. This will let you configure things like usb devices and get you rid of pesky warning messages in VirtualBox's Gtk interface:</p>
<pre><code>adduser your-username vboxusers
</code></pre>
<p>Then logout of your X session and login again so that this new group applies to your session.</p>
<p>Now, there's one final thing that is not mentioned in the howto. I'm using Vagrant as an unprivileged user on my machine, and upon the first invocation, I ran into this error:</p>
<pre><code>/var/lib/gems/1.8/gems/net-ssh-2.1.4/lib/net/ssh/key_factory.rb:38:in `read': Permission denied - /var/lib/gems/1.8/gems/vagrant-0.8.8/keys/vagrant (Errno::EACCES)
[cut... huge ruby backtrace here]
</code></pre>
<p>To fix the above issue, I simply made the file readable to the <em>vboxusers</em> group like so:</p>
<pre><code>sudo chgrp vboxusers /var/lib/gems/1.8/gems/vagrant-0.8.8/keys/vagrant
sudo chmod g+r /var/lib/gems/1.8/gems/vagrant-0.8.8/keys/vagrant
</code></pre>
<h1>Building virtual machine templates</h1>
<p>Once you have Vagrant all setup on your computer, you can <a href="http://www.vagrantbox.es/">go and find base boxes online</a> (e.g. virtual machine templates) that you can download.</p>
<p>However, I wanted to build my own VM templates to customize them to my own liking/needs, so I continued searching info about how to build base boxes. There's some very interesting info on <a href="http://chrisadams.me.uk/2010/05/10/setting-up-a-centos-base-box-for-development-and-testing-with-vagrant/">this site</a> on how to build a base box "manually" for CentOS. I didn't use it directly, but I'm thinking of using it as an example for building Windows XP VMs later.</p>
<p>Now the reason I didn't use the above link is that I found out there was a project that automated everything for a whole bunch of OSes! This project is called "veewee". It's another ruby gem that adds a new subcommand to Vagrant so that you can create base boxes in a really neat (fully automated and scripted -- possible to redo things without typos or forgotten elements) and painless way. Here's <a href="http://www.ducea.com/2011/08/15/building-vagrant-boxes-with-veewee/">an install + usage howto for veewee</a>.</p>
<h2>Creating a puppet client base box</h2>
<pre><code>mkdir vagrant-squeeze && cd vagran-squeeze
vagrant basebox define debian-squeeze 'Debian-6.0.3-amd64-netboot'
vim definitions/debian-squeeze/definition.rb # optional, Set number of CPUs and quantity of RAM and disk here. Note: I also changed :hostiocache to 'on' here since I'm using ext4 on my host computer
vim definitions/debian-squeeze/preseed.cfg # optional, Change language settings, keyboard map and .deb repository mirror hostname here. Some d-i knowledge helps, but you definitely don't need to be an expert to simply change the settings.
vagrant basebox build debian-squeeze
vagrant basebox validate debian-squeeze
vagrant basebox export debian-squeeze
vagrant box add debian-squeeze debian-squeeze.box
vagrant basebox destroy debian-squeeze # Cleanup when we're done
</code></pre>
<p>Ok, you followed me up to this point? We now have a debian-squeeze.box file that contains a basic Debian Squeeze template that we imported as <em>debian-squeeze</em>. Let's build on that to add the Puppet client:</p>
<pre><code>mkdir puppet-squeeze && cd puppet-squeeze
vagrant init debian-squeeze
mkdir manifests
cat > manifests/debian-squeeze.pp <<EOF
package { 'puppet': ensure => installed }
exec { "/bin/hostname puppet-squeeze": }
file { "/etc/hostname":
ensure => present,
content => "puppet-squeeze",
}
exec { "/bin/sed -i 's/debian-squeeze/puppet-squeeze/g' /etc/hosts": }
EOF
sed -i 's/^end$/ config.vm.provision :puppet\nend/' Vagrantfile # This enables puppet provisioning of the VM
vagrant up
</code></pre>
<p>You should now have a VM up and running with puppet installed on it. You can now use <em>vagrant ssh</em> to ssh into it and start playing around with puppet!</p>
<p>To make things easier to re-create later, I strongly recommend putting all of your customizations in the <em>debian-squeeze.pp</em> manifest file (or in other <em>.pp</em> files that you import in <em>debian-squeeze.pp</em>) instead of manually via ssh. You probably want to keep this VM as simple as possible, though, since it's meant as a template. All project-specific customizations can (and probably should) be done later when provisioning a VM with 'puppet-squeeze' as a template.</p>
<p>The last step is to package this up into a box and to import it as a base box:</p>
<pre><code>vagrant package # should gracefully shutdown the VM if it's still running
vagrant box add puppet-squeeze package.box
vagrant destroy
</code></pre>
<p>The resulting <em>package.box</em> is around 300Mb on my laptop. It may be possible to shrink this down a little bit by zeroing out the virtual disk and by asking VirtualBox to compact the disk image file. For more info on how to do this, check out the comments on the Maverick vagrant howto on www.theodo.fr that's linked at the beginning of this post.</p>
<p>As an optional cleanup step, you could get rid of the first template with <em>vagrant box remove debian-squeeze</em>, but depending on what you'll do with Vagrant, you might want to skip this step and just keep the thing laying around. Also, if you ever want to rebuild the puppet-squeeze base box, you'll need the debian-squeeze template.</p>
<p>You're now all set! You can create a new vagrant project with <em>vagrant init puppet-squeeze</em> and start manipulating puppet.</p>