Virtual Linux with NFS-Root under QEMU

Wed, 20 Jun 2012 15:40:25 GMT

Probably, there will be a bright future for Linux Containers. However, so far, I miss a network configuration which is comparably easy to the default networks of Qemu and VirtualBox. I do not know whether the situation is better in OpenVZ, but I somehow do not like OpenVZ much. The main advantage of OpenVZ and Lxc for me is that they use the host filesystem as root filesystem. For incremental backups with rsync, this is vital, but it also makes quick changes to the configuration a lot easier.

So the idea was to actually use Qemu with a small image that boots into an nfsroot. I will try to give a little howto on how to do this with Xubuntu. However, I cannot guarantee that this will work for everyone else. Especially, the command line arguments are constantly changing, so in a few months, this might not work anymore at all. So better take it as a starting point for your own research rather than the final truth.

The packages needed (I hope I did not forget one) are nfs-kernel-server, qemu, qemu-kvm, kvm, debootstrap.

To build an initramfs, I first create a chroot, which I built with debootstrap. I want Debian Wheezy.

# mkdir /qemuChroot
# debootstrap wheezy /qemuChroot http://ftp.de.debian.org/debian/

This does a lot of stuff and takes a while. While it clatters, we can already begin to do our NFS configuration. We need to put the following lines into the /etc/exports file:

/qemuChroot 127.0.0.1(rw,sync,no_subtree_check,no_root_squash,insecure)

"insecure" is necessary, since we want to run the Qemu usermode network stack. Then we have to restart the nfs-kernel-server, to get the new configuration (or use exportfs -a, but I prefer restarting when possible).

Now, hopefully debootstrap ended correctly. Time to chroot into the system:

# chroot /qemuChroot/

In this chroot environment, install a linux-image, and initramfs-tools, and change BOOT=local to BOOT=nfs in your /etc/initramfs-tools/initramfs.conf and run

# update-initramfs -uk all

inside your chroot (please, check twice that you are in the chroot!!!). It will drop a few warnings (we could prevent them by mounting /proc and /sys and /dev) but should work.
To be able to access the machine later, we need to set a root password, using

# passwd

Exit the chroot again. The initramfs and kernel image we need is now in /qemuChroot/boot. For the kernel I use, they are called initrd.img-3.2.0-2-amd64 and vmlinuz-3.2.0-2-amd64.

We now have everything to start Qemu. Firstly, make sure that the kvm-modules are loaded, and that your user has apropriate access to these devices - for me, he needs to belong to the libvirtd-group. To use kvm, we add the argument -enable-kvm to our Qemu-commandline - leave it out if you do not want kvm acceleration, but be aware that this will be slow.
Qemu can directly boot into a linux kernel and initramfs, without the need of an extra image. Just add -initrd /qemuChroot/boot/initrd.img-3.2.0-2-amd64 -kernel /qemuChroot/boot/vmlinuz-3.2.0-2-amd64, or the appropriate paths for you, to the commandline.
We need to inform the kernel of our nfsroot. This is done by kernel arguments, which can be passed by -append "root=/dev/nfs nfsroot=10.0.2.2:/qemuChroot".
Finally, we want virtio-net (to make it faster).

The commandline to start the VM becomes

$ qemu-system-x86_64 -enable-kvm -initrd /qemuChroot/boot/initrd.img-3.2.0-2-amd64 -kernel /qemuChroot/boot/vmlinuz-3.2.0-2-amd64 -append "root=/dev/nfs nfsroot=10.0.2.2:/qemuChroot" -netdev type=user,id=usernet -device netdev=usernet,driver=virtio-net

This should correctly boot you into a virtual machine with the given root. However, for some reason resolvconf does not work properly, so make sure it is not installed on the virtual machine, and change the contents of /etc/resolv.conf into

nameserver 10.0.2.3

It is important to use the internal nameserver, udp packages are not transmitted for some reason.

You may want to export some services from your virtual machine to your host. For me, these are ssh and http. The -netdev argument must be changed into type=user,id=usernet,hostfwd=tcp::2200-:22,hostfwd=tcp::8800-:80 and I leave it to the reader to find out how other ports are forwarded.

Here is a documentation on Qemu-Networking, and here is some information about diskless booting with an Arch Linux rather than Debian. Both sites were very helpful for creating this howto.