HOWTO: Nested KVM based virtual setup with oVirt

I have been working on the oVirt project for almost 3 years now, but I rarely get to experience it from the user’s point of view. Sure there are test days, bug fixes, and other activities that require me to fiddle with oVirt, but I haven’t really experienced it as a user trying to set up his env. Until now…

The good

For those of you who don’t know, oVirt is a rising star in the virtualization world, allowing users to create their own virtual setups with the powerful KVM behind the scenes. Check it out @ – it’s 100% free & open source!

The bad

These last few days I have been trying to install OpenStack using devstack as part of my job – what a nightmare!
First of all, I had to set up a host for the OpenStack environment. “That’s easy”, I thought to myself, “I’ll just grab this mini DELL that’s lying under my feet and install a new OS on it”.
I went to the devstack site and they recommended Ubuntu 12.04 (Precise Pangolin) which is the LTS (Long Term Support) edition – makes sense, you better go with something stable right?

It’s not that i hate Ubuntu, Until now…

I went to the Ubuntu site and saw there’s a few editions, one supposedly including OpenStack – “That’s too easy”, I thought, “I want to control the devstack installation myself and tweak it if necessary”. The server edition seemed the logical choice, but I decided to alsdo grab the desktop edition just in case.. BTW I don’t want to be asked to donate each time I want to download Ubuntu, it’s not that I’m cheap but it’s just annoying to be pushed towards it each time..

Well, let me tell you, Ubuntu 12.04 is utterly uninstallable. You might think it was just my imagination, but each of the images had some corrupt file which halted the installation dead in it’s tracks. I thought maybe it’s because I’m trying to use a USB stick, so I even burned the ISO to a real CD (so 90’s…) but the problem existed even there.. “Right, I must’ve gotten a corrupt download”.. I ran md5sum and compared it with the one they hid on their site.. Completely identical! ARGH!!

As a rather stubborn guy I had to make it work, but how? I found a “minimal ISO” of Ubuntu 12.04, this time no corrupt files as the installation downloads everything – excellent, except that it takes ages! By the time the installation had finished and asked me to reboot, I wanted to go home, so I turned off the computer and said to myself “Eh, I’ll just continue tomorrow”. Lo and behold, the next day the OS booted into a black screen.. Frustration ensued.

At this stage I decided Ubuntu was not my friend in this journey.. Having been scolded by Fedora 19 installation from Live USB, I turned to my last resort: RHEL 6.4.

Virtualization for the win

I decided that I won’t be installing on bare metal anymore. Of course a VM would be a logical choice, one I even considered in the beginning, but because of the need to run VMs on OpenStack I was hesitant at the beginning (before this fiasco). Having spent a couple days work trying to install Ubuntu, while climbing under my desk a lot (It’s rather clean there, but still inconvenient), I decided to give nested virtualization a try.

Of course, you can get a very quick and simple environment running using virt-manager or boxes, but I wanted to:

  • Be able to live migrate the VMs between hosts easily.
  • Be able to take live snapshots of the VMs.
  • Have templates with thin provisioning of disks.
  • Have a centralized place to manage it, that is accessible via web for:
  • When I’m working from home.
  • Someone else from the team wants a VM there.

All the above reasons led me to pick oVirt which I know well, and suits all my needs (and much more).

The good again

Well, turns out it’s rather easy to set up an oVirt env as a user, you’ll probably spend most of the time installing stuff and the administration itself is a breeze. There are some pitfalls to be aware of, but I’ll try to write them all so you know what to avoid.

Setting up oVirt

OK, so for starters I wanted my oVirt host to be virtualized by itself, to be able to back it up and if necessary move to another host. I took my RHEL 6.4 host which also runs Jenkins (probably should move this to a VM also). This one doesn’t have to run nested since we’re not planning to run any VMs on it. For this I used virt-manager + qemu-kvm + libvirt (In the near future, oVirt will have a feature which allows to run itself inside a VM managed by itself, but until then I’ll use plain virt..).
Just install the three on a host that you want to run oVirt on:

sudo yum install -y virt-manager qemu-kvm libvirt \
&& service libvirtd start \
&& chkconfig libvirtd on

If your host doesn’t forward X11 when you ‘ssh -X’ to it, you need to install xauth on it:

sudo yum install xauth

Now start virt-manager and create a new VM, I chose:

  • Disk size 20 GB (this can be resized later anyway)
  • 3 GB RAM (oVirt requires at least 2, but setting exactly 2 will give you trouble)
  • MAC address that probably won’t collide with other MACs in the office network (since we launch VMs they might get the same MAC so we need to make sure not to step on each other’s toes).
  • Fedora 19 image (I installed server packages only, no UI).

Right, install the host and then SSH to it. If you can, give it a DNS or DDNS entry or just remember the IP and make sure it sticks (make it static, or change the DHCP server definition to always give you this IP).

On the host you need to download and install oVirt – just follow the 3 simple steps at

Note: If the installation complains that you don’t have a reverse DNS record, you can add your oVirt VM’s IP and “hostname” that’s used in the installation to /etc/hosts – since you’ll be using a local user, you don’t need a reverse DNS record anyway.

Now that you have oVirt installed you can back it up just to be safe. You can copy the disk image and dump the vm’s XML:

virsh dumpxml [vm name] > /path/to/vm.xml

Later you can easily restore by copying back the disk image and restoring the VM from XML:

virsh define /path/to/vm.xml

However, any changes that were made to the data storage domains will remain there and would need to be cleaned manually.

Setup your basic oVirt environment

First of all, we must disable the MAC anti spoofing configuration value (otherwise the VM inside the VM won’t be able to send traffic since it’s MAC address is different):

engine-config -s EnableMACAntiSpoofingFilterRules=false --cver=3.2 \
&& engine-service restart

Also it’s a good idea to configure a MAC range that won’t interfere with other people (optional):

engine-config -s MacPoolRanges=66:1A:4A:23:11:00-66:1A:4A:23:11:FF \
&& engine-service restart

Next, log into oVirt and add a new data-center in the Data Centers tab. The Default DC is limited a bit (can’t be deleted, etc) so it doesn’t hurt to add another one.
As a bonus, you’ll get the “Guide Me” wizard which walk you through the different steps that you need to do so just follow the steps:

  • Add a cluster.
  • Add hosts to the cluster (Make sure to use F18 or higher).
  • Add a shared storage (NFS most likely).
  • Add a VM.

Now, to have ISO images available you’ll need to import or create an ISO domain (Unless you chose to have one created during installation, which I haven’t tried).
To create an ISO domain:

  • Go to the ‘Storage domains’ tab.
  • Click ‘Add’, the new storage dialog will open.
  • Choose your data center (An active host should be chosen automatically).
  • Choose ‘NFS/ISO’ type.
  • Enter the path to the storage domain, for example ‘’.
  • Click ‘OK’.

The domain will be created and attached to the data center, select it and go to the ‘Data Center’ sub-tab and click on ‘Activate’.
Once the domain is active you can upload ISO images to it. I actually didn’t use the ISO uploader tool, but rather just downloaded the images directly to the ‘images’ folder that was created (it will be in the path you provided, under a folder named after the UUID of the domain). Either way you get the images there is fine.

Enabling nesting on the hosts

Now, we have a working environment, and for regular oVirt usage it would be fine to use like this.
However, to enable nested virtualization, a bit (really just a bit) of manual work has to be done.

Note: If you need to reboot the host, better put it to Maintenance first just to be safe.

Note: You can also use a VM for this, same as you used a VM for the oVirt machine. The only adjustments needed are to enable nested virtualization on the host machine of the VM (explained later on), and have “vmx” required by the CPU of the VMnsel0.

SSH to each host and do the following:

  • To make sure you avoid nested virt bugs (esp. kernel bug that I stumbled upon) better update the host software:
  • yum update -y
  • Make sure nested virtualization is enabled in the KVM:
  • For Intel based hosts (not sure about AMD):
  • sudo yum install -y vdsm-hook-nestedvt

And now you’re basically good to go!

You can check if it works, create a new VM in oVirt and install it, and create another VM inside that VM (You can use virt-manager, boxes, or be adventerous and add the “host” VM as a host in oVirt ;)).

This entry was posted in Uncategorized and tagged , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s