Mon, 19 Feb 2007
Virtualization Overhead
In a followup to my previous article
I want to sum up the speed penalty introduced by the two virtualization
systems, KVM and lguest. The measurements are by no means
meaningful for everyone, just exact data I have measured. And yes,
it sometimes compares apples-to-oranges - read on at your own risk :-)
The test systems were the following:
- Pyrrha
- Dual AMD Opteron 2220 (4 cores total, 2.8GHz), 6GB RAM, 4-way RAID-10 disk, gigabit LAN, Gentoo Linux/AMD64.
- Terminus
- KVM-based virtual AMD64 machine running at Pyrrha, 10 GB disk image, 512 MB RAM, bridged LAN to the main Pyrrha's LAN, Fedora 6/AMD64.
- Scylla
- Pentium 4 3.0GHz HT, 1 GB RAM, single SATA drive, 100baseTx LAN, Fedora Rawhide/IA32.
- Glauke
lguest-based virtual IA32 machine running at Scylla, 10 GB disk image, 512 MB RAM, routed/masqueraded LAN to the main Scylla's LAN, Fedora 6/IA32.
| Test | Pyrrha | Terminus | KVM overhead | Scylla | Glauke | lguest overhead |
|---|---|---|---|---|---|---|
| bc | 6.286 | 5.876 | -6.52 % | 8.130 | 8.240 | 1.35 % |
| wget | 0.441 | 10.885 | 2368.25 % | 3.732 | 3.770 | 1.02 % |
| tar unpack | 15.118 | 20.322 | 34.42 % | 27.566 | 40.701 | 47.65 % |
| rm -rf | 0.538 | 0.634 | 17.84 % | 0.477 | 0.640 | 34.17 % |
| compile | 6.410 | 21.929 | 242.11 % | 126.005 | 184.833 | 46.69 % |
The numbers in the above table are in seconds (so lower is better). I ran each test five times and used the lowest time from these five runs. I did not bother to reboot between the tests or switch the system daemons off.
Description of the tests
- bc
time sh -c 'echo 2^1000000|bc >/dev/null'
A simple test of a CPU-intensive job. Why is Terminus faster than Pyrrha? Maybe the clock skew inside the guest? Or Gentoo-compiledbcbeing slower than Fedora-prebuilt one?- wget
time wget -q -O /dev/null ftp://ftp.linux.cz/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2
Network performance. KVM (having to emulate the real-world NIC) is waaay slower. However, Pyrrha has a gigabit NIC, so the baseline is 10 times off. But still, raw bandwidth used for KVM was ~22 Mbit/s, whilelguesthas filled the Scylla's 100 Mbit pipe without trouble.lguestcould be even faster in the future, if they use a bounce buffer bigger than a single page (which is what they use now).- tar unpack and rm
time tar xjf /usr/src/linux-2.6.20.tar.bz2 ; time rm -rf linux-2.6.20.tar.bz2
A simple filesystem-related test. Nothing to see here (KVM is a bit faster).- compile
make clean; time make modules > /dev/null
A simple part of kernel compile. Both the architecture and kernel config was different between Pyrrha+Terminus and Scylla+Glauke, so do not try to compare the absolute times between those two groups. Interestingly enough, KVM was much slower thanlguest.
From a subjective point of view, lguest feels faster, which was
totally unexpected. I am looking forward to the further development
(especially lguest with AMD64 support). Anyway, James Morris did
a more in-depth
measurements of the lguest network performance.
3 replies for this story:
Adelton wrote: Paravirt being faster unexpectedly ...
I didn't do any exact measures but having worked with both fullvirt and paravirt Xen guests in the last few months, indeed, paravirt feels faster for similar workloads. My naive internal explanation is that paravirtualized kernel knows how to be nice to the hypervisor, while the vanilla kernel does not case and the hypervisor has to try harder. The nice thing about Xen is (besides it being in FC6) that you can easily switch from paravirtual to fully virtual guests -- just boot different kernel from the grub menu.
Adelton wrote: vmware
I don't really know whether you're looking for some particular virtualization solution, but you should definitely check vmware as well -- the server is now free as in beer and having the full computer together with BIOS and PXE boot is just nice.
Yenya wrote: Re: (Adelton)
Yes, when I think about it, it looks obvious that paravirtualization should be faster than full virtualization. As for vmware - from my point of view it is even worse than Xen, being available without source code. Xen has at least some prospect of being included in the vanilla kernel.