Presentations from the Smart Cloud Symposium
San Francisco
I was able to spend time in recently San Francisco with some pretty sophisticated technology operators and data center architects in San Francisco at IBM’s Smart Cloud Symposium. I presented on three topics with two different co-presenters. On the topics of “Open Virtualization” and “Cloud Computing” with KVM, my Co-Presenter was David Hsu, who is Program Director, Linux and Open Virtualization Strategy for IBM. David and I discussed the strength’s and weaknesses, use cases, costs, market positions, and technical foundations of the KVM hypervisor as used in traditional corporate data centers and, more recently, in both public and private cloud data centers.
The third discussion was on KVM Hypervisor Security, and I presented with Ajay Dholakia, Ph.D., who is a Senior Technical Staff Member and Master Inventor responsible for System X Software Strategy.
We put a mix of business-related, architecture, and technical information in the presentations and it turned out to be about equally useful. The audience were technically sophisticated people, weighted toward data center operators with hands-on experience. These were high-end vendors, partners, and technicians (the event was pretty expensive to start with) judging by the questions they asked during the sessions.
IBMSmartCloud_KVM and Security 041712
Open Virtualization with KVM – CIS 26
Cloud Computing with KVM – CIS28
Security and the Kernel Virtual Machine
We hear a lot of different things about KVM being “secure” and “not secure.” Most of the opinions we hear are directly associated with perceptions of security in the Linux kernel. People who have a high opinion of Linux security believe that KVM has high security. People who feel otherwise about Linux tend to also feel otherwise about KVM (that it is not a high-security platform).
We also sometimes hear disparaging opinions of KVM security based on mis-information. Such as that KVM is a “hosted” hypervisor or that it doesn’t use hardware security mechanisms.
Mostly, however, we notice that people don’t have first-hand specific knowledge about KVM’s security features. That’s why some of my colleagues at IBM and I wrote KVM: Hypervisor Security you can Depend Upon. This white paper covers the mandatory access control (MAC) features, security certifications, and secure design elements of the KVM hypervisor. For example, KVM is the only x86 hypervisor to support MAC by default. KVM leverages processor virtualization instructions to enforce guest isolation.
What the white paper does not cover are the secondary issues of coding errors, code review, white and black-hat hacking, and maturity. These are are security issues that transcend specific mechanisms but are related to overall design and implementation. (For example, in 2011 there was a programming error in the Linux kernel that subverted SELinux MAC.)
There are some good things to be said about KVM beyond mechanisms. The KVM code base is carefully and openly reviewed. And its older, more mature code than VMware ESX. In The evolution of an x86 hypervisor the authors who are VMware engineers discuss the substantial re-write and re-design of VMware ESX to support 64-bit operation. From a code maturity perspective, KVM is older, more mature code than VMware, even though KVM is a newer hypervisor.
This is a discussion that isn’t going to end with a white paper.
The KVM Hypervisor, Red Hat Enterprise Virtualization, and IBM Hardware
I’ve uploaded the presentation I made today with Jean Staten of IBM and Chuck Dubuque of Red Hat. There were over 700 registrants for the live Web seminar. Click on the link below to view the presentation. Good set of questions from the participants in the morning session. Many of the questions were concerning tools for automatic migration from VMware to KVM.
There is some content that covers the System X EX5 servers – which you can configure with up to 80 cores and 6 Terabytes of RAM. This is unmatched today on x86 hardware except for some speculative designs. The EX5 architecture uses Intel’s own processor bus to extend the memory and I/O bandwidth of the platform without any material performance penalties.
Red Hat – IBM Webinar KVM and REHV
KVM Reignites Type 1 Versus Type 2 Debate
Some interest in my presentation at the Linux Foundation Collaboration Summit led to an interview and article written by Beth Pariseau of Techtarget. The article is interesting and relevant but the topic of hypervisors is technical and hard to present in a news format. Some of the information provided to Ms. Pariseau by others is not correct, in my opinion but she does a good job presenting opposing viewpoints and a great job overall on the short article.
One particular quote (two years old) from Andi Mann is wrong on virtually every point:
“Xen is run and managed at a lower level (ring 0), even for new virtual machine creation, and guests do not share memory blocks, CPU instructions or any of the underlying (albeit occasionally de-privileged) Linux operating system like KVM does. This means KVM suffers performance, latency, security, scalability, isolation and other issues that do not affect a true bare-metal hypervisor.”
Let’s take these claims one at a time:
- “Xen is run and managed at a lower level (ring 0).” This is egregiously wrong – KVM is run and managed at Ring 0 as are all hypervisors that use Intel VMX or AMD SVM instructions. See slides # 8 and 19 in my presentation kvm-not-what-you-heard or any recent systems’ programmers guide from Intel or AMD.
- “Guests do not share memory blocks, CPU instructions, or any of the underlying Linux operating system like KVM does.” This is also egregiously wrong. Xen guests share memory blocks with Xen paravirtual I/O drivers, which are almost always hosted in the Linux Domain 0 kernel and shared among all guests. The vast majority of all Xen guests all share an entire Linux operating system — Domain 0– that does all I/O and device emulation. The vast majority of all Xen guests share common device drivers. They share the same code emulator. Even the very few Xen geusts that use the Stub domain code share memory blocks with each other and with the Xen kernel using Xenbus. KVM guests also share code, device drivers, and memory blocks. This is not a problem, it is normal, accepted kernel engineering practice. All widely used kernels do this. It is secure because of hardware memory protection.
- “KVM suffers performance, latency, security, scalability, isolation, and other issues that do not affect a true bare-metal hypervisor.” First of all, KVM is a true bare-metal hypervisor. But that doesn’t prove anything, because bare-metal hypervisors may also be vulnerable to “latency, security, scalability, isolation, and other issues” if they are mis-designed or if they suffer from programming errors. So this statement is meaningless. The fact is that KVM has performance advantages over both Xen and VMware (see SPECVirt results, for example.) Xen has a huge isolation issue with Domain 0 (See slides #20-23 in my presentation referenced above).
You should dismiss factual errors, such as the ring 0 statement, out of hand. You should disregard these further statements by Mann unless he provides evidence, data, or explanation.
The Kernel Virtual Machine – Not What you Have Heard
Here’s the presentation I’m giving at the Linux Foundation Collaboration Summit: kvm-not-what-you-heard
Abstract:
Mike will review myths, misstatements, and mistakenly-held notions about KVM, the Linux hypervisor. Starting with the major one: that KVM is not a “bare metal” hypervisor. In addition to the “not a bare metal hypervisor” myth, I’ll also address other prominent myths, including: KVM is not good at running Windows guests, KVM is not as secure as other hypervisors, KVM does not scale, KVM does not over-provision memory and KVM doesn’t have VM_* (meaning VMware apis or features). Whenever possible, I will show the source of the myth and debunk it using source code, published benchmarks, or other mechanisms.
KVM in a Large Data Center
Yesterday I spoke at the Linux Foundation End-user Summit. My talk was about our experiences deploying the KVM hypervisor in the IBM smart business cloud. I spoke largely about our experiences making KVM work well in a series of large data centers, and the benefits of that experience to the upstream KVM development trajectory. Here are my slides (pdf): mdday-kvm-data-center.
Some KVM development community stats
Today I made a presentation (pdf) on the Linux Kernel Virtual Machine to the Red Hat Cloud Computing Forum. I enjoyed the format. All the presentations were short (30 minutes including Q&A) and technical. This is the type of forum I enjoy attending, so its easy to prepare and I am comfortable with the audience.
KVM Developer Participation
One of the topics I covered in the presentation is the level of KVM development activity in 2009. To measure the depth and breadth of participation in KVM development, I used activity on the developer mailing lists for the three primary components of the KVM Hypervisors: KVM, which provides the virtual machine monitor; Qemu, which provides the virtual machine environment; and libvirt, which provides the low-level management interfaces.
I like to think that monitoring the traffic on an open-source project’s mailing list is a lot like gathering intelligence through traffic analysis. You can learn who is working on a project, what specific areas they are working in, and with whom they are working. The volume of traffic is also a good indicator of the weight behind a project and the overall development velocity. If you were really ambitious you could graph the relationships among various projects based on the participation of specific individuals.
Below is a summary of the raw statistics for 2009:
kvm development community statistics 2009
What can we learn from Raw Message Counts?
These three mailings lists are dedicated to development activity. There are three types of messages included in the analysis:
- Source code. All source code changes are submitted as email messages. Per conventions for Linux kernel development, the subject line of these messages usually includes the tag “[PATCH].”
- Source code review. When a developer submits some proposed changes, analysis and discussion of the source code generates replies to the original email. Developers use email clients that support message threading, which makes it easier to follow the discussion.
- Bug reports.
All three of the activities are part of what we typically consider the job description of a software engineer. There is another sub-category of messages that intermix design proposals with source code submissions. These messages usually include the tag “[RFC]” in the subject line.
Because of this, I believe that message counts for mailing lists dedicated to software development provide a good indication of health of a development community. For KVM, the statistics are impressive.
- Almost 400 organizations participated in KVM development, ranging from large corporations such as IBM, Intel, and Red Hat, to academic institutions and individual contributors.
- Approximately 800 unique contributors. This is an extremely broad group of software developers.
- A solid core of “super contributors,” developers who form the top tier of the project contributions.
Top Individual Contributors
It’s also good to look at the top individual contributors. These are the folks who are generally 100% focused on the project and are the most prolific programmers.
KVM-Devel
3810 avi redhat 1261 mst redhat 851 gleb redhat 799 mtosatti redhat 507 ghaskins novell 453 anthony codemonkey 410 lmr redhat 394 agraf suse 362 sheng linux intel 357 jan.kiszka siemens 356 glommer redhat 336 mgoldish redhat 226 amit.shah redhat 223 jan.kiszka web de 209 markmc redhat 208 alex.williamson hp 197 joerg.roedel amd 178 mhiramat redhat
Qemu
1839 anthony codemonkey 1457 kraxel redhat 1447 quintela redhat 961 avi redhat 819 aurelien aurel32 805 lcapitulino redhat 805 blauwirbel gmail 745 mst redhat 617 yamahata valinux 565 agraf suse 558 aliguori ibm 540 jan.kiszka siemens 493 markmc redhat 468 paul codesourcery 425 gleb redhat 418 jamie shareable 407 av1474 comtv 402 armbru redhat 391 glommer redhat 371 kwolf redhat
Comments on Individual Developer Counts
Looking at the individual contributor counts for KVM and Qemu, it is clear that the top contributor on each list is quite a bit more active than next highest contributor. (On the Qemu list, anthony codemonkey and aliguori ibm are the same person posting under two different addresses, which is not uncommon. This means anthony’s actual message count is clost to 2500 messages). Avi Kivity is the KVM maintainer, and Anthony Liguori is the Qemu maintainer. It’s the job of the maintainer to review and accept all code submissions, and to package and announce new releases of the code. So you expect the maintainer of a project to post the most messages.
You can also see from these message counts that there is a large overlap of top contributors to the KVM and Qemu projects. In fact Avi Kivity is a top contributor to Qemu, and Anthony Liguori is a top contributor to KVM.
See for Yourself
You can read the kvm-devel and Qemu mailing lists via the web using Gmane.
Source code for Analysis Tools
I wrote a couple of crude utilities to do this maling list analysis. The are:
string_search.c a utility that understands email address strings and can process them and count instances of specific addresses in a file
mbox-filter.py a python utility that filters an mbox-formatted email file in a number of different ways. I use it, for example, to collect all messages that fall within a certain range of dates.
Perhaps in a future post I’ll document these utilities and enhance them. They are very crude at this point. Once I got the info I needed out of the mbox files I stopped working on them.