Life in Code

Most people in the Information Technology Industry talk a lot these days about OpenStack. They don’t talk as much (or necessarily even know about) oVirt, an imbalance which has presented us with both challenges and opportunities. This was the subject of a presentation that I gave, along with Jean Staten Healy, at the recent Open Stack Summit (youtube video). Jean discusses the different OSS communities that IBM is involved with, how they relate (or not) to OpenStack, and the OpenVirtualization Alliance, a consortium of companies working to increase KVM deployment. I follow with a longer discussion of oVirt and OpenStack services.  A .pdf of our presentation is available at this link: Connected Communities – OpenStack and oVirt – KVM and Cloud Management.

In 2011 when Carl Trieloff of Red Hat and I were working to recruit the charter members of the oVirt community our goals were to (1) create a single ecosystem for KVM virtualization management tools, (2) standardize the market on a single infrastructure for managing KVM virtual machines, and (3) expose and support new KVM features in a timely manner.

The rapid emergence of OpenStack as a production cloud infrastructure has taken a lot of mindshare away from oVirt. OpenStack has also prevented oVirt from forming a viable ecosystem for virtual machine management. At the same time, our goals for a standard infrastructure that exposes all of KVM’s features are being met very well by oVirt.

In addition, oVirt now has some promising opportunities to add value to OpenStack deployments. And we’re not above adopting the same technologies, such as Quantum, for networking services.

Interestingly, a full year before we founded the oVirt community with Red Hat, some of our IBM engineers working on KVM attended the OpenStack summit in San Antonio and really liked what they saw. For several non-technical reasons we didn’t go down that road, and even if we did, we would still have created the oVirt community. But it’s interesting to speculate nonetheless.

As I said at the OpenStack summit: Open Source Communities exist as long as they provide value to users and developers. We’re working hard to make sure that oVirt adds value in an OpenStack world.

This past Saturday I spoke at the Southern California Linux Expo (SCALE).  This is the first time I’ve been to SCALE. It’s a really great show. The conference program was huge and the keynote presentations were relevant–secure boot, the afero GPL–and the expo as great. I spent some time Saturday morning in the IBM booth answering questions, which is always fun. The audience for my talk was standing-room-only and they were very interested in KVM. Franky the interest level of the audience is what makes a speaker good or bad on a particular day.

I covered IBM’s motivations for investing in KVM development, the motivations of our customers for adopting KVM, and our hopes for KVM’s effects on the marketplace. I also covered some characteristics of the KVM development community and some performance and scalability statistics.

I’ve uploaded a copy of my presentation here. advancements-kvm

I still get asked frequently why we changed our development focus from Xen to KVM in 2009. One of the reasons is that the Xen development community was broken in a couple of aspects. First, the maintenance practice. The maintainer, who was an excellent programmer, either merged submissions under his ID, or didn’t merge them and did not provide any comments or review of the submission. Rewriting submissions was particularly problematic for us. It was not unusual for either the maintainer or a XenSource programmer to rewrite an external submission and then to submit it under his or her ID. Second, there was an insistence that important code must be “owned” by a XenSource (later Citrix) engineer. Someone would submit some code, it would languish without a response, and later a XenSource engineer would submit different code that did exactly the same thing, which was merged quickly by the maintainer, usually without comment to the rest of the developers.

Then there was the infamous incident surrounding the release date of the Xen 3.0 upstream code (as a Generally Available snapshot). As the second-largest participant in the project, we  could not get any of the project’s leaders to tell us when the GA of version 3 would occur. Nobody outside of XenSource was able to get an answer. Finally one of the XenSource executives told us (in person) that XenSource was keeping the release schedule secret in order to surprise VMware. (We still didn’t get the information.)

Our transition from Xen development to KVM development occurred because of technical issues, mostly the way that KVM leverages the Linux kernel. But the community issues with Xen added motivation for the transition.

Developer Participation

It’s fair to ask, given the statistics for development mailing list participation, if the stats don’t reflect the actual participation in the community for 2008. The answer lies in message analysis from the xen-devel mailing list. (This is how developers communicate with each other and review code.) The message analysis shows broad participation in development discussions, but this doesn’t translate into broad participation in source code submissions. I believe this shows a real disparity between the hopes of Xen developers and the reality of the Xen source code maintenance practices. This disparity continued through 2010 and then the disparity almost disappeared in 2011, the year when there were some changes in the affiliation and participation of the Citrix developers.

Combined development statistics for Xen, 2008-2011

In the combined chart above we see that breadth of source code submissions is dramatically narrower than the breadth of mailing list participation. For example in 2010 (which showed dramatic improvement from the previous years) there were 861 developers participating in the mailing list discussion, but only 42 developers that were able to get code into the repository under their ID. This is juxtaposed by KVM for the same year, which showed 148 contributors and 741 message correspondents.

2009 Xen Source Code Changes per Domain

2009 KVM Source Code Changes per Domain

The two pie charts above show the diversity of organizations contributing code to Xen and KVM during 2009, which was a good year for Xen in terms of diverse participation. At least partially as a result of the maintenance practices, the breadth of the Xen development suffered from 2005-2010. This doesn’t mean that the community didn’t produce great code (it did) — it means that most of the code is from a single organization. We prefer development communities that have breadth, although this is not a hard-and-fast rule.

Xen Community Improvements beginning in 2011

The maintenance practices and culture of the Xen project improved in 2011, as the Xen founders prepared to leave Citrix for another startup. This created an unprecedented (for Xen)  situation where many of the key developers were “outsiders.” As a result, the community moved to more mainstream practices and a more open culture. You can really see the improvement in the development statistics for 2011, below:

2011 Xen Source Code Contributions per Domain

In terms of  the distribution of code changes (if not total organizations) this chart looks almost identical to KVM’s. Its too bad it took so many years for the Xen maintenance practices to become more open and consistent with the Linux kernel community and other communities with broad participation.

You can download the raw data behind the statistics here. spreadsheet

Continuing my statistical analysis of open-source hypervisor projects, I’ve collected raw data for the years 2008-2011 on the Xen hypervisor. One issue I encountered was obtaining the xen-devel email archive. Both the Xen and KVM communities use public email as the primary development forum. Code submissions, design discussions, bug reports: all are discussed on public mailing lists. This is why a statistical analysis of these projects needs to include email archives.

Back to my issue with Xen: I had partial email archives for the years in question. The archives for a couple of the years were corrupt. The first thing I did was look for a utility program that would download email archives from one of the internet email repositories. I knew, for example, that Gmane had a web API for downloading mbox format archives.

I couldn’t find a utility to download email archives. I spent some more time asking some kernel developers and other open-source colleagues of mine if they knew of a program that did what I needed. I didn’t find anything. (After the fact I was made aware of a nice NNTP utility by Anthony Liguori and Ryan Harper that would suite my needs with some  modifications.)

Instead of manually reconstructing my xen-devel email archives, I wrote a program in Python to download the xen-devel archives from gmane.  It’s a short program and it was not difficult to write nor test. The only problem I had was with gmane’s policy of closing download sessions after they consumed 30 seconds of system time. This would allow me to download January through April, give or take a few weeks of each year, before being cut off by the gmane server.

To solve the gmane throttling problem I simply added some logic to open separate connections for each month when downloading an entire year’s worth of archived emails.

I spent the vast majority of time on this program coding the ability to specify date ranges for the downloaded messages. I had to get creative. Gmane URLs using the “blog” interface include a date encoding. Within the HTML there is a link to the permanent message ID. I ended up scraping the message ID out of that link for both the beginning and ending messages of the date range entered by the user.

def get_msg_id(list, date):

    url = url_base + list + "/" + "day=" + date

    perm_url = ""

    fp = urllib.urlopen(url)

    try:

        data = fp.readline()

        while data:

            match = re.search(r'href=\"http://permalink.gmane.org/([^\'" >]+)', data)

            if match:

                perm_url = match.group(0)

                break

            data = fp.readline()

    finally:

        fp.close()

    if perm_url:

        parsed = urlparse.urlparse(perm_url)

        path = parsed[2]

        pathlist = path.split("/")

        message_id = pathlist[-1]

        return message_id

    else:

        return 0

Once the program finds the beginning and ending message IDs, it retrieves and dumps the messages in mbox format to stdout. To create an mbox archive locally you can use command-line file redirection:

./gmane-mine.py --list gmane.comp.emulators.xen.devel --year 2008 > ~/mail/xen-devel-2008.mbox

Now back to the NNTP utility from Anthony Liguori and Ryan Harper:  Gmane supports NNTP without restriction (it won’t close a session). I added support for the NNTP protocol to my utility by using the techniques that Anthony and Ryan use in their program. To use the NNTP protocol (instead of HTTP) you simply use the –nntp flag on the command line. The NNTP service from Gmane is much slower than the HTTP service. This is probably why it’s unrestricted. But it works reliably, in fact both protocols do.

Here is the usage information:

gmane-mine.py --list <listname> --start yyyymmdd --end yyyymmdd --year <year> --nntp

where <listname> is the gmane name of the list. For example,

gmane.comp.emulators.qemu is the gmane name of the qemu mailing list.

--start and --end are the beginning and ending (non-inclusive) of

the date range in yyyymmdd format. For example, July 4, 1961 is 19610704

 --year 2008 will download all messages from 2008

 --nntp uses the gmane nntp server to download messages

 Here's an example showing how to download qemu messages from

January 1, 2008 through January 9, 2008:
 gmane-mine.py --list gmane.comp.emulators.qemu --start 20080101  --end 20080110

You can get the source code here:  https://github.com/ncultra/kvm-community-stats/blob/master/gmane-mine.py

Roughly two years ago I posted some statistics from 2009 development email traffic for KVM and QEMU. I’m back with more and better data from the years 2008 through 2011.

This data set includes patch submission, lines of code (LOC) statistics, email and traffic analysis for each project for each year. All the data is cross tabulated by individual and organization. It’s a lot of data, and I’ll talk about what it means briefly in just a minute. You can do your own analysis of the data, because I’ve linked to a spreadsheet with all the raw data and the charts.

I’ve also linked to Charts – KVM community statistics 2008-2011 (pdf) which contains charts graphically summarizing the data sets.

Here’s a peak at some of the LOC data, to give you an idea of what you can do with it:

Raw Data Sample: Source Code Lines Changed by Individual

I oriented the mailing list data similarly. Each data set has a table for individual contributions, and a table for contributions per organization.

Here are a couple examples of charts I generated from the data:

Pie chart showing 2009 QEMU lines-of-code changes by Organization

This line chart plots cumulative data for the four-year period of analysis:

Four-Year Cumulative Source Code Statistics for KVM and QEMU

The data shows KVM and QEMU continue to be broad, vital, and dynamic communities. They enjoy the benefit of excellent maintainers. You can derive this last assertion from email traffic analysis and “acked-by” analysis from the source code repository. Both projects continue to enjoy more than 800 regular contributors and more than 300 regular contributing organizations.

Red Hat continues to be the single largest contributor to both projects, but it does not dominate either of the communities.

To gather these statistics I created a new set of tools and modified an existing tool. Having the new tools, I can generate these statistics having a development mailing list and the URL of a source code repository. That’s all I need. The tools download the code, partition the data by the dates you desire, and place comma-separated-value (CSV) files in a directory for each time period. You can open the CSV files with any spreadsheet program.

I had to make three modifications to gitstat. It has always had a “since” option, which allows you to collect statistics after a certain date. But I also needed to bound the statistics by a “before” date, so I added an “until” option.

The second modification I made was to restrict data collection to specific directories. KVM is a discontiguous subset of the Linux source tree, and QEMU includes a lot of directories that KVM does not use. The third modification I made is to collect data filtered by an email regular expression. I added this option so I could generate LOC data by domain.

The email processing tools are much better than two years ago. Here I do most of the work in a “C” program that is smart enough to tabulate by individual and domain. To get good numbers I had to roll up sub-domains, such as de.ibm.com, into their parent.

I automated the entire process using a three shell scripts. One script is the “master of ceremonies” –it invokes all the other processes. The other scripts invoke the email and source repository tools and use awk to warp the output into a nice, neat CSV format with headers.

gitstat.py --one-line -f $@ | awk '{gsub('/[\(\)]/',"",$0); \

gsub('/\\+/', "", $0); \

gsub('/\\//'," ", $0); \

gsub('/\\-/',"", $0) ;   \

gsub('/[[\<\>]]?/',"",$0) ; \

print $0 }' \

| awk '{ if (NF == 7) printf "\"%s %s\", %s, %s, %s, %s\n", $1, $2, $3, $4, $6, $7 ; \

if (NF ==  printf "\"%s %s %s\", %s, %s, %s, %s\n", $1, $2, $3, $4, $5, $7, $8 ;\

if (NF == 6) printf "\"%s\", %s, %s, %s, %s\n", $1, $2, $3, $5, $6}' \

>> $OUTPUT_FILE

Last week I had the pleasure of presenting a keynote talk on the Open Source Solutions Stage at Ohio LinuxFest. I spoke about the unique role of the KVM hypervisor. It is a traditional, bare-metal hypervisor with advanced features, performance, and scalability. At the same time it’s integrated with Linux.

The integration with Linux has meant a lot to us as hypervisor developers over the past several years. Many of the advances we have made with KVM have leveraged the hard work done by Linux kernel engineers. These include:

• Support for new hardware, processors, chip sets, and architectures
• Memory management including large pages, Non-uniform memory access (NUMA), and process migration
• Large system scalability using Read-Copy-Update
• Power Management
• Scheduling and resource control

There are many more examples. We don’t have to write this code, because the Linux kernel community wrote it, tested it, and maintains it. The KVM and Qemu upstream developers can focus on virtualization features, such as resource over-provisioning, distributed block storage, storage offload, and always performance improvements.

I also discussed how our experiences with KVM–specifically how beneficial the leverage provided by Linux has been to us–have begun to inform the greater engineering organization at IBM. The benefits we have seen have been consistent, repeated, and dramatic. They show up in developer efficiency, time-to-feature completion, and reduced resource requirements. I think most developers who have been working with or close to Linux have known this for quite a while. But it’s good to see these lessons influencing a larger organization.

I’ve attached a .pdf of my presentation below.

Advancements in Open-Source Virtualization with KVM

San Francisco

I was able to spend time in recently San Francisco with some pretty sophisticated technology operators and data center architects at IBM’s Smart Cloud Symposium. I presented on three topics with two different co-presenters. On the topics of “Open Virtualization” and “Cloud Computing” with KVM, my Co-Presenter was David Hsu, who is Program Director, Linux and Open Virtualization Strategy for IBM. David and I discussed the strength’s and weaknesses, use cases, costs, market positions, and technical foundations of the KVM hypervisor as used in traditional corporate data centers and, more recently, in both public and private cloud data centers.

The third discussion was on KVM Hypervisor Security, and I presented with Ajay Dholakia, Ph.D., who is a Senior Technical Staff Member and Master Inventor responsible for System X Software Strategy.

We put a mix of business-related, architecture, and technical information in the presentations and it turned out to be about equally useful. The audience were technically sophisticated people, weighted toward data center operators with hands-on experience. These were high-end vendors, partners, and technicians (the event was pretty expensive to start with) judging by the questions they asked during the sessions.

IBMSmartCloud_KVM and Security 041712

Open Virtualization with KVM – CIS 26

Cloud Computing with KVM – CIS28

We hear a lot of different things about KVM being “secure” and “not secure.” Most of the opinions we hear are directly associated with perceptions of security in the Linux kernel. People who have a high opinion of Linux security believe that KVM has high security. People who feel otherwise about Linux tend to also feel otherwise about KVM (that it is not a high-security platform).

We also sometimes hear disparaging opinions of KVM security based on mis-information. Such as that KVM is a “hosted” hypervisor or that it doesn’t use hardware security mechanisms.

Mostly, however, we notice that people don’t have first-hand specific knowledge about KVM’s security features. That’s why some of my colleagues at IBM and I wrote KVM: Hypervisor Security you can Depend Upon. This white paper covers the mandatory access control (MAC) features, security certifications, and secure design elements of the KVM hypervisor. For example, KVM is the only x86 hypervisor to support MAC by default. KVM leverages processor virtualization instructions to enforce guest isolation.

What the white paper does not cover are the secondary issues of coding errors, code review, white and black-hat hacking, and maturity. These are are security issues that transcend specific mechanisms but are related to overall design and implementation. (For example, in 2011 there was a programming error in the Linux kernel that subverted SELinux MAC.)

There are some good things to be said about KVM beyond mechanisms. The KVM code base is carefully and openly reviewed. And its older, more mature code than VMware ESX. In The evolution of an x86 hypervisor the authors who are VMware engineers discuss the substantial re-write and re-design of VMware ESX to support 64-bit operation.  From a code maturity perspective, KVM is older, more mature code than VMware, even though KVM is a newer hypervisor.

This is a discussion that isn’t going to end with a white paper.

I’ve uploaded the presentation I made today with Jean Staten of IBM and Chuck Dubuque of Red Hat. There were over 700 registrants for the live Web seminar. Click on the link below to view the presentation. Good set of questions from the participants in the morning session. Many of the questions were concerning tools for automatic migration from VMware to KVM.

There is some content that covers the System X EX5 servers – which you can configure with up to 80 cores and 6 Terabytes of RAM. This is unmatched today on x86 hardware except for some speculative designs. The EX5 architecture uses Intel’s own processor bus to extend the memory and I/O bandwidth of the platform without any material performance penalties.

Red Hat – IBM Webinar KVM and REHV

Some interest in my presentation at the Linux Foundation Collaboration Summit led to an interview and article written by Beth Pariseau of Techtarget. The article is interesting and relevant but the topic of hypervisors is technical and hard to present in a news format. Some of the information provided to Ms. Pariseau by others is not correct, in my opinion but she does a good job presenting opposing viewpoints and a great job overall on the short article.

One particular quote (two years old) from Andi Mann is wrong on virtually every point:

“Xen is run and managed at a lower level (ring 0), even for new virtual machine creation, and guests do not share memory blocks, CPU instructions or any of the underlying (albeit occasionally de-privileged) Linux operating system like KVM does. This means KVM suffers performance, latency, security, scalability, isolation and other issues that do not affect a true bare-metal hypervisor.”

Let’s take these claims one at a time:

1. “Xen is run and managed at a lower level (ring 0).” This is egregiously wrong – KVM is run and managed at Ring 0 as are all hypervisors that use Intel VMX or AMD SVM instructions. See slides # 8 and 19 in my presentation kvm-not-what-you-heard or any recent systems’ programmers guide from Intel or AMD.
2. “Guests do not share memory blocks, CPU instructions, or any of the underlying Linux operating system like KVM does.” This is also egregiously wrong. Xen guests share memory blocks with Xen paravirtual I/O drivers, which are almost always hosted in the Linux Domain 0 kernel and shared among all guests. The vast majority of all Xen guests all share an entire Linux operating system — Domain 0– that does all I/O and device emulation. The vast majority of all Xen guests share common device drivers. They share the same code emulator. Even the very few Xen geusts that use the Stub domain code share memory blocks with each other and with the Xen kernel using Xenbus. KVM guests also share code, device drivers, and memory blocks. This is not a problem, it is normal, accepted kernel engineering practice. All widely used kernels do this. It is secure because of hardware memory protection.
3. “KVM suffers performance, latency, security, scalability, isolation, and other issues that do not affect a true bare-metal hypervisor.” First of all, KVM is a true bare-metal hypervisor. But that doesn’t prove anything, because bare-metal hypervisors may also be vulnerable to “latency, security, scalability, isolation, and other issues” if they are mis-designed or if they suffer from programming errors. So this statement is meaningless. The fact is that KVM has performance advantages over both Xen and VMware (see SPECVirt results, for example.) Xen has a huge isolation issue with Domain 0 (See slides #20-23 in my presentation referenced above).

You should dismiss factual errors, such as the ring 0 statement, out of hand. You should disregard these further statements by Mann unless he provides evidence, data, or explanation.