An update on OpenStack Nova

This article brought to you by LWN subscribers

Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

The Nova project is targeted at providing compute resources for OpenStack-based clouds. John Garbutt, who is the Nova project team lead (PTL) for the recent Liberty release, as well as the upcoming Mitaka release, presented an update on the project at the Tokyo OpenStack Summit. He looked at the changes that had come over the last few releases, with an emphasis on Liberty, while also giving a brief glimpse into the future of one of the foundational pieces of OpenStack.

He started out by noting that throughout the Liberty cycle the project had been conscious about trying to make sure that others in OpenStack were clear on what Nova was doing—and why. The mission of the project is stated on its web site: "To implement services and associated libraries to provide massively scalable, on demand, self service access to compute resources." The key piece there is that "Nova is all about compute", he said.

Priorities

But in order to get things done, there is a need to focus on particular features and to make time for them. The Nova team has identified a handful of priorities for the project, starting with having a good API with a strong ecosystem around it. In order to build up that ecosystem, the API needs to be the same in all of the different deployments of OpenStack so that other projects can rely upon it. The project is focusing on "doing a better job of that", Garbutt said.

Next up is "making sure we stay robust and reliable". That means when an API call is made, the right thing happens. That requires testing, fixing whatever bugs are found, as well as listening to operator and user groups about where the failures are.

The team has gotten a lot of feedback about the need for upgrades that are easy and that work correctly. It has taken many releases to get to where things are today, where Nova has "quite a good story" for upgrades. This will allow deployments to follow new releases more closely. There is a similar need for "scale out"; as deployments grow, Nova must reliably help by scaling out the compute resources seamlessly.

Maintaining the open culture within OpenStack is another priority for Nova. It is important to continue the innovation that the project has already brought to the table; open source and an open culture are major parts of that.

The final priority is to "focus on not expanding our scope". Nova is already a huge project, Garbutt said, but it needs to stay focused. There are a number of projects that could have been done inside Nova, but were spun out into separate projects. The Heat orchestration component is a good example of that.

Another specific example of avoiding scope creep for Nova happened recently. A "semi-high-availability" feature was proposed that would monitor an instance and bring it back up elsewhere if it crashed. The developers did not want to add it to Nova if that could be avoided, so they added some API calls so that an external high-availability tool could be used to implement it. That way, the problem was solved in the ecosystem, rather than within the Nova project itself.

Changes for Liberty

One of the major changes for the Liberty release "does not sound very sexy", but is actually fairly exciting to Nova developers: lots of architecture evolution. The "bowels of Nova" are being rewritten with an eye toward "maintaining stability while increasing velocity", Garbutt said. He acknowledged that stability and development velocity are at odds to some extent, but the project team is trying to find the right set of tradeoffs there. There are three themes to the evolution: API improvements, work on upgrades, and better scheduling and resource tracking.

The Nova API has evolved over time, but the team "stepped back" to try to better understand the API users and what they need from the API. It identified three types of user, each with different needs from the API.

First is the "absent user", who has some scripts that do what is needed to get their application up and running; they want those scripts to keep running even in the presence of upgrades to their cloud. Second is the "active user", who wants to use all of the new API calls and is happy to rewrite their scripts multiple times a year to do so. Those users want to be able to query what APIs are available so that the scripts can take advantage of the newer features. If the Nova team can't continue to change the API, these active users will get bored, Garbutt noted.

The third user type identified is the "multi-cloud user", who has applications that run in multiple public and private OpenStack clouds. Those users have "magical scripts" and SDKs that work on multiple different versions of OpenStack, including some components that are not even released yet. Nova supports installation from the Git trunk, for example, so there may be disparate versions in the various clouds. Upgrades on those clouds will not happen in lockstep. "That should work too", he said.

Beyond the users, the operations and development staff have an interest here as well. They likely have new problems to solve and need newer features from more recent releases, but they also need to know "who is using what" features, versions, and so on. The staff also would like to know, for example, how many active vs. absent users there are. "It would be great if we found a solution for this too".

APIs

The first Nova API was v2.0—an alias for v1.1—which was all "a bit confusing". In any case, there was this idea that it would consist of a base API plus a whole pile of extensions. Users could query to determine which extensions were available. But it is easy to make mistakes when creating an API and there was no way to evolve v2.0, he said.

So now there is a v2.1 of the API that is exactly the same as v2.0, but it got rid of the idea of extensions—they are bundled in with Nova now. All of the extensions are listed as being present for queries, so there is the same API everywhere. Evolution will be handled with "microversioning", so v2.2 will have some newer calls and potentially some deprecated ones; API users can request a particular minimum version. There will continue to be support for the v2.0 API in Nova, but by default Liberty will use v2.1.

There are also third-party APIs in the tree, such as one for Amazon's EC2, that have been a struggle to keep working and to get people to care about. The team has been working with an external project, which uses the Nova API to interface with EC2, to ensure that the calls needed are available. That will allow Nova to deprecate its EC2 support, which will probably be removed in the Mitaka release.

Upgrades

One of the key tenets of Nova (which it shares with other OpenStack components) is independence for the control and data planes. "That sounds fancy", Garbutt said, but what it really means is that "Nova can die in a fire" and it won't take the hypervisor or virtual machines (VMs) with it. Some downtime on the control plane (Nova) can be tolerated.

Nova supports upgrades from the latest stable branch to the next release, as well as to the next commit within the same cycle. As he noted previously, Nova supports installation from the trunk, which effectively means that it has to support upgrades between commits.

Another important upgrade requirement is that the existing configuration should "just work". It may cause Nova to spit out warnings about problems that will need to be addressed before the next upgrade, but those shouldn't have to be dealt with during the upgrade process. Those warnings are often for deprecated features; the project would rather not deprecate things, but sometimes must as a last resort, he said.

As the Nova PTL, Garbutt felt that he needed to put up a "complicated, scary diagram" of the architecture (which can be seen in the YouTube video of the talk), but he said that what it depicts is actually fairly straightforward. There are REST requests made at the API level that bubble down through the rest of the Nova pieces and eventually result in the creation of a VM (i.e. compute node).

One of those "pieces" is the Nova database, which needs to be dealt with as part of the upgrade process. Both the schema and the data in the database may need to be upgraded. In addition, the Nova control-plane components can be upgraded together, but the compute-node components may be upgraded over time so the conductor (which sits between the compute nodes and the database) must be able to convert the internal remote procedure call (RPC) formats between the old and the new. That requires lots of testing to ensure that "the magic happens" and the upgrade process works.

There is a four-step process for upgrades. First, the database is updated, then the API and control plane are restarted. The database used to be upgraded during that restart, but that turned out to be quite slow, so now the database is dealt with first. Then, the compute nodes (i.e. nova-compute) can be restarted one-by-one at convenient times. Once that is complete, the RPC version that is used can be pinned to the new (upgraded) version. This process has been available since the Kilo release; it stabilized in Liberty and more work is being done to reduce the downtime for Mitaka.

Wrapping up

The project has been working on defining its scope, which will help it avoid the problems of scope creep. For example, Nova does support containers by using LXC and libvirt, but it treats them more like VMs. The Magnum project is geared toward treating containers in a more "container-like" fashion, so that is where the container effort should go. Similarly, nova-docker was removed from the tree because it did not have the testing infrastructure needed to ensure that it did not break when things inside Nova changed.

For Liberty, there has been lots of progress. The architecture evolution has continued and there have been improvements in making upgrades have less of an impact, though there is still lots more work to do there. The API story has gotten better, which should help grow the ecosystem around Nova. In addition, over 60 blueprints were implemented for Liberty and over 400 bugs were fixed.

For Mitaka (and beyond), the Nova team will be working on better support for compute cells, which are meant to support very large deployments. There will be more of a focus on the user experience, which includes better API documentation, better error reporting, as well as improvements to the scheduler. The Nova team will also be working to keep the development process evolving, trying to reduce the review bottlenecks, and to release more often.

[I would like to thank the OpenStack Foundation for travel assistance to Tokyo for the summit.]