Distributions
Rebooting the Rackspace cloud
Rackspace is one company that has a large public cloud deployed—more than one, actually. A Xen security advisory from October 2014 required Rackspace to reboot one of its clouds. Two of its systems engineers reported on that event, and subsequent refinements to the techniques used, in a session at the Tokyo OpenStack Summit.
It took the better part of a week—of 20ish-hour days—for their team to reboot the cloud affected by XSA-108, Joel Preas said to start the talk. He and his colleague Ben Burdick covered a number of facets of the process, including why Rackspace felt this problem was important enough to sacrifice a week of the team's time, how the patch was created for the kernel, how Ansible was used to patch and restart the machines, and the lessons that were learned. Rackspace has had to reboot a few more times since he and Burdick first gave the talk and Preas was glad to be able to say that the team did actually learn the lessons.
The vulnerability was a 3KB or 12KB read of out-of-range memory. Before it was fully understood, it was thought to only allow crashing of the host operating system, which was bad enough, but it turned out to allow read access to memory that should not be accessible. It was small amounts of memory (in Dom0 and other guests) that was constantly changing, so the team was not able to figure out how to do much with it. But it still was considered to be big enough to justify patching and rebooting the affected systems.
The entire diff for the change that needed to be made was tiny: two lines where 0x3ff needed to be changed to 0xff. Before patching, though, the team needed to ensure that the problem could be reproduced. That way, the fix could be verified. The proof of concept was able to find Dom0 memory contents from a guest (three characters of a UUID or something like that, Preas said).
Once that was done, new processes for creating the kernel package needed to be developed and tested, as Rackspace normally gets its kernel packages from Citrix. It was important to Rackspace to fix the entire cloud before the embargo was lifted, Preas said. That amounted to a little over five days, Burdick added.
With that "very aggressive timeline", the team got to work, Preas said. Ansible was used "to drive it all". It was chosen because it has excellent performance, no dependencies, and the YAML files for playbooks and other configuration are easy to read. In addition, Ansible has easy-to-read pass/fail indications, which is important if you are running something on tens of thousands of systems, so that failures can be quickly spotted. That made it "easy-ish to triage" hosts that needed further attention.
There are additional Ansible attributes that made it desirable, Preas said. It is adaptable and collaborative, so multiple people could iterate on a playbook at once. It is easily extensible; modules can be written in Python if they are needed. It is also geared toward idempotency, which means that playbooks can be run multiple times. There are some features, like callback plugins, that were not used in the first rebooting exercise, but have been used since.
The cloud in question is in six different regions with more than a hundred cells, hundreds of cabinets, and thousands of hypervisors (hosts, effectively). All of that means that "flat files are bad" for trying to track dynamic configuration. Some kind of configuration-management database was needed.
Rackspace has its own configuration-management database called "Galaxy"—an unfortunate name collision with Ansible Galaxy—which got its baptism by fire in the first reboot. It collects information from the various "sources of truth" (e.g. switch configurations, Rackspace-internal tracking systems, OpenStack) and allows people to query for all of the information about a particular hypervisor—and get a big JSON object with that information back.
There is a subsystem in Galaxy called Gravity that goes out and polls for this information regularly, storing it all into a MongoDB database. An API has been added on top of that to feed the Observatory, which is a web application for monitoring. It is, effectively, a "giant key-value store with pollers", Preas said. It is also something that Rackspace is planning to release as open source in the "semi-near future", Burdick said.
So the reboot proceeded as an exercise in "divide and conquer", Preas said. There were ten to fifteen people working all at once. Each would pull a portion of the inventory to be worked on using a Galaxy API query. As hosts were patched, they got checked into Galaxy as completed, which was useful to give senior management an insight into the patch propagation.
The Ansible playbook would do some "pre-flight checks" to see what needed to be done. It would look to see if the host had already been patched, if it had the needed kernel modules (some hosts also got a kernel upgrade as part of the fix), made sure that the kernel versions were right, and ensured that all of the guests would automatically restart. Then it would backup the existing kernel, copy the patched kernel, verify the copy operation (even though Ansible already does that), and take a snapshot of the host state (to restore guests or services that didn't come back up for some reason).
After all of that, the reboot is started. The idea is to minimize the down time for the system. The final step is to report the system as completed to Galaxy and to double-check that the cloud block storage has been mounted correctly. A number of redundant steps were taken out of an abundance of caution, essentially. Though the final two steps were the time-honored "???" and "profit".
Burdick then took over to talk some more about Ansible. One of the advantages it has is that a lot of additional logic can be put into the playbooks. That allows tests for things like auto-starting compute nodes, versions of XenServer and Xen, and so on. It allows the playbook to properly back things up and to test that the patch was properly built before doing a reboot.
One of the problems the team ran into was that doing a check-in to Galaxy from the playbook would sometimes time out. Since Ansible is doing each host in a series, waiting for that timeout would slow everything down. In the end, the timeout was drastically reduced and any outliers that didn't check into Galaxy were dealt with separately.
Another problem was that some hosts hung during the shutdown process, Preas said. That would require Ansible to wait for a long timeout before giving an error. One of the team came up with the idea of forcing a reboot using the kernel's sysrq-trigger, but doing so from a playbook would completely kill Ansible (at least for Ansible 1.6). So that task was switched to be asynchronous in the playbook and Ansible was told not to wait around for it.
Some special logic was inserted into the playbooks to ensure that folks were doing what they thought they were doing. For example, there was a variable that needed to be set to true before a host that had guests running on it (thus was likely a real system used by customers) would be rebooted. That way, various tests didn't clobber systems before they were ready to be rebooted.
One of the biggest problems the team ran into was the lack of a good way to track systems with failures of various sorts. Etherpad was used; failures were pasted into the pad. That didn't work well, especially at times when there were feeding frenzies on grabbing hosts out of the list to go fix. In addition, Etherpad was used to track who was handling which hosts, which worked better than with the failures—at least until the public cloud machines it was running on got rebooted as part of the process.
The other big problem was fatigue. The team became tired, hungry, and grumpy at various points, Burdick said. The unsung heroes of the whole exercise were the data center operators, Preas said. They had to physically power cycle multiple machines and such throughout the whole process.
Choose your tools and dependencies wisely, Burdick said, as one of the lessons that had been learned. Make sure to test for all of the different cases, especially in the software versions, before doing the update. Having better tracking for triaging hosts that failed is needed.
To that end, the existing alert workflow (which uses Nagios) was extended to track hosts that failed in the upgrade process. There is an internal dashboard and other tracking tools that can now be used for that purpose.
The first reboot was just the next-generation cloud, Preas said. Subsequent reboots have been both for that cloud and for the legacy cloud. One of the changes that was made was to focus on systems that have running instances (guests) on them. In the first exercise, some amount of time was wasted dealing with systems that customers were not even using.
The future, though, is live patching, Burdick said. That will allow fixing these kinds of problems without a reboot. Some of the hypervisors that Rackspace is running are not capable of being live-patched, so there is at least one more reboot cycle "before we get to the greatness". Hopefully, rebooting the cloud will soon become a thing of the past.
[I would like to thank the OpenStack Foundation for travel assistance to Tokyo for the summit.]
Brief items
Distribution quote of the week
Fedora 23 released
The Fedora 23 release is now available. "We're pleased to bring you the latest incarnations of the three main Fedora editions — Fedora Workstation, Fedora Cloud, and Fedora Server, each built with love by the Fedora community to custom-fit your needs in different areas. Fedora 23 is also available in alternate desktop Spins, curated software Labs, and special images for the ARM processor architecture." See the release notes for details; LWN looked forward to this release in August.
GNU Hurd 0.7, GNU Mach 1.6, GNU MIG 1.6 released
The GNU project has released GNU Hurd 0.7, GNU Mach 1.6, and GNU MIG 1.6. The Mach 3.0 Interface Generator (MIG) translates Remote Procedure Call (RPC) definition files to C code, and is required to compile any packages that are receiving or invoking RPCs, such as GNU Mach, GNU Hurd, and the GNU C Library (glibc) when compiled for the Hurd. GNU Mach is a microkernel, upon which a GNU Hurd system is based. The GNU Hurd is the GNU project's replacement for the Unix kernel. These releases contain improvements and bug fixes.OpenELEC 6.0 released
The OpenELEC team has announced the release of OpenELEC 6.0. "The most visible change is Kodi 15.2 (Isengard). Beginning with Kodi 15.0 most audio encoder, audio decoder, PVR and visualisation addons are no longer pre-bundled into OpenELEC but can be downloaded from the Kodi addon repo if required. PVR backends such as VDR and TVHeadend will install needed dependencies automatically."
OpenSUSE Leap 42.1 released
The openSUSE Leap 42.1 release is now available. "Version 42.1 is the first version of openSUSE Leap that uses source from SUSE Linux Enterprise (SLE) providing a level of stability that will prove to be unmatched by other Linux distributions. Bonding community development and enterprise reliability provides more cohesion for the project and its contributor’s maintenance updates. openSUSE Leap will benefit from the enterprise maintenance effort and will have some of the same packages and updates as SLE, which is different from previous openSUSE versions that created separate maintenance streams." See this June LWN article for some background on this new approach to the openSUSE distribution.
Distribution News
Fedora
Fedora 23 for aarch64 and POWER
Fedora 23 releases for aarch64 and POWER are available. Both support Fedora Server, and POWER also supports Fedora Cloud.Fedora Developer Portal - released
The Fedora Developer Portal is live. "The aim of the portal is to give an overview about important development tools and projects in Fedora (like Docker, Vagrant, DevAssistant). Focus is targeted on beginners, advanced users and developers."
Newsletters and articles of interest
Distribution newsletters
- DistroWatch Weekly, Issue 634 (November 2)
- 5 things in Fedora this week (October 30)
- openSUSE Tumbleweed – Review of the week (October 30)
- Ubuntu Weekly Newsletter, Issue 440 (November 1)
Wind blows Helix Cloud, Pulsar Linux, Rocket RTOS toward IoT (LinuxGizmos)
LinuxGizmos looks at new releases from Wind River. "Intel subsidiary Wind River has released Wind River Pulsar Linux, an IoT-oriented version of its commercial Wind River Linux distribution, as well as a new Wind River Rocket RTOS. Both of these embedded OSes are designed to work with a newly unveiled Wind River Helix Cloud platform for developing, testing, monitoring, and analyzing cloud-connected IoT applications. Wind River Helix Cloud is available in App, Lab, and Device versions, and is said to provide “anytime, anywhere access to development tools, virtual labs, and deployed devices.”"
BackBox Takes Its Security Tools Seriously (LinuxInsider)
LinuxInsider reviews BackBox. "BackBox has its own software repositories. If its toolshed ended there, it would be a solid yet typical old-style Linux operating environment. The developers, however, added a state-of-the-art workroom with the latest stable versions containing some of the most often used and best-known ethical hacking tools."
Page editor: Rebecca Sobol
Next page:
Development>>