Kernel development

Kernel release status

The 4.3 kernel is out, released on November 1. "So on the whole, this remains a rather calm release cycle until the very end. And with the release of 4.3, obviously the merge window for 4.4 is open, and let's keep our fingers crossed that that will be an equally calm release." 4.3 includes the ability to add BPF programs to user-space probes, the "PIDs controller" (an anti-fork-bomb measure), the removal of the ext3 filesystem, support for identifier locator addressing, the ability to handle page faults in user space, and more.

The 4.4 merge window is now open, but has gotten off to a relatively slow start; full merge-window coverage will be found in next week's Kernel Page.

Stable updates: none have been released in the last week.

Comments (none posted)

Quotes of the week

— Peter Hutterer

— Josh Boyer

Comments (7 posted)

The 2015 Kernel Summit

The 2015 Linux Kernel Summit was held October 26-28 in Seoul, South Korea, alongside the Korea Linux Forum. Numerous developers came together to discuss issues of interest to the kernel development community as a whole. This page collects the coverage from that event.

The media minisummit

The only minisummit held this year was for the media subsystem; it happened on October 26. See this report for details on what was discussed there.

The technical day

October 27 was an open day for the discussion of technical topics of general interest; it followed a two-track schedule. Your editor, unsurprisingly, was unable to follow both tracks, so this coverage is not complete. Articles from this day include:

• Running a mainline kernel on a cellphone: why phone handsets run highly patched kernels and what can be done about it.

• Improving (or removing) the kthread freezer. The implementation of the freezer for kernel threads is inconsistent and buggy; it should be fixed. Unless the kernel-thread freezer can just be removed entirely, that is.

• Power-management knobs: many useful power-management features go unused because they are not enabled. Do we need an overall knob to turn on power-efficient operation globally?

• Device dependencies and deferred probing: how to improve the setup of devices that depend on other devices in the system.

• Benchmarking and performance trends: how are we doing with regard to performance regressions, and where are our biggest performance issues now?

• Realtime mainlining: now that the realtime project has funding again, what is the plan for this out-of-tree patch set?

• Security beyond bug fixing; Kees Cook told the assembled developers what can and should be done to increase the security of the kernel.

• Developer workflow security: what practical steps can be made to keep developers' machines secure?

The obligatory unrelated photo

Sunrise in Seoul.

The core day

October 28 was the invitation-only day for core developers and maintainers. Sessions from this day include:

• Kernel testing; various ways of improving automatic testing of the kernel.

• More security discussion and, in particular, whether the kernel community is now more willing to consider merging intrusive security technologies.

• Developer recruitment and outreach: are we bringing in enough new developers, and how can we attract more of them?

• Documentation and the kernel's homegrown documentation-processing tools.

• Restartable sequences; how do we want to support this user-space concurrency mechanism in the kernel?

• Lightning talks: kernel tinification, the year-2038 problem, and out-of-tree code on handsets.

• The stable kernel process: should we make changes to how stable kernels are produced, and what will the next long-term stable kernel be?

• Is Linus happy?: what is the state of the development community, and where is there room for improvement?

The obligatory group photo

Acknowledgments

Your editor would like to thank the Linux Foundation for supporting his travel to the Kernel Summit.

Comments (none posted)

Improving (or removing) the kthread freezer

The process freezer (or just "freezer") holds processes in a suspended state; it is used, for example, when suspending or hibernating the system. Jiri Kosina started his 2015 Kernel Summit session by noting that he had thought he could use the freezer for live patching as well. But, in the process of making that work, he found out that the semantics of the freezer are not particularly well defined; there is no definition of what it means for a process to be "frozen." As a result, the implementation of frozen behavior varies and, in the case of a number of kernel threads ("kthreads"), things are broken entirely.

The original purpose of the freezer was to get processes out of the way so that the system could be suspended and to ensure that no unwanted I/O activity causes filesystem corruption. The freezer also prevents user space from taking kernel locks. Kernel threads, being processes, also are subject to freezing unless they have the special PF_NOFREEZE flag set. A freezable kthread that fails to freeze when requested can block the system from suspending or hibernating.

One of the first things Jiri noticed was that kthreads will call try_to_freeze() (an indication that this would be a good time to freeze the thread if needed) without having first called set_freezable() (which clears the PF_NOFREEZE flag). Such threads can never be frozen, so the try_to_freeze() call will always fail. That suggested to him that the freezer for kthreads is overused and unneeded. Some kthreads (those that participate in the I/O needed to suspend or hibernate the system) cannot be frozen anyway. For the rest, the important part is to get them to the point where they schedule, and that is happening anyway without try_to_freeze().

Jiri said that the arguments for freezing kernel threads do not hold much water when examined. There are some fears that a kthread could hold a lock and deadlock the hibernation process; if that were true, he said, it would deadlock the system anyway. The other reason to freeze kthreads is to prevent them from initiating I/O, but it is, in fact, important for kthread I/O to succeed while the system is changing state. So, rather than worrying about freezing kthreads, Jiri suggested, why not just restrict the freezer to user-space processes? It is unclear why the freezer even exists for kthreads; it appears to serve no purpose and is easy to get wrong.

An alternative idea, Jiri said, would be to annotate I/O that is needed for the suspend or hibernate process to complete. Only specially marked requests would be allowed through; the rest would be blocked until the system is resumed. But, he said, this approach seems complex, fragile, and just as error-prone as what's there now.

The best solution, he said, might be to get rid of the kthread freezer entirely. Its main reason for existing is to ensure that filesystems do not get corrupted when the system suspends and resumes (or fails to resume). That could be just as easily accomplished by freezing the filesystems instead. The virtual filesystem layer already has support for freezing filesystems, and the freeze operation ensures that the filesystems are consistent on disk. Freezing filesystems would also have the advantage of preventing bootloaders from trying to replay the journal — something that overly clever bootloaders evidently do — because there would be no journal to replay.

There was a quick consensus in the room that abolishing the freezer looked like the right way to go. Thus, chances are, we'll see patches toward that goal in a future development cycle.

Comments (none posted)

Power-management knobs

Rafael Wysocki started off his 2015 Kernel Summit session by noting that every generation of hardware promises to be more power-efficient than its predecessor. But that efficiency is not always experienced by users. In an ideal world, systems should run in the most power-efficient mode whenever possible and only employ the less-efficient modes when performance requirements demand it. Real-world systems, though, tend not to run as efficiently as they can. Rafael came with a proposal that, he thought, might improve the situation, but it's not clear that the idea will be implemented.

Any hardware (be it a CPU or a peripheral device) with multiple operating states has to be able to detect when it is safe to change between modes. Almost all hardware has some power-related modes that it is able to employ, but, increasingly, power-efficient operation is not something that a device can choose independently from the rest of the system. A modern system-on-chip (SoC) includes a lot of devices in a single package; these devices share the same power-distribution mechanism. Often, many or all devices must go into a low-power state before the shared power hardware can change state, so a single misconfigured device can keep much of the system in a high-power state. And, as it happens, devices are often configured to not use their lower-power states.

This configuration is done for a number of reasons, but it mostly comes down to safety. Some hardware can misbehave when running in a low-power state, and the consequences of that misbehavior can be severe. Sometimes the problem is in the driver, which does not handle low-power states well. Power-management support tends to be seen, rightly or wrongly, as being experimental, so developers often disable it. And sometimes there are legitimate performance concerns that argue against the use of low-power states.

Power-management configuration can be tricky; there are a lot of knobs in the system that affect it in one way or another. There is usually at least one knob for every driver, often more; few users even know how to find all of these knobs, much less configure them properly. Rafael would like to make life easier for users wanting to tune their systems for power use; to that end, he has proposed the addition of a high-level switch that would enable power management globally. This switch would have to be a command-line parameter, since it needs to be set at boot time.

The switch would have the two obvious settings: "performance" and "power." There could also be a setting to enable power-management settings that are still seen as experimental; braver users could turn that on as well.

There were concerns expressed about the boot-time switch, since it cannot be changed in the powertop utility. A few developers suggested that, rather than adding a global knob, developers should make it easier for users to discover and set power-management options. Standardizing the operation of device-specific power-management knobs would help, as would better documentation of the various knobs found in sysfs. Knobs that are known to be experimental or dangerous should be explicitly marked as such; that might give users more confidence when it comes to tweaking the safer knobs.

Darren Hart suggested that a global power-management policy could be implemented in user space if all of the relevant settings were standardized, discoverable, and well documented. Rafael pointed out that the runtime power management knobs are already standardized, but they still tend to remain set in the "disabled" state. Still, the sense of the group seemed to be that better discoverability and documentation were preferable to a global knob implemented in the kernel. So future work is likely to be focused in that area.

Comments (4 posted)

Device dependencies and deferred probing

A "device" as seen by a user of a modern system is, often as not, a set of multiple independent devices working together. This organization is flexible, but it also can lead to problems for device drivers. How does a driver instantiate a new device if the other devices it depends on are not yet available? One answer is deferred probing — delaying the setup of a device until its dependencies show up. Mark Brown led a somewhat inconclusive Kernel Summit session on deferred probing and possible alternatives.

The core problem, Mark said, is that there is no way to tell the kernel's device model about dependencies between devices. Rafael Wysocki responded that the real issue is dependencies between drivers, but Mark disagreed, saying that a dependency shows itself as one device waiting for another specific device to be instantiated. He noted that there is currently a deferred probing patch set under discussion, but it doesn't address the full problem. In particular, it doesn't address suspend/resume operations (when devices must be suspended and resumed in the proper order), and it does nothing to speed boot time.

Grant Likely said that the problem is with the tree-oriented device model, which is not able to describe the dependencies in real-world systems. This point was generally agreed — it has been understood for a while — but solutions are still hard to come by. Greg Kroah-Hartman described deferred probing as a "hack workaround" for the problem, but didn't immediately offer a less hacky alternative.

There was some talk about the nature of device dependencies. Some of them are described in the system's firmware (or device tree). Others only show up when drivers initialize themselves and can't be seen at boot time. Dependencies can even change over the lifetime of the system as devices are configured in different ways. Tim Bird suggested using the phandles (inter-device references) found in the system's device tree; there is a lot of dependency information to be found there. Most dependencies are "stupid clocks and regulators"; creating a graph of such dependencies prior to device probing would solve a lot of problems, he said.

Greg, despite his earlier comment on deferred probing, now asked whether a more complex dependency graph was needed. Nobody has demonstrated real problems resulting from excessive deferral of device probing. Tim said that readily available dependency information could at least be used to perform hinting; Rafael agreed that pulling together as much information as possible would help the kernel to get the probing order roughly right.

As a whole, the group seemed to agree that the problem is real, and that it should somehow be solved in the driver core. An important first step would be to come up with a way to register dependencies from the information that is already available. Rafael promised to post a proposal with some ideas; he followed through later that day.

Comments (2 posted)

Benchmarking and performance trends

Performance regressions can be an insidious problem for kernel developers. Individual changes may not have a serious performance impact, but thousands of changes over a few development cycles can add up. Chris Mason and Mel Gorman have been paying attention to performance regressions for some time; at the 2015 Kernel Summit, they got up to discuss their most recent findings. The short story seems to be that, while there is always room for improvement, there have been few serious performance issues with recent kernel releases.

Chris started by saying that Facebook is now running 4.0 kernels on 30% of its (numerous) systems. The company currently applies about 90 patches on top of the latest stable update — a relatively small number — and there appear to be no serious problems. The size of Facebook's kernel team has doubled in the last year, but it has only added ten patches to the set; the rest have gone upstream.

On the storage side, Chris said, the story is "fantastic." The multiqueue block layer code has made a significant difference, leading to lower latencies, less system time, and good stability. He noted a few starvation issues, but didn't get into them. He also said that multiqueue is not particularly useful on rotating drives, but it makes a huge difference on solid-state drives.

With regard to networking, most of Facebook's internal traffic is IPv6; some recent improvements in IPv6 performance have helped a lot there. The size of the routing cache has been reduced significantly, taking out a lot of memory overhead. The (bufferbloat-related) work to reduce the hardware packet queues on the interface controllers has helped. There might be, he said, room for some improvement with interrupt batching.

Futex performance has been improved by a patch to reduce contention on the internal bucket lock; it led to a 1% reduction in the amount of system time used. That patch is now in the mainline.

For filesystems, Chris said that there have been no stability problems experienced and no performance regressions. Facebook runs a large variety of workloads, but doesn't have any real issues with any of them.

The scheduler, instead, is the source of most problems that they have experienced at Facebook. There are problems in the wakeup code that lead to less-than-optimal CPU use and latencies. They have put a patch into the 4.3 kernel; it improves things for web-server workloads, but problems remain with other workloads.

In particular, Facebook has a workload where a small number of threads run at nearly 100% CPU time, while something on the order of 100 other threads will occasionally wake up to do some small task. This workload sees 10% higher latencies and gets 2-5% less user-mode CPU time with the 4.0 kernel than it did with 3.10. Somehow processes are just not getting to the CPU quickly when they become runnable; it seems that the scheduler is pushing the workload onto too few CPUs, leaving others idle. There is something in find_busiest_group() that is making the wrong decision, he said.

At this point, Mel took over to say that, from his point of view (watching over performance for SUSE), there have not been that many scheduler problems. His biggest complaint, instead, was with the Intel "pstate" driver, which handles CPU frequency and voltage management on Intel processors. This driver, he said, is making poor decisions. CPUs never seem to go above the minimum frequency on lightly-loaded machines, with results that look like a 10-20% scheduler performance regression, but are really due to pstate. This is, he said, a serious issue; we are at a point where we are extremely efficient at doing nothing, but not so good at actually doing work. As a result, a lot of users are disabling pstate altogether.

Mel also noted problems with client/server workloads that involve a lot of synchronous wakeups (one process wakes the other and waits for some result). The kernel would once try to keep both processes on the same CPU, which would reduce latencies, but that no longer happens. There are good reasons for the change in behavior, he said, but there is also a cost. Performance on database workloads, for example, used to be smooth over time; now it is spiky, though the average throughput is about the same. Synchronous wakeups, he said, have historically been a difficult problem.

Distributions are moving toward enabling multiqueue block I/O as the default, he said, but that can lead to performance regressions on rotating media. Jens Axboe noted that multiqueue is currently controlled by a single "on or off" setting. A near-future likely change is the addition of a flag that only enables multiqueue operation on non-rotational storage. The alternative would be to introduce an I/O scheduler for multiqueue I/O, but that would be a bigger step.

Another problem has surfaced within the memory-management subsystem, Mel said. Storage is now so fast that the kernel's memory reclaim decisions no longer make any sense. A lot of decisions are driven by congestion on backing-store devices but, now, there is never any congestion. The result is that the system ends up thrashing. There are a lot of assumptions in the memory-management code that no longer hold true; fixing those could be a long process.

Mel closed with a problem that looks like a kernel performance regression, but isn't. It seems systemd is configuring all system daemons into the same control group, which causes them to all compete against each other under the block I/O controller. It is, he said, a user-space configuration error that can hurt performance. There was some talk about whether it makes sense to put kernel threads into control groups at all, but Peter Zijlstra pointed out that this capability is useful for people who want to isolate some CPUs from system activity. There may be other ways to mitigate this problem from the kernel side, but it's not clear that they are the best solution. Mel warned the group that there may be a fight about this particular issue coming in the near future.

Comments (4 posted)

Realtime mainlining

The Linux Foundation announced in early October that it had pulled together a collaborative project to support work on the realtime kernel patches. Thomas Gleixner, who has been appointed as a Linux Foundation fellow to lead that work, ran a session at the 2015 Kernel Summit to discuss his plans for realtime Linux in the coming year or so.

The first item on the list would appear to be restarting the work to fix the CPU hotplugging code. Adding or removing a CPU is currently handled by "a notifier mess" that is hard to follow and which implements the plug and unplug operations in an asymmetric manner ("asymmetric" meaning that the steps in the plug and unplug procedures do not match up with each other). It is, Thomas said, the biggest mess that is getting in the way of realtime work at the moment. He had posted a plan for a new CPU hotplug subsystem in 2013; that plan will be picked back up and pushed forward.

The new mechanism involves a (symmetric) array of states in place of the notifier-based scheme. Thomas also plans to move as much code as possible out of the architecture-specific trees and into the core kernel; there are, he said, a lot of duplicated and needlessly inconsistent implementations out there now. Most of the callbacks for a new CPU will be moved to the CPU that is coming up; at the moment, they run on the CPU that initiates the hotplug operation while the new CPU sits and waits.

In summary, CPU hotplug is a nightmare for realtime and for the mainline kernel as well. It breaks regularly and gets fixed with another layer of duct tape; nobody can really say how it works. It is, he said, "25 years of duct tape" that needs to be bulldozed and reimplemented from the beginning. He will do exactly that, and hopes not to break anything in the process. While he is at it, he should be able to make many of the state-transition operations able to run in parallel, speeding the whole thing. The new scheme will also make it possible to intensively test each transition between the (many) states, something that cannot be done now. It will, he said, be "a lot of patches."

The next item on the agenda is to finish the timer wheel reimplementation. He has collected a lot of data on how the timer wheel is actually used; timeouts tend to be spread out evenly up to about 1ms into the future. After that, they are widely scattered and irregular. 93% (or more) timeouts added to the wheel never expire. Given that, the expensive cascading work done by the timer wheel now is mostly wasted.

Thomas's approach is to reduce the granularity of timeouts scheduled far into the future. What that means is that far-future timeouts may expire significantly later than intended; as a general rule, this imprecision is seen as being harmless. There was some concern about adding inaccuracy to user-space timers, but those are all implemented with high-resolution timers in the kernel and do not go through the timer wheel at all. Making far-future kernel timers less accurate seems to make a few kernel developers uncomfortable, but nobody pointed to anything that would actually break.

Sometime next year Thomas plans to get around to what he called the "per-CPU big kernel locks" — the overuse of preemption disabling and local interrupt masking. Disabling preemption has obvious implications for realtime performance, since code that has disabled preemption cannot be preempted by a higher-priority process. But disabling preemption has another problem, in that it is often not clear what is being protected. There is no actual lock involved, so a disabling operation is not tied to any particular data structure. That makes the code hard to change with any sort of confidence.

The plan is to add a new kind of lock, which goes by the name local_lock in the realtime tree — Thomas said he was open to discussions regarding better names. These locks can be tied to the data structures they protect; those are per-CPU variables most of the time. In the realtime tree, a local_lock is a real lock; in mainline, it will simply disable preemption as is done now. The end result will be a solution to the realtime issue, clearer locking that can be verified with the lockdep tool, and, perhaps, the eventual ability to check for missing locking with the sparse utility.

As Thomas reached the end of his talk, Andy Lutomirski asked whether the migration-disable approach would move from the realtime tree to the mainline. The realtime tree is often able to disable migration in places where preemption is disabled in mainline, preserving the ability to preempt the running process. But, Thomas said, the immediate plan is not to move that code to mainline; there are other problems to solve first. Andy would like to see it there so that he can avoid creating future work by adding preempt_disable() calls now. Peter Zijlstra, though, said that disabling migration creates problems for the scheduler; he would rather just have processes disabling preemption. So the migration-disable patches seem likely to stay in the realtime tree for a while yet.

Comments (1 posted)

Developer workflow security

Kernel developers, and subsystem maintainers in particular, are in a trusted position: they are able to direct code into the mainline kernel. They are thus a potential target for anybody who would like to get a malicious change into a kernel used by countless people. How can those developers keep their workflow secure? At the 2015 Kernel Summit, a panel consisting of James Bottomley, Kees Cook, James Morris, Konstantin Ryabitsev, Josh Triplett, and Ted Ts'o talked about personal approaches to workflow security and what other developers should be doing.

Ted started by saying that he is not, in general, worried about attackers trying to steal things from his laptop. He is worried about attempts to steal keys, though, so those are stored on a YubiKey rather than the laptop. He tried SELinux for a while, but found it to be too painful for the amount of security gained. With regard to the patches he handles, he said that the ext4 filesystem doesn't get a huge volume of patches, so he is able to look them over closely before sending them on.

Josh, instead, worries about what will happen if his laptop is stolen. So he uses full-disk encryption and, as a rule, shuts the system down hard rather than suspending it. For the most part, though, he does not expect to be targeted personally, so he mostly worries about defending himself from mass attacks. If you download a tarball, configure, and build it, are you doing that in an isolated environment?

Konstantin started by noting that he has root access to kernel.org, so he sometimes sleeps poorly at night. He feels that he probably is a target, and that he needs to be careful to prevent a compromise of the kernel.org infrastructure. To that end, he and his team have been working on a set of security policies that have been made available on GitHub.

James Morris stated briefly that any developer should feel like they could be target; they are developing software for billions of machines, after all. One special measure he takes is to ensure that his cellphone is not connected to his work in any way.

Kees takes care to ensure that his laptop only authenticates outwardly; it does not accept any kind of incoming connection. He does builds in containers to keep them isolated. His only browser is Chrome, due to the way it uses containment for many of its operations. And he tries to get others to look at patches before accepting them.

James Bottomley said that his process was mostly about "key hygiene." He uses subkeys with short expiration times; his working systems have no access to his main keys. With regard to code acceptance in the SCSI tree, he used to review all patches, but now trusts reviews from certain other developers as well.

Linus jumped in to ask how many developers in the room were carrying their main work machines with them; quite a few were. How many were using disk encryption? Those who aren't might want to look into doing so. Does anybody have the SSH daemon running? "Don't do that" was his advice there.

Most of the rest of this session was spent discussing the GPG keys used to sign tags in Git repositories. Somebody noted that a pull request had been accepted, even though it had been signed by a revoked key. Linus answered that key revocation does not always work as well as one might hope, and that revoked keys will often be validated by GPG. Beyond that, he carries his own keyring; a revocation may not be present there. If you revoke a key, he said, scream loudly so that he knows about it. Linus also said that James's use of subkeys was not necessarily helpful. It is, he said, a case of trusting the tool rather than trusting the person.

Finally, there was a brief discussion about accepting pull requests. Some maintainers will pull directly from another developer's repository, while others want to see patches on a public list. It comes down to a matter of trust; don't pull from a developer you don't trust. And, in the end, as Josh noted, somebody has to be reading and reviewing every patch. If patches are being pulled directly into a repository, the person doing the pulling has to be confident that this review has already taken place.

The session ended without any general conclusions regarding safe developer workflow. One suspects this is a topic that will come around again in the future. A lot depends on the security of a large number of developer laptops and repositories; simply hoping that all of those developers will follow best security practices seems like a path to trouble.

Comments (5 posted)

Kernel testing

Automated testing of the kernel has never been as comprehensive as one might like. To an extent, things will always have to be that way, in that many kernel issues are related to specific hardware or workloads that the developers have no access to. But, even with that in mind, it is not hard to believe that there could be more testing done than happens now. At the 2015 Kernel Summit, Shuah Khan and Masami Hiramatsu ran a session on where kernel testing stands and what is likely to happen in the near future.

Shuah started by noting that the kernel's automated testing framework was merged in the 3.17 development cycle. In the last year, the kselftest target has been added to the main makefile, so running the tests is a matter of a single command. There is also an install target that is useful for regression testing or for testing that requires cross-compilation. New tests have been added for the timekeeping, realtime clock, futex, and ftrace subsystems.

Ben Hutchings asked whether developers should expect all of the tests to pass, given that he has been seeing a lot of failures. It turns out that many of the tests are dependent on specific configuration options, but there is currently no way to limit which tests are run based on how the current kernel is configured. So, for many configurations, test failures are indeed to be expected, and it's hard to know which of those indicate real problems. This problem was mentioned a few times during the session.

Andy Lutomirski added that there is one x86 test that always fails. He found the problem after it had been present for several -rc releases. That indicates to him that not many people are running the tests in any regular way; he would like to see that change.

Masami added that a lot of people now depend on Linux worldwide, so we can't afford to ship kernels that do not work properly. He said that the reporting of bugs is often slow, even when the existing tests catch them. Once again, that suggests that the tests are not being run that often. If developers would make a habit of running the tests before sending in a patch, we would introduce fewer of these bugs in the first place. Linus asked whether the zero-day robot is running at least a subset of tests; the answer was that it is indeed doing so.

Kevin Hilman talked about ARM testing briefly. The ARM world, he said, has a lot of "creative hardware designers" that add a unique challenge of their own. He has been collecting a bunch of hardware test scripts and running automated boot tests, with results that can be seen at kernelci.org. The kselftest tests have been added into the mix. Kevin, too, noted that he has run into trouble from dependencies on configuration settings. He would like to see a better structure for the automatic tests, one that makes the dependencies clear. If a proper test-definition framework could be put together, it would make it easier to run the tests on a broader range of hardware and kernel configurations.

The tests, he said, are being run on as many boards as possible. He is still not sending out results, though, because he is still getting a lot of test failures. The tests are evolving quickly, and he would like them to stabilize a bit before he integrates them into his reports. Luis Rodriguez suggested that some sort of "confidence" tag attached to each test could help in deciding which ones to run. In response to a question from James Bottomley, Kevin said that the tests are run each time linux-next changes, since he does not have the hardware to run them with finer granularity.

Ted Ts'o said that the xfstests suite has a way of annotating tests that have been skipped for some reason; it would be nice, he said, if kselftest had that too. As things stand now, there are lots of failures reported, and that can cause real problems to be missed. Tim Bird added that it would be nice to have a mechanism to turn on all the configuration options needed to run a complete set of tests; there is no way to find out what those options are now. When new tests are added, he said, they should include documentation on which kernel configuration options they need.

Andy suggested annotating how long the tests are expected to take, noting that running the full set can take a while. Shuah said that there is a "quick tests only" option now, but it only works when the tests are built and run on the same system. Adding the option to the install target is on the to-do list. Are there requirements for external libraries to run the tests? There are none currently. There may be a need to add external dependencies in the future, but the tests need to fail gracefully if those dependencies are not available.

James asked about driver testing. Drivers tend to be the hardest thing to test, since they are deeply tied to specific hardware. Dan Williams has evidently been working on mocking hardware resources to allow a certain amount of testing; this work has been used for libnvdimm (persistent memory) development. Herbert Xu said it would be nice to have a simple testing package to give to people who actually have specific hardware available.

Johannes Berg said that there is a certain tension between the desire to add new tests and the need to keep their runtime within limits. The WiFi stack currently has a set of 1,600 tests that "takes forever" to run. These tests also exercise the user-space parts of the WiFi stack, so he is not sure they would be appropriate to add to the kernel's self-test suite. Shuah said that short execution time is not a requirement for tests; we would like to have them all in the kernel, though it is important to be able to separate out the fast ones.

Jan Kara described the "test groups" feature of xfstests. There is one group for tests that run quickly, one for those requiring a hard drive, another for tests that might crash the kernel, and so on. Such a structure could be useful for the kernel's tests as well.

The session ran out of time at this point. There is little disagreement about the need for better tests — and the need for developers to actually run those tests. This is an area that should continue to progress quickly in the coming year.

Comments (2 posted)

Security part 2

On the middle day of the 2015 Kernel Summit, when mostly technical topics were considered, Kees Cook urged kernel developers to take security more seriously. The invitation-only group held a followup discussion one day later regarding what should actually be done to improve the kernel's security. It seems there is a possibility that we will see more security-related features going into the kernel in the near future.

Ted Ts'o started by asking the group whether they were willing to add "painful security stuff" now. Kees suggested starting by taking another look at the PaX and grsecurity patch sets. There are a lot of things that can be done, he said; it's mostly a matter of finding people to work on them. He is one of those people, but he's only one person and the problem is larger than that.

James Bottomley asked if there was one security-related technology that the kernel absolutely should support. Kees answered that we should never execute user-space code in kernel mode.

Linus said that it would indeed be useful for somebody to go through the PaX and grsecurity patches. But that code needs to be looked at carefully. For example, they have implemented protections against executing user-space code, but (according to Linus) they have done so badly. Now hardware offers that protection and we can use it; if we had accepted their code, we would be in a deep well we couldn't dig our way out of. He is glad he refused to take it.

Other parts of those patch sets include "slow, horrible code" that affects fundamental parts of the kernel. He called out in particular the fact that grsecurity adds more use of the high-memory concept, while the kernel developers are instead trying to get rid of it. That said, he allowed that there is probably useful stuff to be found there.

Kees said he would like to document the individual pieces of those patch sets, describing the class of bug addressed by each. The easiest to adopt, he thinks, would be the GCC plugin defending against integer overflows. That would bring the plugin infrastructure into the kernel process and allow developers to experiment with it.

Greg Kroah-Hartman said that the Core Infrastructure Initiative (CII) is willing to fund work improving the security of the kernel, but there is nobody out there to actually do this work. What CII won't do is fund grsecurity to go off in a corner and do its own thing (and, he noted, grsecurity isn't asking for that). The problem is that people who can do this work tend to get quickly hired to do something else.

Kees wants to start by overcoming the kernel community's cultural resistance to the addition of security technologies; he feels that some progress has been made in this area. After that, the next challenge is finding people to do the work. James asserted, though, that cultural resistance is necessary; security is a tradeoff, and we have to find the proper middle ground. So the argument will have to continue.

Ted suggested the addition of kernel-configuration options for the more intrusive security lock-down features. We just have to accept, he said, that some of these features will be painful for developers and will often be disabled on developer machines. They are more than acceptable on handsets, though. It was noted that, in enterprise settings, there is often a need for extensive diagnostic information for support purposes. Security features tend to make such information unavailable, so they can't be enabled; given a choice between security and support, many companies will choose support.

Kees said that support might be an area for some innovation. Kernel symbol information could be made available via an encrypted channel, for example, so it would be available to those who have the right key. James said that the checkpoint/restore work is an example of how things can be done. Initially it required the export of a lot of system information, but things have been closed down over time; features like the kcmp() system call allow checkpointing to be done without leaking important information. It should, he said, be possible to get debugging information out of the kernel without helping attackers.

Ted highlighted live kernel patching as an interesting tradeoff; there is little observable difference between the application of a patch and the installation of a rootkit. So live patching may never be enabled in a number of environments. Masami Hiramatsu said that the module-signing mechanism could be used to verify kernel patches.

The session ended with no clear conclusions. There does appear to have been a bit of a shift toward greater acceptance of security enhancements, but the real proof of that will come when the patches start circulating.

Comments (4 posted)

Developer recruitment and outreach

Greg Kroah-Hartman started off a 2015 Kernel Summit session on developer recruitment by noting that the kernel's rate of change hasn't increased much recently. We might have hit a plateau, he said, but still nobody else can possibly keep up with us; "we won." The only thing that can stop us now is ourselves. To avoid that, we need to keep bringing in lots of new developers. That, in turn, means that we need to be nice to our developers. The future of the project, he said, depends on them.

He took a minute to talk about the widespread hostility toward cleanup patches. The best way to avoid getting such patches, he suggested, was for maintainers to just clean up the code themselves; it shouldn't take that much time for them to do. Failing that, maintainers should point contributors who want to do cleanup patches toward the staging tree. Christoph Hellwig complained that a lot of time has been wasted on "horrible" white-space patches, but Greg responded that getting started in kernel development is hard. We should be nice to developers and let them start with an easy patch. Christoph protested that encouraging developers to do the "wrong thing" was not actually nice, but others disagreed. A quick straw poll of the room verified that a number of maintainers have seen developers who start with cleanup patches move on to more substantive work.

The real recruitment problem, Greg said, has to do with finding intermediate-level tasks for developers. When the kernel got started, the holes in functionality were big, there was lots of stuff to work on, and just about anybody could find a useful task to do. Now the holes are small and it's not always obvious what work should be done. To that end, Greg asked maintainers to send their to-do lists to him so that he can make a combined list available to interested developers.

Greg also suggested participating in programs like Outreachy and the Google Summer of Code (GSoC). We all should be looking to replace ourselves, he said. There is a particular need for more Outreachy mentors. Outreachy, he said, has brought in people with good skills, a number of whom are now doing Linux development work at companies. Keith Packard said that the X.org community has worked with both Outreachy and GSoC. The amount of work required to mentor Outreachy interns was too much, Keith said, but GSoC has been great. There was general agreement among those who have done this work that GSoC is an easier place for mentors to start. Ben Herrenschmidt said that a lot of developers are doing this sort of work within their companies.

Linus said that the request for intermediate tasks was ironic, given what he has seen in the community. He is not against white-space patches, though he is not excited by them either. But he gets annoyed when developers accept the white-space patches, then argue against patches that are real improvements. He mentioned one first-time patch posting from a developer who was trying to improve the code that resulted in a frightening email thread. We need to be nicer to developers who are sending real code improvements, he said, and encourage developers doing white-space work to look at the code instead. Ben noted that this was a classic example of the bikeshed problem: there is no shortage of people who will comment on simple patches.

Christoph said that he prefers to see self-driven people coming into our community. They need to have enthusiasm or they won't stick with it. Dan Williams said that we, as a community, are good at killing enthusiasm, be it by not responding to patches or arguing over little details. We should remember, he said, that maintainership is a service role and act accordingly.

The session wound down with a final suggestion that writing tests might be a good task for beginning kernel developers.

Comments (1 posted)

Kernel documentation

Kernel documentation made the list of topics for discussion during the core day at the 2015 Kernel Summit. Your editor, who has been the documentation maintainer for about one year now, led this discussion. Among other things, that means that notes from this session are, well, nonexistent, so this writeup is entirely from memory. If any inconvenient things have been left out, it's purely accidental.

One thing your editor has found over the course of the last year is that it's often not clear where the responsibility for documentation patches lies. Most kernel subsystems are well contained within their own directory subtrees — except that many of them also have files under Documentation/. Some maintainers want to manage documentation patches that relate to their subsystems, while others are happy to leave it to the documentation maintainer. There is a slow-moving effort underway to document these preferences in the MAINTAINERS file. If nothing else, that will help reduce your editor's email load; thanks to the wonders of get_maintainer.pl, he is copied on every patch that touches anything in the documentation tree.

There was a bit of talk about whether it would make sense to split the documentation out into the various subsystem trees, but people seemed to feel that it would make things harder to find. It seems that kernel developers often use the "grep the documentation subtree" technique to search for information.

The bulk of the session, though, was concerned with the structured documentation found in the DocBook subdirectory. A document here starts as a DocBook template file, which is read by the docproc utility to determine which source-code files to extract documentation comments from. Those files are passed to kernel-doc, which, using its own Perl-based C parser, finds all the symbol names of interest and passes them back to docproc. Then docproc invokes kernel-doc again to actually extract the documentation and do some basic markup. The end result is patched into the template file and passed to xmlto for formatting into HTML files, PDF files, man pages, and more.

Various functionalities have been added to this mechanism; 4.3, for example, adds a simple facility for automatically adding cross-references within a single template file. In the end, the kernel community has spent years slowly building up its own special document-formatting system. With all due respect to the people in the room, your editor said, they just might not be the right crowd for that particular project. The whole thing is a bit of a house of cards; one need not look far to find kernel developers who have given up on making this toolchain actually work.

There is a desire to develop things further, though; a current patch set adds the ability to format the in-code documentation as Markdown text. The patches adding this feature are relatively straightforward, but they depend on the pandoc utility to do the Markdown formatting. An attempt to install pandoc on a Fedora system led to a demand to drag in no less than 70 Haskell-language dependencies. Somehow that didn't seem like something the kernel community would be thrilled about.

Your editor's question was: do we want to merge that work, or maybe consider more wide-ranging changes to the documentation toolchain, preferably in a direction that uses more standard tools supported by others? There was little enthusiasm for adding a pandoc dependency, but little consensus otherwise. Linus said that, while he finds the in-code documentation comments useful, he has never seen any real point in building the formatted manuals. As far as he is concerned, that feature could just be removed. There was some agreement with that position, but others seem to find some value in the formatted documents.

The session was not particularly conclusive in the end. There is general agreement that documentation is good, and a certain preference for documentation that is maintained as comments within the code itself. For the most part, the community will continue to muddle along, producing documentation as well as it can with the time that is available.

Comments (17 posted)

Restartable sequences

As computers incorporate more processors, the concurrency concerns that were once mostly limited to the kernel are pushing out into user space. So user-space developers are increasingly wanting to use many of the techniques found in the kernel for concurrency management. Per-CPU variables are of interest, because they avoid contention between processors, but there is a catch: the kernel's per-CPU variables depend on the ability to disable preemption to serialize access — an ability that user space lacks. An alternative approach is thus needed; one such is restartable sequences, which were covered here in July. At the 2015 Kernel Summit, Andy Lutomirski and Paul Turner led a session about whether support for restartable sequences should be added to the kernel.

Some workloads found at Google make use of per-thread free lists for the malloc() function, Paul said. This technique performs well, but it also eats up a lot of memory; that has led to an interest in using per-CPU free lists instead. The idea is to let threads detect if they have been interrupted while in a critical section and, if so, restart their operation from the beginning. Restartable sequences allow this kind of pattern with no locking and with no need for atomic variables. Paul suggested that it might prove useful for realtime developers as well.

Andy then said that he really didn't like the early attempts at support for restartable sequences. He likes it when debuggers work and context switches have sane semantics; the patches ran counter to both of those. He also was not a fan of accessing user-space memory during scheduling. The work has progressed, though, and could benefit from more review. In particular, there is an interest in having non-x86 developers look at the patches to see whether this functionality could be supported on their architectures.

Chris Mason noted that using restartable sequences cuts memory usage by 20% in a workload he has looked at; he described it as "a big deal." David Howells asked what was required from the kernel to support this functionality. Andy's answer was that user space needs to be able to register a critical section with the kernel. If a process is interrupted while executing within that region, it jumps to a specific recovery address when it resumes executing. That recovery code can then do whatever is needed, which usually is a matter of just restarting the operation from the beginning.

Ben Herrenschmidt asked whether it was possible to register more than one critical section; that would be important for libraries to be able to use this facility. The answer was that critical sections can be nested, so library use should be possible.

Paul noted that a new patch series had been posted that morning. Are there, he asked, any objections to the concept in general or to the patches in particular?

Andy responded that he still doesn't like the idea of context switches having side effects. The current patches seem to be getting better in that regard. Josh Triplett noted that restartable sequences could be useful for timing sections of code; Paul agreed, and said they could be used for user-space read-copy-update implementations as well. In general, objections were scarce, but the real proof will be in how and when the patches are accepted.

Comments (6 posted)

Lightning talks

The lightning talks part of the 2015 Kernel Summit schedule was for topics that were not big enough to justify a full session on their own.

Josh Triplett started by talking about the Kernel Tinification project, which seeks to make it possible to build small kernels for deeply embedded applications. He noted that the zero-day robot, which normally concerns itself with compilation and boot errors, is now able to send out messages for patches that unduly increase the size of the kernel. The news was met with general applause.

Arnd Bergmann talked a bit about the work toward solving the year-2038 problem. He has a patch series that simply removes the definition of time_t from the kernel; that has the effect of breaking the compilation of every file that uses it. He can then go through the wreckage and look at fixing the problems; he has about 200 patches accumulated so far toward this end. There are, he said, still 113 drivers in need of fixing. He is working on introducing new system calls to enable 32-bit systems to survive 2038.

Most of the fixes are relatively easy, but there are some hard ones as well. Some subsystems will need user-space changes to survive the epoch. One of those is the input subsystem, which uses 32-bit timestamps; there is a plan in place for that one. A number of filesystems need work, and ext4 will need an extended inode format. Some filesystems, he said, will almost certainly never be extended. Some drivers will never work properly, since they write 32-bit timestamps directly into hardware registers.

Arnd said that, whenever possible, he wants to avoid breaking things. Linus reinforced the importance of that point, saying that, for example, the FAT filesystem should not be disabled even if its times are wrong. In the end, he asked, who cares about FAT timestamps? Similarly, disabling drivers is not acceptable for a production system, though, as a development tool, it can clearly make sense.

Tim Bird had a quick followup to the previous day's session on running mainline kernels on a cellphone. His slides [PDF] included a table of how much out-of-tree code is found in handset kernels. The numbers vary from 1-3 million lines of code. For the configuration running on those devices, out-of-tree code comprises as much as 2/3 of the entire kernel running there.

In Sony's case, there is a massive delta between the kernel it ships and mainline. Starting with a 3.4 kernel, Sony adds 1,799 patches of its own. But that pales next to the 20,395 patches from Qualcomm and the 2,677 from "other." Sony would like to participate more in community development, he said, but its kernel is just too far away from the mainline for that to be practical.

There was a question about how much common code is found in those out-of-tree patches. There is some, evidently, but also lots of duplication of functionality. Thomas Gleixner noted that different trees often have different drivers for the same hardware.

Tim asserted that the mainline Broadcom wireless driver has never been run on production hardware, but Olof Johansson contradicted him, saying that it is shipped on Chromebooks. Tim added that Sony has backported the driver to the kernels it ships and is working on getting the whole thing working. He also said that nobody has ever charged a mobile device using mainline code; here, Sony is working on a USB-charger framework for the mainline.

Comments (none posted)

The stable kernel process

Sasha Levin maintains the 3.18 stable kernel series for Oracle. At the 2015 Kernel Summit, he led a session discussing the process of creating stable kernels in general and how it can be made more robust. The session ended with a surprise decision on the next long-term support kernel, but, first, Sasha wanted to talk about the problem of too many bugs getting through to kernels that are supposed to not acquire new problems.

In particular, Sasha said, the stable trees often end up pulling in commits that, in truth, are not good enough for the mainline kernel in general (but which land in mainline anyway). Then distributors pick up the stable release and ship new bugs to users. It is not clear what can be done about this problem; the stable maintainers cannot, in general, delay fixes for months to see how well they work. He suggested that, maybe, developers could rank commits by both urgency and "scariness." But Greg Kroah-Hartman, who maintains several stable kernels at any given time, said this would not work; it's the "obviously correct" patches that turn out to be broken in the end.

Beyond that, Greg said, what often happens is that developers will revert a patch that turns out to be broken but forget to tell the stable maintainers about it. James Bottomley asked if it might help to avoid shipping stable updates during the merge window when, presumably, most of the buggy patches are merged. Greg replied that he had tried that and didn't see any difference in the results. Buggy patches can show up at any time.

Sasha complained that there is almost no testing of stable updates at all. He would like to work more closely with the distributors and get them to ship "proposed" updates for wider testing. When asked how many regressions he is talking about, he said there were one or two per release. Greg suggested that number was actually pretty good given the number of patches being shipped, but Sasha disagreed. When asked how many complaints he gets, he said one comes in every few weeks, usually for a problem that has already been fixed in the mainline.

Olof Johannson asked whether there should be -rc releases for the stable updates. Greg replied that, with a stable update coming out about once per week, there is simply no time for -rc releases.

Sasha then turned to a recent complaint of Greg's: there are too many stable kernels and too many stable maintainers out there. Greg said that we now have a situation where people don't know which stable kernel to use. Almost nobody uses more than one, and the benefit to the community as a whole is rather small. He said that Debian does benefit from the maintenance of the 3.2 kernel by Ben Hutchings, but that is the exception rather than the rule.

There was a question about whether the next stable release could be announced ahead of time, since that would allow distributors and other users to plan accordingly. Greg said that, in the past, such announcements have led to everybody trying to push crap into the mainline to get it into the stable release. It was suggested that there might be less pressure to do so now, given that most distributions do not use the long-term releases supported by Greg. The system-on-chip trees do use those releases, though.

How about making 4.4 the next stable kernel? It is too late at this point to queue up a bunch of half-baked code to go in, so that could be a relatively safe announcement to make. Announcing the stable release ahead of time could improve the predictability of the whole process. In the end Greg agreed; he has since announced that the next long-term stable kernel will, indeed, be 4.4.

That led to concerns about Greg maintaining too many kernels. It is about time to drop support for 3.10, though; evidently Willy Tarreau will be picking up extra-long-term maintenance of that kernel for those who want it. Some users, it was said, would really like to have 20-year support for their kernels, but volunteers for that task were scarce in the room.

Comments (8 posted)

Is Linus happy?

It is traditional for kernel summits to close with a session about the development process as whole and, in particular, about any problems that Linus sees. The 2015 event was no exception here. The bottom line is that, while there are always things that can be improved, Linus is pretty happy with how things are working now.

He started by saying that, if the topic of the session was truly "is Linus happy?", the group could all just go home. He had no big issues to complain about. He did take a moment to whine that Dave Airlie tends not to use capitalization in his changelog entries, though.

Linus has one small wish that, he said, seems unlikely to ever happen: he still misses having a maintainer for the list of regressions. The process seems to work well enough without that, but it forces him to go by his gut feelings when deciding whether to delay a release or not. Perhaps it doesn't matter, but he would still prefer to have somebody tracking regressions.

The other thing he wanted to talk about is maintainer groups. Most subsystems are maintained by individuals, but, Linus said, the subsystems that are maintained by groups have worked out well. He would like to see more maintainer groups in the future. Having a group helps to maintain coverage when somebody is busy; it also makes it easier for a maintainer to depart when they want to move on to other things.

Maintainer groups would be especially helpful for certain areas where he has been getting complaints. He singled out the SCSI tree as one in desperate need of group maintainership. When a subsystem isn't working well, the complaints end up with him, and that makes him unhappy. The other area where he gets complaints is the virtual filesystem layer. Al Viro is great when he's around, but he tends to vanish. Finding a co-maintainer for Al is not an easy task, though.

Linus was asked whether he, too, should have a maintainer group. The short answer was that he didn't think it was necessary. He is happy to be there seven days a week, and doesn't have trouble taking time off when he wants to. Being the top-level maintainer is not a big stress for him. So he doesn't think a co-maintainer is needed, though he is happy to talk about it if others disagree. Christoph Hellwig suggested that it should happen eventually, but it is not one of the most urgent problems to solve now.

Tim Bird asked about the problem of unresponsive maintainers in low-level parts of the kernel. What can be done to get rid of a slow maintainer? Linus said that this issue has come up a number of times before; sometimes the maintainers eventually just go away. If that doesn't happen, the next step is to just start sending changes to Linus directly. He's not happy when that happens, but it is a workable fallback when the process is otherwise stuck.

Arnd Bergmann talked a bit about how the arm-soc maintainer group works. They (he and Olof Johannson) have a secret mailing list for patches and pull requests. Things pile up there, and sometimes a couple of weeks can go by where neither maintainer processes things. When somebody finds some time, they acquire the queue lock via a message on the IRC channel and crank through requests. They maintain about six branches that go to Linus, and one that feeds into linux-next.

When asked about testing, Arnd said that they do a lot of build testing of patches. Most of the regressions they encounter are not ARM-related, though. Additionally, Olof runs an automated build-and-boot system to test incoming patches.

Thomas Gleixner said that the management of the x86 tree is similar. They (Thomas, Ingo Molnar, and Peter Anvin) have a huge number of branches; people complain sometimes, but all the branches make life easier. The group has an agreement over who owns each branch at any given time. They, too, employ an IRC locking protocol. Ingo runs a continuous-testing setup.

One important practice for a successful maintainer group, it was agreed, was good communication between maintainers allowing them to present a consistent interface to developers.

Christoph complained a bit about the staging tree. He said that it breaks allmodconfig builds, but that problem was evidently fixed a while ago. He also dislikes the Lustre filesystem, which has been in staging for some time now; Greg agreed and said that he would like to delete it. It was generally agreed that the work being done on Lustre is not substantial enough to justify its continued presence. Christoph also said that the use of the staging tree for code that is about to be deleted could be improved; there are, he said, people doing white-space fixes on doomed code.

Linus highlighted one subsystem that always works well for him, despite its being the largest subsystem in the kernel: the networking tree. Dave Miller admitted to being a bit of a control freak who tries to do everything, but he also said that the networking patchwork system helps a lot. Dave doesn't mind sitting in patchwork and checking boxes all day. Linus jokingly took back his praise shortly thereafter, when he found a big pull request from Dave in his mailbox. The joking stopped after the summit was over when that request elicited a classic Linus rant that has been reproduced all over the net.

The final topic was tree-wide changes, which often prove to be disruptive. Linus's answer was to simply not do them; they are a huge pain for everybody. When they really cannot be avoided, though, there seems to be no "best way" to handle them. Thomas suggested doing them quickly and just getting the whole thing over with. Beyond that, though, there were not a whole lot of suggestions.

At that point the 2015 Kernel Summit broke up and the attendees went off in search of a well-earned beer.

Comments (none posted)