Kernel development

Kernel release status

The current development kernel is 3.14-rc8, released on March 24. Linus said: "I delayed things a day from my normal schedule, hoping I'd feel more comfortable doing a 3.14 release, but that was not to be. So here's an rc8, and I expect to do the final 3.14 release next weekend."

Stable updates: 3.13.7, 3.10.34, and 3.4.84 were released on March 23; 3.12.15 came out on March 26.

Comments (none posted)

The 2014 Linux Filesystem, Storage, and Memory Management Summit

The 2014 Linux Storage, Filesystem, and Memory Management Summit was held March 24 and 25 in Napa, California. Nearly 90 developers met to discuss many issues of interest to the core kernel community; it is one of the most focused and technical events on the calendar. Naturally, LWN was there; articles documenting the discussions held there will be added to this page as they become available.

Plenary sessions

Many of the topics to be discussed were deemed to be relevant to all three groups of developers at the event. Those topics include:

• Page cache issues: in particular, how do we support storage devices with block sizes that are greater than the page cache size?

• Persistent memory and how to support it well under Linux.

• PostgreSQL pain points: developers from the PostgreSQL project came to discuss the places where the Linux kernel causes problems for them and what form solutions might take.

• Facebook and the kernel: how Facebook uses Linux and the problems it faces with the kernel.

• Support for shingled drives; the standards for these drives are evolving, and the kernel will need to evolve with them.

• Toward better testing for filesystems and beyond.

• A revoke() update and more: progress on revoke() along with some work on unifying the different read() and write() variants in the kernel.

• Problems with vmsplice() and how they might be addressed.

The memory management track

There were a number of topics discussed in a smaller setting involving just those developers who are interested in memory management issues:

• Trinity and memory management testing: the carnage that results when fuzz testing hits the memory management subsystem.

• Compressed swap, and improving zswap in particular.

• Memory management locking: improving performance around some highly contended locks.

• Hardware pain points: how could processor vendors make life easier for memory management hackers?

• Volatile ranges: more discussion on this often-revised patch set.

• User-space out-of-memory handling: letting user space decide what to do when there is no memory left.

• NUMA placement problems: how are we doing at improving NUMA performance?

• Memory compaction and how to figure out why it occasionally goes wrong.

• Huge page issues: transparent huge pages improve performance in a number of situations, but hurt in others. How can things be made to perform reliably better?

• Memory accounting and limits: fixing memcg soft limits and the accounting of pinned pages.

The storage and filesystem track

For the most part, the storage and filesystem developers had joint sessions, though there is one exception. Here is what was discussed:

• Ideas for supporting shingled magnetic recording (SMR), a user-space API plus some ideas on splitting filesystems into two parts.

• Data integrity user-space interfaces: support for DIF/DIX draws ever nearer.

• Copy offload: a status report on offloading copy operations to storage arrays and file servers.

• Error handling in the SCSI and block layers.

• Thin provisioning is mostly working, but performance issues need to be addressed.

• Block multi-queue status: progress is being made, but there is still more to do.

• Large-sector drives; what to do about drives with sector sizes >4K?

• Direct I/O status: a rewrite is in progress.

Filesystem-only track

There were also two storage-only track sessions, but those got wrapped up into articles above. Here is what the filesystem developers discussed:

• FedFS, NFS, Samba, and user-space file servers: lots of kernel support needed to support federation and user-space file servers.

Group photo

The traditional group photo was a somewhat disorganized affair. Your editor apologizes to everybody who is not visible in this picture.

Comments (1 posted)

Various page cache issues

The Linux page cache is charged with maintaining the cache of blocks of data from persistent storage devices; it is a fundamental component of both the virtual filesystem and memory management subsystems. The page cache is key to the system's performance as a whole. Unfortunately, it is showing its age in a number of ways. The first technical session at the 2014 Linux Storage, Filesystem and Memory Management Summit discussed a number of page-cache-related problems and mapped a rough path toward their solution.

Large drives on 32-bit systems

One looming problem, introduced by James Bottomley, is that, on 32-bit systems, the page cache cannot work with drives larger than 16TB. The page cache uses the native page size (4KB) to address blocks on devices; with a 32-bit block index, the ability to represent blocks is exhausted at 16TB. The questions that James put to the group were: is this problem worth fixing on 32-bit systems, and, if so, how could be that done?

A number of developers clearly felt that there was no need to solve this problem. They see 32-bit systems as being slowly on their way out; anybody who wants to use large storage devices, it follows, should just get a 64-bit system to run them on. In the real world, though, things might not be that easy. There will be 32-bit devices out there for some time, especially in the embedded world. People will get 16TB USB-connected devices and expect them to work. Others will put a 32-bit processor into a cheap network-attached storage device and want to put large drives into it. There are also manufacturers putting 32-bit processors directly onto drives to allow them to support protocols like iSCSI.

As Dave Chinner pointed out, the core problem is in the page cache itself. If an application does direct I/O (which bypasses the page cache), there are no problems. Where things go bad is when user space attempts buffered I/O on a large device. That happens when filesystems are used, but also before that point: udev does buffered I/O to map out drives, for example. And there is one other little problem: even if a 16TB filesystem could be mounted and used on a 32-bit system, there is not enough address space to run a filesystem checker on it. That problem appears to be intractable on these systems. Even if the kernel is fixed, in other words, the rest of the ecosystem is just broken.

These observations led to a rough consensus in the room. There will be no real attempt to make larger filesystems work on 32-bit systems. But the direct I/O path will work with very large drives and will remain supported. So use cases that depend only on direct I/O — the iSCSI drive case, for example — will work on 32-bit systems. For the rest, it will just be necessary to get a 64-bit CPU.

Large pages in the page cache

Dave, along with Christoph Lameter, then moved the session onto the next topic: storing larger blocks of data in the page cache. It turned out that they want to do this for different reasons which are driving them toward different solutions. It may be possible to solve both problems, but one solution is likely to come sooner than the other.

Christoph is worried about operating system overhead at all levels of the stack; part of his solution is to support larger physical pages in the page cache. Larger physical pages would reduce management overhead in the kernel and the amount of setup and teardown associated with I/O operations on those pages. It may be possible to add an "order" attribute to page cache sizes, allowing differently sized (and larger) pages to be stored there, but there is trouble lurking in a familiar place: maintaining the ability to actually allocate large physical pages after the system has been running for a while.

One solution, Christoph said, is to maintain a reserve of large pages for only this purpose, but that is "awkward" and hard to tune correctly. The other approach is to allow the kernel to move pages around, defragmenting things at run time. Much of the support is already there; the memory-management subsystem has the ability to migrate pages, and that ability is used in the page compaction code for memory defragmentation. The problem is that not all pages are movable; since it only takes one unmovable page to break up a large physical page, that is a significant roadblock.

There are a number of reasons why a particular page might not be movable, but one of the most significant causes of unmovable pages is the allocation of kernel-space objects. If objects obtained from the slab allocators were to be movable, though, this problem would go away. Christoph has patches to add this functionality for some heavily-used caches (the inode cache, for example), but they have not been merged. There was some resistance to this idea; the kernel is full pointers to objects; no object can be moved unless all of those pointers can be changed without introducing race conditions. Christoph maintained that much of the problem could be solved by addressing a few important data structures.

Even then, though, things may not work as well as Christoph might like. Mel Gorman made the point that, even on current systems where attempts are made to segregate movable and unmovable pages, a surprising amount of "movable" memory turns out not to be. Page table pages can be a substantial portion of memory, and they cannot be moved; a patch to fix that added 5-6% overhead, and was thus not merged. Other pages are allegedly movable but are pinned for direct I/O or otherwise locked in memory; even when a lot of care is taken, it can be hard to migrate pages. For now, the main symptom of this problem for most users is that transparent huge page allocations fail; that is not a huge problem. But, Mel said, if we come to really depend on being able to move pages, "it will blow up in our face."

Larger blocks in filesystems

Dave had a related but different question to ask. He is interested in better supporting filesystems with block sizes that are larger than the system's page size. Supporting larger physically contiguous pages in the page cache could help toward that goal, and this approach offers some advantages, primarily being that it requires almost no changes in filesystem code to support. But the same cannot be said for the memory management subsystem, where some rather harder changes would need to be made. Rather than jump into all that work, he suggested, we should consider whether we really need to solve the problem that way.

Part of the impetus toward larger physical pages has been limitations in the kernel's block I/O stack; the maximum size of a single I/O request was, for a long time, too small to get full performance out of high-speed storage devices. Raising the page size would raise that limit, allowing more data to be transferred with the same number of pages, fixing the problem. But the request size limit has long been solved, so there is much less need to solve the page-size problem now. The real problem, he said, is that the page cache knows about the block sizes used within each filesystem. It ends up tracking a lot of filesystem-level information, duplicating the information stored in the filesystems themselves. This duplication leads to coherency issues and occasional nasty bugs; it is also the source of the limit that forces filesystem block sizes to be no larger than the system page size.

Nick Piggin tried to solve this problem some years ago with his fsblock work. The problem with that patch set is that it required extensive changes to every filesystem. Even the ext2 conversion, done as a proof of concept by Nick, needed a lot of changes. So Dave has been looking at a different approach. In short, he would like the page cache to stop maintaining a mapping between pages and filesystem blocks; it would, instead, concern itself with the state of the pages themselves. Everything else is already known in the filesystems; getting that tracking out of the page cache would make the block size limit go away.

This work would also enable the elimination of the use of buffer heads for the tracking of blocks. Buffer heads are painful for filesystem developers to work with; they also add a lot of overhead for little value. The mapping of pages to blocks is much better managed in the extent-tracking information already found in the filesystem code. About all the page cache would need to worry about is whether any given page is up-to-date with respect to permanent store; the rest can more easily and reliably be managed elsewhere.

Christoph came back in to point out that he wants the higher-order page cache to reduce application overhead, especially with I/O requests; the filesystem-level solution described by Dave, he said, does not address his problem. Martin Petersen pointed out that smaller I/O granularity may well be forced by host adapters anyway, but Christoph responded that he uses high-end hardware and doesn't have such problems. Dave said, to widespread agreement, that there are two separate problems being described here, and that, in any case, the large filesystem block solution is needed first.

There were some concerns about interactions between the ELF executable format, which has some alignment assumptions built into it, and a larger filesystem block size. It seems, though, that there are no real problems in this area, especially once one gets away from 32-bit systems.

The consensus at the end of the session seemed to be that Dave should push forward with his work to move filesystem awareness out of the page cache. It is, he said, a surprisingly small change, but still a significant bit of work. The changes will be opt-in; nobody expects the numerous older filesystems supported by Linux to be updated, and those filesystems need to continue to work. There were concerns about how much of this work will be in the generic filesystem code; there is no desire to see duplicated implementations copied into each filesystem.

Meanwhile, Christoph was invited to work on supporting larger physically contiguous pages in the page cache. It was made clear, though, that the request for larger allocations would have to be a hint to the allocator; the system cannot depend on those allocations succeeding. In both cases, the next step will be the posting of code for review.

Comments (3 posted)

Support for shingled magnetic recording devices

One of the plenary sessions on the first day of the Linux Storage, Filesystem, and Memory Management (LSFMM) Summit concerned Linux support for shingled magnetic recording (SMR) devices. These next-generation hard disks have a number of interesting characteristics that will be challenging to fully support. Martin Furuhjelm led a discussion among a few drive vendor representatives and the assembled kernel developers about the latest developments in SMR-land.

There are three types of SMR drives: device managed, host aware, and host managed. Device-managed drives will essentially act just like regular disk drives, though the translation layer in the drive may cause unexpected performance degradation at times (much like flash devices today). Existing drivers don't need to change for device-managed disks. The discussion concentrated mostly on host-aware drives (where the host should try to follow the requirements for shingled regions) and host-managed devices (where the requirements must be followed).

SMR drives will be made up of multiple zones, some that are "normal" and allow random reads and writes throughout the zone, and some that can only be written sequentially. For the sequential zones, there is a write pointer maintained for each zone that corresponds to where the next write must go. Depending on the mode, writing elsewhere in the zone will either be an error (in host-managed devices) or will lead to some kind of remapping of the write (for host-aware devices). That remapping may lead to latency spikes due to garbage collection at some later time.

Two new SCSI commands have been added, one to query what zones exist on the drive and another to reset the write pointer to the beginning of a particular zone. To get the best performance, an SMR-aware driver will need to only write sequentially to the sequential zones (that will likely make up most of the disk), but if it fails to do so, it will be a fatal error only on host-managed drives. For that reason, most of the kernel developers seemed to think the first SMR drives are likely to be host-aware since those will work (though perhaps poorly at times) with today's software.

The T10 technical committee (for SCSI interface standards) is currently working on finishing the standards for SMR, so it is important that Linux developers make any concerns they have with the drafts known soon. Ted Ts'o noted that the drafts are available from the T10 site (Furuhjelm recommended looking for "ZBC"). In addition, more information on SMR and Linux can be found in a writeup from last year's LSFMM.

There were some questions about the zone reporting functionality, but much of that is still up in the air at this point. Currently, all zones are expected to be the same size, though there is a belief that will change before the draft is finalized. There has also been talk of adding a filtering capability on the query, so that only zones fitting a particular category (active, full, sequential-only, etc.) would be returned.

The overall sense was that kernel developers are waiting for hardware before trying to determine how best to support SMR in Linux. No major complaints about the draft interface were heard, but until hardware hits, it will be difficult for anyone to determine where the problems lie.

[ Thanks to the Linux Foundation for travel support to attend LSFMM. ]

Comments (26 posted)

Persistent memory

Matthew Wilcox and Kent Overstreet talked about support for persistent memory in the kernel on the first day of the 2014 Linux Storage, Filesystem, and Memory Management Summit held in Napa, California. There have been, well, persistent rumors of the imminent availability of persistent memory for some time, but Wilcox said you can actually buy some devices now. He wanted to report on some progress he had made on supporting these devices as well as to ask the assembled developers for their thoughts on some unresolved issues.

Persistent memory is supposed to be as fast as DRAM, but to retain its contents even without power. To support these devices, Wilcox has written a "direct access" block layer that is called DAX ("it has an 'X', which is cool", he said—it also is a three-letter acronym that is not used by anything else in the kernel). The idea behind DAX came from the execute-in-place (XIP) code in the kernel, not because the data accessed from persistent memory will be executed, necessarily, but because it should avoid the page cache. XIP originally came from IBM, which wanted to share executables and libraries between virtual machines, but it has also been used in the embedded world to execute directly from ROM or flash.

Since persistent memory is as fast as RAM, it doesn't make sense to put another copy into memory as a page cache entry. XIP seemed like a logical starting point, since it avoided the page cache, but it required a lot of work to make it suitable for persistent memory. So Wilcox rewrote it and renamed it. Filesystems will make calls to the direct_access() block device operation in a DAX driver to access data from the device without it ending up in the page cache. Wilcox would like to see DAX merged, so he was encouraging people in the room to look at the code and comment.

But there are a few problem areas still. Currently, calling msync() to flush a range of memory to persistent storage will actually sync the entire file and metadata. That is not required by POSIX and Wilcox would like to change the behavior to just sync the range in question. Obviously that has a much further reach than just affecting DAX, and Peter Zijlstra cautioned that changing sync behavior can surprise user space, pointing to "fsync() wars from a few years back" as an example. User space often doesn't care what is supposed to be done, instead it depends on the existing semantics, he said.

Wilcox said that kernel developers "suck at implementing [syncing], user space sucks at using it" and concluded that "syncing sucks". The consensus seemed to be that any application that was syncing a range, but depending on the whole file being synced, is broken. Furthermore, Chris Mason was all in favor of fixing msync() for ranges as it would "make filesystem guys look good".

Another problem area is with the MAP_FIXED flag for mmap(). It has two meanings, one of which is not very well known, he said. MAP_FIXED means to map the pages at the address specified, which is expected. But it also means to unmap any pages that are in the way of that mapping, which is surprising. Someone must have wanted that behavior at one time, but no one wants it any more, he said. He has proposed a MAP_WEAK flag that would only map the memory if nothing else is occupying the address range.

The get_user_pages() function cannot be used with persistent memory, because there are no struct page entries created for it. There could be a lot of pages in a persistent memory device, so wasting 64 bytes per page for a mostly unused struct page is not desirable. The call to get_user_pages() is generally for I/O, so Dave Hansen has been working on a get_user_sg() that create a scatter-gather list for doing I/O. The crypto subsystem also wants this capability, Wilcox said.

There is a problem, though. A truncate() operation could remove blocks out from under get_user_sg(), which would leave a mess behind. Wilcox wondered if file truncation could just be blocked until the pages are no longer pinned by the I/O operation. That did not seem popular, but Overstreet had another idea.

Overstreet has been working on a direct I/O rewrite for some time and, in many ways, doing a DAX mapping and a direct I/O look similar, he said. His rewrite would create a new struct bio that would be the container for the I/O. It would get rid of the get_block() callback, which is, he said, a horrible interface. For one thing, it may have to read the mapping from disk, which should be asynchronous, but get_block() isn't. Moving to struct bio would allow the usual block-layer filesystem locking to avoid the truncate().

There were some complaints that making I/O be bio-based was problematic for filesystems like NFS and CIFS that don't use the bio structure. Overstreet said that we may get to a point where buffered I/O lives atop direct I/O, which would help that problem. In addition, Mason did not think that a bio-based interface would really be that big of a problem for NFS and others. A bio is just a container of pages, Overstreet said.

In the end, no really clear conclusions were drawn. It would seem that folks need to review the DAX code (and, eventually, Overstreet's direct I/O rewrite) before reaching those conclusions.

[ Thanks to the Linux Foundation for travel support to attend LSFMM. ]

Comments (10 posted)

Trinity and memory management testing

The Trinity tool is a system call fuzz-testing utility for the Linux kernel. By supplying random data to the kernel in a focused way, Trinity has managed to expose a large number of bugs over the years. Trinity maintainer Dave Jones addressed the memory management track at the 2014 Linux Storage, Filesystem, and Memory Management Summit to talk about how he is using the tool to turn up memory management bugs in particular.

He started by describing an idea he heard from Al Viro: create a memory range with mmap(), unmap a single page in the middle of that range, then pass the result to various system calls and see what happens. As he described it, "all hell broke loose." Large numbers of bugs have been turned up, followed by "heroic efforts" in the memory management community to fix them. As a result of those efforts, he said, he is now unable to find any problems in the 3.14-rc7 kernel's memory management code, which is a good thing.

On the other hand, Sasha Levin has been using Trinity to find bugs in the linux-next tree, where they are rather more plentiful. Those bugs often move into the mainline during the merge window. Bringing more stability to the code in linux-next before it is merged would be a worthwhile thing to do.

In general, Dave said, Trinity is good at finding bugs in the dark corners of the kernel that nobody makes much use of. So areas like huge pages, page migration, and the mbind() system call have been fertile ground. In the case of mbind(), it turned out that all callers were going through a user-space library. That library did argument checking, so, naturally, the system call itself did not. The result was a predictable pile of bugs which have now been fixed.

Lots of parts of the memory management subsystem are, he said, simply not getting adequate testing now. Trinity helps in this area, but its memory management testing is still on the rudimentary side. He wants to develop it further; he plans to work on memory management fuzzing for much of the rest of the year. But, even now, Trinity is finding more bugs in the memory management code than can be dealt with.

Huge pages generate lots of bug reports from Fedora users; they also are the source of lots of problems found by Trinity. Reproducing those bugs in any more standard setting is hard, though; many of them involve applications, like the Java runtime, that Dave is unfamiliar with and uninterested in learning more about. So, for now, transparent huge pages are simply turned off for his Trinity runs.

Reproduction of crashes provoked by Trinity is an ongoing problem in general. The tool can log everything that it does, but the logging is, itself, an expensive operation that can change timings to the point that a lot of problems simply go away. Many crashes are also the result of corrupted internal state in the kernel; the sequence that causes the corruption may happen a long time before that corruption causes the kernel to crash. So establishing the cause of crashes can be difficult.

Dave had a couple of requests for memory management developers. One was for anybody adding new flags to existing system calls; when that happens, he would like to get a note so that he can start testing calls with that flag. He'll often notice them in the patch stream anyway, but an explicit notification is more reliable. The other thing he would like is to see more developers running Trinity on their systems. It is trivial to set up, he said; so there is no real reason not to make use of it.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (6 posted)

Compressed swap

There are a number of projects oriented around improving memory utilization through the compression of memory contents. Two of these, zswap and zram, have found their way into the mainline kernel; they both aim to replace swapping with compressed, in-memory storage of data. They differ in an important way, though. Zram acts like a special block device which can be mounted as a swap device; zswap, instead, uses the "frontswap" hooks to try to avoid swapping altogether.

Bob Liu led a session to talk about this technology with a specific focus on zswap and a performance problem he has encountered with it. Zswap stores "swapped" data by compressing it and placing the result in a special RAM zone maintained by the "zbud" memory allocator. When the zbud pool fills, zswap must respond by evicting pages from that area and pushing them out to a real swap device. That involves decompressing the data, then writing the resulting pages to the swap device. That can slow things down significantly.

Bob had a couple of options that he asked the group to consider. One of those was to turn zswap into a write-through cache; any pages stored in zswap would also be written to the swap device at the same time. That would allow the instant eviction of pages from zswap; since they already exist on the swap device, no further effort would be required. The cost, of course, would be in the form of increased swap device I/O and space usage.

The second option would be to make the zswap area dynamic in size. It is currently a fixed-size region of memory. If it were dynamic, it could grow in response to increased demand. Of course, there would be limits to that growth, after which it would still be necessary to evict pages from the zswap area.

Bob may have hoped for guidance with regard to which direction he should take, but he did not get it. Instead, Mel Gorman made the point that neither zram nor zswap has been well analyzed to quantify the benefits they provide to a running system. When people do run benchmarks, they tend to choose tests like SPECjbb which, he said, is not well suited to the job. Or they pick kernel compiles, which is even worse.

What the compressed swapping subsystems really need, he said, is better demonstration workloads. In fact, they need those workloads so badly that no changes to the behavior of these subsystems will be considered until those workloads have been provided. So the real next step for developers working with compressed swapping is not to worry about how the system responds to pool exhaustion — at least, not until a better way to quantify the performance impact of any changes has been found.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (6 posted)

Memory management locking

Like many parts of the core kernel, the memory management subsystem is highly sensitive to lock contention, which can quickly ruin performance. Davidlohr Bueso has been working on fixing some locking problems in that code; he led a session in the memory management track of the 2014 Linux Storage, Filesystem, and Memory Management Summit to talk about the directions for that work.

Two locks that show contention problems are the anon_vma lock, which controls access to virtual memory areas representing regions of anonymous pages, and the i_mmap_mutex, which protects several fields of the address_space structure. These locks were once mutexes, but the read-mostly access patterns for those data structures led developers to switch them to reader/writer semaphores (rwsems) instead. The only problem is that performance took a significant hit whenever it became necessary to do a non-trivial amount of writing to those data structures.

Some of that performance was regained through the application of "rwsem stealing," whereby a thread that is running can grab a lock ahead of another thread which had been waiting for it. But, Davidlohr said, what was missing was the sort of adaptive spinning found in regular mutexes. Even though a mutex is a sleeping lock, a thread trying to acquire it may spin for a while in the hope that the lock will be released soon; doing so can yield a significant performance boost. Adding spinning to the rwsem implementation gets performance back to previous levels for all workloads. So, Davidlohr asked, is there any opposition to merging that code? The response in the room suggested that no such opposition exists.

In the case of the anon_vma lock, there is a strong desire to avoid using a sleeping lock at all. The rwlock mechanism exists for just that use case, but there are fairness issues with rwlocks. Waiman Long has done some work with queued rwlocks, which address those issues while also improving performance. Peter Zijlstra noted that he has rewritten those patches, but is not quite sure what to do with the results. He likes the fairness, but lacks good benchmarks by which to judge them. There are still problems using these locks on virtualized systems. Even so, he is not opposed to merging this code.

Not everybody feels quite the same way, though. Sagi Grimberg noted that he has code that needs to be able to sleep in functions like invalidate_page(), where the anon_vma lock can be held. So turning that lock into a non-sleeping lock would clearly create problems. This kind of need comes up in areas like InfiniBand and RDMA, where work that can potentially sleep has to be done in settings where this lock is held. Mechanisms like xpmem also have this problem.

Rik van Riel suggested that the best way to avoid problems is to get the relevant code upstream as soon as possible, but Davidlohr protested that the performance cost of using a sleeping lock is severe. Peter added that Linus has had "choice words" for authors of code needing a sleeping anon_vma lock. So, he said, the right thing to do with the non-sleeping lock patches would be to send them to Linus with an explanation of what would break if they were applied. Then we could all see what Linus chooses to do.

Davidlohr went on to say that he would also like to restart the discussion of the mmap_sem semaphore, which protects many parts of a process's address space. It is often held for too long, he said, creating excessive latencies. We are also serializing too much work. It is not necessary, he said, to lock the entire address space if work is being done on a portion of that space. Perhaps it is time to look at range locking as a way to reduce mmap_sem contention?

Michel Lespinasse responded that, while range locking might make sense, it would be better to work on eliminating long hold times for mmap_sem. Rik suggested that, perhaps, it could be turned into a per-virtual-memory-area lock, but Peter responded that this has been tried in the past. The patches have ended up replacing mmap_sem contention with contention for a "big VMA lock" instead.

Jan Kara raised the problem of holding mmap_sem when the memory management subsystem calls into filesystem code. Beyond performance problems, this pattern can create lock inversion issues as well. He has been working on eliminating the places where mmap_sem is held for filesystem calls for some time; that work is getting closer to being ready. There are just a couple of remaining problem areas, one of which is the page fault handling code, but there are solutions to that problem.

The other is calls to get_user_pages(), which requires that the mmap_sem be held by the caller. Jan has been converting callers to get_user_pages_fast(), which does not have that requirement. Most of the easy cases have been handled, but a few of the harder issues remain. Sometimes get_user_pages() is called in situations where mmap_sem has been acquired by higher-level code. The Video4Linux videobuf2 code has some interesting usages of its own which are hard to convert.

But the most worrisome area is uprobes, which needs to be able to place breakpoints into program code (text) pages when they are brought into memory. This code registers a callback on the creation of virtual memory areas; if need be, it installs breakpoints into text pages when they are instantiated. This results in a call into filesystem code from well within an mmap() call. Peter suggested that this one could be fixed by reordering the mmap() code. The initial setup work could be done, after which mmap_sem would be dropped and the page contents could be filled. That would create a window where a program might be able to access some of the mapped pages before their initialization is complete, but, Peter said, no well-behaved program will access pages created by mmap() before that call returns, so there should be no problems.

The session ended with some inconclusive discussion on rationalizing the naming of the growing family of get_user_pages() variants.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (2 posted)

Hardware pain points for memory management

H. Peter Anvin ran a brief session at the 2014 Linux Storage, Filesystem, and Memory Management Summit to ask a simple question: how could hardware (and processors in particular) improve to make the memory management task easier? While he offered no guarantees that any actual hardware changes would result from the discussion, he did say that he would be able to carry any requests back to the hardware people at Intel.

The first complaint had more to do with hardware-specific software in the kernel: Rik van Riel noted that the PowerPC architecture code does not implement the translation lookaside buffer flush functions. Some other architectures (such as SPARC) have similar limitations. That makes it hard to do the right thing in generic code. It would be nice, he said, if something could be done to make it easier for architecture-independent code to update page table entries.

Peter Zijlstra asked for a way to invalidate a range of page table entries on the x86 architecture. Another popular request for x86 was the ability to support 64KB pages. Currently, on that architecture, there is no hardware page size between 4KB and 2MB.

Mel Gorman asked for a fast operation to zero-fill a page of memory. This ability could be especially useful for huge pages, which can take a while to overwrite with zeroes. There was some talk about whether non-temporal stores (which can overwrite memory without pushing other data out of the processor caches) would be helpful in this situation. Somebody suggested zeroing pages in the kernel's idle loop, when nothing else is going on, but Christoph Lameter responded that he has tried that and it does not really help.

Other requests included a version of the iret instruction that is less painful (faster) for the page fault handler. There was talk of the cost of responding to events and passing messages between CPUs; a version of the mwait instruction that works in user space was suggested as being possibly helpful. The end result of the session was a wishlist to be taken back to the hardware developers; what will come of that remains to be seen.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (1 posted)

Volatile ranges

"Volatile ranges" are special regions of memory containing data that the owner application can regenerate if need be. If the system runs short of memory, the kernel is free to evict data from a volatile range, but otherwise the space is usable for activities like caching. The volatile range concept was raised again at the 2014 Linux Storage, Filesystem, and Memory Management Summit, in two separate sessions. This article is a combined look at both discussions.

The first session started with an overview of the latest incarnation of the volatile ranges and MADV_FREE APIs; see this article for an overview of those proposals. One question that came up repeatedly concerned the need for a separate vrange() system call for volatile ranges. Some of the incarnations of that work have used madvise() instead, and some developers think that is the better approach. It turns out that one of the biggest arguments against an madvise() interface has to do with the process of marking pages as no longer being volatile. In that case, the system call needs to return two separate values: (1) how much memory was successfully marked non-volatile, and (2) whether any pages were purged by the kernel while they were marked as volatile. madvise() only allows for one return value, so it cannot be used to create that kind of interface.

Should the interface indicate which pages have been purged when a range is marked non-volatile? The current code returns a single boolean value indicating only whether any pages have been purged at all. Hugh Dickins said that some users would like to have more detailed information. That said, there does not appear to be any plan to extend the interface in that direction at this point.

Another question has to do with page aging. When pages are marked as being volatile, should they be "aged" to look like they have not been referenced for a long time? Aging the pages in that way would cause them to be among the first that are reclaimed if the system encounters memory pressure. There does not seem to be much consensus on whether this kind of aging should be performed; if it is added, it might be under the control of a separate flag allowing user space to select the behavior it wants.

Hugh said he didn't like the vrange() name; he would rather see the name be a verb describing the action that is to be performed. There was also talk of making an madvise2() system call that would be able to provide the needed API. In the end, though, suggestions for better names have been in short supply, and Hugh agreed that, given all the revisions that volatile ranges have been through, keeping that functionality as a separate system call might be the best approach to take.

Keith Packard raised a related use case that he has: graphics drivers can allocate large amounts of memory for caching that they can give up if need be. But the existing shrinker interface is not actually invoked often by the kernel, so he ends up holding memory rather longer than is warranted. Perhaps the volatile range functionality could be made available in a form that could be used by drivers as well?

A couple of other API issues came up toward the end of the session. One had to do with what happens if a process writes to memory that is in a volatile range: in that case, should the memory remain volatile, or should writing the memory automatically make it non-volatile? Some developers would like to see the latter behavior, but John Stultz, the author of current versions of the patch, is uncomfortable with changing the state of pages on writes in that way.

The current interface is memory-based, in that a volatile range is described by a base address and a length. Some versions of the patch have, instead, used a file-based interface, where a volatile range is described as a portion of a file. The Android "ashmem" subsystem, which, it is hoped, can be replaced by volatile ranges someday, uses a file-based interface, but John said that it could be changed internally to use a memory-based method instead. Keith had a bit of a stronger requirement for a file-based interface, though. The graphics system, he said, does not normally have addresses for most of the memory it uses for caching, and mapping all of that memory could create problems on 32-bit systems where there is not a lot of address space available. So he would rather see a file-based API.

In the end, there was little in the way of concrete conclusions from this session. There will certainly be another version of the volatile ranges patch set at some point, but what it will look like is not entirely clear.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (1 posted)

Toward better testing

Dave Chinner and Dave Jones started off the second day of the 2014 Linux Storage, Filesystem, and Memory Management Summit with a discussion of testing tools. What are we doing now, and what can be done better? The state of the art has improved considerably in recent times, but there are always ways to do better yet.

Dave Chinner spoke as the maintainer of the xfstests filesystem testing suite. Despite its name, this suite has not been specific to the XFS filesystem for some time. There are, he said, more people now who are both using and contributing to xfstests, but there is still room for improvement. When a developer finds a filesystem bug, he said, the fix should include a contribution to the test suite to help ensure that the bug does not return in the future.

James Bottomley asked how much code coverage is provided by xfstests now. It seems that the quality assurance people at Red Hat have done some coverage testing; about 75% of the code in the XFS filesystem is exercised by xfstests. Coverage of ext4 is a bit less at 65%; there are currently no tests to exercise the ioctl() code in particular. In general, the common code paths are tested well, but the more esoteric features lack test coverage.

There was a request for the addition of power-failure testing to xfstests. Dave responded that there is a "crashme" script in xfstests now that can be used to randomly reboot the machine; XFS also has a special ioctl() that will immediately cut off the I/O stream, simulating a power failure on the underlying device. So, he said, there is no need to physically remove power to do power-failure testing; it can be done with the software tools that exist already.

Al Viro said that some tests will fail if the underlying storage partition is too small. Dave replied that there is a mechanism in the xfstests harness to specify how much space each test needs. In general, the minimum amount of space is 5-10GB; with that, most of the tests will run. At the other end, he runs some tests on a 100TB device, though, he noted dryly, it is wise to avoid any tests which need to fill the entire filesystem when working at that scale. Al also said that some tests can fail after thousands of operations; it would be nice for debugging to be able to replay an xfstests log and quickly zero in on places where things fail.

In general, Matthew Wilcox said, it is not always easy to figure out why a specific test failed. Dave responded that this situation may not change; the purpose of xfstests is to alert developers that a bug exists, not to actually find that bug. He did say that he would accept patches that provide more hints to developers, but that there is also a reluctance to go back and change existing tests. It is easy to break the test itself, sending developers scrambling to find a filesystem bug that does not actually exist. Things are bad enough even without changing the tests, he said: every couple of years the GNU utilities developers feel the need to change the formats of error strings, causing problems in the test suite.

Zach Brown complained that the discussion was focusing on details, when the most significant resource we have is the fact that Intel is paying people to put together testing infrastructure and actually run the tests on development kernels. Now, when developers introduce a bug, they will often get an automated email informing them of the fact. That is good, since, he said, the xfstests suite is painful to set up and run.

Dave Jones asked if we need a similar test suite for the storage layer. Ric Wheeler responded that storage vendors have such suites, but those suites tend to be kept private. Mike Snitzer has a test suite for the device mapper; among other things, it helped to find problems with the recently merged immutable biovec work. When asked why this tool isn't more widely used, Mike responded that the fact that it is written in Ruby might have something to do with it.

Another developer expressed a desire to coordinate filesystem tests with outside processes; the objective in particular was to create more memory pressure while the tests are running. Dave Chinner agreed that more testing should be done under memory pressure. Dave Jones suggested that the fault injection framework could be used; Dave Chinner agreed, but noted that fault injection, while exercising error paths, does little to exercise the reclaim paths in the kernel. So there is no substitute for real memory pressure. A program found in xfstests now will lock large amounts of memory into place, providing an easy way to add memory pressure to the system.

Moving beyond xfstests, Dave Jones asked the community what kinds of tests are missing in general. James immediately responded that we need better ways of testing for performance regressions. Mel Gorman added that the community is "completely blind" when it comes to I/O performance. He has added some simple I/O tests to the mmtests suite and found some regressions in that area almost right away. But, he said, having the test is not enough, some kinds of problems require looking over the results in a detail-oriented fashion. Performance regressions may manifest themselves as latency spikes that have little effect on overall throughput numbers.

Dave Jones recounted that, during the 3.10 development cycle, RAID5 was broken through the development cycle from the merge window until just before the release. Somebody, he said, should have found the problem sooner. It is also easy, he said, to bring down the kernel when assembling block devices with the device mapper. Developers, he said, simply are not trying to test a lot of this code in any sort of regular way.

Ted Ts'o suggested that not enough developers have come to understand the deep sense of relief that comes from knowing that a set of changes has passed all of the regression tests. He wished he knew of a way to package that feeling and sell it to new developers. In the absence of that ability, he said, maintainers should do more yelling at developers who clearly have not run the available tests on their patches. Once a culture of regular testing sets in, it tends to become persistent.

Dave Jones complained that, while we sometimes write tests for problems that have been experienced, we are not so good at proactively writing tests for functionality that might break sometime in the future. Dave Chinner agreed, saying that the quality assurance organizations run by distributors should really be writing more tests and trying to break things. In most organizations he has worked with, that kind of outside testing is the norm, but we don't do much of it in the kernel community. Developers, he said, tend not to break their own code well enough; we really need outside testers to find new and creative ways to break things.

As the discussion wound down, there was some talk about areas that do not have good tests now. The filesystem notification system calls were mentioned. Some of the more obscure memory management system calls — mremap() or remap_file_pages() for example — don't have much test coverage. More test coverage for the NUMA memory policy code would also be helpful. Developers may eventually write these tests; hopefully others will then run them and let the community know when things break.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (3 posted)

User-space out-of-memory handling

While opinions on how the kernel should respond to out-of-memory (OOM) situations vary, almost everybody seems to agree that what the kernel does now is in need of improvement. A session on the topic during the memory management track at the 2014 Linux Storage, Filesystem, and Memory Management Summit covered some possible improvements, but reached no real conclusions.

David Rientjes used the session to talk about his user-space OOM handling patches and to ask for a green light for their inclusion. He spent a while talking about how these patches work; this introduction can be found in David's article on the subject and will not be repeated here. David has been pushing this work for the last year or so, but it seems clear that the community is still not completely sold on it.

Sasha Levin asked whether it might be better to use the vmpressure mechanism, which sends notifications when memory is getting tight, rather than waiting for a full OOM situation and hoping that user space can handle it. The problem with that approach, as Rik van Riel put it, is that there is no limit to how quickly a system can consume its memory. David added that the vmpressure mechanism does not work as well as one might think. As an illustration of the problem, consider a process that locks many pages into memory; it will consume much of the available memory, but no pressure notifications will result because no reclaim is yet happening. The system can then go from a "no pressure" state to "out of memory" almost instantaneously once reclaim starts; there simply is no opportunity for user space to respond.

As the discussion went on, it became clear that the most discomfort existed around the use of a user-space handler to deal with global OOM situations. If a single control group under the memory controller (a "memcg") runs out of memory, it makes sense to have a user-space handler respond. But, Michal Hocko asked, do we really want to handle global OOM situations (where the system as a whole is out of memory) in user space? He agreed that the current code does not work for everybody, but, he said, pushing responsibility into user space opens up a can of worms and would be hard to maintain in the long term. It would be better, he suggested, to improve the global OOM killer in the kernel instead.

Tim Hockin, speaking about his work at Google (which has driven the user-space OOM handler development), talked about the problems they have had with OOM-handling requirements that have changed over time. Google has a hard time deciding what it wants to have happen in OOM situations; it seems hard to expect the kernel developers to anticipate where those requirements might go in the future. That has led to the desire to push the policy into user space where it can be changed without the need to build and deploy a new kernel — a process which does not happen quickly at Google. He would be happy with an in-kernel mechanism that allowed policies to be changed, but only if it is possible to effect a change without building a new kernel.

Robert Haas agreed that moving the policy into user space gives users the ability to make changes without having to change the kernel itself. Kernel developers, he said, simply are not smart enough to come up with all possible policies. But David said he was willing to try if that was how it would be done, though he suggested that the community might not be happy about the "hundreds of patches" implementing all of the possible policies that would result.

There was also some unhappiness about David's use of the memcg mechanism for global OOM handling. That mechanism will only work if control groups are built into the kernel, but there are still plenty of users who prefer not to enable control groups at all. The motivation for using that interface was to allow per-memcg and global OOM handlers to work with the same interface and be coded the same way. Peter Zijlstra suggested that the same control files could be placed in /proc for global OOM handling, providing something very close to the same interface without needing to enable control groups.

David asked for some guidance on how he could make progress in this area. It has been hard to get a consensus on his user-space OOM handling patches, but no viable alternatives have come forward. So he is somewhat stuck. Unfortunately, no consensus emerged in this session either, so there is still no clear path forward for this project.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (7 posted)

NUMA placement problems

The kernel's handling of task and memory placement has been the subject of a lot of discussion and development in recent years. The pace has slowed for the last few development cycles, but there is still work to be done in this area as can be seen by the discussion on the topic that was held at the 2014 Linux Storage, Filesystem, and Memory Management Summit. The session, led by Rik van Riel, Peter Zijlstra, and Mel Gorman, started with the question: lots of code has been merged, what should happen next? Peter made the observation that, while the code is in the kernel, few people have actually tried to take advantage of it to improve NUMA performance. What is most needed now is user feedback on how things are working and what could be improved.

Davidlohr Bueso said that, on his systems, he can still get much better performance from a carefully hand-tuned configuration than with the automatic NUMA placement code. Rik added that, as far as he can tell, things work close to optimally on four-node systems, but tend to fall apart on systems with more nodes than that. Mel asked why that might be; there was some speculation that the costs of page hinting (tracking who is using each page of memory so that it can be moved to the right node) might be responsible, or perhaps the more complex topology of larger NUMA systems is not being handled well. But it seems that nobody really knows what the problems are.

Mel said that truly understanding NUMA performance issues requires the collection of a lot of data. But that data collection is expensive, to the point that it disrupts the workload under study. It's hard enough for him to run his tests; he hasn't really found a good way for others to do it yet. It seems that Rik, Peter, and Mel each have their own way of measuring NUMA performance; they haven't done much talking among themselves in this area. That is, it was suggested, actually a good thing; each developer is able to find different problems with his particular approach.

Rik noted that, while the NUMA code tries hard to keep anonymous pages on the same node as the processes using them, the same care is not yet applied to page cache pages. His question was: should it be? It is not clear that localization of the page cache would lead to better performance overall; Mel said that this area is pretty much ignored for now.

Johannes Weiner said that a node-local page cache allocation policy might not make sense. If the system tries hard to allocate those pages locally, it could do so at the expense of pushing other useful pages out. At that point, the kernel is buying local pages at the expense of forcing disk I/O for other needed pages — probably not a good bargain. Currently, page reclaim is strongly tied to nodes, so some nodes can reclaim heavily while old pages languish on others. So, he said, it might be good to force some page aging on all nodes even if there isn't memory pressure everywhere. Then interleaving page cache pages across all nodes might be able to increase memory utilization and reduce the aging of useful pages, a win even if it results in more cross-node traffic.

There were complaints that processes communicating over network sockets should be grouped onto the same node, but that doesn't happen now. There is seemingly a bit of a disconnect with the networking developers on how that kind of grouping should be done. There would also be value in moving network-oriented processes onto the NUMA node that holds the network adapter they are using, but there is no I/O awareness in the NUMA code at all currently. Improving the integration of networking and NUMA placement is not going to be an easy task; it will likely involve carrying NUMA information through many layers of the network stack.

The session wound down without a lot in the way of hard conclusions. It seems clear that there is still a lot of work to be done in the area of NUMA placement.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (1 posted)

Memory compaction issues

Memory compaction is the process of relocating active pages in memory in order to create larger, physically contiguous regions — memory defragmentation, in other words. It is useful in a number of ways, not the least of which is making huge pages available. But compaction apparently has some problems of its own; Vlastimil Babka led a brief session in the 2014 Linux Storage, Filesystem, and Memory Management Summit to explore the issue.

After Vlastimil gave a quick overview of how compaction works (also described in this article) and described problems related to compaction overhead, Rik van Riel made the claim that there are two core issues to be looked at in this area: (1) can the compaction code be made to be faster, and (2) when compaction appears to be too expensive, should it just be skipped?

It seems that a number of compaction bugs have been fixed over the years, but some clearly remain. How, it was asked, can they be made easier to find? Writing test programs that reveal compaction problems tends to be hard; these problems arise out of specific workloads that exercise the system in certain ways. There does not appear to be any easy way to abstract the problematic access patterns out of the workloads into separate test programs.

What that means is that the memory management developers don't really have a good understanding of why compaction problems are happening. Some workloads obviously create situations where compaction gets expensive, but how that happens is obscure. So there is clearly a need to gain a better understanding of how the problems come about. One step in that direction might be to add a new counter that is incremented anytime the kernel detects that it has spent a significant amount of time in the compaction code. If that counter starts to increase, that will be a signal that bugs in the compaction code are being tickled. Then, perhaps, it will be possible to try to figure out where those bugs are.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (1 posted)

Huge page issues

Using huge pages can improve performance on a number of workloads, mostly through decreased paging costs and better translation lookaside buffer usage. But supporting huge pages imposes costs of its own on the kernel. The memory management track of the 2014 Linux Storage, Filesystem, and Memory Management Summit set aside some time to talk about those costs and how they might be reduced.

Aneesh Kumar started out by saying that, on architectures with larger page sizes (such as PowerPC), users tend to disable the transparent huge pages feature because it causes performance problems. Those problems might result from the fact that, when normal pages are larger (64KB, for example), huge pages are also larger. They may be getting large enough that internal fragmentation is a concern. One possible solution might be to split huge pages when they are swapped out of memory. That might reduce the amount of I/O required, especially if pages filled with zeroes can be skipped during the swapout process. There might also be some gains to be had by disabling the allocation of huge pages during page fault time, leaving it to the khugepaged daemon to assemble huge pages later on.

A question arises on NUMA systems: if a huge page cannot be allocated locally, is it better to allocate a remote huge page or a local small page? The benefits from using remote huge pages seemingly do not outweigh the costs of doing so. There was some confusion on this issue; Rik van Riel thought that falling back to small pages locally should already be the default course of action. Matthew Wilcox suggested that, if a page must be allocated remotely, it should always be a small page; huge pages should only be allocated on the local node.

There was also some talk of adding a new heuristic that would disable the allocation of huge pages in situations where memory is highly fragmented. When fragmentation happens, it might well be better to try to reduce overall memory usage by sticking with small pages. Once again, khugepaged can collapse things into huge pages later if the resources become available.

Peter Zijlstra had a different suggestion: take the transparent huge page mechanism out of the kernel entirely and leave the associated headaches behind. Andrew Morton agreed that transparent huge pages have "made a mess of the kernel." Hugh Dickins expanded on that thought, noting that the memory management subsystem as a whole has gotten significantly more complex, and that the transparent huge page feature is a big part of the problem. It is a feature that does benefit some workloads, and, he said, it was a magnificent technical achievement. But the way transparent huge pages rely on memory compaction, extensive reference counting, and complex code are downsides to the feature.

Rik noted that much of the complexity comes from the feature having been retrofitted onto the existing memory management code; it may be time to look at simplification through extensive rewriting of the code. Andrew agreed with a focus on simplicity, stating that the memory management code has gone beyond the developers' ability to maintain it.

Davidlohr Bueso shifted the conversation a bit, noting that HPUX has a "zero page daemon" charged with zero-filling huge pages when the CPU is otherwise idle. He would like to add a similar feature to Linux. It would work with the hugetlbfs subsystem only; transparent huge pages would not be zeroed in this way. Working with hugetlbfs is beneficial in that the kernel knows just how many pages have been configured, so there is an automatic bound on how many pages would need to be zeroed.

The benefit of such a mechanism would mainly be in reduced application-startup time. But it is unlikely to find its way into the mainline; Mel stated firmly that he saw it as a bunch of additional code for little real benefit. Adding work to the idle loop would have power-consumption implications, increase memory bandwidth usage, and lead to confusing variability in application performance. It is, he thought, a poor tradeoff overall. Andrew suggested finding a way to do the page zeroing from user space; that would make the overhead visible and put it all under user control.

Transparent huge pages currently only work with anonymous pages; file-backed pages are not covered. Kirill Shutemov talked about his work to extend transparent huge pages to the page cache (described briefly in this article). The work is about one year old and has shown significant improvements on some benchmarks. This improvement has come at the cost of adding a new lock to protect the page cache registry. Things get especially complicated when huge pages need to be split.

At that point, the discussion headed into the question of whether it is ever really necessary to split a huge page in the page cache. With anonymous pages, there are times when splitting is nearly unavoidable; performing copy-on-write on a page within a huge page is one example. But the page cache works differently. It might well be possible for some processes to map parts of a huge page as small pages while others see it as a single huge page cache page. But there are some interesting reference counting questions to be answered.

Reference counts for page cache pages live in the associated struct page in the system memory map. When the page is a huge page, there are many page structures, one for each small page that goes into the huge page. A reference count for a huge page is kept in the first of those page structures, while the reference count in the remaining "tail pages" are not used. But if some threads see only a few of the tail pages rather than the huge page as a whole, where should the reference count for those tail pages be? Rik suggested that the head page should be used to maintain a reference count for the whole huge page; a reference to the huge page would then set the count to 512 (the number of small pages that fit into a huge page). If the page is split, references to individual small pages can be dropped by decrementing the head-page counter, and the kernel still knows how many references to the huge page (or parts thereof) still exist.

Hugh worried that there could be confusion between the head page as representing a huge page and that page on its own as a small page, but Rik thought those issues could be worked out.

Should this work go upstream? Andrew suggested it should if, in the end, it makes things simpler. He also said that, in retrospect, the memory management subsystem should have been designed around variable page sizes from the beginning, but nobody was thinking in those terms. Any future work should be done with that kind of end goal in mind, though; grafting features like huge pages onto the existing memory management code has clearly been a mistake. That, of course, is a tall order for anybody wanting to improve the kernel's management of huge pages; it suggests that we could be seeing some fundamental changes in memory management in the coming years.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (1 posted)

Memory accounting and limits

Two separate sessions in the memory management track of the 2014 Linux Storage, Filesystem, and Memory Management Summit looked at memory accounting and the application of limits to memory usage. One would think that this old problem would have been solved long ago, but it is clear that there are still a number of open issues in this area.

Low limits

At the 2013 Summit, Michal Hocko tried to convince developers that a change in how the "soft" limit in memory control groups ("memcgs") are implemented was needed. He was not successful in that attempt, so, this year, he came back with a variation of that approach: rather than change soft limits, he would like to add a new limit to memcgs called the "low limit."

A soft limit is meant to provide an upper bound on memory consumption when the system is under memory pressure. If there is plenty of memory available, a memcg can consume more than its soft limit would allow, but, when pressure hits, the reclaim code will step in and the memcg's use will be cut back quickly to the soft limit. If the memory pressure persists, processes in memcgs may be cut back even further, well below the soft limit set by the memcg. But sometimes users don't want certain memcgs to go below a minimum amount of memory even when the memory pressure is severe.

That is the purpose of the low limit. If this limit is set on a memcg, the memory management subsystem will not reduce that memcg's usage below the limit even if the system is desperately short of memory. The low limit is meant to be a sort of guarantee; the system takes it seriously enough that it will go into a full out-of-memory condition before it will reduce a memcg below its low limit.

There were a couple of questions that resulted from this presentation. Peter Zijlstra went back to the idea of using the soft limit as a guarantee instead. Since nobody seems to like how soft limits are implemented now, why not just change things? Part of the problem is that the current default soft limit is "unlimited"; using the soft limit as a guarantee would require changing that default to zero. Whether that change (which constitutes an ABI change) would affect users is unclear; as Peter put it, anybody who is actually using soft limits is already changing that value anyway. But Michal, who fought this battle for a while, is nervous about changing that interface now, and he is not the only one.

Other developers questioned the wisdom of setting up a limit mechanism that is designed to push the system into out-of-memory situations. They don't feel that a minimal amount of memory can ever be guaranteed to a memcg, since the total amount of available memory cannot be guaranteed. But, in the end, most seem willing to let Michal try; if users break their systems with it, they get to keep all of the pieces.

But, in contrast to last year's discussion, Michal may well be pushed back toward using the soft limit rather than adding a new one. Some developers don't want to add yet another limit. There is also universal disdain for the current soft limit code, which, it is said, should not be viewed shortly after meals by developers with sensitive stomachs. Changing the way the limits work would enable the removal of much of that code. If soft limits are used, a simple "oom" Boolean flag could be added to allow users to request the "low limit" behavior; this flag would not be set by default. If the current view doesn't change, that is the form that the next version of this patch set will take.

Memory pinning

Peter Zijlstra got up to talk about situations where drivers need to allocate "pinned" pages — pages in a process's address space that cannot be swapped out or even migrated between processors. Pinning is useful for buffers used in RDMA conversations, with the perf events subsystem, and for video frame buffers, among other things. Once upon a time, Peter said, pinned pages were treated much like pages locked into memory with mlock() for accounting purposes. Either type of page would be accounted against the mlock limit, placing an upper bound on the total amount of memory a process could lock down.

More recently, he said, the accounting changed so that pinned pages are counted separately from locked pages. That essentially doubled the amount of memory a process could lock down. On some systems, that meant that processes were now able to push the system into an out-of-memory condition, which is not desirable. So Peter would like to revert the accounting back to the way it was before.

Andrew Morton replied that this could be hard. The kernel has been, for better or worse, changed to be more permissive; going back now could break things for other users. In the end, that view may carry, though no real conclusion was reached in the session.

One reason that Peter is looking at this functionality is that developers in the realtime community are figuring out that mlock() doesn't quite give them the guarantees they would like to have. Locking a page into memory guarantees that it will not be swapped out, but it still gives the kernel some freedom; in particular, the kernel is free to migrate a locked page between locations in RAM. Migration can cause delays and soft page faults for realtime applications, which is not welcomed by realtime developers.

As it happens, the kernel does not currently migrate locked pages, but the memory management developers reserve the right to do so in the future. So Peter is looking at adding a new set of system calls, mpin() and munpin(), that would fully pin pages in memory. When those calls go in, it would be nice to have a clear view of how the accounting will work. At the moment, it appears that pinned pages will go into a different accounting bin than locked pages.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (none posted)

Some vmsplice() issues

Pavel Emelyanov works with the checkpoint-restart in user space (CRIU) project. One of the use cases for CRIU is live migration of processes from one host to another; that involves the moving of a lot of memory to and from sockets. The vmsplice() interface seems like an ideal tool for doing that work without unnecessarily copying the data. But in the process of using vmsplice() for this purpose, Pavel has run into a number of issues. In the final plenary session at the 2014 Linux Storage, Filesystem, and Memory Management workshop, Pavel discussed the problems he has encountered and their possible solutions.

One problem is that using a pipe to move pages of memory — part of the process of using vmsplice() — requires opening two separate file descriptors. CRUI needs to open a lot of pipes, so it tends to run into the limit on the total number of open file descriptors. Al Viro described a possible workaround: find one of the pipe file descriptors under /proc, open it as a read/write file descriptor, then close the two original descriptors. That will cut the number of required file descriptors in half.

vmsplice(), when used with the SPLICE_F_GIFT flag, is meant to hand the indicated pages of data directly to the kernel without copying the data. But, Pavel said, it often ends up copying those pages anyway, even though it seems the copying should not be necessary. Some digging through the commit logs suggests that things were done this way to avoid surprising filesystems with pages of data coming from an unexpected direction. The filesystem developers seemed to agree that the amount of work required to handle such pages would be quite small, so perhaps this behavior could be changed. An action item was taken to try to query Nick Piggin (the original author of this code, who has since disappeared from the kernel community) about whether there are any other subtle issues that might prevent greater use of zero-copy transfers.

Pavel's next problem is that pages sent to files with vmsplice() go into the page cache, but he would rather have them bypass the page cache and be written directly to the target file. It was pointed out that splicing to a file descriptor opened with O_DIRECT should work properly; at that point, the rest of the problem description came out. An O_DIRECT file descriptor does indeed work, but writes are synchronous, slowing things down. Pavel would rather there were a way to do asynchronous O_DIRECT writes via vmsplice(). Al allowed that it might be possible to make this work, but the job "might not be fun."

The final problem had to do with how to send pages out of another process's address space without actually copying them. James Bottomley suggested that some of the machinery behind the fork() system call could be used. The process would not actually be forked, but a copy of its address space would be made so that the migration process could get to its pages directly. The implementation of this functionality could be tricky but, if it could be done, it might make process migration significantly more efficient.

[Your editor would like to thank the Linux Foundation for supporting his travel to the Summit.]

Comments (1 posted)