Allowing small allocations to fail [LWN.net]

Allowing small allocations to fail

Posted Mar 13, 2015 3:16 UTC (Fri) by dlang (guest, #313)
In reply to: Allowing small allocations to fail by neilbrown
Parent article: Allowing small allocations to fail

I use stack space ans an example after seeing the repeated discussions of "XFS + <layers> runs out of stack space" discussions on linux-kernel. I can't talk in detail about the issue, but I know from watching this that the different layers are not as isolated in their effects as you make it sound.

As for the layers, yes, I am saying that these layers can cause things to go wrong, not that they will cause problems. Raid, encryption, compression, snapshots can all require reading in data from disk in order to write data out to disk. Filesystem operations can require read-modify-write cycles that can require memory for the read, iscsi invokes the entire networking stack and needs memory to encapsulate the I/O data, etc.

When the kernel picks a hunk of memory to output to disk (either swap or pending writes), it has no way of knowing what is going to be involved to write this data out. It may be that all of these layers have sufficient memory pre-allocated that they never, ever need to allocate more during the running of the system, but I really have my doubts. I know that at least some of them have enough reserved memory that they can limp along to complete a single request if they can't get a normal allocation, but with the 'too small to fail' logic having been in place, how many of these emergency codepaths have really been tested? And are the allocations ending up in the 'blocking, waiting to succeed' mode when the programmer has actually coded a good failure mode and way to make at least some progress even without the allocation?

The stack I listed above was an off-the-cuff 'bad case', but as people have been challenging it, I've been thinking and don't think it's anywhere near the real worst case.

I can easily see someone having
filesystem
raid
lvm
snapshot
encryption
fuse
virtualization
network (with connection tracking and encryption on the network)

with the possibility that some of these layers may be repeated on the hypervisor level (which shouldn't contribute to memory issues in the guest, but guests could contribute to issues on the host)

I'm probably still not getting the real worst-case situation (it would be interesting to see not just speculation, but real-world information, I'll bet that real-world examples will make the speculation look good)

to post comments

Allowing small allocations to fail

Posted Mar 14, 2015 6:05 UTC (Sat) by neilbrown (subscriber, #359) [Link]

> I know that at least some of them have enough reserved memory that they can limp along to complete a single request if they can't get a normal allocation, but with the 'too small to fail' logic having been in place, how many of these emergency codepaths have really been tested?

Lots of them.

"too small to fail" doesn't apply to all kernel allocations. It does apply to those with the GFP_KERNEL flags set, but not, for example, those with GFP_ATOMIC.

Any code that has been written with a clear emergency fallback almost certainly uses an allocation style that can fail - if it didn't there is a very good chance that it would deadlock. mempools, for example, always set __GFP_NORETRY, so normal allocation *will* fail if no memory is easily available, and the fall-back to the pre-allocated pools is often used.

GFP_NOFS allocations are probably the most problematic. I think they can be treated as "too small to fail", but there can be lots of dirty memory that they cannot touch.

Of all the things in your stack, I think filesystems, fuse, and networking are the most likely to have interesting issues: filesystems because they are complex, fuse because it can depend on userspace behaving correctly, and networking because it is highly optimized for speed and tries to avoid special cases on the fast-paths.
All these can be made to work well, but the problems they might face are really quite independent of what other things in the stack might be doing.