Allowing small allocations to fail [LWN.net]

Allowing small allocations to fail

Posted Mar 11, 2015 21:27 UTC (Wed) by dlang (guest, #313)
In reply to: Allowing small allocations to fail by wahern
Parent article: Allowing small allocations to fail

The problem here is not userspace and overcommit. The problem is in the kernel when it needs memory in order to write data to disk to allow it to free memory.

Causing users programs to fail more frequently, in the hope that developers will get better at handling the failures, is not going to make the slightest bit of difference to this problem.

Eliminating the OOM killer isn't going to address the problem of needing memory in order to free memory.

I'll also point out the backlash against ext4 for loosing data when programmers didn't properly use fsync. This isn't perceived by users as being a problem caused by the programmers of the application they are using, it's perceived as being a failure of the kernel.

to post comments

Allowing small allocations to fail

Posted Mar 11, 2015 22:46 UTC (Wed) by wahern (subscriber, #37304) [Link] (2 responses)

Non-failing allocations in the kernel is part-and-parcel of the OOM killer. The whole point is that some kernel developers assume that if an allocation request fails, the OOM killer will free up some memory.

Of course failing allocations in user space won't fix XFS or other code. It won't fix user space applications, either. (FWIW, malloc could always fail in Linux because of process resource limits.)

But it does change expectations. In an OOM killer world you can make the assumption that if you're called from user space, then a small allocation could never fail because the OOK killer could always kill at least one user space application and thus free up some memory.

Without the OOM killer, XFS would have had to explicitly make arrangements to reserve a bounded amount of memory ahead of time, rather than implicitly relying on fuzzy assumptions.

XFS and similar code exists, and completely refactoring those things is clearly out of the question. But somebody has to pull the short straw and endure a little more pain than the others if and when Linux moves away from the allocations-cannot-fail simplifying assumption.

Allowing small allocations to fail

Posted Mar 12, 2015 0:08 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

Without the OOM killer, XFS would have had to explicitly make arrangements to reserve a bounded amount of memory ahead of time, rather than implicitly relying on fuzzy assumptions.

I think you have the story backward here, at least in regard to XFS. The XFS developers did exactly what you suggested they should do, and reserved a pool of memory in advance in case memory got really tight. XFS would still work in the case that small allocations were allowed to fail, and they're actually the ones who started pushing to allow them to fail. The problem isn't that XFS never considered what to do if memory was really tight, but that their solution never goes into effect because the triggering case- a failed allocation- isn't allowed to happen.

Allowing small allocations to fail

Posted Mar 12, 2015 0:28 UTC (Thu) by dlang (guest, #313) [Link]

It's far more the case that filesystem developers are doing some really complex things that are really hard to roll back cleanly, and so they assume that some other kernel thread is going to be able to make progress and free some memory than that they assume that the OOM killer will free up memory for them.

I think it's a bad thing that this has grown to be as large and as common as it is. I just don't blame it on overcommit.