Allowing small allocations to fail

Allowing small allocations to fail

memory overcommit is not your fundamental problem here. In fact, it help avoid the problem the vast majority of the time.

Isn't this basically the same logic used to justify mapping NULL to a read-only page filled with 0s? Which is what BSD did decades ago. It was a similarly poor design decision from the perspective of resilient systems and it took many years to undo.

Yes, the _majority_ of time you end up with benign behavior. The problem is the cases where you end up with completely wrong behavior. And the only way to solve _all_ the problems was for _everybody_ to write more robust code. The only way to coax code to become more robust was to make it fail spectacularly.

Have we forgotten about the very wise advice, fail fast, fail often?

The problem with overcommit and similar measures is that there's no incentive for people to write robust algorithms. Clearly the XFS people are in this camp. They didn't have to worry about it, so they didn't, and now their code is too complex to change.

You _can_ write robust algorithms resilient against low memory situations. Sometimes it's easy. RAII patterns simplify unwinding state, which is just as easy in C as in C++, minus the automatic destructors. Sometimes it's more difficult and might require rethinking your implementation in terms of an explicit state machine. Other times you must make a tradeoff between CPU cost and memory cost. Linux often already does this--for example using a O(2 log N ), cache-thrashing red-black tree instead of O(1), cache-friendly timing wheel for high resolution timers--but nobody really complains about it not being the absolute fastest possible because raw speed is rarely the _only_ consideration.

Not long ago people said that memory was cheap and worrying about memory constraints was a thing of the past. Then the embedded world exploded.

Properly handling OOM situations in core infrastructure code doesn't solve all the problems. But it's a prerequisite for solving all the problems. And we'll never make any headway as long as people can pass the buck to the OOM killer. The hierarchy of abstraction layers might not be so convoluted and poorly composable if they couldn't rely on simplifying assumption about memory allocation; the interfaces would have to been better designed to make unwinding state easier and make forward progress yieldable at a more fine-grained level.

Look at block devices. The reason we can't have non-blocking disk I/O is because Unix originally made the simplifying assumption that disk writes were per se non-blocking. The assumption was no doubt worthwhile back when it could reduce the complexity of your code by an order of magnitude. But now block device drivers are so complex anyhow that the cost+benefit tradeoff sucks. Same situation wrt to the big kernel lock. I would argue the same thing now applies to the OOM killer. The simplifying assumption has lost it's usefulness, presuming it was ever truly useful.