Allowing small allocations to fail [LWN.net]

Allowing small allocations to fail

Posted Mar 11, 2015 3:59 UTC (Wed) by neilbrown (subscriber, #359)
In reply to: Allowing small allocations to fail by Cyberax
Parent article: Allowing small allocations to fail

> I wonder why they don't approach this problem from the other side - add a special flag __GFP_CAN_FAIL and start annotating the kernel with it.

The flag exists and is spelt "__GFP_NORETRY".

Setting that everywhere could get very noisy.... might be a good idea though.

to post comments

Allowing small allocations to fail

Posted Mar 11, 2015 9:21 UTC (Wed) by ewen (subscriber, #4772) [Link] (2 responses)

So that suggests a way forward:

* Document that allocations for up to 8 pages (order=3) implicitly have __GFP_NOFAIL set on them now, and that in N kernel versions only allocations of up to 4 pages (order=2) will have __GFP_NOFAIL *implicitly* set on them.

* Any code that already has its own recovery mechanism for "allocation might fail", can be amended to include __GFP_NORETRY on it -- or some other flag that means "try some simple stuff, but don't invoke the OOM killer" -- and any code that gets fixed up in advance to cope with the reduced "too small to fail" can have that flag added.

* Any code that cannot be so fixed gets updated to explicitly have __GFP_NOFAIL on it, to document that it cannot cope with running out of memory.

* In N kernel versions change the implicit "too small to fail" protection as promised, and publish another "in N kernel versions it will be changed to..." level.

* Lather, rinse and repeat.

If the clutter of all these extra flags becomes too much create some wrapper functions (macros?) that encapsulate the "do this if possible now" and "keep trying until you an do this", and change to those. As annotations go it doesn't strike me as *that* much more clutter than, eg, likely()/unlikely().

Plus of course test all these changes with the fault injection framework. Because it sounds like a lot of those "allocation failed" handlers probably haven't been tested in anger due to "too small to fail".

Ewen

Allowing small allocations to fail

Posted Mar 11, 2015 11:24 UTC (Wed) by vbabka (subscriber, #91706) [Link] (1 responses)

I don't see the benefit of annotating all places where allocation can fail, if the default is going to change later anyway. Sounds like a lot of work that will in the end just clutter the code needlessly. What's needed is annotating (or fixing, if possible) places where allocation *cannot* fail.

Allowing small allocations to fail

Posted Mar 11, 2015 12:01 UTC (Wed) by cesarb (subscriber, #6266) [Link]

> Sounds like a lot of work that will in the end just clutter the code needlessly.

It's a good way to change a default. In the initial state, you have a lot of code which might rely on the old default. Gradually you change each call site to explicitly state what it really needs (cannot fail, indifferent, can fail, can fail and must do so quickly), so after a while nothing depends on the default (except the ones explicitly verified as "indifferent"). Then you change the default, which will now have no effect (except for the call sites explicitly verified as "don't care"). Finally, you gradually remove the now redundant annotations corresponding to the new default.

Yes, it's more work, but it's also a safer path. Each step is small enough that it can be individually verified, and what would otherwise be a large step (the change in the default) becomes a minor step. It's similar to the concept of a "reversible process" in physics.