Brian Kernighan on the origins of Unix [LWN.net]

Brian Kernighan on the origins of Unix

Posted Jan 28, 2022 6:05 UTC (Fri) by wahern (subscriber, #37304)
In reply to: Brian Kernighan on the origins of Unix by farnz
Parent article: Brian Kernighan on the origins of Unix

cgroups doesn't solve the memory sharing problem any better, though admittedly VMs do currently compound the problem. virtio-balloon, which is widely supported (including by typical Linux hosts and guests) permits a guest to return memory (e.g. filesystem buffer caches) to the host. The problem (AFAIU) is that currently there's no implementation on either side (host or guest) that communicates memory pressure back-and-forth, opportunistically shifting memory around. See defunct auto-ballooning project at https://www.linux-kvm.org/page/Projects/auto-ballooning. So, just as with cgroups, if the memory accounting budget is overcommitted and the VM manager hits a hard OOM condition, it's already too late to guarantee ideal behavior; that is, it's too late to first recover caches and other any pages across all processes/guests that aren't strictly necessary for correct continuation (if at all possible) of every process/guest.

The alternative in each case--VMs and cgroups--is to preallocate memory so that the sum is no more than available backing space. But the Kubernetes clusters I've worked on, for example, all overcommitted cgroups memory--the sum of all pod cgroup memory limits was greater than physical memory (and machines had no swap space, despite my protests). Use of cgroups couldn't prevent the OOM killer from unnecessarily shooting down processes when under high aggregate load; it only helped blunt edge cases where individual pods allocated much more memory than ever expected (e.g. because of a memory leak). This was especially true if a JVM was running in a pod, as for whatever reasons the Java applications people liked to run on those clusters aggressively used anonymous memory. Whenever some other team complained about their pod being OOM killed despite not going over their cgroup budget, the host invariably had a JVM running in another, unrelated pod. (This was especially problematic for the period, 2019-2020, when Linux kernels had a regression that made it *much* more likely that the OOM killer couldn't flush buffer caches faster than processes would dirty them. Worse, the result was the OOM killer spinning, sleeping, and eventually timing out in its flush routine; and often times multiple processes could end up in that routine--or end up trying to take a lock held by that routine--effectively resulting in a situation where the entire system could lock-up for an indeterminate amount of time. Even if you could get an SSH session, poke the wrong process and your session would lock-up, too.)

to post comments

Brian Kernighan on the origins of Unix

Posted Jan 28, 2022 14:48 UTC (Fri) by farnz (subscriber, #17727) [Link]

Cgroups have two things that a virtual machine setup does not have:

Limits to determine which containers swap more than others. By setting memory.low to your "fair" share without overcommiting, and memory.high to your limit with overcommit, you bias the kernel's paging to page out memory hogs (both to swap and paging out their executable pages), and to slow down processes that can't reclaim memory to stay in their lane.
An interface for a management process to use to identify cgroups that are using more than their fair share. You can monitor memory.events for changes, and use the notifications to tell containers to reduce their memory use.

Now, if you're dealing with people who overcommit physical RAM but who've not read and understood In defence of swap, you're in trouble anyway - they're working on a faulty model of how the OS allocates memory to processes to begin with, and both models are going to break because they're expecting "magic" to save them. But with competent people, cgroups already does better than VMs; and because cgroups has more visibility into what the workload is doing, it's more likely that cgroups will improve over time.

As an example, EPOC32 had low memory signals it sent to the applications to let them know that the system was low on memory, and that it was out of memory. Applications were expected to handle these signals by discarding caches, performing an early GC etc - and could also (ab)use them for better performance (e.g. don't run GC until the system is low on memory). You can get similar signals in a cgroup-aware application by having the application monitor memory.events, and treating a breach of the "low" threshold as "memory is running low" and a breach of the "high" threshold as "out of memory". Then, your JVM could be aware of this, and whenever it breaches the "low" threshold (container using too much memory), it does a GC run, while when it breaches the "high" threshold, it also discards any cached JITted sections etc.

This interacts in a good way with the kernel's preference to swap stuff that breaches low more than it swaps stuff that breaches high - when you breach low, you try to come back under control, when you breach high, you try to avoid being killed.