Fighting Spectre with cache flushes

This article brought to you by LWN subscribers

Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

Speculative-execution vulnerabilities are only exploitable if they leave a sign somewhere else in the system. As a general rule, that "somewhere else" is the CPU's memory cache. Speculative execution can be used to load data into the cache (or not) depending on the value of the data the attacker is trying to exfiltrate; timing attacks can then be employed to query the state of the cache and complete the attack. This side channel is a necessary part of any speculative-execution exploit.

It has thus been clear from the beginning that one way of blocking these attacks is to flush the memory caches at well-chosen times, clearing out the exfiltrated information before the attacker can get to it. That is, unfortunately, an expensive thing to do. Flushing the cache after every system call would likely block a wide range of speculative attacks, but it would also slow the system to the point that users would be looking for ways to turn the mechanism off. Security is all-important — except when you have to get some work done.

Kristen Carlson Accardi recently posted a patch that is based on an interesting observation. Attacks using speculative execution involve convincing the processor to speculate down a path that non-speculative execution will not follow. For example, a kernel function may contain a bounds check that will prevent the code from accessing beyond the end of an array, causing an error to be returned instead. An attack using the Spectre vulnerability will bypass that check speculatively, accessing data that the code was specifically (and correctly) written not to access.

In other words, the attack is doing something speculatively that, when the speculation is unwound, results in an error return to the calling program — but, by then, the damage is done. The error return is a clue that there maybe something inappropriate going on. So Accardi's patch will, in the case of certain error returns from system calls, flush the L1 processor cache before returning to user space. In particular, the core of the change looks like this:

    __visible inline void l1_cache_flush(struct pt_regs *regs)
    {
	if (IS_ENABLED(CONFIG_SYSCALL_FLUSH) &&
	    static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
	    if (regs->ax == 0 || regs->ax == -EAGAIN ||
		regs->ax == -EEXIST || regs->ax == -ENOENT ||
		regs->ax == -EXDEV || regs->ax == -ETIMEDOUT ||
		regs->ax == -ENOTCONN || regs->ax == -EINPROGRESS)
			return;

	    wrmsrl(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
	}
    }

The code exempts some of the most common errors from the cache-flush policy, which makes sense. Errors like EAGAIN and ENOENT are common in normal program execution but are not the sort of errors that are likely to be generated by speculative attacks; one would expect an error like EINVAL in such cases. So exempting those errors should significantly reduce the cost of this mitigation without significantly reducing the protection that it provides.

(Of course, the code as written above doesn't quite work right, as was pointed out by Thomas Gleixner, but the fix is easy and the posted patch shows the desired result.)

Alan Cox argued for this patch, saying:

Andy Lutomirski is not convinced, though. He argued that there are a number of possible ways around this protection. An attacker running on a hyperthreaded sibling could attempt to get the data out of the L1 cache between the speculative exploit and the cache flush, though Cox said that the time window available would be difficult to hit. Fancier techniques, such as loading the cache lines of interest onto a different CPU and watching to see when they are "stolen" by the CPU running the attack could be attempted. Or perhaps the data of interest is still in the L2 cache and could be probed for there. In the end, he said:

Answering Lutomirski's criticisms is probably necessary to get this patch set merged. Doing so would require providing some numbers for what the overhead of this change really is; Cox claimed that it is "pretty much zero" but no hard numbers have been posted. The other useful piece would be to show some current exploits that would be blocked by this change. If that information can be provided, though (and the bug in the patch fixed), flushing the L1 cache could yet prove to be a relatively cheap and effective way to block Spectre exploits that have not yet been fixed by more direct means. As a way of hardening the system overall, it seems worthy of consideration.

Note: you can avoid this step in the future by logging into your LWN account.