Filesystems and crash resistance [LWN.net]

Please consider subscribing to LWN
Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

By Jake Edge
May 21, 2019

LSFMM

The "guarantees" that existing filesystems make with regard to persistence in the face of a system crash was the subject of a session led by Amir Goldstein at the 2019 Linux Storage, Filesystem, and Memory-Management Summit (LSFMM). The problem is that filesystem developers are not willing to make much in the way of guarantees unless applications call fsync()—something that is not popular with application developers, who want a cheaper option.

Currently, there are applications that create and populate a temporary file, set the attributes desired, then rename it, Goldstein said. The developers think that the file is written persistently to the filesystem, which is not true, but mostly works. The official answer is that you must use fsync(), but it is a "terrible answer" because it has impacts all over the system.

He wondered if there could be an API that gives the application developers what they are looking for. The CrashMonkey developers did a bunch of testing on filesystem behavior after crashes, then brought some of the problems they found to the Btrfs developers, who said that they were not bugs. So the CrashMonkey folks wanted to document the expected behavior, then test and file bugs for filesystems that did not conform; that didn't work either, he said. He said it resulted in a long discussion between Dave Chinner, Ted Ts'o, and the Btrfs developers about the expected behavior, but there was a concern about committing to the existing behavior.

So, perhaps there is a middle ground, Goldstein said. The kernel could add a new API via an ioctl() command or perhaps called rename_barrier() that provides the guarantees of the behavior that filesystems already provide today. Later, if there is an optimization that changes the existing behavior, simply do not add that optimization to the path that the new API uses.

There are two types of barriers that he is talking about. The first would be used by overlayfs; it sets extended attributes (xattrs) on a file, then renames it. Overlayfs expects that, if the rename is observed, then the metadata has been written persistently. The other barrier is for data to be persistently written, which can be done today using the FIEMAP ioctl() command (with the FIEMAP_FLAG_SYNC flag), at least for XFS and ext4, he asserted.

But Chris Mason said that won't work for sparse files even on ext4 and XFS. Jan Kara said that it is a side effect of how ext4 and XFS do their journaling; the data will get to disk and the metadata will go into the journal. It is cheaper than an fsync(), he said.

Ric Wheeler is concerned that filesystem developers have "spent decades" trying to explain how to use fsync() to application developers without success, so any new mechanism will simply be "so mysterious" that no one will use it. But Goldstein disagreed, saying that the mechanism is "totally natural". Wheeler, however, was not convinced; if he was to put things in a "more mean" way, he would say that the application developers do not understand what they expect.

Kara said that he agreed with Wheeler that a new API is not going to solve the problem. Even if it were all documented, Ts'o is skeptical that application authors will read and internalize the documentation; "we have ample evidence" that they do not. But Goldstein said that he is an application developer and he needs an API for this. Right now he is using FIEMAP with the sync flag, but that is not an API, he said; FIEMAP was added for an entirely different purpose and the flag was only meant to ensure the extents had been allocated. Mason said that he "would love it" for rename to work the way the application developers want it to, but it is too expensive. Adding a new way to accomplish what he thinks the applications want, but costs more than a rename, would be fine with him.

Ts'o said that kernel developers need to carefully document whatever they are going to do; even then "we are still going to lose". He doesn't want to repeat the experience of sync_file_range(), which didn't do exactly what people expected it to do, but applications ended up depending on things it didn't actually do. Two new operations, fbarrier() and fdatabarrier(), have been proposed in several academic papers lately, but the semantics differ among the papers. Before committing to a name and the behavior, it may be worthwhile to look at these papers, he said.

Goldstein said that the CrashMonkey developers have already documented some of what is needed, but Ts'o said that he hated that document; "that is not a starting point", he said. It is based on what the developers observed, rather than on a specification of a new system call and what guarantees it provides—and does not provide. Observing what ext4, XFS, and Btrfs do today and expecting that to continue into the future "is not the way to do this".

The idea is just to have an API that guarantees what ext4 already does today, Goldstein said; there should be no implementation cost. But Ts'o cautioned that should not be how it is framed. Instead, a specification should be written first, then debated, before an implementation is done. He is not in favor of the feature, but if people want to proceed, this is the way to "minimize the blast radius". Ts'o believes the feature will be "misused more than it is used".

Wheeler is worried about making a new API that will be hard to explain; right now, there are ways to accomplish the same thing without the API. But Goldstein said it is a weird situation to be hiding this kind of thing from users. Ewan Milne said that presenting a new API that will be seen as a "faster fsync()" that doesn't do everything that fsync() does will just serve to confuse.

The problem is getting worse, Goldstein said, as fsync() is getting more expensive. Since there are already ways to get the behavior that he and other application developers want, without changing anything, it makes sense to expose it. There are advanced users of the filesystems (e.g. overlayfs, PostgreSQL, Git) that will benefit from it.

In the end, the outcome of the discussion seemed rather inconclusive. Later in the conference, though, Goldstein made a point of letting me know that some discussions with Ts'o and others after the session did come to some level of agreement on the path forward.

Index entries for this article
Kernel	Filesystems
Conference	Storage, Filesystem, and Memory-Management Summit/2019

to post comments

Filesystems and crash resistance

Posted May 22, 2019 3:03 UTC (Wed) by jkingweb (subscriber, #113039) [Link] (5 responses)

> The official answer is that you must use fsync(), but it is a "terrible answer" because it has impacts all over the system.

Could someone please clarify what is meant by this? It's implied it's more than just slowness, but the article is never explicit about what the problems with fsync are. I can only assume it's obvious to people in the know, but it's not obvious to me. I'm more a user of software than a programmer of software.

Filesystems and crash resistance

Posted May 22, 2019 3:24 UTC (Wed) by jake (editor, #205) [Link] (4 responses)

> It's implied it's more than just slowness, but the article is never explicit about what the problems with fsync are.

My apologies. I should have linked this in somewhere, perhaps even in the first sentence:

https://lwn.net/Articles/322823/

This also gives some useful background:

https://lwn.net/Articles/351422/

From that article:

> At the same time, fsync() is often more expensive than it absolutely needs to be. The easiest way to implement fsync() is to force out every outstanding write to the file system, regardless of whether it is a journaling file system, a COW file system, or a file system with no crash recovery mechanism whatsoever. This is because it is very difficult to map backward from a given file to the dirty file system blocks needing to be written to disk in order to create a consistent file system containing those changes. For example, the block containing the bitmap for newly allocated file data blocks may also have been changed by a later allocation for a different file, which then requires that we also write out the indirect blocks pointing to the data for that second file, which changes another bitmap block... When you solve the problem of tracing specific dependencies of any particular write, you end up with the complexity of soft updates. No surprise then, that most file systems take the brute force approach, with the result that fsync() commonly takes time proportional to all outstanding writes to the file system.

jake

Filesystems and crash resistance

Posted May 22, 2019 15:04 UTC (Wed) by jkingweb (subscriber, #113039) [Link]

Thanks, Jake. Much clearer now.

Filesystems and crash resistance

Posted May 22, 2019 15:40 UTC (Wed) by rweikusat2 (subscriber, #117920) [Link] (2 responses)

This text is ten years old and entirely generic. Additionally, the conclusion

This is because it is very difficult to map backward from a given file to the dirty file system blocks needing to be written to disk in order to create a consistent file system containing those changes. For example, the block containing the bitmap for newly allocated file data blocks may also have been changed by a later allocation for a different file, which then requires that we also write out the indirect blocks pointing to the data for that second file

is just plain wrong. Nobody asked for "a consistent filesystem containing those changes" aka "for a sync" (ultimatively). The request was that changes to a specific file are to be made persistent. Obviously, the 'easiest' way to implement that is to ignore this and write everything. Which filesystems did this back then? Which filesystems didn't? And what's the situation today? This could be useful information for choosing a particular filesystem.

Filesystems and crash resistance

Posted May 23, 2019 18:41 UTC (Thu) by matthias (subscriber, #94967) [Link]

Writing everything is the awfully slow solution that is not wanted. The problem with writing out bitmap blocks for unrelated files is, that -- if a crash would happen in the wrong moment -- the metadata would be inconsistent. And this is what should never happen. Of course, not everything needs to hit the disk, but the subset that hits the disk has to be consistent. Therefore it is problematic if accidentally some unrelated metadata changes are written.

Filesystems and crash resistance

Posted Jun 21, 2019 3:47 UTC (Fri) by nivedita76 (subscriber, #121790) [Link]

What do you see as the difference between "consistent file system containing those changes" and "changes to a specific file are to be made persistent"? Those "changes to a specific file", the one that got fsync'd are exactly what that first quote is referring to.

Filesystems and crash resistance

Posted May 22, 2019 10:58 UTC (Wed) by kilobyte (subscriber, #108024) [Link] (2 responses)

rename_barrier() would be extremely situational. A much better solution has been proposed as fsbarrier(fdA, fdB) -- once the call returns, any writes to fdA completed before the call are no less durable than any writes to fdB started after the call. Useful cases include fdA==fdB; fdB being the directory holding fdA; fdB being dpkg's status file; etc...

Such a call would provide consistency and ordering but no immediate persistency (until a later fsync() or natural writeout). Even when immediate persistency is desired, many filesystems can optimize ordering much better than a pair of fsyncs. Example: you're on the orbit of Pluto, writing to a filesystem mounted over nbd on a server on Earth (half-ping: 9 hours). A typical operation requires two fsyncs: this takes 36 hours before your thread can continue. With fsbarrier(), you can report success before the packets even leave the network card -- then the final persist of any number of operations takes 18 hours.

Filesystems and crash resistance

Posted May 22, 2019 20:12 UTC (Wed) by luto (subscriber, #39314) [Link] (1 responses)

How would this interact with mmap?

Filesystems and crash resistance

Posted May 23, 2019 15:40 UTC (Thu) by kilobyte (subscriber, #108024) [Link]

Good question. I see that when fdA!=fdB and there's no loop back to fdA, no special handling is needed: no problem if some later writes will catch the ordered train.

Only when fdA is ordered with regard to itself, you'd need to block any write page faults until the flush is done.

The implementation that was discussed for btrfs involved flushoncommit specific to that file, ensuring all dirty pages for the file will be included in the next transaction. It's ok if a newer version of a page gets committed, as long as no newer version can go live before all barriered ones get in.

Filesystems and crash resistance

Posted May 22, 2019 19:30 UTC (Wed) by ikm (guest, #493) [Link] (4 responses)

I'm surprised that crash resistance is linked to persistence here. I would argue that in an event of a crash what is much more important is consistency - I don't mind if I lose the latest changes to the file I'm editing, but I would be very upset if a file is left filled with all zeroes afterwards. If I lose just the latest changes, I can just redo the last several seconds of the work, but if the file is left in an inconsistent state, or worse it, wiped clean, *all* the work is lost. I've seen this happen many times with xfs, and never with ext4, for example. To me, those two filesystems seem to provide entirely different guarantees in practice, even though in theory maybe none of those are supposed to be guaranteed in the first place, as no fsync() may be called at all.

Filesystems and crash resistance

Posted May 23, 2019 6:05 UTC (Thu) by amir73il (subscriber, #66165) [Link]

You have several correct observations:
1. People keep mixing persistence/durability with crash consistency
2. In the absence of fsync(), ext4/xfs exhibit entirely different *behavior* in practice
3. In the absence of fsync(), ext4/xfs provide no guaranties about consistency of file data
ext4 has some heuristics in place to reduce the number of support calls from people that did not issue fsync() as documented,
but those are NOT to be confused with guaranties.

Filesystems and crash resistance

Posted May 23, 2019 8:23 UTC (Thu) by nilsmeyer (guest, #122604) [Link] (2 responses)

> If I lose just the latest changes, I can just redo the last several seconds of the work,

And how would you store those latest changes? You would need some durable storage for that.

> To me, those two filesystems seem to provide entirely different guarantees in practice, even though in theory maybe none of those are supposed to be guaranteed in the first place, as no fsync() may be called at all.

That's likely the root of the problem. Also with ext4 it depends on mount/mkfs options, data=ordered will behave differently from data=writeback or data=journal. That makes it very difficult for application developers to handle, especially if they also target platforms other than Linux.

Filesystems and crash resistance

Posted May 23, 2019 11:28 UTC (Thu) by rweikusat2 (subscriber, #117920) [Link] (1 responses)

The usually sensible strategy for "handling" this is "don't". People who are (for more or less good reasons) paranoid about data loss in case of a sudden, unplanned shutdown can mount their filesystems synchronously, albeit this still doesn't guarantee anything because writes take a finite amount of time and if power drops at the wrong time, something is going to break (I've had damaged file systems mounted with this option).

Or "listen to Microsoft" (voices of reason alone are usually disregarded) and use a suitable storage engine aka DBMS.

Filesystems and crash resistance

Posted May 31, 2019 0:58 UTC (Fri) by Wol (subscriber, #4433) [Link]

Use a dbms? And watch the dbms get corrupted when the power failure corrupts its writes.

There's nothing magic about a dbms, it's just another application. And it's just as prone to being corrupted as any other. One only has to look at the hoops Postgres has to jump through to TRY and ensure integrity.

I've always thought write barriers make sense. If I can guarantee that all writes before the barrier land before any ones after the barrier get written, I can reason about that and - more importantly - make sure that either (a) my logs are consistent, or (b) my data is consistent. More importantly, I can then recover my data successfully to pre- or post- write condition. I may not have a choice which, but at least the data is consistent.

Cheers,
Wol

Filesystems and crash resistance

Posted May 23, 2019 22:00 UTC (Thu) by k3ninho (subscriber, #50375) [Link]

The answer of 'it's coming soon to eBPF' doesn't apply here -- can we use 'the grace periods of RCU'?

(Only part joking: the grace period expiry and expiry failure notification of RCU give an answer for which copy of the blocks are or aren't on disk and worst-case two controlled states if failure happened during the grace period. The fs recovery tools get one mechanism; application developers gain something like fire-and-forget.)

K3n.