Temporary files: RAM or disk? [LWN.net]

Temporary files: RAM or disk?

Posted Jun 2, 2012 21:38 UTC (Sat) by dlang (guest, #313)
Parent article: Temporary files: RAM or disk?

I've used Solaris quite a bit and the default /tmp in ram behaviour ends up meaning that a lot of things that should be put in /tmp end up elsewhere because it's just too easy to run out of ram.

as others have noted, /tmp shouldn't create that much I/O as most of the files don't actually need to hit disk. However, since ext3 is so badly behaved in the face of fsyncs, this is not the case on ext3 based systems.

but instead of changing how the system works, they should switch to a different filesystem that doesn't have the bad behaviour of ext3. Use either ext4 or XFS

to post comments

Temporary files: RAM or disk?

Posted Jun 4, 2012 7:19 UTC (Mon) by Serge (guest, #84957) [Link] (8 responses)

> However, since ext3 is so badly behaved in the face of fsyncs, this is not the case on ext3 based systems.

As long as no real-world applications fsync files in /tmp... who cares?

Temporary files: RAM or disk?

Posted Jun 4, 2012 7:22 UTC (Mon) by dlang (guest, #313) [Link] (4 responses)

the way fsync is broken on ext3, that would only matter if /tmp was a separate filesystem. a fsync anywhere on the filesystem triggers this bug

Temporary files: RAM or disk?

Posted Jun 4, 2012 10:44 UTC (Mon) by Serge (guest, #84957) [Link] (3 responses)

Not sure how this can affect /tmp in real-world then. Could you write some detailed description for this use case so that it could be tested?

Temporary files: RAM or disk?

Posted Jun 4, 2012 23:05 UTC (Mon) by dlang (guest, #313) [Link] (2 responses)

create one program that does massive sequential writes to a file in one directory.

then do a fsync on some file in a different directory (with a small change to the file)

watch the fsync take a long time to complete on ext3, and almost no time on any other filesystem.

because of this behavior, users and sloppy programmers have been conditioned that fsync calls make their program pause unexpectedly for a potentially long time period (I think I've seen Ted Tso report that he's seen delays longer than 30 seconds)

If you go back the the blog messages about fsync and data reliability when people were claiming that ext4 was eating their KDE configuration data, you will see detailed discussions about this.

Temporary files: RAM or disk?

Posted Jun 5, 2012 5:24 UTC (Tue) by Serge (guest, #84957) [Link] (1 responses)

> watch the fsync take a long time to complete on ext3, and almost no time on any other filesystem.

Thanks for a detailed description.

I could not notice a major difference on my system, it could be my HDD is too fast for that (100MB/s, usual rotating hdd, no raids, no LVMs). I thought this "bug" was fixed a few years ago with Linus's blessing, btw.

But the question is: how is this related to /tmp? Nobody fsync()s file in /tmp. This won't work for small short-lived files as well, since there's no chance they're created at the moment of fsync (and even if they are, you won't notice the difference, because they're small). So there must be program writing large file in /tmp. It must be large enough so that fsync on another partition was noticeably delayed. But large file will trigger dirty*ratio and start writing to disk, thus not delaying fsync() much anyway.

BTW, even if fsync was delayed, what is the application where you could notice this delay? I'm trying to say that I can't think of any real-world use cases, that /tmp on ext3 is not good for.

> I think I've seen Ted Tso report that he's seen delays longer than 30 seconds

Technically it may be possible (if it still was not fixed 3 years ago). You need a machine with a lot of RAM, very slow HDD, increase dirty*ratio to 90%, write a few GBs and then call fsync(). But that would be useless, because it's not related to any real-world use cases.

> If you go back the the blog messages about fsync and data reliability when people were claiming that ext4 was eating their KDE configuration data, you will see detailed discussions about this.

Those were different ext4-specific problems of recently modified files lost on crash (usual thing, actually, official xfs "feature"), not related to fsync(), and definitely not related to /tmp, as far as I remember. And those were fixed a few years ago anyway.

Temporary files: RAM or disk?

Posted Jun 5, 2012 7:12 UTC (Tue) by dlang (guest, #313) [Link]

as I understand the problem, it's a design flaw in ext3, not something that can be fixed without a major re-write.

something along the lines that the ext3 journal doesn't know what blocks are related to the metadata, so to avoid revealing old data that may be on disk, the filesystem is required to flush all pending writes.

the XFS/ext4 'problem' you refer to is the way every filesystem other than ext3 works. If you don't do a fsync, the data isn't safe and a crash at the wrong time can give you grief.

the functionality that makes this less of a problem on ext3 is the same functionality that makes it behave so horribly when you do a fsync

this is good for crash-prone desktop systems running software that wasn't written to be crash safe (note that it doesn't make the systems safe, it just reduces the probability of data loss)

but if you are running any software that is written to be crash safe, ext3 is about the worst filesystem you could use (in some cases worse than ext2 or other non-journaled filesystem).

Temporary files: RAM or disk?

Posted Jun 4, 2012 18:34 UTC (Mon) by raven667 (subscriber, #5198) [Link] (2 responses)

fsync on tmp is uniquely pointless since the contents are not intended to survive a normal restart let alone a system fault. Data loss in /tmp is OK and expected during a restart, for persistant data try /var/tmp

Temporary files: RAM or disk?

Posted Jun 5, 2012 12:34 UTC (Tue) by roblucid (guest, #48964) [Link] (1 responses)

Unfortunately, one of the practical issues with /tmp being TMPFS based is the lack of consensus on the reboot semantics (UNIX & Linux). It was quite common in past for /tmp files only to be cleaned when older than 24hrs.

There was even debate in openSUSE's FATE, when it was proposed to default to the FHS behaviour and delete /tmp files by default (due to misfeature of old SuSE Linux automatic deletion needed to be explicitly turned on, which caused maintenance issues as developers who "knew" /tmp is automatically cleaned out, wouldn't ensure cleanup on process crashes and so on).

Temporary files: RAM or disk?

Posted Jun 5, 2012 15:40 UTC (Tue) by nybble41 (subscriber, #55106) [Link]

> It was quite common in past for /tmp files only to be cleaned when older than 24hrs.

The distinction which I learned, which may very well be non-universal, was that /tmp is for temporary files which are tied to a particular process. These files can be removed as soon as the process exits, including on reboot. The /var/tmp directory is for temporary files which may need to outlast a given process, possibly across reboots, and should thus be cleared out much more sparingly based on the ages of the files.

Of course, not all processes follow this distinction.