Insane number of files created per second [LWN.net]

Insane number of files created per second

Posted Aug 25, 2016 2:47 UTC (Thu) by pr1268 (guest, #24648)
Parent article: Btrfs and high-speed devices

Is it just me, or does the creation of e.g. 175,000 or 200,000 files per second on a single computer (albeit one with multiple cores) on (presumably) a single filesystem seem just the slightest bit outrageous?

Granted, this is Facebook, and their data storage requirements are colossal (to say the least), but I just imagine a company, even of FB size, to distribute massive file creation of that order to at least, say, 10 or 20 computers.

;-)

P.S. A huge thanks to Mr. Mason for his contributions to BTRFS and Linux in general.

Insane number of files created per second

Posted Aug 25, 2016 7:51 UTC (Thu) by farnz (subscriber, #17727) [Link]

To put that sort of number into context, it's on the order of 1 file per second for each person in 0.02% of Facebook's daily active user base. Spread that userbase across 5,000 machines, and it's still only one file per second per user per machine - and that's assuming it's the sort of data that doesn't benefit from locality of access, so can be spread sensibly.

Gives you a sense of how badly intuition can break down at unusual scales...

Insane number of files created per second

Posted Aug 25, 2016 18:00 UTC (Thu) by josefbacik (subscriber, #90083) [Link] (2 responses)

We just use these benchmarks as a way to classify our metadata performance overhead. So while we don't personally _need_ 200k files/sec, the workload quickly shows us where we have pain points that would cause us problems with real world workloads. So seeing the tree lock show up in this test is a good indication that we could hit latencies with our normal day to day workloads, so it's something we work on.

Insane number of files created per second

Posted Aug 27, 2016 12:07 UTC (Sat) by walex (guest, #69836) [Link] (1 responses)

«we don't personally _need_ 200k files/sec, the workload quickly shows us where we have pain points that would cause us problems with real world workloads.»

That is about absolute speed of metadata operations, and that's not the real issue. The real issue being described here is that metadata operations don't scale with hardware capacity, regardless of the absolute speed desires, that is the real issue is about design.

The real issue exists because it is relatively easy to have scalable data speeds: just choose a domain which is "easily parallelizable" and throw more disks more RAM more threads at it. For data, RAID scales up speed pretty nicely.

By contrast metadata operations are not easily parallelizable, because there are dependencies across metadata, both structural dependencies and ordering dependencies, and therefore fairly fine grained locking must be used (ordering) and RAID does not work as well (structural).

The biggest problem with hard-to-parallelize metadata is not even file creation rates, it is whole-tree scans, like fsck or RSYNC scans. I have seen a lot of cases where some "clever" person designed a storage subsystem for average data workloads, and that became catastrophes during peak metadata workloads, which must happen quite periodically, one way or another.

Insane number of files created per second

Posted Aug 29, 2016 17:19 UTC (Mon) by SEJeff (guest, #51588) [Link]

I'm pretty sure the GP, one of the lead developers of BTRFS and mentioned in this article, is fully aware of this. No need to reiterate :)

Insane number of files created per second

Posted Aug 25, 2016 18:42 UTC (Thu) by ott (guest, #110845) [Link] (3 responses)

Say for example you want to check-out a source repository that has millions of small files, file creation can be a serious bottleneck.

Insane number of files created per second

Posted Aug 25, 2016 23:00 UTC (Thu) by gerdesj (subscriber, #5446) [Link] (2 responses)

I'm intrigued. Do you mean that checking out on a btrfs is slowed down as opposed to say ext4 or xfs?

I've just checked out Libre Office a few times and copied it around a bit. As a long time Gentoo user on a fair few systems, I'm quite familiar with how some pretty large software projects behave from source to binary.

LO is "only" about 82,000 files at 2.2GB. On my laptop with a reasonably modern Core i7 quad core + HT, with 16GB RAM and 1 x SSD + 1 x spinning disc, it takes a fair old while to compile and needs rather a lot of space. On previous laptops it used to be an overnight thing. Check out times are the least of my worries.

Back to your assertion, even if you are checking out over a 10GBs-1 connection I doubt that the fs is holding you back. What kind of projects involve millions of files? Also what sort of repo are you using?

Cheers
Jon

Insane number of files created per second

Posted Aug 26, 2016 3:25 UTC (Fri) by ott (guest, #110845) [Link]

Disclaimer: I work at Facebook but not in that team, so I don't know the specifics. However Google has released some numbers about their monorepo (http://cacm.acm.org/magazines/2016/7/204032-why-google-st...), and it's about 9M files.
It was just to say, it's not "outrageous" to have a directory with million of files, which could be checked out all at once. The FS performance definitely plays a major role there.

Insane number of files created per second

Posted Aug 26, 2016 15:55 UTC (Fri) by cwillu (guest, #67268) [Link]

Again though, this isn't _about_ file creation, it's about metadata ops; file creation is a convenient way to exercise it, while also being relevant to some usages.

Grepping a tree for the first time today? Atime updates. "make clean" in a large repository? combinations of mtime and deletions. "apt-get upgrade" with a bunch of pending updates? Oh, you better believe there's a shittonne (SI technical unit) of file creations, fsyncs, mv's and other metadata updates.

It ends up being a major but sometimes hidden determinant of how fast you can get shit done.

Insane number of files created per second

Posted Sep 7, 2016 21:44 UTC (Wed) by Pc5Y9sbv (guest, #41328) [Link]

Many applications may need to multiplex or demultiplex files out of container formats. In the case of scientific imaging, we have cases today where we want to transcode a proprietary image, unpack its constituent image tiles, and manipulate the files with open source tools which do not understand the proprietary container format.

A single microscope slide image might have 100k to 200k tiles in it, totalling a few hundred GB of space. We often want to unpack and host each tile as an individual JPEG file on a static http file server, where a client-side pan/zoom viewer can retrieve just the tiles it need as a user navigates the viewport. If we are transcoding the tiles, we may be CPU limited but if we are simply extracting them without changing the codec format, we are limited by the metadata rates on the filesystem.

Conversely, time-series imagery might be produced as a sequence of image frames from data acquisition tools and later multiplexed and/or re-compressed into a movie container format. An hour at 60 fps is 216k frames. However, scientists may want to apply other batch processing steps to each image frame before converting it to a movie file for archving or distribution. These jobs could run much faster than real-time, and the metadata rates can become the bottleneck. Such processing is often too exploratory or ad hoc to justify a custom, tuned implementation where you would get your hands on libraries of all needed algorithms, plan your buffer pipeline, and avoid bounding data through external commands with file I/O.