Insane number of files created per second

Many applications may need to multiplex or demultiplex files out of container formats. In the case of scientific imaging, we have cases today where we want to transcode a proprietary image, unpack its constituent image tiles, and manipulate the files with open source tools which do not understand the proprietary container format.

A single microscope slide image might have 100k to 200k tiles in it, totalling a few hundred GB of space. We often want to unpack and host each tile as an individual JPEG file on a static http file server, where a client-side pan/zoom viewer can retrieve just the tiles it need as a user navigates the viewport. If we are transcoding the tiles, we may be CPU limited but if we are simply extracting them without changing the codec format, we are limited by the metadata rates on the filesystem.

Conversely, time-series imagery might be produced as a sequence of image frames from data acquisition tools and later multiplexed and/or re-compressed into a movie container format. An hour at 60 fps is 216k frames. However, scientists may want to apply other batch processing steps to each image frame before converting it to a movie file for archving or distribution. These jobs could run much faster than real-time, and the metadata rates can become the bottleneck. Such processing is often too exploratory or ad hoc to justify a custom, tuned implementation where you would get your hands on libraries of all needed algorithms, plan your buffer pipeline, and avoid bounding data through external commands with file I/O.