An update on storage standards

We're bad at marketing

We can admit it, marketing is not our strong suit. Our strength is writing the kind of articles that developers, administrators, and free-software supporters depend on to know what is going on in the Linux world. Please subscribe today to help us keep doing that, and so we don’t have to get good at marketing.

In a second-day plenary session at the 2017 Linux Storage, Filesystem, and Memory-Management Summit, Fred Knight updated the attendees on what has happened in the storage standards world over the last year. While the transports (e.g. Fibre Channel, Ethernet) and the SCSI protocol have not seen a ton of changes over the last year, the NVM Express (NVMe) standards have had a lot of action.

On the transport standards side, Fibre Channel has some new speeds available (32Gb/128Gb) and some in the works (64Gb/256Gb). Terabit Fibre Channel is on the roadmap, Knight said, "it will be interesting to see if they can reach it". Ethernet has added new speeds (from 2.5Gb up through terabit) as well as new markets (e.g. automotive). NVMe has a new command set and multiple connectivity options (e.g. PCIe, RDMA over Ethernet, InfiniBand).

SCSI

For SCSI, a simplified binding status command has been added (TEST BIND). The WRITE ATOMIC command has been added; it will allow writes that either write all of the data or none of it. In addition, a 32-byte variant of WRITE SCATTER has been added. The combination of atomic with scatter writes, which was discussed at LSFMM 2016, is not likely to make an appearance, Knight said. Some storage companies are objecting to it; they have figured out that Linux gets this right and they might get it wrong, so they want to leave it alone, he said.

The WRITE STREAM command for stream IDs has been added to the standard. In addition, the BACKGROUND CONTROL command has been added to allow some control over the background tasks that a storage device may be doing (e.g. garbage collection). That background processing can impact I/O at times, so users have wanted some ability to change the parameters governing it.

Some predefined feature sets (e.g. "2010 Basic", "2016 Basic", "Maintenance", "Provisioning") have been established that storage devices can advertise and hosts can check to see what features are available. New feature sets will be added over time, Knight said, so it will be an ongoing process. Ted Ts'o asked if devices can advertise some feature set but implement additional features beyond those in the set; Knight said they could.

The biggest thing in the SCSI world over the last year has been drive depopulation, Knight said. The way that drives fail is different than what was expected, so it is often just a part of the drive that goes bad. If you have an 8TB drive where one head goes bad, you can just disable that portion of the drive and turn it into a 6TB drive, for example.

For both SCSI and ATA, the "repurposing" support is available. The drive will be available at the reduced capacity and the data that was there will either no longer be available or will be at a different logical block address (LBA) than it was before. The GET PHYSICAL ELEMENT STATUS command can be used to determine what element is failing and what capacity it has. The REMOVE ELEMENT AND TRUNCATE command can then be used to disable the failing piece. The drive can be reformatted; if another element fails, the process can be repeated.

There has been longtime support in the standard for reading the whole drive to try to recover the data that is still there. There are commands that will provide information that the next N LBAs are bad, so they can be skipped. That will allow the host to recover as much data as it can after an element failure.

There was some talk last year about a "data preserving" mode for depopulation. It is complex, however, so it will take a lot longer to be added if it is at all. There are "a bunch of people" asking if there really is a need for that, so it could "end up on the cutting room floor", Knight said.

NVMe

It has been a busy year for the NVMe group, he said. The formal specifications for the fabric were published. The "sanitize" command has been formalized; it can be used to clean the drive before it is repurposed. Other mechanisms, such as crypto erase and block erase, will ruin SSDs, so those devices do not implement them.

A device crash-dump facility, called "telemetry", has been added to the specification. Stream IDs have been added as well. Some support for persistent reservation has been added, but it is incompatible with the SCSI feature, which is not what was wanted. A compatible version is currently a work in progress.

Virtualized/emulated controllers are another feature that has been added. This allows virtual machines to think they have their own dedicated NVMe controller. Each guest gets a dedicated queue pair mediated by the hypervisor. That allows multiple guests to share a single physical NVMe adapter.

The NVMe group has around 80 people on a two-hour call each week, while the T10 (SCSI), T11 (Fibre Channel), and T13 (ATA) committees have three-day meetings every few months. There is some concern that those groups are running out of things to do. The NVMe group is ramping up, he said, while the others are much more stable than they were a few years ago.

Mathew Wilcox asked how Knight felt about all the different errors that can be reported by storage devices essentially all boiling down to EIO (I/O error) in Linux. Knight chuckled and pointed to all of the kinds of sense codes and media errors that are reported by storage devices. But as Martin Petersen pointed out, for POSIX they really all do need to be mapped to EIO because applications don't understand anything else.