Block multi-queue status

Please consider subscribing to LWN

Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

The block layer multi-queue work was the subject of a discussion led by Nic Bellinger at this year's Linux Storage, Filesystem, and Memory Management (LSFMM) Summit. One might have expected Jens Axboe and Christoph Hellwig to be part of any discussion of that sort, but Axboe was ill and Hellwig was boycotting the location, Bellinger said. That left it up to him, though Axboe did provide some notes for the block multi-queue work.

From those notes, which Bellinger also provided to me, he said that the initial multi-queue work was merged for the 3.13 kernel. It only supported the virtio_block driver and "mostly worked". There have been changes since that time, but overall it appears to be architecturally sound.

A basic conversion of the Micron mtip32xx SSD driver has been done. The existing driver is a single queue with shared tags. After the conversion, there are eight queues available. It runs at about 1.8 million I/O operations per second (IOPS), which is about the same as the unpatched driver. It works well on a two-socket system, but falls down on a four-socket machine.

Part of the problem is a lack of tags. The percpu-ida code is not going to cut it for tag assignment. An audience member said they had replaced percpu-ida recently, which eliminated the tags problem. Matthew Wilcox noted another tag problem: the Linux implementation makes them unique per logical unit number (LUN), while the specification says they need only be unique per target. In addition, James Bottomley said, the specification allows for 16-bit tags, rather than the 8-bit tags currently being used.

Bellinger then moved on to his and Hellwig's work on adding multiple queues to the SCSI subsystem. Since 2008, there have been reports of small-block random I/O performance issues in the SCSI core. Most of that is due to cache-line bouncing of the locks. It limits the performance of that type of I/O to 250K IOPS. Getting performance to 1 million IOPS using multiple SCSI hosts was taking up to 1/3 of the CPU on spinlock contention.

So Bellinger used the block multi-queue infrastructure to preallocate SCSI commands, sense buffers, protection information, and requests. His initial prototype had no error handling, any errors would oops the system. But he was able to get 1.7 million IOPS out of that prototype code. Hellwig got error handling working and has been driving it to something that could be merged.

There are plans for an initial merge, but Bottomley was concerned that Bellinger and Hellwig did not agree on whether the faster IOPS mode was the default case or not, with Bellinger on the side of it being an exception. Bellinger said that there had been no agreement yet on that, which would make merging difficult, Bottomley said.

Converting drivers should be fairly easy, Bellinger said, though Bottomley cautioned that there would need to be a lot of work done on lock elimination in the drivers. There is also a question of per-queue vs. per-host mailboxes, Bottomley said. There is work to do to determine which submission model will work best, he said.

[ Thanks to the Linux Foundation for travel support to attend LSFMM. ]