io_uring vs pread: How Batched NVMe I/O Actually Scales

Modern solid-state storage subsystems have shifted the primary bottleneck of input/output operations from mechanical medium traversal to host CPU operating system overhead. Solid-state drives (SSDs) operating across PCI Express (PCIe) buses using the Non-Volatile Memory Express (NVMe) interface deliver raw physical latencies under 10 microseconds for 4 KiB block transfers. At these hardware speeds, the legacy software abstractions that mediated block I/O throughout the Unix era introduce measurable, unsustainable latency penalties.

When comparing io_uring vs pread for high-throughput random read workloads, developers confront the fundamental divergence between synchronous single-request dispatch and batched asynchronous ring architectures. The traditional POSIX pread(2) system call forces a synchronous context switch, traps into the kernel, allocates and submits a block I/O request, and puts the calling thread to sleep until hardware completion. In contrast, Linux's modern io_uring interface separates submission from consumption using shared circular memory rings, allowing user space to bypass recurring system call invocation and maintain deep hardware queue occupancy.

Bypassing the operating system page cache using the O_DIRECT flag exposes the mechanical reality of these two approaches. When serial 4 KiB reads are dispatched across a storage interface, execution alternates between hardware activity and CPU context manipulation. Saturating enterprise NVMe drives requires keeping their internal flash channels, dies, and controller cores perpetually saturated, which synchronous I/O loops simply cannot achieve.

The Anatomy of a Read: Synchronous pread Syscall Datapath

The execution path of a synchronous pread(2) operation reveals how legacy kernel conventions tax the host CPU. A single 4 KiB read traversing the synchronous path executes an extensive sequence of kernel operations before any data reaches the user-allocated memory buffer:

[ User Space ]
     │
     │  1. pread(fd, buf, 4096, offset)
     ▼
[ Kernel: System Call Handler ]
     │  2. Trap: sys_enter (Context switch, register spills, KPTI page table swap)
     │  3. VFS dispatch: vfs_read() -> blkdev_read_iter()
     ▼
[ Kernel: Block Layer (blk-mq) ]
     │  4. bio allocation & initialization (O_DIRECT)
     │  5. Direct mapping of user pages to scatter-gather list
     │  6. Request dispatch: tag allocation from hardware queue bitmap
     ▼
[ Kernel: NVMe Driver ]
     │  7. Build NVMe command (SQE) in host DRAM
     │  8. MMIO write: Doorbell register across PCIe bus (~1 µs overhead)
     ▼
[ NVMe Controller Hardware ]
     │  9. DMA fetch command -> Flash translation layer (FTL) -> NAND read
     │ 10. DMA transfer data directly to user-space host physical memory
     │ 11. Controller post CQE -> Fire MSI-X hardware interrupt
     ▼
[ Host CPU Interrupt & Wakeup ]
     │ 12. Interrupt Service Routine (ISR) -> softirq completion
     │ 13. bio completion callback -> wake_up_process(thread)
     │ 14. Scheduler activates thread (wait queue removal, runqueue append)
     │ 15. Context switch back to user mode -> sys_exit
     ▼
[ User Space Thread Resumes ]

Every single synchronous read forces the calling thread through this entire lifecycle. Step 2 requires entering Ring 0 via the syscall instruction. Following vulnerabilities such as Meltdown and Spectre, modern Linux kernels running with Kernel Page Table Isolation (KPTI) must swap user and kernel page tables upon kernel entry and exit. This operation invalidates un-tagged entries in the translation lookaside buffer (TLB), elevating memory access latencies for instructions executed immediately after the context transition.

At Step 6, the Linux block multi-queue layer (blk-mq) maps the target block to a software staging queue and attempts to obtain a unique hardware tag. Once allocated, the NVMe driver writes a 64-byte command into the host memory submission queue allocated for that CPU core. However, to inform the physical NVMe controller that a command is waiting, the driver executes an explicit Memory-Mapped I/O (MMIO) register write to the NVMe Submission Queue Tail Doorbell register (Step 8). PCIe MMIO writes are un-cached and posted, frequently stalling the issuing CPU core pipeline for several hundred nanoseconds while traversing the PCIe root complex.

Most critically, at Step 8, the calling thread can make no further forward progress. The kernel invokes io_schedule(), changing the task state to TASK_UNINTERRUPTIBLE and yielding the CPU core to the scheduler. When the hardware finishes retrieving the 4 KiB page from flash memory, the controller fires an MSI-X interrupt. The CPU processes the interrupt, marks the task runnable, executes a second context switch, and restores user-space registers.

This model yields a rigid throughput ceiling governed by Little's Law:

$$\text{IOPS} = \frac{\text{Queue Depth}}{\text{Latency}}$$

With serial pread(), Queue Depth is permanently fixed at 1. If an ultra-fast enterprise NVMe drive sustains a round-trip latency of 12 microseconds (encompassing flash cell access, DMA transfer, interrupt handling, and two context switches), a single thread using serial pread() can never exceed:

$$\text{IOPS}_{\text{max}} = \frac{1}{0.000012,\text{s}} \approx 83,333\text{ IOPS}$$

The SSD controller remains largely idle, waiting for the host CPU to crawl through its scheduling machinery between individual reads.

io_uring vs pread: Execution Model and NVMe Queue Depth Scaling

The architectural distinction between io_uring vs pread is rooted in how each paradigm models device parallelism. Modern NVMe storage controllers do not operate as single-command serial pipelines. Instead, they expose up to 64,000 independent hardware queues, each capable of holding up to 64,000 commands concurrently. Internally, modern SSDs partition NAND flash into multiple packages, dies, planes, and channels. To achieve manufacturer-rated throughputs—often exceeding 800,000 to 1,500,000 random read IOPS—the storage controller must receive multiple in-flight read operations simultaneously, allowing its internal Flash Translation Layer (FTL) to stripe requests across all parallel NAND dies.

Understanding this hardware reality clarifies why database architectures such as How LSM Tree Leveled Compaction Internals Actually Work consistently refactor their storage engines away from synchronous block reads toward asynchronous request pipelines.

The io_uring subsystem breaks the coupling between system call invocation and device dispatch. Instead of utilizing synchronous function signatures, it instantiates two lockless ring buffers allocated in kernel memory and mapped directly into the process's user-space address space via mmap(2):

1. Submission Queue (SQ): A ring buffer containing 64-byte Submission Queue Entries (SQEs). The application writes read descriptors into this buffer without transitioning into the kernel.
2. Completion Queue (CQ): A ring buffer containing 16-byte (or 32-byte in extended variants) Completion Queue Entries (CQEs). The kernel appends results here; the application reads completions via simple memory pointer dereferencing.

       USER SPACE                               KERNEL SPACE
┌─────────────────────────┐               ┌─────────────────────────┐
│ Application Task Engine │               │   Linux Block (blk-mq)  │
│                         │               │                         │
│   ┌─────────────────┐   │               │   ┌─────────────────┐   │
│   │ SQ Ring Buffer  │───┼──Shared Memory┼──>│ SQ Ring Buffer  │   │
│   │ (Head / Tail)   │   │               │   │ (Head / Tail)   │   │
│   └─────────────────┘   │               │   └─────────────────┘   │
│            │            │               │            │            │
│      enqueue 32 SQEs    │               │    dequeue batch of 32  │
│            │            │               │            │            │
│            ▼            │               │            ▼            │
│   ┌─────────────────┐   │               │   ┌─────────────────┐   │
│   │ io_uring_enter()│───┼─Single Syscall┼──>│ NVMe SQ Doorbell│   │
│   │ (Batched Enter) │   │               │   │ (1 MMIO write)  │   │
│   └─────────────────┘   │               │   └─────────────────┘   │
│            ▲            │               │            │            │
│      reap 32 CQEs       │               │    process 32 ISR/DMAs  │
│            │            │               │            │            │
│   ┌─────────────────┐   │               │   ┌─────────────────┐   │
│   │ CQ Ring Buffer  │<──┼──Shared Memory┼───│ CQ Ring Buffer  │   │
│   │ (Head / Tail)   │   │               │   │ (Head / Tail)   │   │
│   └─────────────────┘   │               │   └─────────────────┘   │
└─────────────────────────┘               └─────────────────────────┘

When an application wants to sustain a queue depth of 32 random reads using io_uring, it writes 32 SQEs into the shared ring buffer, updates the user-space tail pointer, and executes a single io_uring_enter(2) system call.

The consequences for hardware efficiency are immediate:

• Amortized System Call Overhead: A single transition into Ring 0 dispatches 32 operations. The per-request system call overhead drops by a factor of 32.
• MMIO Doorbell Coalescing: The kernel's NVMe driver processes the batched requests and submits them to the hardware controller in a unified burst, updating the physical PCIe NVMe doorbell register once for the entire batch rather than 32 separate times.
• Continuous Device Saturation: The NVMe controller's internal scheduler receives 32 distinct physical block addresses in immediate succession. It can reorder the internal NAND reads to minimize command-to-data bus collisions across dies, achieving maximum flash concurrency.
• Interrupt Aggregation: Because multiple commands complete in close temporal proximity, the NVMe controller's MSI-X interrupt mitigation logic coalesces completions, triggering a single interrupt vector for multiple I/O events.

For specialized workloads requiring even lower submission latency, io_uring also supports the IORING_SETUP_SQPOLL flag, which spawns an internal kernel thread that polls the shared submission ring directly. This architecture, analyzed in detail in How io_uring Submission Queue Polling Actually Works, eliminates the io_uring_enter(2) system call entirely from the steady-state fast path.

Architectural Comparison: Data Paths and Queue Management

To understand how these differences manifest at scale, observe the mechanical progression of both systems operating over equivalent workloads. When executing 32 random reads of 4 KiB each, pread processes them sequentially ($QD=1$), while io_uring issues them in parallel ($QD=32$).

Synchronous Serial Path (pread, QD=1)
Time ────────────────────────────────────────────────────────────────────────►
Thread:  [Syscall][Block Wait...][Syscall][Block Wait...][Syscall][Wait...]
PCIe:       │▲                │▲    │▲                │▲    │▲       │▲
            ▼│                ▼│    ▼│                ▼│    ▼│       ▼│
NVMe HW:    [Flash Read 1]    [Flash Read 2]    [Flash Read 3]    ... (x32)
Die Concurrency: 1 Die active at any moment; remaining 31 flash channels idle.

Asynchronous Batched Path (io_uring, QD=32)
Time ────────────────────────────────────────────────────────────────────────►
Thread:  [Enqueue 32 SQEs][Syscall: io_uring_enter][Harvest 32 CQEs]
PCIe:       ││││ ... (Batched Doorbell)        ▲▲▲▲ ... (Coalesced MSI-X)
            ▼▼▼▼                               ││││
NVMe HW:    [Flash Read 1 - Die 0 ]────────────┘│││
            [Flash Read 2 - Die 1 ]─────────────┘││
            [Flash Read 3 - Die 2 ]──────────────┘│
            [Flash Read 4 - Die 3 ]───────────────┘
Die Concurrency: All internal NAND channels and dies actively transferring.

The difference extends beyond instruction counts to the memory bus. In synchronous serial pread(), the kernel allocates a new struct bio and request descriptor for every read, walks the page tables to pin the target user buffer, and frees these kernel allocations upon return to user space.

Under io_uring, the memory footprint is stable and reusable. Applications can pre-register their target data buffers using io_uring_register(..., IORING_REGISTER_BUFFERS, ...). This optimization maps the user-space virtual buffers into kernel space once during initialization, locking the underlying physical pages (via pinned page frames). Subsequent reads bypass the standard virtual-to-physical address translation (get_user_pages_fast) on every request, allowing the host CPU to stream physical addresses straight into the device's scatter-gather DMA descriptors. This high-efficiency direct memory architecture parallels optimizations found in real-time gaming engines; for instance, How DirectStorage Asset Streaming Works in Modern Games employs an identical philosophy of saturated hardware queues and direct DMA paths to stream gigabytes of asset textures without hitching the main thread.

Implementing High-Throughput Random Reads with liburing

The standard, production-grade interface to io_uring is provided through liburing, a low-overhead userspace library authored by the kernel subsystem maintainers. Below is a minimal, complete implementation demonstrating how to structure an asynchronous random read engine maintaining a concurrent queue depth of 32 using direct I/O (O_DIRECT).

#define _GNU_SOURCE
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#define QUEUE_DEPTH 32
#define BLOCK_SIZE  4096

struct io_request {
    uint64_t offset;
    int index;
};

int run_io_uring_reader(const char *filepath, size_t total_reads) {
    struct io_uring ring;
    int fd = open(filepath, O_RDONLY | O_DIRECT);
    if (fd < 0) {
        perror("open(O_DIRECT)");
        return 1;
    }

    // Initialize the io_uring instance
    if (io_uring_queue_init(QUEUE_DEPTH, &ring, 0) < 0) {
        perror("io_uring_queue_init");
        close(fd);
        return 1;
    }

    // O_DIRECT requires memory buffers aligned to physical sector boundaries (typically 4096)
    void *buffers[QUEUE_DEPTH];
    for (int i = 0; i < QUEUE_DEPTH; i++) {
        if (posix_memalign(&buffers[i], BLOCK_SIZE, BLOCK_SIZE) != 0) {
            perror("posix_memalign");
            return 1;
        }
    }

    struct io_request req_meta[QUEUE_DEPTH];
    size_t submitted = 0;
    size_t completed = 0;

    // Seed initial pipeline to saturate target Queue Depth (32)
    for (int i = 0; i < QUEUE_DEPTH && submitted < total_reads; i++) {
        struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
        if (!sqe) break;

        req_meta[i].offset = (submitted * BLOCK_SIZE);
        req_meta[i].index = i;

        io_uring_prep_read(sqe, fd, buffers[i], BLOCK_SIZE, req_meta[i].offset);
        io_uring_sqe_set_data(sqe, &req_meta[i]);
        submitted++;
    }

    // Submit the initial batch to the device
    io_uring_submit(&ring);

    // Event loop: maintain exactly QUEUE_DEPTH operations in flight
    while (completed < total_reads) {
        struct io_uring_cqe *cqe;
        
        // Wait for at least one completion event
        int ret = io_uring_wait_cqe(&ring, &cqe);
        if (ret < 0) {
            perror("io_uring_wait_cqe");
            break;
        }

        struct io_request *req = (struct io_request *)io_uring_cqe_get_data(cqe);
        if (cqe->res < 0) {
            fprintf(stderr, "Async Read failed: %s\n", strerror(-cqe->res));
        }

        completed++;
        int slot = req->index;
        io_uring_cqe_seen(&ring, cqe);

        // Immediately replenish completed slot if more work remains
        if (submitted < total_reads) {
            struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
            if (sqe) {
                req_meta[slot].offset = (submitted * BLOCK_SIZE);
                req_meta[slot].index = slot;

                io_uring_prep_read(sqe, fd, buffers[slot], BLOCK_SIZE, req_meta[slot].offset);
                io_uring_sqe_set_data(sqe, &req_meta[slot]);
                submitted++;
                
                // Submit immediately to maintain hardware pipe saturation
                io_uring_submit(&ring);
            }
        }
    }

    // Clean up allocated resources
    for (int i = 0; i < QUEUE_DEPTH; i++) {
        free(buffers[i]);
    }
    io_uring_queue_exit(&ring);
    close(fd);
    return 0;
}

Contrast this implementation with the standard serial loop required by pread():

int run_pread_reader(const char *filepath, size_t total_reads) {
    int fd = open(filepath, O_RDONLY | O_DIRECT);
    if (fd < 0) return 1;

    void *buf;
    posix_memalign(&buf, BLOCK_SIZE, BLOCK_SIZE);

    for (size_t i = 0; i < total_reads; i++) {
        off_t offset = i * BLOCK_SIZE;
        ssize_t bytes_read = pread(fd, buf, BLOCK_SIZE, offset);
        if (bytes_read != BLOCK_SIZE) {
            // Handle error or EOF
            break;
        }
    }

    free(buf);
    close(fd);
    return 0;
}

While the pread() loop is simple to write, it fundamentally starves the underlying hardware. The single execution thread repeatedly yields its core to the operating system scheduler, spends significant execution time processing system call entry/exit routines, and enforces a strict serial barrier between block requests.

Latency Tails, CPU Utilization, and Block Layer Overhead

Throughput measurements capture only part of the operational contrast. The structural differences between synchronous system calls and ring buffers alter the latency distribution profiles of production servers.

Synchronous pread (QD=1):
Latencies:  [ p50: ~12 µs ] ──────────────────── [ p99: ~45 µs ] ─── [ p99.9: ~180 µs ]
Variance: High latency dispersion caused by CPU scheduling context jitter.

io_uring Batched Pipeline (QD=32):
Latencies:  [ p50: ~160 µs (Batch) ] ─────────── [ p99: ~210 µs ] ── [ p99.9: ~260 µs ]
Variance: Deterministic tail profile; device hardware internal parallelism maximized.

In the synchronous pread() architecture, individual read latencies at the 50th percentile ($p50$) can appear deceptively low—typically 10 to 14 microseconds on modern enterprise NVMe drives. Because the hardware queue depth is 1, there is zero queueing delay inside the NVMe controller; the request is handled immediately by the first available flash channel.

However, the tail latency ($p99$ and $p99.9$) for pread() frequently balloons by an order of magnitude. This latency amplification does not stem from flash medium degradation, but from operating system scheduling artifacts. Because the thread voluntarily relinquishes the CPU via io_schedule() upon every request, it depends on the Linux CFS (Completely Fair Scheduler) or EEVDF (Earliest Eligible Virtual Deadline First) scheduler to re-acquire CPU execution rights when the hardware interrupt fires. If competing threads occupy the core, or if the interrupt routing targets a CPU socket different from the thread's local NUMA node, cache invalidation and scheduling latency directly lengthen the observed read duration.

With io_uring running at $QD=32$, the latency profile shifts. The per-request amortized turnaround increases (often to 150–200 microseconds) because each request now incurs intentional queueing latency within the SSD's controller queue. However, the tail latency envelope remains remarkably bounded. The hardware controller operates within its deterministic throughput sweet spot, and the host thread retains its core allocation without repeatedly dropping into deep scheduling wait states.

From a CPU utilization standpoint, serial pread spends an enormous proportion of its total cycle budget performing unproductive bookkeeping:

• Register spilling to the kernel stack frame.
• Context switching between user virtual memory and kernel space.
• Allocating, mapping, and freeing individual struct bio instances inside the blk-mq layer.
• Uncoordinated individual MMIO register writes across the PCIe root complex.

By contrast, io_uring structures work as a continuous assembly line. The host CPU acts as an orchestrator, filling circular memory entries that the kernel consumes in optimized vector sweeps. By delegating request aggregation to memory structures rather than system call trap boundaries, io_uring converts wasted CPU execution cycles back into usable computational bandwidth.

Choosing the Right Abstraction

Despite the clear architectural performance advantages of io_uring at scale, pread() remains appropriate in specific systems scenarios:

1. Deterministic Single-Thread CLI Tools: Utilities that scan configuration files or read linear segments where the data is either already cached in the Linux Page Cache or where operational throughput is entirely secondary to simplicity.
2. Predictable Low-Queue Memory Mapped Scenarios: Applications performing occasional, purely random single-block reads where spinning up kernel submission queues adds unnecessary complexity.
3. Restricted Sandboxes: Environments where restrictive seccomp security filters disallow io_uring system call primitives (io_uring_setup, io_uring_enter, io_uring_register) due to historical attack surface considerations in older kernel versions.

For high-performance systems development, high-density key-value stores, distributed storage backends, and database engines reading non-volatile media directly with O_DIRECT, relying on pread() guarantees that hardware potential is squandered.

Conclusion

The evolution from pread() to io_uring represents a structural shift in Linux storage architecture. While POSIX synchronous file access was conceived for mechanical media with milliseconds of seek latency—where system call overhead was mathematically insignificant—it imposes a strict ceiling on PCIe-attached NVMe storage.

By replacing synchronous context switches with shared user-kernel memory ring buffers, io_uring bridges the gap between processor performance and solid-state hardware parallelism. It decouples the application from the operating system scheduler, pools hardware interactions into batched operations, and maximizes NVMe queue depths without requiring user-space drivers to abandon the Linux storage stack. For modern software architectures demanding millions of random read operations per second, batched asynchronous rings are no longer merely an optimization; they are the baseline mechanism required to exploit non-volatile storage hardware.

Measured on our own hardware: io_uring vs pread: Random Reads at Queue Depth 32

How many random 4 KiB reads per second can one thread sustain with serial pread() syscalls, compared with io_uring submitting 32 at a time, when the page cache is taken out of the picture?

We ran it. The numbers below come from a program executed on the server hosting this site on 2026-09-13 — an AMD EPYC 9354P 32-Core Processor with 8 cores visible, 31.3 GB of memory, Linux 6.8.0-139-generic.

Metric	Value
io uring speedup factor	8.12
block bytes	4096
blocks read	20000
io uring best seconds	0.5032
io uring iops	39748
o direct	true
pread best seconds	4.0855
pread iops	4895
pread mean latency us	204.28
queue depth	32
repeats	3

This is a shared virtual server, not an isolated test rig, so treat the absolute figures as indicative and the ratio between the two cases as the finding. The full method, the machine specification, and the complete source code are on the io_uring vs pread: Random Reads at Queue Depth 32 benchmark page, so you can check the method or run it yourself.

References

• Axboe, J. (2019). Efficient IO with io_uring. Linux Kernel Documentation. https://kernel.dk/io_uring.pdf
• NVM Express Workgroup. (2021). NVM Express Base Specification, Revision 2.0. https://nvmexpress.org/specifications/
• Linux Kernel Organization. (2023). Block Layer Multiqueue (blk-mq) Architecture. https://www.kernel.org/doc/html/latest/block/blk-mq.html