We Replaced mmap with io_uring in Our Rust Query Engine. It Got Slower.

In the beginning, there was mmap. It was convenient: it let us lazily read huge numbers of Arrow IPC files from disk without managing memory ourselves. It fit our file format perfectly — Arrow IPC’s layout is designed for zero-copy random access, and mmap gives you exactly that.

Then we deployed to production, ran real concurrent query loads, and mmap became a real problem.

Our Workload

At Conviva, we analyze trillions of events a day to pinpoint and diagnose end user experience. At the core of our architecture is an event and pattern analysis engine built on DataFusion, Arrow, Rust, Rayon, and Tokio. Raw events get transformed, encoded in a proprietary mostly-numeric format, and stored in the cloud. We copy them to local NVMe and read large (~3–5 GB) Arrow IPC files. We chose Arrow IPC for simplicity and speed — its memory and disk layouts are identical, so decode cost is minimal, and mmap gives us zero-copy reads natively supported by arrow-rust. A typical query touches 6 columns across 8 batch files (one batch per file), ~1.6 GB per batch, ~13 GB total per day of data.

The Test Setup

Hardware: 192-core box, ~750 GB RAM. Two disk configs during the investigation: 2× NVMe LVM-striped (~5.5 GB/s fio ceiling) and 32× NVMe RAID-0 (~21 GB/s fio ceiling). Kernel 5.15 during investigation, 6.x in production.

The Production Symptom

At lighter loads, mmap worked well — fast, serving queries from raw events in seconds. The trouble started under heavier concurrency. Some latency increase under load is expected — more queries competing for the same CPU. But we saw p95s and p99s spike well beyond what linear scaling would predict, with rows scanned per core dropping sharply even after accounting for concurrency:

• OS page cache shrank — each pod consumed more memory as private allocations, less as shared cache
• A huge number of page faults
• p95 spiked from ~30s to 150s+ under real concurrent load
• Adding pods made it worse, not better

That pointed at mmap page-cache thrashing under memory pressure.

Controlled Benchmark: 1 Pod vs. 4 Pods

To isolate the effect, we ran a controlled test: 1 pod vs. 4 pods on the same host, same concurrent query load. We expected 4 pods to win — more parallelism, better isolation. We were wrong.

For 14-day queries — long enough to fill the page cache — 1 pod beat 4 pods by a real margin: 41% faster at max, >20% at p95. The mmap page cache lives on the host and is shared across pods, so the 4 pods weren’t fighting each other for CPU — they were fighting for page cache. perf record on the same run showed 100% lock contention at the kernel level.

The core issue: mmap’s page cache is implicit shared state. Every process on the host shares one cache, one lock hierarchy, one eviction policy. No single pod controls the resource that matters most for read latency, and as concurrency rises, everyone’s slice of it shrinks.

A Storm of Page Faults

We didn’t want to just guess, so we dug into the mmap mechanics and the page fault stats. When you mmap a file, the application gets a region of memory backed by the file on disk. Here’s what happens on access:

1. The CPU touches a Virtual Memory Address (VMA) with no physical page attached, and throws a page fault.
2. The kernel handles the exception: looks up the VMA, checks ownership, and acquires a lock, since another thread might be modifying or unmapping that virtual space concurrently.
3. Linux 6.4+ has a fast per-VMA lock; earlier kernels fall back to the slower mmap_lock.
4. Once the kernel confirms the VMA is file-backed, it triggers a file fault — a minor fault if the bytes are already in warm page cache from read-ahead, or a major fault if it must trigger physical I/O.

Recommended reading: mmap_lock scalability (LWN) and per-VMA locks (LWN, Suren Baghdasaryan’s design).

Under heavy page cache contention, read-ahead runs out of room and major faults spike. Here’s what that looked like — pidstat on one process during a stressful run:

23:26:46  RSS = 652 GB (87.88%)
23:27:47  RSS = 734 GB (98.91%)   ← peak, nearly all RAM
23:27:48  RSS starts dropping     ← kernel begins evicting
23:28:05  major faults appear: 571/s, 1352/s, 975/s

RSS grows to 98.91% of RAM → the kernel has no choice but to evict pages still needed → evicted pages get touched again → a major fault storm as they’re read back from disk. Early on, read-ahead keeps faults mostly minor and fast; as concurrent queries pile up, read-ahead stops keeping up and major faults spike.

Minor faults, meanwhile, ran sustained in the millions per second:

23:27:09   1,255,709 minor faults/sec
23:27:41   2,124,327 minor faults/sec
23:27:46   2,354,383 minor faults/sec

Each minor fault touches a cache line via atomics — at 2 million faults/sec, that’s enough to thrash L1/L2 entirely, which is deadly for an application that leans on large cache-resident lookup tables. Faults can also trigger TLB shootdowns, and CPUs only hold a few thousand TLB entries. (You can’t eliminate page faults, but you can manage them better — more on that in Part 2.)

An uncontended minor fault costs roughly 0.5–1 microsecond, so 2 million/sec is close to the ceiling of what mmap can sustain — and under real contention, the thread-visible delay runs well past that.

Virtual address space, meanwhile, had grown to ~3 TB from mmap’ing so many Arrow files:

Start:  3,125,750,740 KB (~2.9 TB virtual)
Peak:   3,209,184,828 KB (~2.98 TB virtual)

Modern kernels handle large VMA trees, but not for free — every fault does a VMA lookup, and every lookup takes the mmap lock (fast path notwithstanding).

Context switches told the same story:

cs = 2,106,576/sec
cs = 2,025,726/sec
cs = 1,944,397/sec
cs = 1,524,279/sec

Over 2 million context switches/sec, versus 14K/sec on a warm-cache run — 150x more. Every thread was constantly blocking on page faults, getting descheduled, and rescheduled once pages arrived.

Perf Top and Off-CPU Analysis

To confirm the link between page faults and lock contention, we compared perf top on cold vs. warm runs of the same query:

__filemap_add_folio adds a page to the page cache. It barely shows up warm, since the data’s already there; cold, under memory pressure, it dominates because pages are constantly evicted and re-inserted. Our actual query code drops from ~45% of CPU (warm) to ~5% (cold) — not because it’s doing less work, but because the kernel is doing so much more.

Off-CPU time via bpftrace (actionable time only, excluding idle Rayon threads):

• Futex: 30.9% (1,172s) — threads blocked on synchronization, queued behind another thread’s page-fault handler
• Preempted: 29.3% (1,109s) — surprisingly high for 12 threads on 192 cores; the kernel’s page-fault work (readahead kthreads) was preempting our worker threads
• Disk I/O: 6.9% (262s) — actual NVMe latency was small next to the machinery above it
• mmap_sem: 0.9% (33.5s) — the explicit VMA lock; small only because it captures the wait, not the cascading futex wakes from threads queued behind it

The picture: under load, page-cache thrashing and kernel-level lock contention — not disk I/O — were the bottleneck. This isn’t unique to mmap; any buffered I/O path can hit similar page-cache and lock contention.

The Fio Ceiling

fio with the io_uring engine — 4 processes, iodepth 32, 4 MiB blocks, O_DIRECT:

READ: bw=20.2 GiB/s (21.7 GB/s)
All 32 NVMe drives at ~99.75% utilization
md0 util = 99.95%

What mmap actually delivered, peak, from vmstat during the stressful runs: 3.44 GB/s — about 16% of what the hardware could do. That gap was the size of the prize.

Enter io_uring

io_uring has earned its hype. Beyond async kernel I/O, part of its promise is direct user I/O that bypasses the page cache entirely — the thing causing most of our problems above. Worth reading:

1. “io_uring for high performance DBMS” — a good overview of optimizations, though focused on traditional DBMSes with 4KB page buffers, so many don’t translate. IOPOLL needs specific block-device access not really available from containers; SQPOLL had no measurable effect in our Arrow-based testing.
2. LanceDB’s io_uring post — oriented around small 4KB (vector search) reads. Key takeaway: without better scheduling and concurrency, io_uring by itself doesn’t help.

The plan: bypass the page cache with O_DIRECT, submit reads via io_uring, coordinate with Tokio, decode Arrow inline. We used compio, a Rust-native io_uring wrapper (executor + futures + reactor built around io_uring). The first cut leaned on compio’s async futures — one future per Arrow column read, all 40 columns (8 batches × 5 columns) submitted concurrently, awaiting completions to yield decoded Arrow buffers.

Here’s how we expected io_uring to answer mmap’s problems:

Initial laptop testing wasn’t encouraging

Looking back, maybe we should have written this post before building anything — our first design didn’t deliver on most of that last column. Proper io_uring design takes real work, and we wanted to iterate fast, so we started small.

We started on macOS, which has no io_uring — kqueue is a different beast entirely — but it let us sanity-check the compio abstraction and our batching logic: does it compile, are we producing more or fewer page faults than mmap? A laptop can’t prove correctness, but it can catch obvious regressions before booking time on the big Linux boxes.

Total runtime was slower, but major faults collapsed nearly 70×. That’s the expected shape of bypassing the page cache — no OS-managed faults, because there’s nothing to fault. We told ourselves the extra latency was because macOS lacks real io_uring under the hood, and that Linux would deliver the throughput win.

The Linux reality check

Major faults dropped from 128,957 to 3,647 — a 35× reduction. We’d solved exactly the thing io_uring is supposed to solve: the kernel was no longer thrashing on major page-ins.

But minor faults went up 8×, and total query time went from 13.6s to 21.8s — io_uring was ~60% slower than the mmap baseline it was supposed to replace. We’d traded one class of fault for another and lost on the trade. The internet is full of posts declaring io_uring wins over mmap. We had an implementation that had gone backwards.

Before we get to what went wrong, it’s worth walking through what we’d actually built — the mistake only makes sense once you see the architecture around it.

The Batch Materialization Layer

Arrow IPC organizes data as batches — contiguous row chunks, each with its own on-disk layout, with columns stored in their own byte ranges. In our setup, every file holds one large batch, and a day of data spans roughly 8 files. A query spanning one day touches all 8 files, pulls ~5–6 columns from each, and ends up issuing around 40 individual column reads.

With mmap, the read count barely matters — point the query engine at the file, it appears as memory, and the kernel pages in whatever bytes a query touches. With io_uring, every read has to be submitted explicitly, which is more code and more places to get the timing wrong. So we built a layer between the io_uring plumbing and the query engine — the Batch Materialization Layer (BMT in our logs) — with a simple API:

• prefetch(batch, columns) — fire io_uring reads for a set of columns, populate a per-column cache with OnceCell-style slots
• materialize(batch, column) — return cached bytes, or await the in-flight read

The design felt right on a whiteboard. Prefetching maps directly to how the query engine wants to hide I/O latency behind CPU work: start reads early, do other things while they land, come back for bytes when needed. The cache decouples the query engine from io_uring semantics entirely — queries just call materialize and either get bytes immediately or wait.

What we didn’t see at the time was how much this one layer was doing. Everything ran on a single BMT thread: accepting prefetch/materialize calls, submitting compio futures for every requested column, awaiting completions, decoding bytes into Arrow buffers, populating the cache, and handing materialized columns back to the query engine. I/O coordination, Arrow decode, cache management, and query-facing API, all on one call stack and one async runtime. It felt like clean separation of concerns at the time. It was actually one layer doing five jobs, where a fix in one would ripple into the others. More on that in Part 2.

The first cut fired all ~40 column reads at once — ~40 concurrent compio futures the moment a query wanted a file. That’s what “prefetching” meant to us then: fire everything, let async coordinate, wait for it all.

The Meandering

The io_uring papers all emphasize O_DIRECT, which we weren’t using — our reads were still flowing through the page cache. So the first thing we tried was turning O_DIRECT on: direct DMA into our buffers, no kernel caching, none of the page-cache machinery that had been our whole problem under mmap. Runtime dropped from 21.8s to about 19s — real, but modest, and not the leap the blog posts had led us to expect.

Next was Arrow, which was showing up prominently in perf. Samples sat on Buffer::from_slice_ref, arrow-rs’s default way to build a Buffer from a byte slice — it allocates fresh memory and memcpys into it. Every 4 KiB destination page that memcpy touches needs the kernel to zero and map it — a minor fault per page — and 8 million minor faults over ~13 GB of reads lines up almost exactly with that math. Our Arrow layer was, in effect, forcing the kernel to redo the memory-management work we thought we’d bypassed by moving to io_uring in the first place.

We worked around it by constructing the Buffer directly from the io_uring-owned bytes, skipping the copy. Runtime dropped from 19s to about 16s, but the code was ugly enough that any reviewer would flag it — manual Buffer construction sidesteps arrow-rs invariants in ways that are hard to follow six months later. We logged the real fix as future work: a reusable buffer pool where io_uring writes into pre-allocated destination memory and Arrow copies from that — you’d still pay the memcpy, but into pre-faulted pages, which is close to free.

16 seconds — better than our first cut, still worse than mmap. We’d done the obvious things — O_DIRECT on, Arrow’s default copy worked around — and the blogs had promised near-hardware-ceiling throughput. We weren’t close, and at that point, nothing else in the design looked obviously wrong to us.

To be continued…

We’d tried the obvious fixes and talked through more theories with Claude than I want to count. We were sitting on an implementation 60% slower than mmap, with a Batch Materialization Layer that looked reasonable on paper. What we didn’t have — and needed, to make any more progress — was actual data about what our io_uring submissions were doing moment to moment.

Time to add logs and instrumentation, and dig in at a micro level, the old-fashioned way.

Part 2 picks up here: the logs, the arguments with Claude, the moment we realized 40 concurrent SQEs was the actual problem, the architectural rethink that followed, and the memory-management story that ended up mattering as much as any of it.

Learn more at P99 CONF

I’ll be doing a deep dive on this topic at P99 CONF, online October 21–22, 2026 — a conference for developers who care about p99 percentiles and high-performance, low-latency applications.

Register here.