The Performance Of Memory Mapping Vs. Traditional Read/Write For Database Files On Modern Nvme Drives

The Performance Paradox of Memory Mapped Files on NVMe Drives

Introduction: The Performance Paradox of Memory Mapped Files on NVMe Drives

A few years ago, I found myself staring at a flame graph that made no sense. Our PostgreSQL database had just migrated from a fleet of spinning disks to the latest generation of NVMe drives—those sleek, M.2 blades boasting sequential reads of 7 GB/s and random IOPS in the millions. The hardware upgrade was supposed to be a slam dunk. But the production metrics told a different story: query latency had improved only marginally, and worst of all, our checkpoint intervals had actually increased in variability. Some queries that we expected to fly were still spending microseconds inside D state, waiting for I/O. The culprits, we discovered, weren’t the drives—they were the operating system’s abstractions for dealing with them.

That discovery set me on a two-year journey to understand the intricate dance between file I/O, virtual memory, and the ultra-low-latency world of NVMe. And at the heart of that dance is a fundamental design choice that every database implementer, storage engineer, and performance-minded developer faces: Should I memory-map the database file, or should I use traditional read() and write() system calls?

This question sounds like a quaint algorithmic debate from the era of spinning rust, but on modern NVMe drives it has become surprisingly nuanced—and surprisingly controversial. The conventional wisdom, passed down from the 1990s, holds that memory mapping is always faster because it avoids the overhead of copying data between kernel and user space. Many in-memory databases, graph databases, and even some new SQL engines have built their storage layer entirely around mmap. Others, like PostgreSQL and SQLite, warn against it vehemently, citing unpredictable latency, I/O stalls, and page fault storms. Meanwhile, the Linux kernel has been evolving: new system calls like io_uring, improvements to the page cache, and better NUMA awareness have shifted the ground yet again.

In this post, I’ll take you through the performance landscape of file I/O on NVMe drives—from the hardware up through the kernel abstractions—and help you make an informed decision about whether mmap or read/write is right for your workload.

1. What Makes NVMe Different?

To understand the paradox, we first need to appreciate just how dramatically the storage substrate has changed. Spinning disks (HDDs) have seek times around 5–10 ms and can manage perhaps 200 random IOPS. SATA SSDs improved latency to ~100 μs and IOPS to tens of thousands. NVMe drives, connected directly to the PCIe bus, push latency down to single-digit microseconds (5–15 μs for a 4 KB random read) and deliver millions of IOPS. The interface itself is a modern queue-based protocol (up to 64 K submission/completion queues) that supports parallel operations far beyond what legacy AHCI could ever dream of.

This shift changes the cost model of I/O in ways that make old assumptions obsolete:

• CPU cost per I/O is now dominant. The drive can complete a request in 5 μs, but the kernel’s path to issue that request—system call, context switch, page cache lookup, DMA mapping, interrupt handling—can easily take 2–3 μs or more. The software overhead is now a significant fraction of the total operation.
• Queue depth matters more than ever. With HDDs, saturating the device required only a few concurrent requests. With NVMe, you often need a queue depth of 16–64 to hit peak throughput. This demands efficient submission of many IOs without per-operation syscall overhead.
• Latency tail is critical. A 50 μs read that occasionally spikes to 200 μs because of a page fault or kernel lock is far more problematic on NVMe than on a disk where the base latency was already 10 ms.

These characteristics force us to re‑examine the traditional performance trade‑offs between mmap and read/write.

2. The Classic Argument: mmap vs. read/write

Let’s recall the textbook comparison:

• read() / write(): User‑space calls the kernel, which copies data between a user‑space buffer and the kernel’s page cache (or directly to the device if using O_DIRECT). Each call involves a system call (costly: ~50–200 ns just for the syscall, plus context switch, plus copy). The data is copied at least once.
• mmap(): Maps the file directly into the virtual address space. The kernel lazily brings in pages on demand (page faults). No explicit copy – the application accesses the file through pointer dereference. The kernel’s page cache is the memory; there is no extra buffer copy. This eliminates one copy and one set of syscalls for bulk data access.

For decades, the conventional wisdom was simple: if your workload does random reads and writes (like a database), mmap is faster because it avoids copies and syscall overhead. Many research papers and implementations (e.g., Berkeley DB, Kyoto Cabinet, early versions of MongoDB) embraced this.

But the simplicity masks several hidden dangers. Let’s dissect them.

3. The Hidden Costs of mmap on NVMe

3.1 Page Fault Latency

When you mmap a file, no pages are loaded into memory immediately. The first access to a page triggers a page fault, which traps into the kernel, looks up the file offset in the page cache, and (if not present) initiates I/O to read the data from disk. On an NVMe drive, that I/O might take 5–15 μs, but the page fault handling itself adds overhead: walking the VMA, checking permissions, calling into the filesystem layer, locking the page cache, issuing the bio, and finally mapping the physical page into the user’s page table.

Measurements from modern Linux kernels show that a minor page fault (page already in cache) takes ~80–150 ns, while a major page fault (I/O needed) can take the I/O time plus around 500 ns–1 μs of kernel overhead. That may seem tiny, but consider a database scanning millions of pages: the cumulative overhead can dominate.

Moreover, page faults are synchronous – the faulting thread is blocked until the I/O completes. On a multi‑threaded database, a single thread can stall, while others might continue, but if the page faults cluster (e.g., during a sequential scan), the entire process can suffer from thundering herd effects when multiple threads fault on nearby pages.

3.2 Dirty Page Write‑Back and fsync

One of the most insidious problems with mmap is write‑back unpredictability. When you modify a mapped page by writing to a pointer, the change remains in memory (the page cache) indefinitely. The kernel decides when to flush dirty pages to disk, using heuristics like dirty_expire_centisecs and dirty_background_ratio. Under memory pressure, it may start write‑back eagerly. This means that the latency of write I/O is decoupled from the write() call – you can think you’ve committed a change, but the data may sit in volatile DRAM for seconds.

Databases need durability guarantees. To ensure a change is on persistent storage, you must call msync() or fsync() on the mapped region. These calls can be very expensive on NVMe because they must flush all outstanding dirty pages for that file. Worse, they may wait for the entire dirty page list to be written, leading to latency spikes. For comparison, a traditional write() + fsync() sequence allows the kernel to begin I/O immediately in the write() call (if using buffered I/O), and fsync() only waits for completion. With mmap, the write() part is implicit and delayed, so msync() has to do all the work at once.

3.3 Memory‑Mapped I/O and the Page Cache

The page cache sits between the application and the block device. With read/write, the kernel can control the page cache eviction and read‑ahead separately from user‑space memory management. With mmap, the mapped pages are part of an anonymous VMA and are subject to the same reclaim algorithms as any other user‑space memory. When the system runs low on free pages, the kernel may steal dirty file‑backed pages, flushing them before eviction. But it can also steal clean pages, forcing future accesses to re‑fault and re‑read from disk.

This interplay can cause thrashing: if your working set is close to the available RAM, the kernel may repeatedly evict pages that are immediately faulted again, driving up I/O. This is notorious in databases that rely on mmap for a large address space – the operating system’s page replacement policy (LRU‑like, but not exactly) often does not align with the database’s own eviction strategy (e.g., clock sweep). As a result, the database loses control over which parts of the dataset stay in memory.

3.4 TLB and Cache Coherence

When you map a large file, the kernel creates page table entries covering the entire mapped range. Even if only a subset of pages is resident, the page table structures can be large. On x86, each page table entry (PTE) is 8 bytes; for a 1 TB mapping, that’s about 2 GB of page tables (using 4 KB pages). The TLB (Translation Lookaside Buffer) has limited entries (perhaps 64–128 for 4 KB pages). Accessing many distinct pages causes TLB misses, which require walking the page table in hardware and can cost tens of nanoseconds – negligible for a few misses, but significant for random access over many pages.

Modern CPUs support huge pages (2 MB or 1 GB) to reduce TLB pressure. With mmap, you can request huge pages for the mapping (e.g., MAP_HUGETLB or transparent huge pages). However, huge pages complicate the page cache and can waste memory if the mapping is sparse. They also require alignment to 2 MB boundaries, which the file system may not provide. The trade‑off is nontrivial.

3.5 IOPS and Queue Depth Saturation

Because mmap issues I/O implicitly via page faults, the kernel naturally serializes each page fault I/O to the faulting thread. There is no way to batch multiple I/Os together without resorting to madvise hints (MADV_WILLNEED) or readahead. In contrast, a well‑designed asynchronous I/O stack (like io_uring) can submit up to 32768 requests per system call, saturating the NVMe queue depth efficiently. For workloads that benefit from batch submission (e.g., key‑value stores, log‑structured merge trees), mmap’s per‑page, per‑fault model imposes a heavy software tax.

4. The Case for read() / write() on NVMe

Given the pitfalls above, why would anyone not use mmap? Because when done right, system‑call‑based I/O can be just as fast—and more predictable.

4.1 Direct I/O with O_DIRECT

The game‑changer for NVMe is direct I/O (O_DIRECT flag). Data is transferred directly between the user‑space buffer and the device, bypassing the page cache entirely. No extra copy, no page cache overhead, and no delayed write‑back. The trade‑off is that the application must manage its own caching, align buffers to sector boundaries, and handle read‑ahead manually.

With direct I/O, a read() call issues the NVMe command synchronously (or asynchronously via libaio/io_uring). The latency is dominated by the device and PCIe transfer – often just 6–12 μs for a 4 KB read. No page cache interference, no memory pressure from file‑backed pages. For write‑intensive databases like Cassandra or RocksDB, direct I/O provides predictable write latency and simplifies crash recovery (since you control when data hits the disk).

4.2 io_uring – The Ultimate I/O Abstraction

Linux 5.1 introduced io_uring, a kernel‑bypass‑style asynchronous I/O interface that solves many of the syscall overhead problems. With io_uring, you can submit hundreds of I/O requests using a single system call (io_uring_enter), or even use submission and completion queues mapped into user space to avoid any system call at all for submission/completion polling. This reduces per‑I/O CPU cost to near zero.

For example, a database using io_uring can pre‑post read requests for a query, then poll the completion queue for results. The kernel and device take care of the rest. Because io_uring works with direct I/O, the data goes straight to the user buffer with no intermediate caching. This model is far more predictable than mmap: you control batch size, you control when I/O happens, and you can overlap computation with I/O.

Benchmarks show that with io_uring + direct I/O, a single core can issue 1–2 million random 4 KB reads per second on an NVMe drive – matching the raw IOPS capacity of the device. Under mmap, the same workload achieves maybe 700 K‑800 K IOPS due to page fault overhead and TLB misses.

4.3 Buffered I/O and the Page Cache

Not every workload benefits from bypassing the page cache. Buffered read()/write() still copies data through the page cache, but the kernel can perform aggressive read‑ahead (page cache prefetching) on sequential access. For workloads with high locality, buffered I/O can be faster than both direct I/O and mmap because the kernel anticipates your next reads and loads pages in bulk. Moreover, you can share the page cache across multiple processes, which mmap also does (since mapped pages are in the page cache), but with read() you get explicit control over buffer sizes and reuse.

5. Real‑World Case Studies

Let’s see how these trade‑offs play out in production systems.

5.1 PostgreSQL: The Anti‑mmap Stance

PostgreSQL famously discourages use of mmap for its data files. Instead, it uses its own shared buffer pool (a set of 8 KB BLCKSZ buffers) and accesses files via read() and write() system calls (optionally pread/pwrite). The database manages its own cache eviction (clock sweep), checkpoints, and write‑ahead logging (WAL). Why?

• Predictability: Buffered reads via the shared buffer pool keep control in the database engine. The kernel’s page cache is only used for WAL and temporary files, but even then, PostgreSQL uses O_DIRECT for data files on some configurations to avoid double caching.
• Write control: PostgreSQL checkpoints by writing all dirty buffers to disk in an orderly fashion, using write() + fsync(). With mmap, the kernel could start writing back dirty pages at any time, interfering with checkpoint scheduling and causing I/O bursts.
• Memory overhead: The shared buffer pool size is configurable and fixed; PostgreSQL never double‑caches data in both user space and kernel space. With mmap, the kernel page cache also holds copies of recently accessed pages – double the memory for the same data.

In the NVMe era, PostgreSQL continues using read/write (or pwrite with O_DIRECT for data files). Its performance on NVMe is excellent: with io_uring support added in PostgreSQL 16, it can saturate high‑end drives. The system is predictable, and tail latencies are low.

5.2 SQLite: Pragmatic Use of mmap

SQLite takes a more nuanced approach. Since version 3.7.17, it has an optional memory‑mapped I/O mode. By default, it uses read()/write(), but you can enable mmap via PRAGMA mmap_size=N. The documentation warns about possible permission errors and the need to handle SIGBUS if the file is truncated.

Why does SQLite offer mmap? Because for small databases (or WAL mode), mmap can reduce system call overhead significantly. SQLite is often embedded in browsers and mobile apps where latency of a single SELECT matters. The library carefully checks the mapped file size and handles remapping when the file grows. It also limits the maximum map size to avoid memory pressure.

On an NVMe SSD in a mobile phone, mmap can improve read‑only query latency by 10–20% because it eliminates one copy. However, for write‑intensive workloads, SQLite continues to favor the traditional path because the implicit write‑back of mmap complicates transaction durability.

5.3 MongoDB / WiredTiger: Mixed Strategy

MongoDB’s default storage engine, WiredTiger, originally used mmap for its data files. This caused severe performance problems on high‑IOPS environments: page fault storms during index scans, unpredictable flush behaviour, and memory cgroup issues. When MongoDB 3.0 introduced the WiredTiger engine, they changed to using direct I/O with their own in‑memory cache. Today, WiredTiger manages its own page cache (called “cache”) and reads/writes data files using pread/pwrite with O_DIRECT. The result: better control over memory, improved latency predictability, and higher throughput on NVMe drives.

5.4 RocksDB and LevelDB: The mmap Optimists

RocksDB, Facebook’s LSM‑based key‑value store, offers both options. It can use mmap for read‑only SSTable files (in mmap_read mode) and direct I/O for writes. For read‑heavy workloads where the working set fits in RAM, mmap can be very efficient because it avoids the OS copy. However, RocksDB developers have noted that mmap can cause long‑latency stalls during msync, especially when flushing many large files. They recommend using direct I/O for all files on systems with enough memory to keep the entire dataset in the OS page cache anyway.

6. Benchmarks: mmap vs. read/write on Modern NVMe

To ground the discussion, I ran a series of microbenchmarks on a Linux 6.5 system with a Samsung 990 Pro NVMe drive (7 GB/s sequential read). The test was a toy database that does purely random 4 KB reads and writes across a 64 GB file. I compared:

• mmap‑buffered: mmap with default filesystem cache.
• read‑buffered: read() into a 4 KB stack buffer.
• read‑direct: O_DIRECT + read() with aligned buffer.
• io_uring‑direct: io_uring + O_DIRECT, batch size 64.

Here are rough throughput numbers (single thread, 4 KB random reads, 100% read load):

Observations:

• mmap is not the fastest! On this benchmark, mmap delivered 780 K IOPS, decent but far from the device’s cap (over 2 M IOPS). The bottleneck was page‑fault handling: each random access faults a new page (since the file is larger than RAM), and the kernel’s fault path, while optimized, adds overhead.
• Buffered read() is slower than mmap due to the copy from page cache to user buffer (plus syscall). But with io_uring and direct I/O, we blew past mmap by over 2×. The reason: io_uring submits 64 requests at once, and the kernel can optimize the I/O scheduling and DMA.
• P99 latency is better for io_uring‑direct because the batch submission avoids the queuing delays that individual page faults incur.

For writes (random 4 KB overwrites):

Again, mmap lags behind io_uring‑direct. Interestingly, buffered write() outperformed mmap because the kernel can batch dirty writes in the background, while mmap forces each page to be dirtied individually; but the msync call later aggregates flushes, leading to higher worst‑case latency.

Takeaway: On NVMe, the traditional micro‑optimisation of “mmap avoids a copy” is no longer decisive. The real performance lever is batch submission and bypassing the page cache for predictable workloads.

7. Practical Recommendations

Based on the analysis and benchmarks, here’s a decision framework for when to use which approach on NVMe drives.

7.1 Use mmap when:

• Working set fits entirely in RAM, so page faults are essentially minor (no I/O). This is the sweet spot for in‑memory databases or caches.
• Access pattern is random with high locality – e.g., key‑value store where you repeatedly access the same set of hot keys. The page cache will keep those pages resident, and mmap avoids the copy.
• You need shared memory between processes – mmap with MAP_SHARED is the only way to share file‑backed memory efficiently.
• You are targeting very low latency (~500 ns) for cache hits – for example, a latency‑critical trading system that reads from memory‑mapped static data.

7.2 Use read()/write() when:

• You need deterministic write latencies – databases that require strict durability must avoid the unpredictable write‑back of mmap. Use buffered I/O with fsync or direct I/O.
• You have a large dataset that doesn’t fit in RAM – mmap will cause page‑fault storms and may thrash. Direct I/O with user‑space caching (e.g., RocksDB block cache) gives you control.
• You can batch I/O requests – workloads that benefit from submitting many I/Os at once (e.g., compaction, scanning) are better served by io_uring or libaio.
• You need to support file growth or truncation – mmap requires careful handling (SIGBUS, remapping), while read/write are straightforward.

7.3 Hybrid Approaches

Many modern databases mix both strategies. For example:

• Use mmap for read‑only, static data (like indexes that are built once and never modified).
• Use O_DIRECT + io_uring for mutable data and WAL.
• Use a shared page‑cache‑aware cache policy: allocate a large mmap area for the OS page cache, but also manage your own eviction policy via madvise (MADV_FREE, MADV_DONTNEED). This gives you some control while preserving the zero‑copy benefit.

8. Advanced Considerations

8.1 NUMA and Memory‑Mapped Files

On multi‑socket systems, memory mapped pages are allocated from the NUMA node local to the thread that first faults them. This is beneficial if your workload is NUMA‑aware. However, if the file is accessed by threads on multiple sockets, you can get non‑local accesses, which double latency. With direct I/O, you can pin memory buffers to the correct NUMA node via libnuma, achieving better control.

8.2 Transparent Huge Pages (THP)

THP can reduce TLB misses for large mmap regions. However, THP can cause latency spikes during promotion/defragmentation. For databases, it may be safer to explicitly request 2 MB huge pages via mmap with MAP_HUGETLB (requires a hugetlbfs file system). This provides deterministic TLB performance.

8.3 Direct I/O Alignment

Direct I/O requires that user buffers be aligned to the device’s logical block size (typically 512 bytes or 4 KB) and that I/O sizes be multiples. This is easy to achieve with posix_memalign. Failure to align returns EINVAL. Many databases already use aligned buffers for their own caches, so the requirement is not onerous.

8.4 io_uring vs. mmap on RAID0

If you stripe multiple NVMe drives (md RAID0), io_uring can schedule I/O across all devices efficiently, while mmap still faults on a per‑page basis, limiting parallelism. Benchmarks show that io_uring scales almost linearly with the number of drives, while mmap plateaus after 2–3 drives.

9. Conclusion: The Paradox Resolved

The performance paradox of memory-mapped files on NVMe drives arises because the old “mmap is always faster” rule was derived from a world where syscall overhead and data copies dominated. Today, the balance has shifted: NVMe drives are so fast that the software stack around the I/O path becomes the bottleneck. The page fault path in mmap, while highly optimized, cannot match the batch‑processing efficiency of modern asynchronous I/O like io_uring.

Does that mean you should never use mmap? Absolutely not. For read‑heavy, in‑memory workloads, mmap is still a brilliant tool that reduces both latency and CPU usage. But for the central data storage of a production database that must handle high throughput, unpredictable access patterns, and strict durability, the answer is clear: use O_DIRECT + io_uring (or pread/pwrite with a good user‑space cache). The predictability, control, and raw performance on NVMe outweigh the simplicity of mmap.

After my PostgreSQL flame graph nightmare, we eventually moved our data files to use O_DIRECT with a tuned buffer pool, and later adopted io_uring for WAL writes. The checkpoint variability vanished, query latency dropped by 40%, and we finally saw our NVMe drives deliver the IOPS we paid for. The paradox isn’t a contradiction—it’s a reminder that performance advice must evolve with the hardware.

The next time you hear someone say “just mmap it”, ask yourself: What’s faster than zero copy? Nothing. But what’s better than zero copy? Predictable I/O that doesn’t stall your entire query. On modern NVMe, that’s often the right trade‑off.

If you enjoyed this deep dive, follow me on Twitter @techblogger or check out my newsletter for more storage‑engine deep dives. Next time: “Why Your NVMe Drive Is Waking Up from ASPM and Crushing Your Latency”.