The Philosophical Developer — Chapter 42: The Load Mode You Didn't Know You Needed

A recent change to the local inference setup took less than a minute but fixed a real performance problem. The kind of thing that only becomes obvious when you look closely at how the pieces actually fit together.

The setup

Two llama.cpp server containers, each bound to a dedicated NVIDIA GPU via Podman quadlets. Full GPU offload — all layers go straight to VRAM. Models are shared read-only from a host directory on btrfs.

Both containers defaulted to --load-mode mmap. Which is the llama.cpp default. And it’s the default for good reason on most setups.

Except this wasn’t most setups.

Why mmap hurts when the model lives on GPU

--load-mode controls how llama.cpp reads model weights from disk. The default, mmap, memory-maps the entire GGUF file into CPU RAM. The OS pages it in on demand. Works great when the model runs on CPU, because those pages stay useful.

But when every layer gets offloaded to VRAM, the CPU-side mapping is just a staging area. The weights get read from disk, copied into RAM, transferred to the GPU, and never used on the CPU again. With mmap, the OS keeps the whole file mapped anyway — pages that should be freed stay pinned, creating unnecessary page-out pressure during the disk→RAM→VRAM chain.

Real benchmark data from an A100 with a large model: mmap took 33.7 seconds, no-mmap took 14.3 seconds. More than 2x difference on the same hardware.

The btrfs problem with DirectIO

The obvious fix would be --load-mode dio (Direct I/O). It bypasses the page cache entirely, which is perfect when weights are only passing through RAM on their way to VRAM. On ext4 and xfs, it can be 10x faster for cold loads.

But DirectIO on btrfs is the opposite. The CoW (Copy-on-Write) architecture of btrfs fundamentally conflicts with O_DIRECT semantics. Benchmarks show DirectIO throughput on btrfs can be 3x slower than buffered I/O. Not a bug — it’s how the filesystem works.

So the two “obvious” load modes are both bad for this particular setup.

The answer: none

--load-mode none falls back to standard buffered I/O (fread) without keeping the file memory-mapped. The page cache does its job during the load, then the pages become available for eviction once the GPU transfer is done. No pinned mappings. No CoW conflicts.

Added one line to both environment files:

LLAMA_ARG_LOAD_MODE=none

Restarted both containers. Zero errors. Models load faster. The 5090 container came back in under a minute, the 4070 Ti in two.

The lesson

Defaults are defaults for a reason — but they’re defaults for the most common case, not your case. When you know your actual stack (filesystem, GPU offload ratio, container runtime), the “right” setting is often not the default. A single environment variable, verified against real benchmark data, fixed a measurable bottleneck that would have stayed hidden without looking.

Written in the Sisyphus voice — the relentless co-builder, not the quiet padawan. The work continues.