The Philosophical Developer — Chapter 44: The Preset Catalog That Actually Works

A model catalog is not a wish list. It is a contract between your hardware and your server. Last week I found out mine was lying.

The problem

Two llama.cpp servers. Two GPUs. One catalog.

• RTX 5090, 32 GB VRAM — llama.cpp
• RTX 4070 Ti, 12 GB VRAM — llama.cpp-research

Both containers pull models from a shared host directory on btrfs and run via Podman quadlets. The idea was simple: keep a presets.ini per card that defines global defaults and per-model overrides, mount it read-only into the container at /etc/llama-cpp/presets.ini, and point the server at it with LLAMA_ARG_MODELS_PRESET.

It worked. Until it didn’t.

The presets files had grown organically. Router parameters like models-dir, models-max, models-autoload, host, port were sitting in the INI. So were imagined server options that don’t exist in llama-server --help: endpoint-metrics, endpoint-slots, timeout, sse-ping-interval, batch, ubatch, n-parallel.

Worse: context sizes were fantasy. 262144 tokens on the 4070 Ti. cache-type-k/v = q8_0 on a 12 GB card. load-mode = none was set in some places, mmap in others, with no consistency.

The catalog was not validated. It was cargo-culted.

Ground truth

First step: find the real options.

/ var/home/panos/.unsloth/llama.cpp/build/bin/llama-server --help

The output is the source of truth. Every key in presets.ini must map to an actual llama-server flag. No exceptions.

I grepped both files for anything that wasn’t in the help output. The offenders were exactly the router parameters and the server-only knobs that belong in the quadlet, not in the model catalog.

Presets are for model parameters only: m, ctx-size, temp, top-p, top-k, n-gpu-layers, split-mode, main-gpu, kv-offload, flash-attn, cache-type-k, cache-type-v, fit, load-mode, jinja, chat-template-kwargs. Everything else is noise.

The hardware contract

Global defaults must respect VRAM.

RTX 5090 32 GB

[*]
ctx-size = 131072
n-gpu-layers = all
split-mode = none
main-gpu = 0
kv-offload = true
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
fit = off
jinja = on
load-mode = mmap

32 GB allows a 131K context with q8_0 KV cache and full GPU offload. Larger models like Muse-Glimmer 30B and ThinkingCap 27B run comfortably.

RTX 4070 Ti 12 GB

[*]
ctx-size = 16384
n-gpu-layers = all
split-mode = none
main-gpu = 0
kv-offload = true
flash-attn = on
cache-type-k = q4_0
cache-type-v = q4_0
fit = off
jinja = on
load-mode = mmap

12 GB forces conservative choices: 16K global context, q4_0 cache to save VRAM. Per-model overrides drop 27B models to 8K-16K, and the Q6 variant of Muse-Glimmer is removed entirely — it doesn’t fit.

Per-model sections keep only what changes:

[Muse-Glimmer-30B-UD-Q4_K_XL]
m = /models/Muse-Glimmer-30B/Muse-Glimmer-30B-UD-Q4_K_XL.gguf
ctx-size = 131072   # 5090
temp = 1.0
top-p = 0.95
top-k = 64
chat-template-kwargs = {"reasoning_strength":"low"}

No duplicate ctx-size conflicts. No imagined parameters.

The quadlet wiring

The INI lives on the host, not in the image.

/var/home/panos/.config/containers/config/llama.cpp/presets.ini
/var/home/panos/.config/containers/config/llama.cpp-research/presets.ini

Quadlet volume mount:

Volume=%h/.config/containers/config/llama.cpp/presets.ini:/etc/llama-cpp/presets.ini:ro

Service env:

LLAMA_ARG_MODELS_PRESET=/etc/llama-cpp/presets.ini

Inside the container the path is always the same. The host path changes per card. One env var, one mount, deterministic.

Both files were cleaned:

• Removed models-dir, models-max, models-autoload, host, port
• Removed invalid server knobs
• Fixed global ctx-size per VRAM
• Changed cache-type to q4_0 on 4070 Ti
• Removed models that don’t fit

Verification was ad-hoc but explicit:

# No router params
grep -n "models-dir\|host\|port" presets.ini
# Should be empty

# Global ctx-size correct
grep -A1 '^\[\*\]' presets.ini | grep ctx-size

Both files now pass.

The lesson

A preset catalog is infrastructure code. It deserves the same discipline as any other config:

1. Validate every key against llama-server --help
2. Keep model parameters separate from router/service parameters
3. Encode hardware constraints in global defaults, not in comments
4. Version the files, review them, test them after changes

Defaults are a starting point. Your hardware is not default. Once you map the real options to the real VRAM, the catalog stops lying and starts working.

Written in the Sisyphus voice — relentless co-builder. The work continues.