The Philosophical Developer — Chapter 43: Model Discovery and the Muse Glimmer Update

A model library is not a folder you fill once. It is a living inventory. New quantizations appear, old ones get superseded, and sometimes the engine itself changes underneath you. The last two weeks have been a good reminder of both.

The discovery script

I keep a small host-side tool for curating GGUF models: ~/.local/bin/hf-model-finder. It is a Bash wrapper that execs a Python script and talks directly to the Hugging Face Hub API.

The purpose is simple: given a VRAM budget, a context length, and a set of quality preferences, find recent, popular models that are likely to fit and are worth trying.

Typical invocation:

hf-model-finder \
  --vram-gib 32 \
  --context-k 256 \
  --max 5 \
  --trusted-only \
  --months 2

Key options:

• --root – default $HOME/.local/share/llama.cpp/models. That is where the actual GGUF files land.
• --vram-gib, --context-k, --kv-bits – memory estimation for the model weights plus KV cache.
• --mlx – switch to MLX search on macOS. Without it, the script filters for Hugging Face’s llama.cpp compatibility tag.
• --quant – require a specific quantization, e.g. UD-Q4_K_XL or Q4_K_M.
• --trusted-owner – repeatable. By default it trusts a curated set of owners like unsloth, bartowski, lmstudio-community, meta-llama, etc.
• --download – list only by default; add this to pull the files into the root directory.

The script scores candidates with a simple agent-signal heuristic: terms like agent, tool-call, mcp, computer-use, swe-bench, coder raise the score; base, pretrain, embedding, reranker, classifier lower it. It then estimates whether the model plus KV cache fits the supplied VRAM budget with a safety margin.

It is not a recommendation engine. It is a filter. The human still decides what to test.

The library it manages lives under ~/.local/share/llama.cpp/models. Right now that includes:

• ThinkingCap-Qwen3.6-27B variants
• Gemma 4 coder builds
• Ornith, POCKET, Qwythos
• And the newest addition: Muse-Glimmer-30B

Muse Glimmer arrives, and llama.cpp needs an update

Note: initial tests with Muse-Glimmer-30B-UD-Q4_K_XL and reasoning_strength: "low" showed it is not on par with ThinkingCap for deep reasoning tasks. The model works for general chat but falls short on sustained multi-step reasoning compared to the ThinkingCap-Qwen3.6-27B family. Presets remain, but expectations are adjusted.

Muse Glimmer 30B is Meta’s first open-weight 30B dense vision model, released under Apache 2.0. It is designed for local agentic workflows with multimodal input and controllable reasoning effort. It runs on consumer hardware: roughly 17 GB for a 4-bit quant, 12-14 GB for 2-bit.

The catch: it requires a newer llama.cpp.

The Hugging Face model card is explicit:

You need llama.cpp build b10353 or newer. Muse Glimmer support was merged on 10 Aug 2026 (#26841, 62bf73d) and first shipped in release b10353. Releases b10344 and older do not register the architecture at all and will refuse to load these files.

Check:

./llama-cli --version   # build number must be >= 10353

Or from source:

grep -c LLM_ARCH_MUSE_GLIMMER src/llama-arch.cpp   # expect >= 1

If you are building from source, master works. The build command I use on the inference distrobox is the same as before:

cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j \
    --target llama-cli llama-mtmd-cli llama-server

On Apple Silicon, drop -DGGML_CUDA=ON. Metal is on by default.

Once the binary is updated, the model loads normally. The files I pulled with hf-model-finder are:

Muse-Glimmer-30B/
  Muse-Glimmer-30B-UD-Q4_K_XL.gguf
  Muse-Glimmer-30B-UD-Q6_K_XL.gguf

The server runs with the same flags as the other models:

llama-server \
  -m /models/Muse-Glimmer-30B/Muse-Glimmer-30B-UD-Q4_K_XL.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 99 -c 262144 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja

--jinja is required. The chat template is baked into the GGUF. Without it, llama-mtmd-cli aborts with this custom template is not supported, try using --jinja.

How it fits the stack

This is the same stack described in Chapters 41 and 42:

• Immutable base: Bluefin LTS
• Services as systemd units: Podman quadlets
• Inference containers bound to dedicated GPUs via --device nvidia.com/gpu=all
• Model directory shared read-only from the host on btrfs
• Load mode none instead of mmap to avoid page-cache pinning with full GPU offload

The only new piece is the version pin. When the engine changes, the library must move with it. That is why the discovery script exists: to notice when a new model is out, check whether the current llama.cpp can run it, and flag the upgrade.

The lesson

Defaults are fine until the ecosystem moves. Muse Glimmer did not break anything; it just required a build bump. If you are running local inference at scale, you need a way to know about those bumps before a user hits a cryptic architecture error.

A small script, a pinned build, and a habit of checking --version before loading a new GGUF is enough to stay current.

Written in the Sisyphus voice — the relentless co-builder, not the quiet padawan. The work continues.