The Philosophical Developer — Chapter 26: Local Inference and the Agent Toolchain
2026-07-13 · 6 min read
Chapter 25 was the alpha milestone. We had a bootable container image, a CI pipeline that built it, and a single agent container running Hermes on first boot. It was enough to validate the idea that an operating system could be the interface for an AI agent.
But one agent is not a platform. And cloud-only inference is not sovereignty.
So I sat down with my Padawan, OrsonRius, and we extended Chaossynergy in two directions that turned it from a prototype into something far more interesting.
Two new Architecture Decision Records capture the changes: ADR-011 for local inference and ADR-012 for the agent toolchain expansion. Both are drafted, reviewed, and implemented in the justfile on the build image.
ADR-011: Local inference matters
I have been running Hermes through OpenRouter since the beginning. Remote inference works. But it comes with costs that compound the more you use the system.
Privacy is the first one. Every prompt, every code review, every conversation leaves the machine. For a tool that lives inside my OS and sees everything I do, that is a hard constraint. Latency is the second. A round trip to an API provider adds hundreds of milliseconds to every inference, even for simple operations. Offline dependence is the third. If the internet goes down, the agent goes silent. Cost is the fourth. Per-token billing adds up when the agent is running all day.
The Chaossynergy host ships with NVIDIA drivers pre-baked per ADR-001. The hardware I run it on has an RTX 5090 with 32 GB of VRAM. That is enough to run a 27B parameter model at Q4_K_M quantization with room left over for KV cache. The default model is ThinkingCap-Qwen3.6-27B, a reasoning-enhanced fine-tune that fits comfortably in the available memory.
The design decision that took the most thought was choosing the inference engine. I ended up with two.
Llama.cpp is the universal Swiss Army knife. A single binary, zero Python dependencies, CPU and GPU hybrid support. It auto-downloads GGUFs from Hugging Face on first request. Install it with a curl pipe. It is the discovery and fallback engine. When you want to try a new model, you run a command and it works in seconds.
VLLM is the production serving engine. PagedAttention, continuous batching, tensor parallelism. It is Python-based and pip-installable. Once you settle on a default model, vLLM serves it with high throughput and Hermes connects to it as a custom provider via the OpenAI-compatible API.
They are mutually exclusive. The VRAM on a single GPU cannot run both simultaneously. You pick one at setup time. Llama.cpp listens on port 8080. VLLM listens on 8000. The inference server runs in its own dedicated distrobox container, isolated from the Hermes agent and independently restartable.
The setup is delivered through ujust recipes: ujust setup-inference-llama or ujust setup-inference-vllm. Each command creates the distrobox, installs the engine, configures the systemd user service, and wires Hermes to the local endpoint.
ADR-012: A multi-agent toolchain
Chaossynergy shipped with one agent: Hermes. That was always meant to be the starting point, not the destination. ADR-004 established the pattern for multiple agents in isolated distrobox containers. ADR-012 finally fills in the rest.
Three coding agents are now available as optional installs.
OpenCode is provider-agnostic, open source, and standards-compliant. It works with any LLM provider, has a clean TUI, and is strong at feature implementation and code review. It ships as an npm package called opencode-ai.
Pi from pi.dev is something different. It is a minimalist coding agent harness designed to be self-modifying. You can ask Pi to build an extension for its own capabilities. Tree-structured session history, four operating modes, 15 plus provider support. The extensibility model is what drew me to it. Instead of configuring Pi, you teach it to grow.
Claude Code is Anthropic’s official coding agent. Terminal-first with a TUI, deep Sonnet integration, and the strongest reasoning of the three. It requires an Anthropic API key but delivers research-grade code work when you need it.
Each agent gets its own distrobox container. No config conflicts, no shared runtime dependencies, independently upgradeable and resettable. Hermes remains the orchestrator. When you present a task, Hermes decides which coding agent to delegate to based on the task type. Feature work goes to OpenCode. Custom workflows go to Pi. Complex reasoning goes to Claude Code. Infrastructure stays with Hermes itself.
The ujust delivery mechanism
This is worth calling out separately because it shaped the whole architecture.
The original plan used systemd template services for everything. setup-agent@.service would fire at boot and install the selected agents. I moved away from that for a simple reason: first boot should be fast.
Instead, everything except Hermes itself is installed on demand through ujust. Bluefin’s ujust system discovers justfiles in /usr/share/ublue-os/just/ and presents them as interactive recipes. The user decides when to install the inference server, when to add OpenCode, when to try Pi. Each recipe is idempotent. Run it once and the agent is ready. Run it again and it skips.
This decision applies the minimal host principle from ADR-005 to the agent layer itself. The OS image is clean. Nothing runs until you ask for it. Herdr does not care how an agent was installed. It just shows whatever harness is running in a pane.
The justfile at chaossynergy.just now contains recipes for setup-inference-llama, setup-inference-vllm, setup-opencode, setup-pi, setup-claude, list-agents, remove-agent, and inference management commands like inference-logs, inference-status, and inference-model for switching between quantizations.
What this means
Chaossynergy is no longer a single-agent OS. It is an agent orchestration platform with local inference running on bare metal.
The inference container gives the system independence from cloud providers. Your code never leaves the machine. Your prompts are private. The agent works offline. There is no per-token cost.
The toolchain expansion gives the system depth. One agent cannot be good at everything. Hermes orchestrates, OpenCode implements, Pi extends, Claude reasons. Each one in its own sandbox, each one replaceable.
The build pipeline is still live. GitHub Actions on the chaos branch builds the container image with buildah, signs it with cosign, and publishes to GHCR. The justfile changes ship with every build. Users who pull the latest image get the new ujust recipes automatically.
The alpha from Chapter 25 was a proof that an agent could own the OS substrate. Chapter 26 turns that proof into a platform. Local inference for sovereignty. Multiple agents for specialization. On-demand installation for a clean first boot.
Here I geek out with my young Padawan, OrsonRius.
Repos: