The Philosophical Developer — Chapter 41: Containers as Systemd Units, on an Immutable Base

An immutable operating system changes how you run services. When you cannot (and should not) install packages onto the host, everything you want to run has to live in a container — and the most reliable way I have found to run those containers is to make them native systemd units. This is the story of how I containerise the tools I use every day on an immutable base, and why it is the groundwork for Chaossynergy.

Why an immutable base changes everything

My desktop runs Bluefin, an immutable Silverblue-based Fedora. The host filesystem is read-only and image-based. You do not dnf install your way into a working workstation, and you do not fight the base when it updates — because the updates replace the whole image.

That is a philosophy, not an inconvenience. An OS you cannot silently drift is an OS you can trust to be the same tomorrow as it is today. But it forces a decision: every tool that is not part of the base image must be delivered some other way. Containers are the answer that respects the boundary.

The interesting part is how you run them. A script full of podman run invocations works once and stops being the source of truth the moment you forget what was running. What I wanted was declarative, starts at boot, restarts on failure, and lives in version control — the same discipline I apply to infrastructure and code.

Quadlet: containers declared as systemd units

Podman ships a generator called quadlet. You write a small file that declares a container — its image, its network, its volumes, its environment — and systemd turns that declaration into a native unit. systemctl start and systemctl stop just work. systemctl enable makes it start at boot. Restart=on-failure gives you the resilience of a real service manager for free.

The beautiful part is what this buys you. Container lifecycle becomes systemd lifecycle. Logs go to the journal. Restart policies are systemd policies. Dependencies between containers are expressed as unit dependencies (After=, Wants=) rather than custom glue. You get the operational surface of a service manager without pretending you need a cluster.

The unit files live in the user’s systemd container config directory. systemd watches that directory, and the declaration — not a sidecar bash script — is the thing that describes what should be running. Version-control it, review it, and you can rebuild the exact same set of services on a fresh machine.

The network topology

Containers need to talk to each other, and they need a single, intentional doorway to the outside. I define a small podman network that all the service containers join. Inside that network they can reach each other by name — caddy, ollama, llama-cpp — no IP chasing, no host networking, no port collisions.

One container is the doorway: a reverse proxy that owns the published ports. Every service is reachable through it on its own hostname, with TLS handled at the edge. Only that proxy is exposed to the host and the LAN; everything else stays on the private network. That is a clean, defensible shape: a handful of internal services, one controlled ingress, nothing reachable by accident.

Volumes across the boundary

Immutable hosts bring a nice wrinkle: the container wants to persist data, and the host filesystem is read-only. The answer is explicit named or host volumes — model libraries, data directories, config — the container mounts exactly the directories it is allowed to keep.

The subtlety is ownership. Under rootless podman the container user is remapped into a subordinate UID range, so a volume owned by your desktop user can be unreadable inside the container even when the numeric UID looks the same. The clean fix is keep-id: map the container’s user back to your host identity so a shared directory is writable from both sides. One line in the declaration, no host-side permission games.

Building your own, from a Containerfile

Not every tool ships as a ready-made image. For the ones I extend — a diagramming app I’ve been working on, for instance — I keep the build definition beside the rest of the container config. A Containerfile describes the build; podman builds it into a local image; the quadlet unit references that local image. The whole chain — build definition, runtime declaration, network, proxy config — lives in one place and one repo.

The payoff is a workstation I can rehydrate. Cloned onto fresh hardware, the same declarations produce the same running services. That is the property I actually care about: not the cleverness of any single file, but the reproducibility of the whole.

The groundwork for Chaossynergy

This is more than a convenient way to run tools. It is the skeleton of Chaossynergy — the immutable, AI-augmented OS I keep describing in these chapters. Bluefin gives me the immutable base; the philosophical developer methodology gives me the discipline; this quadlet pattern gives me the delivery mechanism for every agent, model server, and service the OS will need.

An OS built from an image, extended only through declared, version-controlled containers, managed by systemd, with a single intentional network and one edge proxy — that is an OS I can reason about, secure, and update. The tools I run today fit that shape because they have to. The OS I am building makes that shape the default.

The boulder keeps rolling. Each chapter is one more revolution, and this one is the layer underneath everything else that follows.

Repos:

• github.com/dark5un/sketchlab.app (the containerised tool referenced here)
• github.com/dark5un/Chaossynergy (the OS these containers are groundwork for)

Written in the Sisyphus voice — the relentless co-builder, not the quiet padawan. The work continues.