The Philosophical Developer – Chapter 34: The SRE Watchtower

I sat down with my Padawan, OrsonRius, to plan the next phase of the work.

We had proved two things. First, that infrastructure-as-code could be testable, modular, and portable across clouds using OpenTofu. Second, that the same methodology transferred to Pulumi and Go, proving it was tool-independent, not HCL-specific.

The local-cloud arc was about building infrastructure correctly. The next arc needed to be about operating it correctly.

The Problem with Automation

Automation has a well-known failure mode. It works beautifully for the cases you anticipated. It fails silently for the cases you did not.

A threshold-based alert fires when CPU hits 95%. A playbook restarts the service. The playbook works. But the one time the CPU spike is caused by a memory leak that only manifests after a database migration, the playbook does not catch it. The alert fires, the service restarts, the metrics look clean, and the root cause is still there.

The SRE discipline calls this the “automation tax.” You pay for the automation in complexity, and if the automation fails in an unexpected way, you pay again in debugging time.

The alternative is the human on-call. A human sees the alert, investigates, correlates, and fixes. The human is flexible. The human is slow. The human is expensive. The human makes mistakes when tired.

We needed something in between. A system that is as flexible as a human and as reliable as automation. A system that can investigate, correlate, and decide, but respects the boundaries we set for it.

The Inspiration

The idea came from two places.

The first was the Concourse CI philosophy. In Concourse, every action is a build. The build runs in a fresh container. The credentials are injected at runtime. The build is logged, traceable, and reproducible. Nothing happens outside the pipeline. No SSH keys on laptops. No “works on my machine.”

The second was the Philosophical Developer methodology itself. Testable infrastructure. Mock-driven development. CI-verified code. The same discipline that made our infrastructure reliable should also make our operations reliable.

We asked ourselves: what if the agent that watches the infrastructure is built using the same methodology as the infrastructure itself?

The Design Constraints

We had three constraints.

Zero paid services. Everything runs on local infrastructure or free GitHub APIs. No Datadog, no PagerDuty, no OpsGenie. The agent polls GitHub Issues. The pipeline writes to GitHub Issues. The issue is the single source of truth.

No direct SSH access. The agent never holds credentials for production systems. It triggers workflows. The workflows run in GitHub’s ephemeral Actions runners. The credentials are injected at runtime. The agent reads the results, not the secrets.

Incremental trust. The agent starts with no action permissions. It investigates and reports. If we trust the reports, we add one action at a time. The agent earns trust the same way a junior engineer does – by demonstrating good judgment on low-stakes tasks before being given access to production.

The Four-Stage Loop

The watchtower runs a loop with four stages.

Watch. The pipeline fails. The pipeline’s on_failure step creates a GitHub Issue with the sre-watchtower label. The issue body contains the run ID, workflow name, branch, and commit hash. The deduplication check prevents duplicate issues for the same run.

Acknowledge. The agent polls for open issues with the sre-watchtower label. It assigns itself to the issue and comments “I see this failure. I am investigating.” The human team knows the agent is on it.

Investigate. The agent reads the workflow run logs via the GitHub API. It checks the diff of the triggering commit. It looks for patterns: flaky tests, infrastructure timeouts, code regressions. It calls an LLM to analyze the logs and correlate the findings.

Report. The agent writes a structured comment on the issue. The comment includes the root cause, the severity, and a recommendation. If the agent is allowed to act, it triggers the appropriate workflow. If not, it waits for human input.

The State Machine

The issue body carries a status field that controls the agent’s behavior.

pending – the issue is new. The agent assigns itself and starts investigating. investigating – the agent is working on it. It continues reading logs and correlating. needs_input – the agent needs human help. It asks a specific question. resolved – the agent confirmed the fix. It closes the issue. wont_fix – the failure is a false positive or intentional. The agent closes with a note.

The status is stored in the issue body, not in the agent’s memory. If a human changes the status, the agent respects it on the next poll. The issue is the record.

The Architecture

The system has three components.

The responder workflow runs on GitHub Actions. It fires on workflow_run completion. If the run failed, it creates a GitHub Issue with the failure context. The deduplication check prevents duplicates.

The agent package is a Python library that polls for issues, investigates failures, and writes reports. It is testable, mock-driven, and CI-verified. The LLM calls are behind an interface that can be mocked in tests.

The manifest is a YAML file in the repo that configures the agent’s behavior. Which repos to watch, which workflows to respond to, what actions are allowed. The agent reads the manifest fresh every run. No caching, no stale state.

The Repo

The code lives in a new repo: github.com/dark5un/sre-watchtower. It is a Python package with a test suite, a CI pipeline, and a set of GitHub Actions workflows that demonstrate the pattern.

The repo is the implementation. The chapter is the methodology. The next two chapters will show the watchtower in action – first on a deliberately flaky workflow, then on a real CI pipeline.

Here I geek out with my young Padawan, OrsonRius.

Repos:

• github.com/dark5un/sre-watchtower

LinkedIn:

• The Philosophical Developer – Chapter 34: The SRE Watchtower