Quarantine Your Flaky Agent — Replay Until It Confesses

Part 6 counted the money. Part 7 hunts the flake: pulse flake replays a trace N times dry-run and diffs per-step signatures — stable steps read 3/3, diverging steps get named by index. The output feeds quarantine directly: mock step 7, constrain step 12.

The run

uv run pulse flake --trace corpus/cost-unit-test.py --runs 3

stable          3/3  cost-unit-test.py
stable          3/3  cost-unit-test.py
stable          3/3  cost-unit-test.py

Boring — and boring is the point. Dry-run replay is deterministic (cached responses, no model calls), so 3/3 stable is the baseline that says the harness is sound. Flakiness found here would mean the trace itself is nondeterministic, which is a different, rarer bug worth knowing about.

Reading a real flake

On a trace that diverges, the same command names names:

flaky           2/3  some-trace.py diverging=[7, 12]

Step 7 and step 12 differ across runs. The signature is kind + name/text — never timing, so slow steps don’t false-positive. The quarantine decision writes itself: mock the tool at step 7 in tests, constrain the model’s freedom at step 12 (fewer tools, tighter prompt), re-run until 3/3.

The layering that matters

Dry-run first (catches harness nondeterminism), live separately (catches model nondeterminism) — never mixed. pulse flake is deliberately dry-run-only; live variance is a separate experiment with a separate budget, run through pulse replay --live and compared with pulse compare. Confusing the two layers is how teams “fix” flakes that were never in their code.

Deterministic-then-live, quarantine what diverges, re-run until stable. That’s the whole workflow, and it fits in one command.

Next post: Ship Your Trace as a Paper Artifact — manifests, bundles, and the reviewer who reruns everything.

Written from the workshop — stable or named, no third option.