Prompt A/B With Distributions — Stop Comparing Single Runs

Part 1 built the gym. Part 2 answers the question every prompt tweak raises: is variant B actually better, or did you get lucky once? pulse compare scores N traces per variant and reports distributions — mean, median, p25/p75, cost delta — with a plain-English verdict that says “provisional n=5” instead of pretending five runs are science.

The dishonest default

Most prompt A/B today: run prompt A once, run prompt B once, squint at two outputs, declare victory. Single runs of nondeterministic systems prove nothing — the same prompt replays differently. The honest version needs distributions over traces with cost attached, which is exactly what the gym already produces.

The recipe

Capture N traces per prompt variant (same task, same model — only the prompt changes), then:

uv run pulse compare --a traces_prompt_a/ --b traces_prompt_b/

Here with 5 scored traces per side from the local corpus:

/tmp/cmp_a (n=5) vs /tmp/cmp_b (n=5)
           mean median    p25    p75
A         100.0  100.0  100.0  100.0
B          98.0  100.0  100.0  100.0
score delta (A-B): +2.0 pts
cost delta (A-B):  +0.0000 USD mean
A wins on quality (+2.0 pts, cost +0.0000, provisional n=5)

Read it carefully: medians tie at 100, the mean gap comes from one 90 in B, cost is identical. “A wins” with n=5 and a +2.0 delta is a hint, not a finding — and the verdict labels itself provisional with the N attached, so nobody mistakes it for one.

What “provisional” buys you

Every threshold in Pulse is labeled provisional until ~100-session calibration, and the compare verdict follows the same rule: effect size + N, no significance testing at v1. The discipline is in what the tool refuses to do — no p-values computed on n=5, no “statistically significant” badge for a two-point gap. When your N grows, the same command grows with it; the verdict format doesn’t change, only the confidence behind it.

The tie rule is explicit too: within ±1 point, the verdict says “ties” rather than crowning a winner on noise.

When to promote B anyway

Small-N comparison still decides things, honestly:

• Cost delta dominates. Same quality, half the tokens? Ship B at n=5 — the cost column is measured, not sampled.
• Worst-case moves. p25 up 10 points while the mean ties? That is robustness, and distributions show it where single runs hide it.
• Direction for the next N. A provisional win tells you which variant deserves the next 20 traces, not which deserves production.

The full A/B loop

# capture N traces per variant (same task, same model)
uv run pulse compare --a traces_prompt_a/ --b traces_prompt_b/ --json  # machine-readable
uv run python scripts/pulse_gate.py --baseline traces_prompt_a --candidate traces_prompt_b  # merge check

Compare for the analysis, gate for the merge. Both read the same sidecars.

Next post: Your 3am Flight Recorder — one command that prints the timeline, the scores, and the counterfactual when the agent goes off the rails.

Written from the workshop — distributions or it didn’t happen.