Session Gym in 20 Minutes — 10 Traces to a Quality Gate

This is the first of eight hands-on posts covering every scenario from the Pulse coverage analysis — each one copy-paste runnable against real commands and real output. We start with A1, the session gym: 10 traces in, a regression corpus out, replay on model change, and a merge gate that fails the build when quality drops. Twenty minutes, one terminal, zero Hermes install — pip install hermes-pulse plus a directory of trace files is the whole setup.

What you need

A directory of unroll trace files. Mine is the local Pulse corpus — 10 traces captured from real Hermes sessions:

pip install hermes-pulse
git clone https://github.com/dark5un/pulse && cd pulse
ls corpus/ | head -4

20260905_124136_5eba1d.py
20260905_124136_5eba1d.score.json
20260905_124321_640a40.py
20260905_124321_640a40.score.json

Each trace ships with a *.score.json sidecar (score, penalty, model, task type, cost, signals with evidence). Everything below reads sidecars and trace files only — no session database, no config, no daemon.

Step 1: Build the corpus (2 min)

Score every trace, keep the worst 5 as your regression set:

uv run python scripts/build_corpus.py --traces corpus --out /tmp/corpus_demo --keep 5

 90  $0.0000  20260905_124510_3b6a81.py  ['latency_regression']
100  $0.0000  20260905_124136_5eba1d.py  []
100  $0.0000  20260905_124321_640a40.py  []
100  $0.0000  20260905_124321_8d8106.py  []
100  $0.0000  20260905_124321_ba5e40.py  []
kept 5/10 in /tmp/corpus_demo

The bottom of the barrel floats to the top: the one trace with a latency_regression signal (score 90) leads the list. That is the whole point — the corpus is your worst sessions, the ones a model change is most likely to break further.

(Flag note: --traces and --out may point at the same directory — refresh mode rescores in place instead of copying. Same-dir rescues instead of SameFileError.)

Step 2: Replay the corpus (5 min)

pulse replay runs every trace dry-run-from-cache (default) or live (--live executes real LLM calls), fanned out with per-trace timeout:

uv run pulse replay --corpus corpus --jobs 4

PASS      20260905_124136_5eba1d.py
PASS      20260905_124321_640a40.py
PASS      20260905_124321_8d8106.py
PASS      20260905_124321_ba5e40.py
PASS      20260905_124321_cc3f5f.py
PASS      20260905_124321_d803de.py
PASS      20260905_124321_ee1135.py
FAIL(1)   20260905_124510_3b6a81.py
FAIL(1)   cost-syntax-check.py
PASS      cost-unit-test.py
8/10 passed, 2 failed

Two failures — and they are honest failures worth reading. --json gives the output tails:

• 20260905_124510_3b6a81.py dies in its own structured-output printer: UnicodeEncodeError: surrogates not allowed — the trace captured lone surrogates from a model stream and its json.dumps(..., ensure_ascii=False) chokes on stdout. A capture-side encoding bug, surfaced by replay. That is the gym working.
• cost-syntax-check.py dies with NameError: name 'null' is not defined — a hand-written synthetic trace with JSON null pasted as Python. A bad test fixture, surfaced by replay. Also the gym working.

Neither is a Pulse bug; both are exactly what “replay on model change” is for. Exit code is 1 when anything fails, 0 when all pass — CI-ready.

Step 3: Rank it (2 min)

uv run pulse leaderboard --corpus corpus

== chat (10 traces) ==
  BEST:
    100  008e785bc8ee  meta/muse-spark-1.3
    100  97d020c11e8b  meta/muse-spark-1.3
    100  777790ac1029  meta/muse-spark-1.3
  WORST:
     90  6a71658a24e2  meta/muse-spark-1.3 latency_regression
    100  008e785bc8ee  meta/muse-spark-1.3
    100  97d020c11e8b  meta/muse-spark-1.3

Session IDs are sha256-anonymized (first 12 chars); ties break toward lower cost. The worst slot names the signal — latency_regression — so you know why before opening anything.

Step 4: Gate it (3 min)

The model-change loop closes with the merge check. Baseline corpus vs candidate corpus; fail when any task-type mean drops more than tolerance (default 5 points):

uv run python scripts/pulse_gate.py --baseline /tmp/gate_base --candidate /tmp/gate_cand

Pulse gate: PASS  baseline=99.0 candidate=98.89 delta=-0.11 (tolerance=5.0)
task         baseline candidate   drop
chat             99.0     98.89   0.11

Drop it into CI as one line:

- name: Pulse quality gate
  run: uv run python scripts/pulse_gate.py --baseline corpus/main --candidate corpus/pr

The loop, on one line

Replay → refresh (build_corpus.py --traces corpus --out corpus) → leaderboard → gate. New model version? Run the loop. Prompt change? Run the loop. Green means ship it — with the evidence attached, not vibes.

Next post: Prompt A/B With Distributions — two prompt variants, N traces each, and reading a verdict that says “provisional n=5” instead of pretending.

Written from the workshop — the gym is open, bring your worst sessions.