The Session Gym, Part 1: Leaderboards and CI Gates

Pulse v0.2 learned to score unroll traces. Pulse v0.3 asks what you do with a whole directory of scored traces: rank them like a leaderboard, and fail the build when a candidate model scores worse than baseline. Both ship as standalone tools — pip install hermes-pulse plus a trace directory, no Hermes install — because the gym lives on the decoupled side of the architecture: plugins capture, traces and scores are open artifacts, and the quality lab runs anywhere.

This is Part 1: gamified quality and quality-as-a-merge-check. Part 2 covers adversarial calmness and skill portability.

The layering that makes the gym possible

Three layers, each dumber about the others than you’d expect:

1. Capture layer (Hermes-specific, fine). The unroll plugin writes each session as a standalone Python trace; the Pulse plugin scores it. They do their job and stop.
2. Artifact layer (open). Trace .py files plus *.score.json sidecars — plain files carrying score, penalty, model, task type, cost, active skills, and signals with evidence. Any harness, any lab, any laptop can read them.
3. Gym layer (portable). Everything in this post operates on sidecars and trace files only. No session database, no env vars, no config file. Fresh clone plus one command equals output.

The enabling refactor for v0.3 was extracting the scorer both the corpus builder and the gym tools share (trace_score.py) — one scoring path, so the leaderboard, the gate, and the corpus can never disagree about what a score means.

Leaderboards: gamified session quality

pulse leaderboard reads the sidecars in ./corpus (scoring any trace that’s missing one, live) and prints the top and bottom 3 per task type. Session IDs are sha256-anonymized; score ties break toward lower cost, so efficiency wins arguments. Here is real output from the local corpus, ten scored traces:

== chat (10 traces) ==
  BEST:
    100  008e785bc8ee  meta/muse-spark-1.3
    100  97d020c11e8b  meta/muse-spark-1.3
    100  777790ac1029  meta/muse-spark-1.3
  WORST:
     90  6a71658a24e2  meta/muse-spark-1.3 latency_regression
    100  008e785bc8ee  meta/muse-spark-1.3
    100  97d020c11e8b  meta/muse-spark-1.3

Two things worth noticing. First, the worst-of-ten still scores 90 — this corpus is the bottom ten of production traces, and the floor is high; the leaderboard tells you where the floor is, which is the number that matters for regression tracking. Second, the single flagged signal (latency_regression — one step slower than 5s) is exactly the kind of evidence-backed detail that turns “session felt slow” into a checkable claim. --json gives the same ranking for machines.

The gamification angle is deliberate, not decorative: a leaderboard that updates every week gives a team a visible, shared definition of “good session” — and onboarding from excellence (“read these five top-scored debugging sessions”) beats a wiki page.

CI gates: quality as a merge check

The gate compares mean score per task type between a baseline corpus and a candidate corpus, and fails when any task-type mean drops more than a tolerance (default 5 points):

uv run python scripts/pulse_gate.py --baseline corpus/main --candidate corpus/pr

Pulse gate: PASS  baseline=99.0 candidate=99.0 delta=0.0 (tolerance=5.0)
task         baseline candidate   drop
chat             99.0      99.0    0.0

Exit 0 on pass, exit 1 on fail — and the fail path was verified too (a candidate degraded by 20 points prints FAIL ... delta=-20.0 and exits 1). The intended use is a GitHub Actions step on every model or provider change:

- name: Pulse quality gate
  run: uv run python scripts/pulse_gate.py --baseline corpus/main --candidate corpus/pr

This is CI for agent behavior: replay the regression corpus live under the candidate, re-score, green means ship it. The per-task breakdown matters because a candidate that improves brainstorming while regressing coding should not average its way to a pass — any single task-type drop beyond tolerance fails the gate.

Zero-config as a design rule

Every gym tool runs with no arguments from the repo root and does the right thing: default traces dir, default corpus dir, flags override but are never required. If a downstream tool ever needs a field the sidecar lacks, the fix goes into the sidecar writer — never into a config file or a new flag. That discipline is what keeps the gym portable: the same commands run in CI, on a colleague’s laptop, or in another lab entirely, against any directory of traces.

Part 2 takes the same machinery into adversarial territory: twelve red-team prompts across four nasty categories, a calmness ranking per model, and the skill portability ledger — which skills travel between models, and which are just lore.

Pulse v0.3.0 is released on GitHub. uv run pulse leaderboard from the repo root reproduces the output above.