Pulse Meets Unroll — Turning Agent Traces into Coaching, Regression Tests, and Research Data

Two plugins I built for Hermes Agent grew up side by side. hermes-unroll captures every session as a runnable Python program — the trace is the program. Pulse scores session quality across 26 signals, attributing friction to human, agent, or system. v0.4.0 of unroll and the planned Pulse v0.2 close the loop between them: capture → structure → evaluate → improve, with no manual step.

This post is the map of that loop: what Pulse gets for free from unroll trace files today, the new signals they unlock together, the closed-loop workflows that fall out, and where researchers and engineers can take them next.

What Pulse gets for free from unroll v0.4

Today Pulse parses conversation text from the session database. Unroll hands it structured evidence instead:

• Measured timing, not estimates. Every replayed step carries original_offset_ms vs replay_offset_ms plus per-step durations. Reasoning loops and tool repetition become timing-backed claims — “3 identical terminal calls, 450ms each” — instead of keyword guesses.
• Dollar costs on every signal. Each trace carries COST = {model, cost_usd, input/output_tokens}. “This correction chain cost $0.41” is the sentence that makes quality work legible to managers.
• Real dependency graphs. DEPENDENCIES maps each step to the steps it actually depended on, so correction-chain attribution follows the true graph instead of positional heuristics. Edit step 1, watch steps 1–3 invalidate transitively, blame flows along real edges.
• State graphs with subagent subgraphs. Goal drift becomes graph divergence: the walked chain vs the intended chain. Delegated work scores separately from the main loop instead of being blended into one number.
• Ground truth on tools and skills. TOOL_SCHEMAS (what existed), the dispatch record (what ran), ACTIVE_SKILLS (what the agent loaded) — tool-diversity and skill-effectiveness signals finally measure instead of inferring.
• Redaction receipts. Traces are redacted before they hit disk, and the redacted fields are listed. Sharing signal vectors computed over already-redacted traces — the federated pool Pulse plans — becomes a credible privacy story instead of a promise.

Concretely: Pulse gains an --unroll mode that reads a trace file with a safe AST loader (literals only, never executed) instead of the session DB. Existing detectors run unchanged on the reconstructed messages; new detectors run on the timing, cost, graph, and skill fields. The score lands back in a sidecar next to the trace. The capture hook on unroll’s side already exists — opt in, and every session is scored at write time.

New signals the combination unlocks

• Latency regression. Per-step replay time vs original time, tracked across runs of the same trace in CI. Catches model slowdowns, not just wrongness.
• Cost anomaly. Session cost z-scored per task type. A brainstorm that bills like a full coding session is a signal.
• Skill deadweight. A skill in ACTIVE_SKILLS with zero downstream use, followed by corrections. Direct feedback for skill authors — load without benefit is measurable now.
• Subagent waste. Delegated subgraphs with high step counts whose summaries are never referenced downstream. Delegation that went nowhere.
• Cache divergence. Live replay vs response-cache replay, diffed step by step. Diverging steps mark nondeterminism hotspots — a flakiness map of your agent setup.
• Destructive proximity. How often the agent reached for a dangerous tool, even when the dry-run guard caught it. Habitual reaching is the leading indicator; the incident is the lagging one.

Closed-loop workflows

The session gym. Collect production traces, score them all, keep the lowest-scoring ten as a regression corpus. Every model or provider change replays the corpus live and re-scores. Green means ship it. This is CI for agent behavior, and both halves already exist.

Prompt A/B with teeth. Same task, two system prompts, N traces each. Compare score distributions plus cost plus timing — not vibes. The counterfactual --edit engine even tests “what if the first prompt had said X” without rerunning from scratch.

The skill ROI ledger. Every skill gets a card: times loaded, correction rate after load, cost delta vs sessions without it. Kill or fix skills with data instead of opinion.

The automated weekly review. A scheduled job scores the week’s traces and delivers one digest: worst session with evidence quotes, cost total, top recurring signal. The review writes itself; you just read it.

For AI researchers

• Fine-tuning data, nearly free. Traces are message lists with tool calls; Pulse scores are quality labels. High-score traces become SFT data. Correction chains become preference pairs — pre-correction as rejected, post-correction as chosen — for DPO. The labeling pipeline you would otherwise build by hand falls out of tools that already run.
• Dense reward labels. Two dozen signals with evidence spans per session is a reward-model training set. Train a small judge on trace-to-signal pairs; validate against human review.
• Experiments with real numbers. Tool-repetition rate across four providers on identical traces. The cost-vs-quality frontier for live replay. Cross-model skill portability. The corpus, the scorer, and the replayer form a complete experimental apparatus — the lab bench for the landscape the paper-analysis post surveyed.
• Reproducibility artifacts. A trace file plus its score sidecar is self-contained: runnable, scored, costed. Ship it as supplementary material and reviewers can actually rerun your agent sessions.

For engineers and production teams

• Incident postmortems. The trace is the flight recorder. Replay it step by step, substitute the bad tool result, watch the counterfactual. The score says whether the session was already off the rails before the incident or broke at exactly one step.
• Honest LLM bills. The cost ledger aggregated by session metadata is the first per-feature LLM bill most teams ever see. Attribute spend to teams, features, or task types.
• Flaky-agent quarantine. Replay the same trace live five times; the divergence map shows which steps are nondeterministic. Treat them like flaky tests — mock, constrain, or quarantine.
• Onboarding from excellence. New team members read top-scored traces for their task type instead of absorbing tribal knowledge. “Here are five excellent debugging sessions, with scores explaining why” beats a wiki page.
• Quality as a merge check. Fail the build when a candidate model scores worse than baseline on the regression corpus. Pulse as a CI gate turns session quality into a property the pipeline enforces.

Wilder ideas, cheap to prototype

A trace leaderboard per task type (engineers compete to produce clean traces — gamification works on engineers). An adversarial red-team corpus measuring which models stay calm under deliberately confusing prompts. Cross-model skill portability cards showing which skills are portable wisdom vs model-specific lore. All three are queries over data the pipeline already produces.

A note on how this was built

Much of the work described here — the unroll v0.3.0 and v0.4.0 implementation phases, the timed replay engine, the safety guards, the counterfactual mode — was written with muse-spark-1.3, orchestrated through Hermes Agent’s subagent-driven development: phased plans, test-driven implementation per task, and review before commit. The loop the post describes (plan → build → verify → improve) is also the loop the software was built with. Credit where it is due: the model did the bulk of the implementation work; the human set the direction and held the quality bar.

What ships first

The integration plan is written: Pulse --unroll mode (loader, message reconstruction, CLI flag), then three detectors — latency, cost, skill-deadweight — all pure functions over fields that exist in v0.4 trace files today. Then the session-gym cron, then the research pipeline. Deterministic detectors first; the LLM-judge deep mode can follow without blocking any of it.

The deeper point: neither tool is the product. The loop is the product. Capture every run, score every run, feed the scores back into prompts, skills, and model choices, and the agent compound-interest curve starts working for you. Two plugins, one loop, and sessions that get better the more you use them.

Written from the workshop — the loop is the product, and it keeps turning.