The Landscape of Reproducible Agent Traces — A Survey of Related Work

The companion post to Unrolling the Agent Loop mapped the problem and the implementation. This one maps the academic and project landscape — what exists, what’s missing, and where the synthesis sits.

The question that motivated this survey: has anyone built the trace-as-program idea before?

The short answer is: in pieces, but not the synthesis.

The landscape

I found ten projects and papers that touch on trace reproducibility for agent systems. Each one solves a piece of the puzzle, but none of them emits a self-contained Python program that is both the trace and the executable.

Each row represents real engineering — most of these projects are substantial. The point isn’t that they’re “incomplete” but that the gap they collectively leave is the one hermes-unroll fills.

Let me walk through each one with more context.

Execution Lineage (arXiv 2505.06365, 2025)

Execution Lineage builds a DAG of artifact-producing nodes — every tool call that generates a file, every API response that feeds back into the prompt, every intermediate result. The lineage tracks which node produced which artifact and what depended on it. Replay is identity-based: you can re-run a specific node if its dependencies changed.

Where it lands: The trace is metadata stored alongside artifacts. You can navigate the DAG and re-run selective nodes, but the trace itself is not a standalone program. You need the Execution Lineage tooling to interpret it.

TraceCompiler (arXiv 2508.02680, 2025)

TraceCompiler takes a different approach: mine N agent traces for the same task, identify recurring patterns (same tool sequence, same conditional branches), and compile those intents into a deterministic workflow you can checkpoint and resume.

Where it lands: This is post-hoc mining. You need multiple traces to find the pattern, and the compiled output is a workflow definition, not the trace itself. The initial run is still ephemeral.

Shepherd (Stanford/Northeastern, 2025)

Shepherd is the closest conceptually to what I wanted. It implements reversible Git-like execution traces — you can fork from a decision point, revert to a previous state, and explore counterfactuals. The trace is an immutable effect stream where every decision is recorded.

Where it lands: The trace is a first-class object, but it’s an effect stream in Shepherd’s format, not a Python file. You need Shepherd’s tooling to navigate and replay it. And Shepherd is a research prototype, not something you can drop into an existing agent harness today.

CompileAgent (2025)

CompileAgent compiles agent reasoning traces into an intermediate representation (IR), then uses a deterministic executor to re-run steps. The goal is to separate the “reasoning” from the “execution” so you can re-run with different models.

Where it lands: The IR is a DSL, not Python. The trace is compiled into bytecode for a custom executor. This is powerful for its use case (cross-model evaluation) but doesn’t give you a Python file you can import, edit, and diff.

Hindsight (2025)

Hindsight is a full pipeline: record a trace, replay it deterministically using a response cache, attribute failures to specific steps, fix them, and verify by re-replaying. The deterministic replay uses cached LLM responses so re-runs are idempotent.

Where it lands: The trace is stored in Hindsight’s own format. The replay is deterministic via the cache, but the trace is not itself a program. You run Hindsight’s replay, not python trace.py.

AgentReplay (2025)

AgentReplay defines a language-neutral trace protocol and uses it as a regression harness. Store traces in the protocol format, then replay them against new model versions in CI. If the output diverges, the CI gate fails.

Where it lands: This is the closest to the CI/CD use case. The gap is that AgentReplay is a harness — it defines a protocol and a runner. The trace is a protocol-recorded blob, not a self-contained Python file. You need AgentReplay’s runner to interpret it.

DSPy (Stanford, 2024)

DSPy is a framework for compiling declarative LM pipelines into optimized programs. You define the pipeline as a graph of modules, and DSPy optimizes the prompts, few-shot examples, and tool calls.

Where it lands: DSPy compiles a graph into an execution plan, but it doesn’t unroll a runtime trace. The compilation is before execution, not after. There’s no record of what actually happened during a run — only the plan for how to run it.

Heimdall / stepback (2025)

Heimdall is a reversible debugger for agent runs. It records every state transition and lets you step backwards, inspect intermediate states, and run counterfactuals.

Where it lands: The trace is a record for the debugger. It’s not a standalone program. You need Heimdall to navigate it. The focus is on interactive debugging, not on producing an artifact you can version, diff, and distribute.

Hermes-Trace plugin (hlothaire, 2025)

This is the most directly relevant: an existing Hermes plugin that registers 18 hooks and captures every agent event as a directed graph. It provides rich observability — you can see the graph structure of a session and navigate individual nodes.

Where it lands: Observability. The graph is inspectable but not executable. There’s no code generation step. The plugin demonstrates that Hermes’s hook system can capture deep event data, but it stops at the visualization layer.

Hermes Flight Recorder (BunsDev, 2025)

Flight Recorder takes Hermes trajectories and produces scorecards, static reports, and CI gates. It’s focused on deterministic evaluation — running the same trajectory against multiple models and comparing the outcomes.

Where it lands: Evaluation, not replay. The trajectory is consumed by an evaluator, not compiled into a program. The output is a scorecard, not a .py file.

Where hermes-unroll fits

The gap that all of these leave is at the intersection of three properties:

1. Self-contained — the trace is a standalone artifact, not a record in a format requiring specific tooling to interpret
2. Executable — python trace.py reproduces the session without needing a special runner
3. Editable — you can import the trace, modify it, and use it as the starting point for a new conversation

None of the existing projects achieve all three. Execution Lineage is not self-contained. Shepherd is not executable without its runtime. AgentReplay is not editable (you don’t modify protocol blobs by hand). DSPy doesn’t produce a runtime trace at all.

hermes-unroll takes the simplest possible route to all three: emit Python source code. The format is human-readable, machine-executable, and universally editable because every developer has a Python interpreter.

Why Python source code?

This is the key design decision. The alternatives considered were:

• Protocol buffers / JSONL — fast to write, fast to read, but opaque. You need a decoder to inspect traces, and you can’t edit them naturally.
• YAML / TOML — human-readable but not executable. You’d need a separate runner.
• SQLite — suitable for large datasets but requires a query interface. Not human-inspectable.
• Custom IR / bytecode — what CompileAgent does. Powerful for deterministic execution but requires custom tooling.

Python source code hits the sweet spot: it’s the same language the harness is written in, every developer has a runtime, and the output is self-documenting because the data structure IS the representation.

The cross-harness question

Could this approach apply to other agent harnesses? I surveyed five:

Hermes was the natural starting point because of the plugin system — it’s the only harness where a tracing plugin can be implemented without core modifications. For LangGraph or Pi-Agent, you’d need to instrument the loop manually. For Claude Code, it’s impossible without API access to the runtime.

The larger pattern

One theme across every project in this survey — Execution Lineage, TraceCompiler, Shepherd, Hindsight, AgentReplay — is that they’re all solving the record problem but not the format problem. They each define a record format, a replay runner, and a set of tools. None of them treat the format as a first-class programming language artifact.

hermes-unroll is the opposite: emit to a format that already has universal tooling (Python), and the replay problem evaporates because the format itself is the tooling.

This is a deliberately simple approach. The complexity is in the ecosystem, not the format.

The main announcement post: Unrolling the Agent Loop covers the implementation, code, and usage. The full specification and source are at github.com/dark5un/hermes-unroll.