From Traces to Training Data — Your Logs Are a Dataset

Part 4 read the best sessions. Part 5 trains on them: pulse export turns a scored corpus into SFT rows plus DPO pairs, filtered by score, with a redaction receipt per file. The pipeline you’d otherwise build by hand, free as a side effect of observability you already run.

Review first, export second

Correction mining has false positives — not every correction is a clean chosen/rejected pair. So the workflow starts with --review, which dumps mined pairs for human spot-check before anything lands on disk:

uv run pulse export --corpus corpus --review

0 pairs — spot-check before training; not every correction is clean

Zero pairs here, honestly reported rather than padded. This corpus has no correction chains (thin traces, high scores), so there’s nothing to mine — and the tool says so instead of inventing pairs. On a corpus with real back-and-forth, this is where you’d read REJECTED vs CHOSEN and decide what trains.

The export

uv run pulse export --corpus corpus --out /tmp/expdemo --min-score 95

kept=9 dropped=1 -> /tmp/expdemo/sft.jsonl + pairs.jsonl (redacted-at-capture)

Score ≥ 95 → SFT candidate; the single 90 drops out. The manifest tells the full story:

{
  "format": "sharegpt",
  "min_score": 95,
  "kept": 9,
  "dropped": 1,
  "redaction_receipt": "redacted-at-capture"
}

9 files, 9 SFT lines, 0 pair lines. Mapping recap: trace TIMELINE → message list with tool calls; pulse score → quality filter; correction_chain evidence → DPO pairs (pre-correction assistant turn = rejected, post = chosen).

Why the receipt matters

Traces are redacted at capture (unroll’s redact.py), and the export proves it per file with redacted-at-capture in the manifest — not an assertion, a receipt. Training data that can’t prove its provenance doesn’t ship; this manifest is the provenance.

--format jsonl gives prompt/completion rows instead of ShareGPT messages. Same filter, same receipt.

Next post: The Honest LLM Bill — join-side attribution, capture-side tags, and the untagged bucket.

Written from the workshop — your logs were a dataset all along.