What a Remediation Plan Taught Two Plugins About Quality

An independent review landed on my two Hermes plugins — Pulse, the session-health coach, and hermes-unroll, the trace-to-program recorder — and I spent the last cycle working through all twelve of its work units. Thirteen branches, hundreds of new tests, every finding closed or deliberately scoped. This post is about what that process surfaced: the deep loader and lifecycle knowledge that changed both plugins, and why each change makes them measurably better tools.

The short version: both plugins were built on plausible guesses about how the host behaves. The remediation replaced guesses with evidence read from the live source — and almost every guess turned out to be wrong in an interesting way.

Verify inspects, never executes

The sharpest finding was also the smallest. Pulse’s verify command — the thing you run against a downloaded artifact — executed the trace file in a subprocess to check it “replays”. Trace files are generated Python programs. Verifying an artifact was arbitrary host code execution, one verify away from whoever checked it.

The fix deletes the execution path entirely. verify now reports three structural facts: the trace parses via the safe AST loader and defines its required constants, its bytes match the sha256 pinned at bundle time, and rescoring it reproduces the pinned sidecar score. The old replays key (“the script exited 0”) is gone with the schema bump, because it measured the wrong thing. Replay still exists — it just lives behind the explicit pulse replay path the operator opts into per trace, never inside verification.

Quality effect: a trust boundary where there wasn’t one. Reviewers, CI jobs, and curious strangers can now check artifacts without running them.

The loader was never what we assumed

Both plugins carried a sys.path prologue and absolute self-imports, following the plugin skill’s guidance. Reading the actual loader (hermes_cli/plugins.py) showed directory plugins import as real packages — submodule_search_locations set, __path__ pointing at the plugin dir, stale submodules evicted, per-home namespacing. Relative imports work natively. The sys.path hack wasn’t just unnecessary; it was the cause of a cwd-dependent ModuleNotFoundError that only reproduced from the plugin’s parent directory.

Pulse is now entirely on relative imports, the hack is deleted, and install.sh writes modules to exactly one location instead of the same module at two paths. The skill that taught the old pattern got patched in the same cycle, so the next plugin doesn’t inherit the defect.

Quality effect: installs that work from any directory, one canonical module location, and guidance that agrees with the code.

Sessions are concurrent — state must be keyed

Unroll kept one process-global recorder, replaced on every session start. The gateway runs sessions concurrently across many platforms — this was live cross-session corruption, not a hypothetical. Every hook now resolves a SessionContext from its own session_id under a lock; unknown sessions are logged no-ops, never fallbacks; subagent events belong to the parent.

The research gift here was that the host already passes everything needed: post_tool_call carries session_id, tool_call_id, turn_id, status, and error_type — all previously swallowed into **kwargs. No upstream change required, and the previously-planned “stable identity” work got its key for free.

Quality effect: two interleaved sessions now produce two clean traces. The isolation tests fail if a single event crosses over.

Per-turn hooks and the leak they kept rewriting

Two measurements changed severity ratings. First, on_session_end fires once per turn, not once per session — unroll was regenerating a 600KB+ artifact with a non-atomic write on every turn. Second, the redactor covered event payloads but never session metadata, so generated files contained the full system prompt — memory, profile, registration details — at mode 0644, while correctly showing redaction markers in event data. The regex was fine; it was never called.

So finalization is now exactly-once (state accumulates per turn, one atomic write at on_session_finalize, sealed contexts, byte-identical duplicates), redaction covers every generated string field and fails closed, files go out 0600 into 0700 directories with content-hash filenames. And because old artifacts are still on disk, both changelogs ship re-permission guidance — a code fix alone would leave the leaked data sitting there.

Quality effect: no more O(turns) rewrite storms against a 10-second finalize budget, and shareable artifacts that are actually safe to share.

Replay identity: from 20–35% to exact

The response cache keyed lookups by a positional counter over 2 event kinds while steps were numbered over 13 — measured hit rate 20–35%, with the “hits” being coincidental collisions serving the wrong event’s result. Silent data corruption in a tool whose purpose is faithful replay, masked by a quiet default fallback.

Events now get a stable id at record time (preferring the host’s own tool_call_id/turn_id), the cache is keyed by it, misses raise loudly, recorded tool arguments are emitted as dispatch defaults, and --stop-at bounds execution instead of truncating display. The counterfactual path applies edits to input before replay rather than mutating results after it, and live replay is honestly scoped as model-only — tools stay cached, because faithful live tool execution belongs behind the host’s permission stack, not inside a standalone artifact.

Quality effect: dry-run replay resolves every lookup to its own event, and the documentation describes the replay modes that actually exist.

Feedback that can’t cross wires, scores that agree

On the Pulse side: deep-mode failures previously vanished (only the success path persisted), so analyses disappeared from trends. Every deep exit now writes exactly one row with its own run mode. Feedback verbs bound to a global latest row — two concurrent sessions cross-rated each other; they now bind to the session just analyzed. The judge input path got the same treatment as unroll’s redaction work: secrets redacted before prompt construction, bounded transcripts with disclosed truncation, key resolution honoring the active Hermes home, and an explicit notice on the CLI before any transcript leaves the machine. The phantom deep_context_drift weight is gone, replaced by entries for the four real deep signals, and every scoring entrypoint now agrees on one canonical result.

Quality effect: trends distinguish deterministic, failed, and successful deep runs; feedback lands on the right session; nothing scores differently depending on which door you entered through.

The meta-lesson

Twelve work units, and the pattern repeats: read the host source before building on it. Hook payloads aren’t documented — they’re discoverable by grepping dispatch sites. Hook names lie (on_session_end is per turn). Budgets are real (10s finalize, bare except: pass). The plan’s five “open questions” all resolved without a single upstream change, because the answers were already in the code.

Both plugins now describe themselves as experimental research tooling, with version-consistency gates and loader-level integration harnesses to keep them honest. The branches are all pushed and linked below — thirteen of them, one per unit, each with its failing-first tests.

Pulse: fix/artifact-no-exec, fix/input-validation, fix/persistence-session-binding, fix/scoring-canonical-privacy, fix/install-contract, test/loader-integration, chore/version-consistency. Unroll: fix/session-context, fix/lifecycle-redaction, fix/replay-identity, fix/live-semantics, test/loader-integration, chore/version-consistency.

Written from the workshop — the guesses are gone, the evidence stayed.