feat: emulo --coach reports how you use the model, without mining - #37
Merged
Conversation
The first substantive reply to the star outreach asked for a mode focused on improving Claude Code usage rather than extracting conventions. The data was already there: failure modes are mined today, but they are written to the agent under "protect this person from these" and never addressed to the person. --coach reads the same local logs and answers the other question. Asks sent three times in a row unchanged, context re-explained after the agent lost it, runs of rephrasing the same request, and the rate of turns that open by correcting the last answer. No model call, no corpus written, so it runs in seconds before anyone has decided whether mining is worth it. --source claude narrows it to Claude Code, --json emits the report for other tools. Every finding prints the dated messages behind it, windowed onto the match when the evidence sits deep in a long message, because a receipt you cannot read is not a receipt. Checks that come in under their bar print their counts, so a clean result reads as measured rather than as silence. Thresholds came from measuring a real 2,271 session corpus, and the measurement changed the feature three times. Without a word floor the repeat check was almost entirely "ok" and "yes" sent three times, which is approval. "no need for the repo" was counting as a correction. A 1.4% correction rate was being reported as a problem, so corrections now clear a rate bar and the rate prints either way. Also fixes two harness preambles that were being mined as user prose: "# Context from my IDE setup:" and "# Files mentioned by the user", 2,543 messages on one real corpus. They repeat near-verbatim while a file stays open, so mining read them as deeply held rules. That fixes you.md quality, not just this report. Found by reading the receipts --coach printed: the loudest finding in the first run was an IDE preamble counted as the same ask sent twelve times. The reword-loop check ships unproven and the changelog says so. It has fired once ever, on a pasted pygame banner, now excluded by a line guard. 444 tests, 4 skipped, 0 failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The plugin security scanner failed on the branch while unittest passed. The
new redaction test carried a literal matching emulo's own OPENAI_KEY pattern,
sk-[A-Za-z0-9]{20,}, and a scanner cannot tell a fixture key from a real one
that leaked. Assembling the same string at runtime keeps the test exercising
the identical redaction path with no key-shaped literal in the source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first substantive reply to the star outreach asked for a mode focused on improving Claude Code usage rather than extracting conventions. This is that mode, in the open source CLI.
No model call, no corpus written, finishes in seconds. That matters for installs: someone gets a real result before deciding whether mining is worth it.
What it reports
Asks sent three times in a row unchanged, context re-explained after the agent lost it, runs of rephrasing the same request, and the rate of turns that open by correcting the last answer. Every finding prints the dated messages behind it, windowed onto the match when the evidence sits deep in a long message, because a receipt you cannot read is not a receipt. Checks that come in under their bar print their counts, so a clean result reads as measured rather than as silence.
Measuring it changed it three times
Thresholds came from a real 2,271 session corpus, not from guessing.
ok/yes/ok do itsent three times. That is approval, not a loop. With a four word floor the same corpus yields one genuine run.no need for the repowas counting as a correction.It also found a real mining bug
Two harness preambles were reaching the corpus as user prose:
# Context from my IDE setup:(an editor stapling the open file and tab list onto the turn) and# Files mentioned by the user(Codex listing attachments). That is 2,543 messages on one real corpus, and because they repeat near-verbatim for as long as a file stays open, mining read them as deeply held rules. Both are now inINJECTED_CONTEXT_PREFIXES, which fixesyou.mdquality and not just this report.Found by reading the receipts
--coachprinted rather than trusting its counts. The loudest finding in the first run was an IDE preamble counted as the same ask sent twelve times.Shipping something unproven, on purpose
The reword-loop check has no confirmed true positive. Across every corpus available to test it, it fired exactly once, on a pasted pygame banner, now excluded by a line guard. It ships because the behaviour it looks for is real and cheap to check, and because a check that finds nothing prints its zero rather than staying silent. The changelog says to trust it less than the other three.
Verified
444 tests, 4 skipped, 0 failures, up from 415. The 29 new tests cover the loop detectors, the exclusions, receipt quality, the injected preambles, redaction of receipts, and the promise that
--coachleaves no files behind.Stated in the report itself: it reads only the messages you typed, which is all Emulo keeps, so it cannot see cost, tokens, tool calls, or whether the agent was right, and it does not score them.
🤖 Generated with Claude Code