Skip to content

feat: emulo --coach reports how you use the model, without mining - #37

Merged
ohad6k merged 3 commits into
mainfrom
feat/usage-report
Jul 28, 2026
Merged

feat: emulo --coach reports how you use the model, without mining#37
ohad6k merged 3 commits into
mainfrom
feat/usage-report

Conversation

@ohad6k

@ohad6k ohad6k commented Jul 28, 2026

Copy link
Copy Markdown
Owner

The first substantive reply to the star outreach asked for a mode focused on improving Claude Code usage rather than extracting conventions. This is that mode, in the open source CLI.

python emulo.py --coach                   # every source it can find
python emulo.py --coach --source claude   # Claude Code only
python emulo.py --coach --json            # for other tools

No model call, no corpus written, finishes in seconds. That matters for installs: someone gets a real result before deciding whether mining is worth it.

What it reports

Asks sent three times in a row unchanged, context re-explained after the agent lost it, runs of rephrasing the same request, and the rate of turns that open by correcting the last answer. Every finding prints the dated messages behind it, windowed onto the match when the evidence sits deep in a long message, because a receipt you cannot read is not a receipt. Checks that come in under their bar print their counts, so a clean result reads as measured rather than as silence.

Measuring it changed it three times

Thresholds came from a real 2,271 session corpus, not from guessing.

  • Without a word floor, the repeat check was almost entirely ok / yes / ok do it sent three times. That is approval, not a loop. With a four word floor the same corpus yields one genuine run.
  • Widening from consecutive sends to sends within three turns added only filler repeated 84 turns apart, so the rule stayed strict.
  • no need for the repo was counting as a correction.
  • A 1.4% correction rate was being reported as a problem. Corrections now clear a rate bar, and the rate prints either way so anyone can disagree with the bar.

It also found a real mining bug

Two harness preambles were reaching the corpus as user prose: # Context from my IDE setup: (an editor stapling the open file and tab list onto the turn) and # Files mentioned by the user (Codex listing attachments). That is 2,543 messages on one real corpus, and because they repeat near-verbatim for as long as a file stays open, mining read them as deeply held rules. Both are now in INJECTED_CONTEXT_PREFIXES, which fixes you.md quality and not just this report.

Found by reading the receipts --coach printed rather than trusting its counts. The loudest finding in the first run was an IDE preamble counted as the same ask sent twelve times.

Shipping something unproven, on purpose

The reword-loop check has no confirmed true positive. Across every corpus available to test it, it fired exactly once, on a pasted pygame banner, now excluded by a line guard. It ships because the behaviour it looks for is real and cheap to check, and because a check that finds nothing prints its zero rather than staying silent. The changelog says to trust it less than the other three.

Verified

444 tests, 4 skipped, 0 failures, up from 415. The 29 new tests cover the loop detectors, the exclusions, receipt quality, the injected preambles, redaction of receipts, and the promise that --coach leaves no files behind.

Stated in the report itself: it reads only the messages you typed, which is all Emulo keeps, so it cannot see cost, tokens, tool calls, or whether the agent was right, and it does not score them.

🤖 Generated with Claude Code

ohad6k and others added 2 commits July 28, 2026 23:24
The first substantive reply to the star outreach asked for a mode focused
on improving Claude Code usage rather than extracting conventions. The data
was already there: failure modes are mined today, but they are written to
the agent under "protect this person from these" and never addressed to the
person.

--coach reads the same local logs and answers the other question. Asks sent
three times in a row unchanged, context re-explained after the agent lost
it, runs of rephrasing the same request, and the rate of turns that open by
correcting the last answer. No model call, no corpus written, so it runs in
seconds before anyone has decided whether mining is worth it. --source
claude narrows it to Claude Code, --json emits the report for other tools.

Every finding prints the dated messages behind it, windowed onto the match
when the evidence sits deep in a long message, because a receipt you cannot
read is not a receipt. Checks that come in under their bar print their
counts, so a clean result reads as measured rather than as silence.

Thresholds came from measuring a real 2,271 session corpus, and the
measurement changed the feature three times. Without a word floor the
repeat check was almost entirely "ok" and "yes" sent three times, which is
approval. "no need for the repo" was counting as a correction. A 1.4%
correction rate was being reported as a problem, so corrections now clear a
rate bar and the rate prints either way.

Also fixes two harness preambles that were being mined as user prose:
"# Context from my IDE setup:" and "# Files mentioned by the user", 2,543
messages on one real corpus. They repeat near-verbatim while a file stays
open, so mining read them as deeply held rules. That fixes you.md quality,
not just this report. Found by reading the receipts --coach printed: the
loudest finding in the first run was an IDE preamble counted as the same
ask sent twelve times.

The reword-loop check ships unproven and the changelog says so. It has
fired once ever, on a pasted pygame banner, now excluded by a line guard.

444 tests, 4 skipped, 0 failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
ditto Ready Ready Preview, Comment Jul 28, 2026 10:44pm

The plugin security scanner failed on the branch while unittest passed. The
new redaction test carried a literal matching emulo's own OPENAI_KEY pattern,
sk-[A-Za-z0-9]{20,}, and a scanner cannot tell a fixture key from a real one
that leaked. Assembling the same string at runtime keeps the test exercising
the identical redaction path with no key-shaped literal in the source.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ohad6k
ohad6k merged commit 2dace19 into main Jul 28, 2026
6 checks passed
@ohad6k
ohad6k deleted the feat/usage-report branch July 28, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant