feat(manual-qa): metrics-collection hooks + tokenomics export/HTML report by olexis-st · Pull Request #48 · arozumenko/sdlc-skills

olexis-st · 2026-07-22T18:20:02Z

Summary

Optional add-on to the manual-qa bundle: collects token/cost/timing/pass-rate
metrics on every test-run-lead-driven session (purely observational — doesn't
touch any agent or skill, doesn't change how the team runs), plus two manual,
on-demand scripts that reshape those metrics into the EPAM cross-factory
factories-tokenomics-dataset schema and, optionally, a self-contained HTML
report from it.

hooks/ — 5 Claude Code hook events (SessionStart, PreToolUse,
PostToolUse, SubagentStop, SessionEnd) + 10 scripts. Ephemeral state
files are session_id-scoped so concurrent test-run-lead sessions in one
project don't clobber each other's metrics; orphaned files older than 7 days
are swept on the next fresh session.
build-tokenomics-report.mjs / build-tokenomics-html.mjs — verified
end-to-end against a real run (elitea-testing RUN-2026-07-22-005) before
landing here. hooks/templates/ carries the one-time per-project setup
(factory-profile.json + two READMEs) these two scripts need, since the
bundle installer only copies hooks/scripts/.
test-run-lead — normalizes each test case's id before dispatch (source
of truth: the TC file's own name), so tokens/cost stay attributable per TC
even when a case's frontmatter id and filename briefly disagree.
tasks/baseline-web/ — 8 baseline TCs this add-on's own benchmarking was
developed and verified against.
knowledge/metrics-format.md and knowledge/test-run-report-format.md
document every field the current scripts actually read/produce (turns,
subagent_dispatches, orchestrator_cost_pct, tokens_by_model,
tokens_by_agent, and the optional tokenomics-export frontmatter keys), with
a real, internally-consistent sample run throughout.

Squashed from the original feat/manual-qa-challenges branch (10 iterative
commits, developed over several sessions) onto current main as a single
clean commit. That branch's history included an entire first implementation
of this same feature (top-level hooks/scripts, build-run-metrics.mjs,
etc., plus a stray pre-run-ccusage hunk in the shared hooks/session-start)
that got superseded by the bundle-based design partway through but was never
actually deleted — this PR omits it rather than asking reviewers to read past
dead code marked "deprecated" inline. A straight rebase was tried first and
hit a real conflict on the very first of 12 commits (.gitignore, from how
far main had moved on) — squashed instead via a clean three-way merge onto
fresh main, which correctly preserved the unrelated changes main had made
to bundles/manual-qa/README.md/BUNDLE.md in the meantime.

Test plan

node --check on every .mjs script in this PR
build-tokenomics-report.mjs and build-tokenomics-html.mjs run
end-to-end in an isolated scratch directory against a real captured run
(RUN-2026-07-22-005), producing output identical to the project-local
copies they were ported from
Verified live on a real elitea-testing session: reports/metrics/*.json,
tokens_by_model, tokens_by_agent, and the tokenomics export all
populated correctly end-to-end
JSON schema example in knowledge/metrics-format.md parses and is
internally consistent (tokens_by_agent sums to session.total_tokens)
Fresh npx github:arozumenko/sdlc-skills init --bundle manual-qa install
+ one real run, on a clean project (not yet done from this exact PR
branch — please confirm on your end before merging, per the existing
"First-install self-check" in hooks/README.md)
macOS/Linux hook wrapper — still only reasoned through from the code,
not confirmed on real Mac/Linux hardware (flagged as a known caveat in
hooks/README.md)

🤖 Generated with Claude Code

…port Optional add-on to the manual-qa bundle: collects token/cost/timing/ pass-rate metrics on every test-run-lead-driven session (purely observational — doesn't touch any agent or skill), plus two manual, on-demand scripts that reshape those metrics into the EPAM cross-factory factories-tokenomics-dataset schema and, optionally, a self-contained HTML report from it. - hooks/ — 5 Claude Code hook events (SessionStart, PreToolUse, PostToolUse, SubagentStop, SessionEnd) + 10 scripts. Ephemeral state files are session_id-scoped so concurrent test-run-lead sessions in one project don't clobber each other's metrics; orphaned files older than 7 days are swept on the next fresh session. - build-tokenomics-report.mjs / build-tokenomics-html.mjs — verified end-to-end against a real run (elitea-testing RUN-2026-07-22-005) before landing here; hooks/templates/ carries the one-time per-project setup (factory-profile.json + two READMEs) these two scripts need, since the bundle installer only copies hooks/scripts/. - test-run-lead — normalizes each test case's id before dispatch (source of truth: the TC file's own name), so tokens/cost stay attributable per TC even when a case's frontmatter id and filename briefly disagree. - tasks/baseline-web/ — 8 baseline TCs this add-on's own benchmarking was developed and verified against. - knowledge/metrics-format.md and knowledge/test-run-report-format.md document every field the current scripts actually read/produce (turns, subagent_dispatches, orchestrator_cost_pct, tokens_by_model, tokens_by_agent, and the optional tokenomics-export frontmatter keys), with a real, internally-consistent sample run throughout. Squashed from the original feat/manual-qa-challenges branch (10 iterative commits plus today's fixes) onto current main — that branch's history included an entire first implementation of this same feature (top-level hooks/scripts/build-run-metrics.mjs etc., plus a stray pre-run-ccusage hunk in the shared hooks/session-start) that got superseded by the bundle-based design two commits later but was never actually deleted; this squash omits it rather than asking reviewers to read past dead code marked "deprecated" inline. A straight `git rebase origin/main` was tried first and hit a real conflict on the very first of 12 commits (.gitignore) — not worth 11 more rounds of that for history nobody needs to re-read.

…ows, and the audit/retro toolchain Squashed from 41 commits onto current main. The history is omitted rather than replayed because a third of it builds a state board — 11 files under scripts/board/ — that a later commit deletes: a commit-by-commit reviewer would read an entire implementation that no longer exists, then read its removal. The net diff is the change. (Same reasoning as arozumenko#48.) WHY NO BOARD. The board recorded progress, and progress only needs recording if something reads it mid-run. Nothing did: the runtime already persists every agent's return to journal.jsonl, and resume replays from cache. What the board added was cost — every transition a clerk dispatch — and a second version of the truth that drifted: on one campaign 4 of 12 merged cases still sat at `approved-static`. Replaced by OUTCOMES: a case ends somewhere, it does not travel a state machine, and `findings[]` is orthogonal so a green case can still report a defect it found. Batch pipeline - One workflow per batch: analysts in K lanes → builds sequenced in the one working tree → parallel static review → integrate → hardening gate (N consecutive green) → one report at close. Campaign conductor on top: heads analysis → foundation (built, reviewed, mini-gated) → waves. - Two branch levels: case branches under a batch trunk, ONE PR to base — gating a trunk and then merging N case PRs proves one object and ships another. - The fix loop runs until the reviewer APPROVES. It stops only when every surviving blocker is `persists` or `external`; anything still `unaddressed` earns another round, because "forgotten" and "impossible" look identical in a finding list and demand opposite responses. Round caps are runaway backstops, not the working control. - Red-by-design is declared, not discovered: a ticketed product defect keeps its loud failure, the gate runs it without counting it, and the case reports `blocked` on the ticket rather than a green it never earned. - Contracts hold on every host. Where a script runs the loop on Claude Code, the orchestrator is the loop elsewhere; reviewer-contract.md and implementer-contract.md carry the same rules for hosts with no workflow tool. Recovery and bookkeeping - SubagentStop hook (bundle-scoped, workflow-only, async) records each agent's structured return under .agents/automation/_returns/. fold-returns.mjs reads them — including the agents that ended WITHOUT a structured result, which are exactly the dispatches a resume must re-run. Receipts are append-only facts; the fold is derived on demand and never persisted, or the board is back. - resume-report.mjs demoted from verdict to evidence: it now scopes the journal search to THIS project (it was taking the newest journal on the machine, from a different repository, and case ids collide), reads PR state (it fetched `state` and discarded it, so a CLOSED abandoned PR read as `built`), and says plainly what it cannot interpret. - Long jobs run in the FOREGROUND, every slot. An agent that ends its turn waiting on a background job is not waiting, it is done — measured: one such stall held a finished branch and a whole conductor for 12 minutes with no error anywhere. efficiency-audit / session-retrospective - Cost per case measured, not typed: --resolved-from reads the pipeline's own run reports, reports $/spec-delivered AND $/case-examined, folds re-entries, and refuses the ratio when nothing ties the spend to the cases. - GitHub Copilot support on both skills; self-contained HTML report + CSV. - Correction detection rewritten — the old regex was anchored to `^` and matched 1 of 160 user turns on a real session; now unanchored, ranked, labelled by kind, with interrupts counted. - Transcript discovery fixed: a repo-local .claude/projects is searched (a project can relocate its whole store there), and the project-dir encoding now maps every separator Claude Code maps — 6 of 28 real projects resolved before, 28 of 28 after. 291 tests, `npm run validate` clean.

…ows, and the audit/retro toolchain Squashed from 41 commits onto current main. The history is omitted rather than replayed because a third of it builds a state board — 11 files under scripts/board/ — that a later commit deletes: a commit-by-commit reviewer would read an entire implementation that no longer exists, then read its removal. The net diff is the change. (Same reasoning as arozumenko#48.) WHY NO BOARD. The board recorded progress, and progress only needs recording if something reads it mid-run. Nothing did: the runtime already persists every agent's return to journal.jsonl, and resume replays from cache. What the board added was cost — every transition a clerk dispatch — and a second version of the truth that drifted: on one campaign 4 of 12 merged cases still sat at `approved-static`. Replaced by OUTCOMES: a case ends somewhere, it does not travel a state machine, and `findings[]` is orthogonal so a green case can still report a defect it found. Batch pipeline - One workflow per batch: analysts in K lanes → builds sequenced in the one working tree → parallel static review → integrate → hardening gate (N consecutive green) → one report at close. Campaign conductor on top: heads analysis → foundation (built, reviewed, mini-gated) → waves. - Two branch levels: case branches under a batch trunk, ONE PR to base — gating a trunk and then merging N case PRs proves one object and ships another. - The fix loop runs until the reviewer APPROVES. It stops only when every surviving blocker is `persists` or `external`; anything still `unaddressed` earns another round, because "forgotten" and "impossible" look identical in a finding list and demand opposite responses. Round caps are runaway backstops, not the working control. - Red-by-design is declared, not discovered: a ticketed product defect keeps its loud failure, the gate runs it without counting it, and the case reports `blocked` on the ticket rather than a green it never earned. - Contracts hold on every host. Where a script runs the loop on Claude Code, the orchestrator is the loop elsewhere; reviewer-contract.md and implementer-contract.md carry the same rules for hosts with no workflow tool. Recovery and bookkeeping - SubagentStop hook (bundle-scoped, workflow-only, async) records each agent's structured return under .agents/automation/_returns/. fold-returns.mjs reads them — including the agents that ended WITHOUT a structured result, which are exactly the dispatches a resume must re-run. Receipts are append-only facts; the fold is derived on demand and never persisted, or the board is back. - resume-report.mjs demoted from verdict to evidence: it now scopes the journal search to THIS project (it was taking the newest journal on the machine, from a different repository, and case ids collide), reads PR state (it fetched `state` and discarded it, so a CLOSED abandoned PR read as `built`), and says plainly what it cannot interpret. - Long jobs run in the FOREGROUND, every slot. An agent that ends its turn waiting on a background job is not waiting, it is done — measured: one such stall held a finished branch and a whole conductor for 12 minutes with no error anywhere. efficiency-audit / session-retrospective - Cost per case measured, not typed: --resolved-from reads the pipeline's own run reports, reports $/spec-delivered AND $/case-examined, folds re-entries, and refuses the ratio when nothing ties the spend to the cases. - GitHub Copilot support on both skills; self-contained HTML report + CSV. - Correction detection rewritten — the old regex was anchored to `^` and matched 1 of 160 user turns on a real session; now unanchored, ranked, labelled by kind, with interrupts counted. - Transcript discovery fixed: a repo-local .claude/projects is searched (a project can relocate its whole store there), and the project-dir encoding now maps every separator Claude Code maps — 6 of 28 real projects resolved before, 28 of 28 after. Scope: bundles/test-automation/ plus the shared hooks/ (which serve every bundle by design). The feature-development bundle is deliberately untouched — its own session-retrospective copy still carries the old parser, and fixing it belongs in a branch scoped to that bundle. 285 tests, `npm run validate` clean.

arozumenko merged commit 7908614 into main Jul 22, 2026
2 checks passed

Navigation Menu

feat(manual-qa): metrics-collection hooks + tokenomics export/HTML report - #48

feat(manual-qa): metrics-collection hooks + tokenomics export/HTML report#48
arozumenko merged 1 commit into
mainfrom
feat/manual-qa-metrics

olexis-st commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

olexis-st commented Jul 22, 2026

Summary

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants