[engine-accuracy] EPIC: fundamentally-accurate classification at scale

[k08200]

Tracking issue for making Klorn's 4-tier classification fundamentally accurate and scale-safe. Diagnosis from a hard-measured + adversarial audit (2026-07-01).

The core finding

The accuracy number is currently unfalsifiable: CI runs 50 synthetic, body-less, single-labeler emails through a judge with EMPTY_JUDGE_CONTEXT — a thinner pipeline than production (which feeds body + corrections + sender facts + traits). Measured deterministic recall: PUSH 46% (6/13), AUTO 0% (0/4). The only accuracy instrument that scales with users is the per-user DecisionLabel override ledger (real, body-bearing, grows automatically) — not a synthetic set.

Order matters: build the measuring stick first; tuning thresholds/prompt blind overfits the synthetic blind spots.

Already shipped this session (7 PRs merged, all flag-gated / safe)

• fix(reliability): surface 25 silently-swallowed catches + guard gmail-push payload #634 reliability (silent-failure signals) · feat(push): one-tap tier override from notification actions (Later/Mute) #637 1-tap tier override · feat(calibration): risk-coverage AUTO confidence calibration from override ledger #640 risk-coverage AUTO calibration · feat(judge): semantic re-ranking of correction few-shots (free, local-by-default) #642 semantic correction re-ranking (free/Ollama) · feat(firewall): opt-in AUTO-tier execution — wire AUTO to the autonomous agent #643 opt-in AUTO-tier execution · feat(judge): exact prompt→result cache for the LLM scorer (COGS) #645 exact judge cache (COGS) · feat(judge): fleet-wide fallback-rate tripwire for silent accuracy degradation #647 fleet fallback-rate tripwire (accuracy-at-scale alarm).

Remaining work

Instrument (do first):

• [engine-accuracy] Build a real, body-bearing, multi-labeled eval set (the #1 accuracy blocker) #648 real body-bearing multi-labeled eval set (DATA — the docs: add terminal and GitHub Actions screenshots to README #1 blocker, founder/dogfood)
• [engine-accuracy] Full per-tier real-accuracy report from the DecisionLabel override ledger #649 full per-tier real-accuracy from the override ledger (code, safe now)
• [engine-accuracy] Run the eval through the PRODUCTION buildJudgeContext (body + context) #650 run the eval through production buildJudgeContext (code, safe now)

After the instrument (no blind tuning):

• [engine-accuracy] Fix the AUTO prompt↔threshold contradiction (senderTrust 0.3 vs 0.5) #651 fix the AUTO prompt↔threshold contradiction (0.3 vs 0.5) + PUSH urgency
• [engine-accuracy] Per-user thresholds + promote real-override accuracy to a release gate #652 per-user thresholds + promote real-override metric to a release gate (needs volume)
• [engine-accuracy][ops] Validate + flip the built engine flags against the eval instrument #653 validate + flip the built engine flags (ops)
• [engine-accuracy] Expose getJudgeHealth via an admin endpoint + fallback-rate follow-ups #654 getJudgeHealth admin endpoint + sender-prior guard hardening

Bottleneck: #648 (real override/label volume via dogfooding) gates the fundamental accuracy gains — this is a distribution/data problem, not a code one. The engine infrastructure is already per-user, indexed, honest-by-design, and now has a degradation tripwire.