feat: a harder trap dataset - the grounding claim now measures +22.2 points

[kiyeonjeon21]

Why

The first A/B came back inconclusive (+2.8 points, inside the noise floor, metrics disagreeing on direction) because the dataset had stopped discriminating: 8 of its 12 cases passed 3/3 in both arms. A frontier model does not fall for a 7-table fan-out trap. Stage 3 of 3.

The result

Same configuration as before (12 cases, repeat 3, verify on, maxSteps 12). Validity checks clean - neither arm hit the turn budget, both baseline controls passed 3/3 in both arms:

	grounded	raw-sql
run pass rate	29/36 (80.6%)	21/36 (58.3%)
strict cases	9/12	6/12
mean tool steps	1.5	4.5

+22.2 points, and both metrics now agree in direction. Two cases go 3/3 versus 0/3.

The control arm's failures collapse to a single legible cause: it never excludes void invoices, returning 11,275.50 / 1,297.50 / 2,535 / 1,128 where the answers are 10,944 / 1,020 / 2,257.50 / 1,074. That is a business rule the schema cannot express and only a glossary carries - precisely what the semantic layer is for.

Two results kept in the open

• Grounding lost revenue-by-category 0/3 vs 1/3. It summed inv.net_amt after joining down to inv_line, double-counting each invoice across its lines (2520.5 vs 1074). I checked whether this was a bad case rather than a real failure: line-level and header totals are both exactly 10,944, so the question is unambiguous and this is a genuine grain error that the grounding invited by naming a measure with no grain. Recorded as ROADMAP step 6.2 with a candidate fix; the case was deliberately not changed.
• The fan-out case is 0/3 in both arms. Both write the naive double join and inflate one customer 4x (8,156 vs 2,039). Grounding does not prevent fan-out once the agent leaves query_metric and hand-writes SQL.

The dataset

11 tables of deliberately bad naming plus a committed glossary. Traps were chosen against the engine's measured solvable envelope: an FK is only discoverable when it shares a token with "<keyTable> <keyColumn>", so cust_ref and owner were rejected - the engine cannot solve them either, so they would fail in both arms and discriminate nothing. What remains sits where the engine infers a join at 92-100% from value overlap while a name-reading model must guess.

Wrong routes land far from right: revenue is 10,944 correct, 10,507.50 via the stale legacy column, 11,275.50 including void, 49,829 including the decoy load buffer.

It also found a real engine defect before a single agent case ran. On first derivation, money columns whose values coincided with a price lookup were inferred as foreign keys at 81/68/54%, and since relationship endpoints are excluded from measures, inv.net_amt silently stopped being a measure at all. The fixture was made realistic (negotiated prices, which real invoices have) so the intended traps work, and the defect is recorded as ROADMAP step 6.1 rather than worked around.

Verification

• npm run check exit 0; npm run test:cli 140 pass / 0 fail
• npm run eval:engine 18/18; npm run eval:engine:hard 25/25, now gating CI
• The glossary chain is provably load-bearing: drop --glossary and exactly the five hard-term-* cases fail (20/25). A test asserts that, so the enrichment chain cannot silently rot
• Every expectedSql was run against the CSVs before being committed, and a test re-runs all twelve so ground truth cannot drift
• Per AGENTS.md, nothing was tuned after seeing the numbers

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKZ9UVgw2Mhiu6T5YAG7k6