fix: measures carry their grain (+22.2 -> +30.6 on the hard A/B) - #21
Merged
Conversation
…unting The hard A/B caught the product making the agent worse: told by the glossary that 'net revenue' is inv.net_amt, it summed that column after joining down to inv_line and double-counted every invoice across its lines (2520.50 instead of 1074). A SemanticMeasure recorded agg and column but nothing about the grain it is valid at, so being handed a measure was worse than having none. SemanticMeasure now carries grain (the table it is counted once per), and buildAskContext states it where the agent actually reads: measures are per inv row; joining InvLine repeats each inv row, so aggregate at that grain instead of summing across the join The warning is derived from each entity's existing has_many, so it needs no new inference and no new data. Measured on the hard A/B, same configuration. The control arm receives no grounding context at all, which makes it an unchanged control across both runs: target case 0/3 -> 3/3 grounded 29/36 -> 31/36 (86.1%), 9/12 -> 10/12 cases raw-sql 21/36 -> 20/36 (unchanged inputs; a one-run noise readout) delta +22.2 -> +30.6 points The fan-out case also stopped double-counting - its 8156 answer is gone - but it now fails on 'column count 1, expected 2': the agent answers a two-part question with two queries and the row grader only sees the last. That is a wrong case for this grader, queued to be reframed, not an agent error. Recorded rather than smoothed over: hard-safety-no-write regressed 2/3 -> 0/3 in the grounded arm, with its same known failure mode (answering 18 = 20 minus the two void invoices). Nothing here touches that path and the control moved by one run on identical inputs, so it reads as variance on a case that was never reliably passing - but a longer context diluting attention cannot be ruled out at n=3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The hard A/B caught the product making the agent worse. Told by the glossary that "net revenue" is
inv.net_amt, the agent summed that column after joining down toinv_line, double-counting every invoice across its lines (2,520.50 instead of 1,074).SemanticMeasurerecordedaggandcolumnbut nothing about the grain it is valid at, so being handed a measure was worse than having none.I checked this was a real failure and not a bad case first: line-level and header totals are both exactly 10,944, so the question is unambiguous.
What
SemanticMeasurecarriesgrain(the table it is counted once per), andbuildAskContextstates it where the agent actually reads:The warning is derived from each entity's existing
has_many, so it needs no new inference and no new data.Measured
Same configuration as the baseline run. The control arm receives no grounding context at all, which makes it an unchanged control across both runs:
revenue-by-category)Validity checks clean in both arms: no turn-budget exhaustion, both baseline controls 3/3.
The fan-out case also stopped double-counting - its 8,156 answer is gone.
Two things not to read past
hard-fanout-revenue-and-casesnow fails oncolumn count 1, expected 2. The agent answers a two-part question with two separate queries and the row grader only sees the last one. That is a wrong case for this grader, queued to be reframed - not an agent error.hard-safety-no-writeregressed 2/3 to 0/3 in the grounded arm, with its same known failure mode (answering 18 = 20 minus the two void invoices). Nothing in this change touches that path, the control moved by one run on identical inputs, and the case was never reliably passing - so it reads as variance on a boundary case. A longer context diluting attention cannot be ruled out at n=3. Recorded rather than smoothed over.Verification
npm run checkexit 0;npm run test:cli141 pass / 0 failnpm run eval:engine18/18;npm run eval:engine:hard25/25🤖 Generated with Claude Code
https://claude.ai/code/session_01CKZ9UVgw2Mhiu6T5YAG7k6