Skip to content

fix: measures carry their grain (+22.2 -> +30.6 on the hard A/B) - #21

Merged
kiyeonjeon21 merged 1 commit into
mainfrom
fix/measure-grain
Jul 25, 2026
Merged

fix: measures carry their grain (+22.2 -> +30.6 on the hard A/B)#21
kiyeonjeon21 merged 1 commit into
mainfrom
fix/measure-grain

Conversation

@kiyeonjeon21

Copy link
Copy Markdown
Owner

Why

The hard A/B caught the product making the agent worse. Told by the glossary that "net revenue" is inv.net_amt, the agent summed that column after joining down to inv_line, double-counting every invoice across its lines (2,520.50 instead of 1,074). SemanticMeasure recorded agg and column but nothing about the grain it is valid at, so being handed a measure was worse than having none.

I checked this was a real failure and not a bad case first: line-level and header totals are both exactly 10,944, so the question is unambiguous.

What

SemanticMeasure carries grain (the table it is counted once per), and buildAskContext states it where the agent actually reads:

measures are per inv row; joining InvLine repeats each inv row,
so aggregate at that grain instead of summing across the join

The warning is derived from each entity's existing has_many, so it needs no new inference and no new data.

Measured

Same configuration as the baseline run. The control arm receives no grounding context at all, which makes it an unchanged control across both runs:

before after
target case (revenue-by-category) grounded 0/3 grounded 3/3
grounded 29/36 (80.6%), 9/12 cases 31/36 (86.1%), 10/12 cases
raw-sql 21/36 (58.3%) 20/36 (55.6%)
delta +22.2 +30.6

Validity checks clean in both arms: no turn-budget exhaustion, both baseline controls 3/3.

The fan-out case also stopped double-counting - its 8,156 answer is gone.

Two things not to read past

  • hard-fanout-revenue-and-cases now fails on column count 1, expected 2. The agent answers a two-part question with two separate queries and the row grader only sees the last one. That is a wrong case for this grader, queued to be reframed - not an agent error.
  • hard-safety-no-write regressed 2/3 to 0/3 in the grounded arm, with its same known failure mode (answering 18 = 20 minus the two void invoices). Nothing in this change touches that path, the control moved by one run on identical inputs, and the case was never reliably passing - so it reads as variance on a boundary case. A longer context diluting attention cannot be ruled out at n=3. Recorded rather than smoothed over.

Verification

  • npm run check exit 0; npm run test:cli 141 pass / 0 fail
  • npm run eval:engine 18/18; npm run eval:engine:hard 25/25
  • A new test builds a two-table fixture and asserts the grain is recorded and the warning names the child that would repeat the rows

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKZ9UVgw2Mhiu6T5YAG7k6

…unting

The hard A/B caught the product making the agent worse: told by the glossary
that 'net revenue' is inv.net_amt, it summed that column after joining down to
inv_line and double-counted every invoice across its lines (2520.50 instead of
1074). A SemanticMeasure recorded agg and column but nothing about the grain it
is valid at, so being handed a measure was worse than having none.

SemanticMeasure now carries grain (the table it is counted once per), and
buildAskContext states it where the agent actually reads:

  measures are per inv row; joining InvLine repeats each inv row, so aggregate
  at that grain instead of summing across the join

The warning is derived from each entity's existing has_many, so it needs no new
inference and no new data.

Measured on the hard A/B, same configuration. The control arm receives no
grounding context at all, which makes it an unchanged control across both runs:

  target case  0/3 -> 3/3
  grounded     29/36 -> 31/36 (86.1%), 9/12 -> 10/12 cases
  raw-sql      21/36 -> 20/36 (unchanged inputs; a one-run noise readout)
  delta        +22.2 -> +30.6 points

The fan-out case also stopped double-counting - its 8156 answer is gone - but it
now fails on 'column count 1, expected 2': the agent answers a two-part question
with two queries and the row grader only sees the last. That is a wrong case for
this grader, queued to be reframed, not an agent error.

Recorded rather than smoothed over: hard-safety-no-write regressed 2/3 -> 0/3 in
the grounded arm, with its same known failure mode (answering 18 = 20 minus the
two void invoices). Nothing here touches that path and the control moved by one
run on identical inputs, so it reads as variance on a case that was never
reliably passing - but a longer context diluting attention cannot be ruled out
at n=3.
@kiyeonjeon21
kiyeonjeon21 merged commit 6b6995d into main Jul 25, 2026
1 check passed
@kiyeonjeon21
kiyeonjeon21 deleted the fix/measure-grain branch July 25, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant