fix: unique measure names, and a gradeable fan-out case - #23
Merged
Conversation
Two cleanups that close out the defect list the hard dataset surfaced. 1. Colliding measure names (ROADMAP 6.3). Two tables with the same numeric column both produced the same measure name, and compileMetric resolved a name by scanning entities in order - so it silently computed the first table's measure. This was not hypothetical: the committed hard dataset already had sum_net_amt on both inv and inv_staging, and its engine cases were passing only because inv sorts before inv_staging. The measurement infrastructure had a coin flip in it. Measure names are now unique across the model, table-qualifying EVERY side of a collision so the outcome never depends on table order (inv_sum_net_amt, inv_staging_sum_net_amt). Names that do not collide are untouched; the original dataset has zero renames. Unique names rather than refuse-on-ambiguity, which would have matched the compiler's usual style: a catalog keyed by name cannot hold duplicates, and resolve_terms was offering two identical-looking entries pointing at different tables, which no user could disambiguate. 2. The fan-out case asked for two numbers, the agent answered with two queries, and the row grader only ever saw the last one - so it failed for harness reasons rather than agent ones. It now asks for one row per customer with a support case: a single gradeable result set that keeps the trap sharp. The naive double join still inflates Acme to 8156 vs 2039 and Umbrella to 4515 vs 2257.50.
The hard A/B is unchanged at +30.6 (grounded 31/36, raw-sql 20/36) across two runs at different code states - a useful reproducibility signal for the harness. The reframed fan-out case still fails 0/3, but now for a real reason: it returns [2, 3405] against an expected [1702.5, 2], which is exactly the 2x inflation for a customer with two tickets. Before the reframe it failed on 'column count 1, expected 2', a grading artifact. So the reframe worked - the case now measures what it was built to measure - and the underlying fan-out failure is genuine. Worth recording for step 7: the grain warning added in 6.2 is present in the context for this exact entity and did not prevent the double count. A passive warning is not sufficient here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two cleanups that close out the defect list the hard dataset surfaced.
1. Colliding measure names (ROADMAP 6.3). Two tables with the same numeric column both produced the same measure name, and
compileMetricresolved a name by scanning entities in order - so it silently computed the first table's measure.This was not hypothetical: the committed hard dataset already had
sum_net_amton bothinvandinv_staging, and its engine cases were passing only becauseinvsorts beforeinv_staging. The measurement infrastructure had a coin flip in it.Measure names are now unique across the model, table-qualifying every side of a collision so the outcome never depends on table order (
inv_sum_net_amt,inv_staging_sum_net_amt). Names that do not collide are untouched - the original dataset has zero renames.I chose unique names over refuse-on-ambiguity, which would have matched the compiler's usual refuse-don't-guess style, because a catalog keyed by name cannot hold duplicates and
resolve_termswas offering two identical-looking entries pointing at different tables.2. The fan-out case was failing for harness reasons. It asked for two numbers, the agent answered with two queries, and the row grader only ever saw the last one. It now asks for one row per customer with a support case: a single gradeable result set that keeps the trap sharp.
Re-measured
Identical across two runs at different code states - a useful reproducibility signal for the harness itself.
The reframed fan-out case still fails 0/3, but now for a real reason:
[2, 3405]against an expected[1702.5, 2], which is exactly the 2x inflation for a customer with two tickets. Before the reframe it failed oncolumn count 1, expected 2, a grading artifact. So the reframe did its job - the case now measures what it was built to measure.Recorded for step 7: the grain warning added in 6.2 is present in the context for this exact entity and did not prevent the double count, so a passive warning is not sufficient here.
Verification
npm run checkexit 0;npm run test:cli143 pass / 0 failnpm run eval:engine18/18;npm run eval:engine:hard25/25 (cases updated for the qualified name)🤖 Generated with Claude Code
https://claude.ai/code/session_01CKZ9UVgw2Mhiu6T5YAG7k6