The divergence
scripts/roll-benchmark-record.py promises in its own docstring that the gate
and the rollup can never disagree:
The allowlist is CANONICAL_SECTIONS in scripts/check-public-doc-tables.py, read
from there so this script and the CI gate can never disagree.
They can, and the allowlist is not why. _h2_headers in
scripts/check-public-doc-tables.py is a bare startswith("## ") scan with no
fence tracking, while split_sections in scripts/roll-benchmark-record.py
tracks fences correctly. A heading-shaped line inside a code fence is therefore
a SECTION to the gate and NOT a section to the script the gate tells you to run.
Reproduced on 918c568a
text = "# T\n\n## At a glance\n\n```text\n## CLAIM-FOO 2026-08-04 result\n```\n"
check_public_doc_tables._h2_headers(text) # ['At a glance', 'CLAIM-FOO 2026-08-04 result']
roll_benchmark_record.split_sections(text) # [('At a glance', ...)]
The consequence is a failure whose named remedy does nothing:
ERROR: docs/BENCHMARKS.md has 1 non-canonical H2 section(s) (CLAIM-FOO 2026-08-04
result); ... Run scripts/roll-benchmark-record.py --apply to move them verbatim
into .agents/benchmark-record.md
$ scripts/roll-benchmark-record.py --apply
OK: every docs/BENCHMARKS.md section is canonical, nothing to roll.
An author hitting this has to work out for themselves that the "section" is
sample output inside a fence and edit it by hand, which is the opposite of what
the mechanical rollup exists for.
Not fixed in #494, deliberately
Found while doing #460 (PR #494), which adds a fence-aware _headings() for its
own regrowth guard and leaves _h2_headers untouched. Correcting _h2_headers
changes what an existing gate counts as a section, so per AGENTS.md it needs its
own spec, a red-before test and green-after evidence, rather than riding along
in a PR scoped to something else.
Neither shipped page has a fenced heading today, so nothing is currently
misreported. docs/BENCHMARKS.md does carry fenced sample output in
## Reproduce, so it is one pasted line away.
Repair
Make _h2_headers fence-aware, most simply by deriving it from the _headings
scanner #494 adds. Red-before is the reproduction above, asserted as agreement
between the two functions rather than against a hardcoded list, so they cannot
drift apart again.
The divergence
scripts/roll-benchmark-record.pypromises in its own docstring that the gateand the rollup can never disagree:
They can, and the allowlist is not why.
_h2_headersinscripts/check-public-doc-tables.pyis a barestartswith("## ")scan with nofence tracking, while
split_sectionsinscripts/roll-benchmark-record.pytracks fences correctly. A heading-shaped line inside a code fence is therefore
a SECTION to the gate and NOT a section to the script the gate tells you to run.
Reproduced on
918c568aThe consequence is a failure whose named remedy does nothing:
An author hitting this has to work out for themselves that the "section" is
sample output inside a fence and edit it by hand, which is the opposite of what
the mechanical rollup exists for.
Not fixed in #494, deliberately
Found while doing #460 (PR #494), which adds a fence-aware
_headings()for itsown regrowth guard and leaves
_h2_headersuntouched. Correcting_h2_headerschanges what an existing gate counts as a section, so per AGENTS.md it needs its
own spec, a red-before test and green-after evidence, rather than riding along
in a PR scoped to something else.
Neither shipped page has a fenced heading today, so nothing is currently
misreported.
docs/BENCHMARKS.mddoes carry fenced sample output in## Reproduce, so it is one pasted line away.Repair
Make
_h2_headersfence-aware, most simply by deriving it from the_headingsscanner #494 adds. Red-before is the reproduction above, asserted as agreement
between the two functions rather than against a hardcoded list, so they cannot
drift apart again.