Skip to content

check-public-doc-tables and roll-benchmark-record disagree on what a section is: _h2_headers ignores code fences #495

Description

@localai-bot

The divergence

scripts/roll-benchmark-record.py promises in its own docstring that the gate
and the rollup can never disagree:

The allowlist is CANONICAL_SECTIONS in scripts/check-public-doc-tables.py, read
from there so this script and the CI gate can never disagree.

They can, and the allowlist is not why. _h2_headers in
scripts/check-public-doc-tables.py is a bare startswith("## ") scan with no
fence tracking, while split_sections in scripts/roll-benchmark-record.py
tracks fences correctly. A heading-shaped line inside a code fence is therefore
a SECTION to the gate and NOT a section to the script the gate tells you to run.

Reproduced on 918c568a

text = "# T\n\n## At a glance\n\n```text\n## CLAIM-FOO 2026-08-04 result\n```\n"

check_public_doc_tables._h2_headers(text)   # ['At a glance', 'CLAIM-FOO 2026-08-04 result']
roll_benchmark_record.split_sections(text)  # [('At a glance', ...)]

The consequence is a failure whose named remedy does nothing:

ERROR: docs/BENCHMARKS.md has 1 non-canonical H2 section(s) (CLAIM-FOO 2026-08-04
result); ... Run scripts/roll-benchmark-record.py --apply to move them verbatim
into .agents/benchmark-record.md

$ scripts/roll-benchmark-record.py --apply
OK: every docs/BENCHMARKS.md section is canonical, nothing to roll.

An author hitting this has to work out for themselves that the "section" is
sample output inside a fence and edit it by hand, which is the opposite of what
the mechanical rollup exists for.

Not fixed in #494, deliberately

Found while doing #460 (PR #494), which adds a fence-aware _headings() for its
own regrowth guard and leaves _h2_headers untouched. Correcting _h2_headers
changes what an existing gate counts as a section, so per AGENTS.md it needs its
own spec, a red-before test and green-after evidence, rather than riding along
in a PR scoped to something else.

Neither shipped page has a fenced heading today, so nothing is currently
misreported. docs/BENCHMARKS.md does carry fenced sample output in
## Reproduce, so it is one pasted line away.

Repair

Make _h2_headers fence-aware, most simply by deriving it from the _headings
scanner #494 adds. Red-before is the reproduction above, asserted as agreement
between the two functions rather than against a hardcoded list, so they cannot
drift apart again.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions