ci: give main a baseline lane that can actually finish (#274)

[localai-bot]

Closes the second half of #274: main is not verified by its own CI, so nobody can name the last SHA where it was green, and a red check on a PR reads as someone else's problem. Spec: .agents/specs/main-verifiability.md. Issue table row added in .agents/roadmap_v1.md.

The six sanitizer failures are NOT touched (that is #274 finding #1 / #301, in flight in parallel). The first baseline run will therefore be RED, on exactly those six. That is the correct first verdict and the reason not to mute them: a baseline that hides known failures certifies nothing.

The stated mechanism was wrong, and that changes the fix

#274's third comment checked the workflow-level concurrency block, found it already correct for push, concluded the cancellation must be external — "a repository/org-level runner concurrency or spending cap … an automation cancelling older runs" — and advised the next person not to touch the concurrency config.

The workflow-level block is indeed correct. The nine job-level groups are the cause. Each is keyed on ${{ github.ref }} with cancel-in-progress: true, and every push to main shares the constant refs/heads/main.

Measured, 3 for 3 — the cancel instant equals the next push's start instant:

Run	grouped jobs cancelled at	next run	created at
31485402200	11:46:33	31488132224	11:46:32
31482845117	11:05:12	31485054749	11:05:11
31478575965	09:39:10	31478659162	09:39:09

It also explains the partial cancellation that comment found puzzling: a job-level group kills individual jobs, so finished ones keep their result and the two DIFF-scoped jobs that carry no group at all always survive. Short-vs-long was the right observation; the cause is runtime crossing the next-push interval, not an external canceller.

40 consecutive main runs at 0eb049f7: 26 cancelled, 12 failure, 1 success, all event=push, spanning 17.43 h — 55 pushes/day against a 99-minute suite.

The trap that shaped the design

sanitize-cpu is continue-on-error: true, so a run's conclusion can be success while it is red. The single completed run in that window — 31448896841 at 5812b8b6 — is exactly that: run success, both sanitizer lanes failure. Anything reading run.conclusion would publish that SHA as the known-good baseline. So nothing here reads it; the verdict is per-job, always.

What lands

• .github/workflows/ci.yml — schedule: 17 */4 * * * + workflow_dispatch. The nine job groups and the workflow group gain ${{ github.event_name }}; without it a push cancels the scheduled baseline exactly as it cancels the previous push. For push and pull_request that token is a constant, so their grouping is unchanged. cancel-in-progress is false on the baseline lane, so a second baseline queues instead of killing the first. The two DIFF-scoped jobs are skipped there (a schedule payload has no github.event.before, so their range would be empty) and still carry no group. New baseline-summary publishes the verdict.
• scripts/main-baseline.py — answers the question. Derived at read time; writes nothing. A committed LAST_GREEN.md would be the worst instance of the surface AGENTS.md forbids — one line every merge wants to rewrite — so this mirrors scripts/now.py, including REMOTE_UNVERIFIED rather than rendering an absence as "nothing failed".
• tests/scripts/test_main_baseline.py — 44 offline tests. The workflow half evaluates the workflow rather than grepping it: see Review repair below.
• .agents/verification.md — how to read it, next to the other false-green traps.

baseline-summary fails when any covered job is red, sanitizers included. It can never run on a push or a pull_request, so it blocks no contributor and continue-on-error is untouched for those lanes. It only refuses to call a baseline green while a job it covered is red.

What a reader can answer after this

$ scripts/main-baseline.py --run-id 31448896841
2026-08-11T01:18:27Z  5812b8b667ea  RED    run 31448896841
    failed:    sanitize-cpu (thread), sanitize-cpu (address,undefined)
    not run:   pr-size

NEWEST BASELINE: RED at 5812b8b667eabcb7fe6f12767176a48c04ff7b26
LAST FULLY GREEN: none -- no fully green baseline in the last 1 run(s).

That is live output against the real API, on the run whose own conclusion says success.

Rejected: option (a), a non-cancellable run per merge

At 55 pushes/day that is ~55 overlapping 99-minute runs, ~190 h/day of job time, on a pool where run 31485402200 already sat queued 37 minutes and was cancelled without a single job starting — it would make PR feedback slower for everyone, to answer a question asked a few times a day. It also reverses a deliberate, documented decision (ci.yml:9-14) with no new evidence against it. And at that push rate the newest complete baseline would still be several commits stale, which is the property (a) was supposed to buy. workflow_dispatch keeps the useful half: pin a baseline on a SHA you care about, right after merging it.

Also rejected: a separate workflow calling ci.yml via workflow_call (github.ref is still the caller's, so it needs the identical discriminator, plus workflow_call semantics for no gain); duplicating the job definitions (two suites that drift); daily (~55-commit range) and hourly (24 × 99 min, overlapping runs) cadences.

Review repair (fresh review returned FAIL on 36fa56d0)

The concurrency change itself was proved SAFE by that review — all ten group keys extracted from both revisions, resolved per event, zero partition mismatches — and is untouched. Seven findings are repaired in d496f006.

F1 (HIGH, blocking) — the lane could never have reported GREEN

agent-record is in baseline-summary's needs:, carried no event guard, and embedded the same diff-scoped github.event.before logic that 36fa56d0 correctly guarded out of documentation-checkpoint and commit-protocol-tag. On a schedule payload PUSH_BASE renders empty, and under set -eu:

range=[..36fa56d0...]
check-commit-trailers.py: error: argument --range: range must be exactly BASE..HEAD
STEP_EXIT=2

Reproduced before touching anything. So the lane would have published RED forever, for a reason with nothing to do with the tree — the whole row would have certified nothing.

The two range-scoped calls are guarded in place rather than the job being skipped: the rest of agent-record is tree-scoped and the baseline needs it. check-role-discipline.py gets the same guard for a different reason — it would not have aborted, because --base "" makes commits_in_range fall back to [head] and print OK: every change on main arrived on a task branch (scripts/check-role-discipline.py:328-333). A vacuous PASS covering one commit while looking like it covered a range is worse than a loud skip. That checker's own behaviour is untouched (a checker-semantics change needs its own spec and red-before evidence) and is recorded as risk 10 in the spec — finding F7, record-only.

F2 (HIGH, blocking) — the suite guarding every claim here ran on no machine

grep -rn test_main_baseline . returned exactly one hit: the file itself. It was in neither scripts/agent-preflight.sh's SUITES array nor the agent-record job's explicit list, which is how the 35 sibling suites run. Registered in both, and asserted by SuiteRegistrationTests.

check-test-registration.py cannot see this class: its REQUIRED_TESTS is a fixed map naming one C++ target, and wiring_errors() only proves that checker guards itself. Changing its semantics needs a spec, new mutations in test_check_test_registration.py and a re-pinned MUTATION_MANIFEST_SHA256, so the class is filed as #408 — 12 of 54 suites, 11 still unwired after this — rather than folded in.

F3 (MEDIUM) — four mutations SURVIVED the original suite

All four were substring assertions where a resolved value was needed.

(b) is also a correction this PR owes. 36fa56d0's commit message credits test_push_and_pull_request_grouping_is_unchanged with establishing that push/PR grouping is unchanged. It does not. It compared the key against no baseline at all and blocked only github.run_id; adding ${{ github.sha }} — which gives every run its own group and disables push/PR dedupe entirely — walked straight through it. The claim was true; the test did not establish it. It does now, as an equality against the base revision's key with the constant -<event> inserted, for all four events, plus a by-name blocklist of run-varying tokens.

F4-F6 — scripts/main-baseline.py

• F4. verdict() had no expected-job set, so it was purely subtractive: called with eight of nine covered jobs simply absent it printed GREEN with jobs covered: 1. EXPECTED_JOBS is pinned and cross-checked against baseline-summary's needs: list, matrix lanes match by id prefix (sanitize-cpu (thread)), a skipped expected job counts as never-ran, and any absence is RED.
• F5. Unfinished jobs (conclusion is None) were bucketed under failed:. Fail-closed was right; the label was a false statement. They get a pending bucket.
• F6. "No completed baseline run found on main" exited 0 while REMOTE_UNVERIFIED exited 1, so main-baseline.py && echo ok read absence as success — the exact confusion the module docstring says it prevents. Absence is now non-zero.

Mutation table — 13 re-run, 13 caught, 0 survived

Each applied to a scratch copy of the four files, suite re-run, tree restored.

#	Mutation	Verdict	Caught by
a	invert cancel-in-progress polarity on all nine job groups	CAUGHT	test_cancellation_resolves_to_the_right_boolean_for_every_event
b	add ${{ github.sha }} to a job group key	CAUGHT	test_each_resolved_group_is_the_base_key_plus_the_event_constant, test_no_group_key_carries_a_run_varying_token
c	append || true to baseline-summary's --emit-summary step	CAUGHT	test_the_verdict_step_never_masks_its_own_exit_status
d	add continue-on-error: true to baseline-summary	CAUGHT	test_the_verdict_job_is_not_continue_on_error
F1a	drop the trailer-walk guard (the shipped defect)	CAUGHT	test_the_baseline_lane_invokes_no_range_scoped_checker_and_exits_zero
F1b	drop the role-discipline guard	CAUGHT	same
F1c	delete the range-scoped checkers instead of guarding them	CAUGHT	test_push_and_pull_request_still_get_the_full_range_scoped_checks
F2a	unregister the suite from preflight SUITES	CAUGHT	test_registered_in_the_preflight_suite_array
F2b	unregister the suite from the agent-record CI job	CAUGHT	test_registered_in_the_agent_record_ci_job
F4a	verdict() ignores the expected-job set	CAUGHT	test_an_expected_job_the_payload_never_mentions_is_red, +3
F4b	EXPECTED_JOBS silently narrowed to one job	CAUGHT	test_expected_jobs_is_pinned_against_the_workflow_needs_list, +3
F5	in-progress jobs bucketed under failing again	CAUGHT	test_an_unfinished_job_is_pending_not_failed
F6	absence of any baseline exits 0	CAUGHT	test_no_completed_baseline_run_exits_nonzero

AgentRecordDiffRangeTests executes both agent-record step bodies under a python3 argv shim on all four events and asserts both directions — no range-scoped call where there is no range, and the range-scoped calls still happen on push and pull_request — which is why F1c (deleting rather than guarding) is caught too.

Evidence, and what it does NOT establish

RED-first: a stub reading run.conclusion fails test_run_level_success_with_red_sanitizers_is_not_green with AssertionError: True is not false; before the workflow edits, all workflow-lane assertions failed. Green now: pytest tests/scripts/test_main_baseline.py -q 44 passed / 171 subtests, check-agent-record, test_agent_record, test_doc_checkpoint, test_agent_gates, check-test-registration, check-release-workflow, check-container-workflow, check-container-matrix, check-role-discipline, yaml.safe_load of ci.yml, and scripts/agent-preflight.sh --staged exit 0.

A CI-config change cannot be proven by this PR's own CI run. That run is a pull_request event, so it exercises neither the schedule trigger (GitHub fires cron only for the workflow committed on the default branch) nor main-branch grouping. Everything green above is static. Six claims are owed a post-merge observation, and the spec's ## Outcome stays unwritten until they are made:

1. a scheduled run actually fires on the chosen cron;
2. it survives concurrent pushes to main instead of being cancelled;
3. baseline-summary obtains the Actions API result for its own run with the default GITHUB_TOKEN and actions: read;
4. scripts/main-baseline.py finds and parses real baseline runs;
5. every job in baseline-summary's needs: can reach a green conclusion on a schedule payload. agent-record demonstrably could not (F1). No static gate here distinguishes "green on the push lane" from "green on the baseline lane", so the first scheduled run is where each of the nine is observed for the first time;
6. the verdict's missing bucket stays empty against a real payload — the job names the API reports still match EXPECTED_JOBS.

After merge, the fastest way to settle all four is gh workflow run ci.yml --ref main rather than waiting for the cron.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]