ci: fix the #7205 relapse on the scheduled gate arm, and reap dead PR runs (#7966) - #7969
Merged
Conversation
…runs #7856 moved ten expensive gates' main-line arm from `push: branches: [main]` to a staggered six-hourly `schedule:`. Their concurrency groups key on `github.event_name == 'push' && github.sha || github.ref` -- #7205's fix, guarded on the event being `push`. With the main-line arm now `schedule`, the guard stops matching and the group falls through to `github.ref`, constant `refs/heads/main`. GitHub keeps at most one PENDING run per concurrency group and cancels the previously pending one when a new run enters, regardless of `cancel-in-progress`. Measured 2026-08-12, identically on all ten gates: oldest run `queued` holding the group, the next two `cancelled` with `jobs: 0`, newest `pending`. `gate-freshness` itself -- the alarm for this -- was cancelled the same way. Groups are now keyed on `github.run_id` for every non-pull-request event. PR runs keep the shared per-ref group and keep coalescing. `gc_gate_wiring_check.py` gains `check_schedule_group`, swept over all 31 workflows rather than just the GC gates. It found ten further workflows with the same latent constant group -- including `test.yml`'s nightly safety net and `soak-autofix`, whose group was a bare constant string -- all fixed here. Five new self-test cases; the first is the sabotage case, since the existing CLEAN fixture carries the bad shape. Capacity half: 1,529 runs queued against 12-14 concurrent, of which 794 were `pull_request` runs over 63 head branches -- 61 of which no longer existed. Roughly 790 runs, 51% of the queue, were dead work for already-merged PRs sitting in front of the `main` gates. `scripts/reap_stale_ci_runs.py` + `ci-queue-reaper.yml` cancel QUEUED pull-request runs with no open PR; dry-run by default, `--max` capped, and structurally unable to touch a push/schedule/tag/dispatch run. `zizmor`'s `push: main` arm is path-filtered to `.github/**` and gains the concurrency block it never had. Refs #7966
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (29)
📝 WalkthroughWalkthroughThis change updates CI concurrency groups, adds scheduled concurrency validation, introduces stale queued pull-request run cleanup, and documents the gate-starvation incident and related workflow behavior. ChangesCI gate reliability
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant ci-queue-reaper
participant reap_stale_ci_runs.py
participant GitHubAPI
GitHubActions->>ci-queue-reaper: schedule or manual dispatch
ci-queue-reaper->>reap_stale_ci_runs.py: run self-test and cleanup
reap_stale_ci_runs.py->>GitHubAPI: retrieve open PR branches and queued runs
GitHubAPI-->>reap_stale_ci_runs.py: return branch and run data
reap_stale_ci_runs.py->>GitHubAPI: cancel eligible runs when apply is enabled
Possibly related PRs
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
proggeramlug
marked this pull request as ready for review
August 12, 2026 15:27
proggeramlug
pushed a commit
that referenced
this pull request
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the diagnosis in #7966. The issue's hypothesis was starvation. That is half of it; the other half is a bug this repo has now shipped three times.
1. #7205 relapsed on the arm #7856 created
Every scheduled gate carries:
The
github.shaarm is #7205's fix and it is guarded on the event beingpush— correct while the main-line arm waspush: branches: [main]. #7856 moved that arm toschedule:. The guard stopped matching, the expression fell through togithub.ref(constantrefs/heads/main), and every scheduled run of a gate shared one group again.GitHub keeps at most one PENDING run per group and cancels the previously pending one when a new run enters, regardless of
cancel-in-progress— the finding #7205 was measured by. Observed 2026-08-12, identically on all ten gates:jobs: 0is the exact zero-execution signature of #7205. The oldest run holds the group; every newer one is cancelled on arrival. The gate cannot run again until that one run drains.gate-freshnessitself — documented as built so it "cannot be starved by the condition it is alarming about" — was cancelled the same way at 13:38Z.Groups are now keyed on
github.run_idfor every non-pull-request event. PR runs keep the shared per-ref group and keep coalescing.2. The guard, so there is no fourth relapse
scripts/gc_gate_wiring_check.pygainscheck_schedule_group, swept over all 31 workflows rather than just the GC gates (the hazard reachedgate-freshnessandtest.ymltoo). It requiresgithub.run_idin the concurrency group of any workflow with aschedule:trigger.It immediately found ten more workflows already carrying the same latent constant group — including
test.yml's nightly full-workspace safety net andsoak-autofix, whose group was the bare constant stringsoak-autofix. All fixed here.Five new self-test cases (16 total). The first is the sabotage case: the existing
CLEANfixture has the bad shape, so a checker that could not fail on it would be worthless.lintis required, so a relapse is now a red build.3. The capacity half — 51% of the queue is dead work
pull_request, 181push, 19scheduleGitHub does not reliably cancel a queued run when its PR merges and the branch auto-deletes. ~790 runs — 51% of the entire queue — were work for already-merged PRs, pinned in front of ten
maingates. A six-hourly sweep cannot help when the queue in front of it is half garbage.scripts/reap_stale_ci_runs.py+ci-queue-reaper.ymlcancel QUEUEDpull_requestruns with no open PR. Deliberately timid: dry-run by default,--maxcap, and it only ever touchesevent == pull_request+status == queued+ no open PR — a push/schedule/tag/dispatch run is structurally unreachable, and an in-flight run is left alone. Keyed on open PRs, not branch existence, which is what keeps fork PRs safe. Self-tested with two sabotage cases.Live dry run:
781 reapable of 1,000 sampled.Also:
zizmor'spush: mainarm was unfiltered while its PR arm was already scoped to.github/**— 90 queued runs whose subject could not have changed. Path-filtered, plus the concurrency block it never had.Not starved — actually broken
Two of the eleven are not a scheduling problem:
gc-native-rootshas never had a single successful run, any branch, any event. Three of four arms fail with three distinct causes (aarch64-linux SIGSEGV underPERRY_STACKMAP_WALKER=verify; Windows Rust panic; macos-14gc_evacuation_liveness_assert.pyreporting 0 copying minors / 0 objects copied).llvm-inprocessfailed its last threemainruns, and worse, its PR "successes" shownative-backend: skipped— the path filter skips the only real job and the workflow reports green. Hazard 4.Filed separately; neither is fixed here.
What this does NOT close
python3 scripts/reap_stale_ci_runs.py --apply(dry run is the default). I did not run it — it cancels ~780 runs on shared infrastructure and is a maintainer call.parityandcompile-smokeare required contexts whose jobs carryif: github.event_name == 'push' || …whiletest.yml'spush:is tags only — so on an ordinary PR they never report, and a required context that never reports blocks the merge button forever. Every merge therefore needs an admin bypass, which bypasses the required contexts that do work. That is upstream of this whole incident: bypass is the normal path, so a pending gate stopped looking unusual. Server-side state, admin only.gc-ratchetwould not have caught it even had it run: its gating metrics have no full-mark-sweep count, the counter that found it (collection_kind: "full"0 → 2) is not one of them and no gate in the repo ratchets one, the two dimensions that moved (wall time, RSS) are explicitly"gating": falsein theshared_ciprofile CI uses, and its probe corpus is not the gc-handoff workloads that showed it. The human counter census was not a lucky substitute for a starved gate — it was the only instrument covering that dimension.No new required contexts
Nothing here is promoted to required.
ci-queue-reaperis a janitor, not a gate, and cannot fail a merge. The only change to a required context's file issecurity-audit.yml's concurrency group andtest.yml's — neither alters what those workflows assert or when their PR arm runs.Validation
Full measurement log:
gc-handoff/GATES-NOTES.md. Scheduling doc updated — including a correction to its own "Those blocks are correct. Do not 'fix' them again", which was true when written and false three days later, and is exactly the sentence that would send the next reader past this bug.Summary by CodeRabbit
New Features
Bug Fixes
Documentation