All macos-14 GC gates have been starved of runners for 2+ days: gc-ratchet and tls-budget have 0 successful main runs, 27 queued

[proggeramlug]

Summary

Every GC gate that runs on macos-14 has produced zero results on main for over a day. They are not failing and not cancelled — they never reach a runner. The Ubuntu-hosted gates in the same repo run normally, which localises this to macOS runner capacity rather than to any workflow's configuration.

Measured 2026-08-11:

workflow	runner	successful main runs in last 24h
gc-ratchet	macos-14	0
tls-budget	macos-14	0
gc-root-dominance	ubuntu-latest	17

gc-ratchet's last successful run on main was 2026-08-09. Of its 30 most recent main runs, 27 are queued and 3 completed. The oldest still-queued run was created at 05:24 and was still waiting ~10 hours later.

Queue depth is repo-wide across the macOS gates:

Security Audit                   9 queued
gc-native-roots                  9
GC Root Dominance                9
GC Ratchet                       9
TLS Budget                       8
GC Ptr<Shape> OFF-arm witness    8

Why this is worse than a slow gate

Two days of GC-heavy merges landed with the GC gates dark. #7799, #7809, #7812, #7834 and #7839 all touch the collector, the allocation path, or write barriers. gc-ratchet is the primary defence for exactly that surface, and it has not run on main since before any of them merged.

The consequence is already visible: #7843 reports 7 gating rows genuinely red on main (proven code-drift, not host, by a pin-binary/main-binary A/B on one host, bisected to at-or-before #7812). Nobody saw them, because the gate that would have said so never ran. Separately, two probes widely believed red (04_dead_after_deep_stack, 11_collect_at_depth) turn out to be green — a stale belief that also survived because there was no signal to correct it.

This is not the concurrency bug, and that matters

gc-ratchet.yml's concurrency block is already correct and carries a long comment documenting two prior attempts (#7205): a shared group with unconditional cancel-in-progress cancelled three consecutive main runs; scoping cancel-in-progress to pull requests did not fix it either, because GitHub allows at most one pending run per group. The current config keys the group on github.sha for push events, so no two main runs can contend.

That fix worked, and the failure mode simply moved. Runs no longer cancel each other — they queue instead, and never drain. The gate is dark either way. This is a third variant of CLAUDE.md's "four ways a gate can be unable to fail": not continue-on-error, not missing from required contexts, not cancelled — starved.

Suggested directions

1. Move what can move to ubuntu-latest. gc-root-dominance already does and is unaffected. Anything that does not genuinely need macOS semantics (Mach-O, _tlv_get_addr, the Darwin allocator) is a candidate.
2. Reduce macOS trigger frequency on main — for instance, run the macOS arm on a schedule plus tag pushes rather than every merge, so the queue can drain. A gate that runs four times a day and completes is worth more than one that runs on every merge and never does.
3. Check the account's macOS concurrency limit against the current merge cadence. Six workflows × every merge, at 40+ minutes queued each, may simply exceed it.
4. Whatever is chosen, alert on staleness: "no successful main run in N hours" should be visible, because the current failure is silent by construction — an empty result set looks identical to a healthy one that nobody has looked at.

Found by

Investigating why gc-ratchet's red rows (#7843) went unnoticed for two days during a GC-heavy merge window.

[proggeramlug]

Measured before fixing. The constraint is repo-wide total Actions concurrency, not macOS runner availability — PR #7860.

At 2026-08-11 15:07 UTC: 9 jobs running repo-wide (6 ubuntu-latest, 3 macos-14) against 100+ queued runs. Queued jobs across the 40 newest runs: 45 ubuntu-latest, 14 macos-14, 2 windows. zizmor (ubuntu) was queued in the same second as gc-ratchet (macOS). 600 runs were created that day by 15:07 (392 pull_request + 208 push), and merge cadence went 10 → 32 → 58/day across 08-09 → 08-11. Zero self-hosted runners.

Two claims in this issue did not survive checking, and both change the remedy:

1. gc-root-dominance runs macos-14, not ubuntu-latest (two jobs). Its healthy-looking run count was pull-request runs, which drain because they supersede each other; its main arm was queued like all the others. So the ubuntu-vs-macOS contrast that localised this to macOS was comparing a PR arm against a main arm, not Linux against Darwin.
2. Ubuntu was starved harder in absolute terms, so suggestion 1 (move what can move to ubuntu-latest) would not have helped. Independently, gc-ratchet cannot move anyway: its baseline is keyed darwin-arm64 and the checker refuses a platform mismatch, so the move turns the gate red rather than relieving it.

Only cutting total job demand works, which is what #7860 does: the post-merge arm of ten gates becomes a staggered six-hourly sweep plus release tags (suggestion 2), removing 19 jobs from every merge — ~1,100 job-starts/day → ~76. PR arms are untouched, so every PR is still measured, and no probe or threshold moved. The concurrency: blocks are left alone.

The staleness alarm (suggestion 4) is gate-freshness.yml. Its first live run found two things beyond this issue's scope: llvm-inprocess has been dark for 171.8h, and security-audit — a required context, still on every merge — is itself stale at 17.4h.