From 7cf15bb160ea53d71a1adac1fd43c0dfc2fab9b0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sun, 16 Aug 2026 07:55:21 +0200 Subject: [PATCH 1/8] ci: restructure CI into PR gate / main sweep / full suite tiers One workflow, three tiers, one policy file. `scripts/ci_plan.py` decides what a run of test.yml executes; every job is `needs: plan` + a plan flag; the fan-in job (`pr-gate` / `main-gate` / `full-suite-gate`) is the single status the outside world keys on. * PR tier: lint, check (clippy x2 + api-docs-drift), warnings, scoped cargo-test, 4-shard fast-mode gap suite, gc-stress (PR subset), e2e-scoped, security-audit when deps changed. Docs-only PRs run lint only. `pr-gate` is the only required context. * Sweep tier: every push to main, coalesced (constant concurrency group, cancel-in-progress off); PR tier unscoped + windows x2, gc-stress full, compiler-output-regression, repsel-census, harmonyos-smoke, binary-size, security-audit. Sweep-only jobs chain behind `check`. * Full tier: nightly, tags, dispatch, `run-extended-tests` label; adds parity, compile-smoke (no more continue-on-error), 8-shard auto-optimize gap suite, doc-tests, package smokes, native-abi-evidence-packet. release-packages.yml dispatches `tier=full` and requires a successful `full-suite-gate` job on the SHA. * Satellite gates (gc-*, tls-budget, auto-opt, eh-transport, llvm-inprocess, ext-link, container-tests): PR arm is opt-in via the `run-extended-tests` label; six-hourly / nightly main-line arms and tag arms unchanged. ext-link gains a nightly arm; container-tests drops its push:main arm. * sccache saves only from main-line runs (PR saves were ~200 GB/day into a 10 GB budget and unreadable by other PRs anyway); cache-warm.yml removed (the sweep is the cache-producing build on main now). * security-audit.yml becomes workflow_call + weekly schedule. Measured before: 14 workflows / 48 jobs / ~650 runner-min per PR push on a 20-slot org; 0/66 PR runs of Tests concluded; last 12 merges all bypassed protection. Docs: docs/src/testing/ci-tiers.md. --- .claude/skills/release/SKILL.md | 6 +- .github/workflows/auto-opt-app-patterns.yml | 8 + .github/workflows/cache-warm.yml | 133 --- .github/workflows/container-tests.yml | 30 +- .github/workflows/eh-transport.yml | 8 + .github/workflows/ext-link.yml | 15 + .github/workflows/gc-moving-witnesses.yml | 8 + .github/workflows/gc-native-roots.yml | 10 +- .github/workflows/gc-parse-churn-gate.yml | 8 + .../workflows/gc-ptr-shape-off-witness.yml | 8 + .github/workflows/gc-ratchet.yml | 8 + .github/workflows/gc-root-dominance.yml | 10 + .github/workflows/llvm-inprocess.yml | 10 +- .github/workflows/release-packages.yml | 42 +- .github/workflows/security-audit.yml | 11 +- .github/workflows/test.yml | 830 ++++++++++++------ .github/workflows/tls-budget.yml | 8 + CLAUDE.md | 6 +- CONTRIBUTING.md | 9 +- docs/src/SUMMARY.md | 1 + docs/src/contributing/releasing.md | 31 +- docs/src/testing/ci-gate-scheduling.md | 8 + docs/src/testing/ci-tiers.md | 158 ++++ scripts/ci_plan.py | 446 ++++++++++ 24 files changed, 1357 insertions(+), 455 deletions(-) delete mode 100644 .github/workflows/cache-warm.yml create mode 100644 docs/src/testing/ci-tiers.md create mode 100755 scripts/ci_plan.py diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index c63d8525a6..57bd2c2e9e 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -13,7 +13,7 @@ allowed-tools: Bash, Read, Edit, Write, Glob, Grep Perry releases are **tag-last**: nothing public happens until the test gate and every build leg are green. You dispatch `release-packages.yml` with `cut_release=true` on a branch pinned at the release candidate; the workflow then: 1. `preflight` — resolves `vX.Y.Z` from `Cargo.toml` on that SHA, fails fast if the tag already exists, if CLAUDE.md's `**Current Version:**` disagrees, or if `changelog.d/` has no fragments. -2. `await-tests` — dispatches `test.yml` + `simctl-tests.yml` on the branch if no run exists on the SHA yet, then polls by head SHA until both are green. +2. `await-tests` — dispatches `test.yml` (with `tier=full`) + `simctl-tests.yml` on the branch if no suitable run exists on the SHA yet, then polls by head SHA until both are green. For `test.yml` only a run whose `full-suite-gate` job succeeded counts — a green push-to-main sweep or PR-tier run on the same SHA is not release-grade (see docs/src/testing/ci-tiers.md). 3. `build` + `build-cross` — all release binaries, archives as workflow artifacts. 4. `create-release` — **only now** creates the tag + GitHub Release (notes concatenated from `changelog.d/` fragments via `cut_release_notes.sh --notes-only`), and dispatches the tag-rider workflows (docs, benchmark, container-tests) on the new tag. 5. `publish-assets` → homebrew / apt / apt-repo / winget / npm / update-workers. @@ -44,9 +44,9 @@ git push origin "release/v$VERSION" gh workflow run release-packages.yml --ref "release/v$VERSION" -f cut_release=true ``` -The pinned branch matters for two reasons: `workflow_dispatch` always runs on a ref's **tip** (so `main` moving would shift the SHA under you), and `test.yml`'s `test-` concurrency group means dispatching tests on `main` cancels a running nightly. +The pinned branch matters because `workflow_dispatch` always runs on a ref's **tip** (so `main` moving would shift the SHA under you). (`test.yml`'s dispatch runs are keyed per run-id, so a dispatch on `main` no longer cancels a running nightly — but pin anyway.) -Optional pre-warm: dispatch `test.yml` + `simctl-tests.yml` on the branch yourself right away — the gate matches any run on the SHA, so pre-flighted runs subtract their ~30 min from the critical path. If you skip this, `await-tests` dispatches them for you. +Optional pre-warm: dispatch `test.yml` (`-f tier=full`) + `simctl-tests.yml` on the branch yourself right away — the gate matches any full-tier run on the SHA, so pre-flighted runs subtract their time from the critical path. If you skip this, `await-tests` dispatches them for you. ### 4. Watch diff --git a/.github/workflows/auto-opt-app-patterns.yml b/.github/workflows/auto-opt-app-patterns.yml index 27f6a276d8..7d23283db1 100644 --- a/.github/workflows/auto-opt-app-patterns.yml +++ b/.github/workflows/auto-opt-app-patterns.yml @@ -47,6 +47,12 @@ name: Auto-Optimize App Patterns on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. # `push: branches: [main]` starved this gate and nine others for two days # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job @@ -83,6 +89,8 @@ env: jobs: auto-opt-app-patterns: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') runs-on: ubuntu-latest # The auto-optimize rebuild is a second full release build of # perry-runtime + perry-stdlib on top of the workspace build, so this is a diff --git a/.github/workflows/cache-warm.yml b/.github/workflows/cache-warm.yml deleted file mode 100644 index 7f4162e1ef..0000000000 --- a/.github/workflows/cache-warm.yml +++ /dev/null @@ -1,133 +0,0 @@ -name: Warm CI cache - -# WHY THIS EXISTS -# --------------- -# The `Tests` workflow (test.yml) runs ONLY on pull_request + version tags — -# never on push to main. GitHub Actions cache scoping means a run can only -# restore caches saved on its OWN branch or on the DEFAULT branch (main); one -# PR can never read another PR's cache. So with nothing running the cache- -# producing build on main, no main-scoped cache ever existed, and every PR -# started cold: -# - Swatinem/rust-cache in test.yml has `save-if: refs/heads/main`, but -# test.yml never runs on main → it never saved target/. -# - sccache (now a persisted disk cache, v0.5.1179) likewise only saw -# PR-scoped saves, invisible to other PRs. -# Result: cargo-test recompiled the whole dependency graph cold every run -# (~90-103 min), tipping over its timeout on perry-runtime/perry-codegen PRs. -# -# This workflow runs the cache-producing build ONCE per main merge so the -# main-scoped rust-cache (target/) and sccache disk cache exist for every PR -# to restore. It uses the SAME shared-key / sccache key prefix as test.yml's -# jobs, so those jobs warm-restore from here. It is best-effort -# (continue-on-error), never a gate. -on: - push: - branches: [main] - paths: - - 'crates/**' - - 'Cargo.toml' - - 'Cargo.lock' - - '.github/workflows/cache-warm.yml' - workflow_dispatch: - -concurrency: - # Only the latest main commit needs a warm cache; cancel older in-flight warms. - group: cache-warm-${{ github.ref }} - cancel-in-progress: true - -jobs: - warm: - runs-on: ubuntu-latest - # Best-effort cache population — must never block or fail anything. - continue-on-error: true - # Comfortably above the cold first run; later runs are far shorter. - # 150: the release-compiler + gap-suite variant warm below adds up to - # ~60 min on a cold variant set (it is the same bill the smoke job was - # paying per-PR; here it is paid once per merge, off the critical path). - timeout-minutes: 150 - env: - RUSTC_WRAPPER: sccache - # Mirror test.yml's sccache config EXACTLY (local disk cache, NOT the GHA - # backend) so the cache this job saves is byte-compatible with what the - # cargo-test / api-docs-drift / compiler-output-regression jobs restore. - SCCACHE_GHA_ENABLED: "false" - SCCACHE_DIR: ${{ github.workspace }}/.sccache - SCCACHE_CACHE_SIZE: "12G" - CARGO_INCREMENTAL: "0" - # Keep artifacts small so building the heavy crates' test binaries does - # not exhaust the shared-runner disk (same reason test.yml sets these). - CARGO_PROFILE_TEST_DEBUG: "0" - CARGO_PROFILE_DEV_DEBUG: "0" - CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS: "-C linker-features=-lld" - steps: - - uses: actions/checkout@v7 - - - name: Install Rust toolchain - uses: dtolnay/rust-toolchain@stable - - uses: ./.github/actions/setup-llvm22 - - - name: Install sccache - uses: mozilla-actions/sccache-action@v0.0.11 - - # Save the sccache disk cache under the SAME prefix test.yml restores - # (`sccache--perry-`). This runs on main, so the saved entry is - # main-scoped → restorable by every PR. run_id keeps the key unique. - - name: Cache sccache objects - uses: actions/cache@v6 - with: - path: ${{ github.workspace }}/.sccache - key: sccache-${{ runner.os }}-perry-warm-${{ github.run_id }} - restore-keys: | - sccache-${{ runner.os }}-perry- - - # Same shared-key as test.yml; save-if true because this only runs on main. - - uses: Swatinem/rust-cache@v2 - with: - shared-key: "${{ runner.os }}-perry" - save-if: "true" - - # Compile (do NOT run) the test binaries of the heaviest crates. This - # pulls in essentially the entire third-party dependency graph plus the - # big first-party crates — the bulk of cargo-test's compile time — so the - # cheap ext-* crates that follow in cargo-test reuse the cached units. - # `--no-run` skips execution (this is a cache warm, not a test run); the - # per-crate prune keeps target/ under the runner's disk budget. - - name: Warm build (heavy crates) - run: | - for pkg in perry-runtime perry-stdlib perry-codegen perry; do - echo "::group::cargo test -p $pkg --no-run" - cargo test -p "$pkg" --no-run - echo "::endgroup::" - find target/debug/deps -maxdepth 1 -type f -perm -111 ! -name '*.so' -delete || true - done - - # Warm the RELEASE compiler + the auto-optimize archive variants for the - # conformance-smoke job. Diagnosis (2026-07-04, cancelled smoke job - # 85123031828): smoke restored this cache with a FULL match yet still - # timed out at 60 min — the debug units above share nothing with its - # release build (~10 min), and every gap test whose feature combo - # (`auto_optimized_cache_key`: regex/temporal/url/… × stdlib set) had no - # `target/perry-auto-` archive triggered an in-job cargo build of - # the runtime — orphan `cargo`/`rustc` processes were still compiling at - # the timeout. Running the gap suite ONCE here builds every variant the - # suite needs; they land in `target/perry-auto-*`, which rust-cache - # saves under the shared key. Variant keys embed CARGO_PKG_VERSION, and - # code-only PRs carry main's version, so PR smokes restore exact - # matches until the next merge bumps the version — at which point THIS - # workflow re-warms on that very push. `|| true`: this is a cache warm, - # not a gate — the smoke job is where gap failures are reported. - - name: Warm release compiler + gap-suite auto-opt variants - run: | - echo "::group::cargo build --release -p perry" - cargo build --release -p perry - echo "::endgroup::" - echo "::group::run_gap_tests (variant warm)" - ./scripts/run_gap_tests.sh || true - echo "::endgroup::" - echo "perry-auto variants warmed:" - du -sh target/perry-auto-* 2>/dev/null || echo "(none)" - df -h /home/runner 2>/dev/null | tail -1 || true - - - name: sccache stats - if: always() - run: sccache --show-stats || true diff --git a/.github/workflows/container-tests.yml b/.github/workflows/container-tests.yml index d8bf1dbd75..549e9ac2ea 100644 --- a/.github/workflows/container-tests.yml +++ b/.github/workflows/container-tests.yml @@ -18,21 +18,16 @@ name: Container Tests on: push: - branches: [main] tags: ['v*'] - paths: - - 'crates/perry-container-compose/**' - - 'crates/perry-container-e2e/**' - - 'crates/perry-stdlib/src/container/**' - - 'crates/perry-stdlib/tests/container_*' - - 'crates/perry-hir/src/lower.rs' - - 'crates/perry-hir/src/ir.rs' - - 'crates/perry-codegen/src/lower_call.rs' - - 'tests/e2e/**' - - 'types/perry/{container,compose,workloads}/**' - - '.github/workflows/container-tests.yml' pull_request: branches: [main] + # PR arm is OPT-IN via the `run-extended-tests` label (job-level `if:` + # below); the paths filter still decides whether a run is created at + # all. Post-merge coverage is the nightly `schedule` -- the old + # `push: main` arm was 5 jobs (2 macOS) per merge that touched + # lower.rs / lower_call.rs, and the workflow has been red on every + # scheduled run since 2026-08-07 regardless. + types: [opened, synchronize, reopened, labeled] paths: - 'crates/perry-container-compose/**' - 'crates/perry-container-e2e/**' @@ -82,6 +77,8 @@ jobs: # Layer A + B: hermetic tests (no runtime). Every PR. # =========================================================================== hermetic: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') name: Hermetic Tests (Layer A + B) strategy: fail-fast: false @@ -163,7 +160,7 @@ jobs: integration-macos-apple: name: Layer C — Integration (macOS / apple/container) runs-on: macos-14 - if: github.event_name != 'pull_request' || github.base_ref == 'main' + if: (github.event_name != 'pull_request' || github.base_ref == 'main') && (github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) steps: - uses: actions/checkout@v7 @@ -210,7 +207,7 @@ jobs: integration-linux-podman: name: Layer C — Integration (Linux / podman) runs-on: ubuntu-24.04 - if: github.event_name != 'pull_request' || github.base_ref == 'main' + if: (github.event_name != 'pull_request' || github.base_ref == 'main') && (github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) steps: - uses: actions/checkout@v7 @@ -250,8 +247,9 @@ jobs: e2e-linux: name: Layer D — E2E (Linux / docker) runs-on: ubuntu-24.04 + # e2e-linux never ran on pull requests (main/tags/manual only) and the + # `push: main` arm is gone, so this is tags + manual now. if: | - github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v') || github.event.inputs.run_e2e == 'true' steps: @@ -347,7 +345,7 @@ jobs: name: Container Tests Gate runs-on: ubuntu-24.04 needs: [hermetic] - if: always() + if: (always()) && (github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) steps: - name: Check required jobs run: | diff --git a/.github/workflows/eh-transport.yml b/.github/workflows/eh-transport.yml index db8a027efd..557a1242a6 100644 --- a/.github/workflows/eh-transport.yml +++ b/.github/workflows/eh-transport.yml @@ -28,6 +28,12 @@ name: eh-transport # `changes` job: a job skipped by `if:` still reports a check run. on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. # `push: branches: [main]` starved this gate and nine others for two days # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job @@ -64,6 +70,8 @@ concurrency: jobs: changes: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') runs-on: ubuntu-latest timeout-minutes: 5 permissions: diff --git a/.github/workflows/ext-link.yml b/.github/workflows/ext-link.yml index 00c5a185d6..fb010a828c 100644 --- a/.github/workflows/ext-link.yml +++ b/.github/workflows/ext-link.yml @@ -30,6 +30,19 @@ name: ext-link on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] + # Main-line arm: nightly + release tags. It used to have none -- the PR arm + # was the only execution and it was red on every PR (#8155) at 90-130 min a + # run. Staggered off the six-hourly GC gates' minutes. + schedule: + - cron: "52 4 * * *" + push: + tags: ["v*"] workflow_dispatch: concurrency: @@ -42,6 +55,8 @@ permissions: jobs: ext-link: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') runs-on: ubuntu-latest # Measured on an arm64 dev Mac. COLD release target, the five crates from # #7650 only: 7:24, 7 test binaries. All 38 ext crates with perry-runtime diff --git a/.github/workflows/gc-moving-witnesses.yml b/.github/workflows/gc-moving-witnesses.yml index ae9db34abf..f3b4aec393 100644 --- a/.github/workflows/gc-moving-witnesses.yml +++ b/.github/workflows/gc-moving-witnesses.yml @@ -108,6 +108,12 @@ name: GC Moving Witnesses on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. # `push: branches: [main]` starved this gate and nine others for two days # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job @@ -167,6 +173,8 @@ env: jobs: gc-moving-witnesses: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') # ubuntu-latest, matching `gc-stress` in test.yml: the matrix already runs # there, the queue is shallower than the macOS pool, and the cargo cache is # shared with that job (same `shared-key`) so the build is usually warm. diff --git a/.github/workflows/gc-native-roots.yml b/.github/workflows/gc-native-roots.yml index 10bcd389b6..55e2765f96 100644 --- a/.github/workflows/gc-native-roots.yml +++ b/.github/workflows/gc-native-roots.yml @@ -174,6 +174,12 @@ on: push: tags: ["v*"] pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] workflow_dispatch: concurrency: @@ -239,6 +245,8 @@ jobs: # not a parse error but a wrong answer: two bytes of drift per field # silently relocates every root that follows, and every arm stays green. native-roots-rs4gc: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') strategy: fail-fast: false matrix: @@ -728,7 +736,7 @@ jobs: # Deliberately cheap: one probe, no runtime, no oracle. gc-native-roots-complete: needs: [native-roots-rs4gc] - if: always() + if: (always()) && (github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) runs-on: ubuntu-latest timeout-minutes: 5 steps: diff --git a/.github/workflows/gc-parse-churn-gate.yml b/.github/workflows/gc-parse-churn-gate.yml index 3b88af1414..252cbf4102 100644 --- a/.github/workflows/gc-parse-churn-gate.yml +++ b/.github/workflows/gc-parse-churn-gate.yml @@ -76,6 +76,12 @@ name: GC Parse-Churn Layout Gate on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. # `push: branches: [main]` starved this gate and nine others for two days # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job @@ -131,6 +137,8 @@ jobs: run: python3 scripts/gc_parse_churn_layout_check.py --self-test gc-parse-churn-gate: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') runs-on: ubuntu-latest # Matches gc-moving-witnesses/gc-stress: the witness run itself is # minutes, but a cold cargo cache makes the compiler build the whole cost diff --git a/.github/workflows/gc-ptr-shape-off-witness.yml b/.github/workflows/gc-ptr-shape-off-witness.yml index 894ef0e766..48ca612ec8 100644 --- a/.github/workflows/gc-ptr-shape-off-witness.yml +++ b/.github/workflows/gc-ptr-shape-off-witness.yml @@ -74,6 +74,12 @@ name: GC Ptr OFF-arm witness on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. # `push: branches: [main]` starved this gate and nine others for two days # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job @@ -117,6 +123,8 @@ env: jobs: gc-ptr-shape-off-witness: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') # ubuntu-latest, matching gc-stress: the matrix already runs there, and # the cargo cache is shared with that job (same `shared-key`) so this # build is usually warm. diff --git a/.github/workflows/gc-ratchet.yml b/.github/workflows/gc-ratchet.yml index fac8cfc322..c556535e82 100644 --- a/.github/workflows/gc-ratchet.yml +++ b/.github/workflows/gc-ratchet.yml @@ -15,6 +15,12 @@ name: GC Ratchet on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] # POST-MERGE ARM: a staggered six-hourly sweep of `main`, NOT one run per merge. # # `push: branches: [main]` is what took this gate dark for two days (#7856). @@ -98,6 +104,8 @@ env: jobs: gc-ratchet: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') # macos-14 is arm64, which matches the `darwin-arm64` platform key the # baseline was captured under. The checker refuses a platform mismatch # rather than comparing numbers that are not comparable, so moving this to diff --git a/.github/workflows/gc-root-dominance.yml b/.github/workflows/gc-root-dominance.yml index 78fe145dab..6c87339ed5 100644 --- a/.github/workflows/gc-root-dominance.yml +++ b/.github/workflows/gc-root-dominance.yml @@ -61,6 +61,12 @@ name: GC Root Dominance on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. # `push: branches: [main]` starved this gate and nine others for two days # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job @@ -109,6 +115,8 @@ env: jobs: gc-root-dominance: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') runs-on: macos-14 timeout-minutes: 90 steps: @@ -435,6 +443,8 @@ jobs: # first, then promote. Promotion is an admin action and is tracked separately # from the PR that adds this. gc-root-dominance-statepoints: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') runs-on: macos-14 timeout-minutes: 90 steps: diff --git a/.github/workflows/llvm-inprocess.yml b/.github/workflows/llvm-inprocess.yml index bbbd4cd6fa..2c11677fef 100644 --- a/.github/workflows/llvm-inprocess.yml +++ b/.github/workflows/llvm-inprocess.yml @@ -16,6 +16,12 @@ name: llvm-inprocess # run (conclusion: skipped), which branch protection accepts. on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. # `push: branches: [main]` starved this gate and nine others for two days # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job @@ -55,6 +61,8 @@ jobs: # action). Pushes to main always run — main executions are the gate's # anchor and the promotion prerequisite. changes: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') runs-on: ubuntu-latest timeout-minutes: 5 permissions: @@ -306,7 +314,7 @@ jobs: # gate for eight days without anyone noticing. llvm-inprocess-complete: needs: [changes, native-backend] - if: always() + if: (always()) && (github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) runs-on: ubuntu-latest timeout-minutes: 5 steps: diff --git a/.github/workflows/release-packages.yml b/.github/workflows/release-packages.yml index 7b5a06d96f..8c3d18484a 100644 --- a/.github/workflows/release-packages.yml +++ b/.github/workflows/release-packages.yml @@ -161,9 +161,19 @@ jobs: # nightly. The /release skill dispatches cut_release runs on a # pinned release/vX.Y.Z branch for exactly that reason. for wf_file in "test.yml" "simctl-tests.yml"; do - count=$(gh api \ - "/repos/$REPO/actions/workflows/$wf_file/runs?head_sha=$SHA&per_page=1" \ - --jq '.total_count') + # test.yml: only a FULL-tier run counts (see the poll below), so + # look for one of those rather than any run on the SHA -- a + # push-to-main sweep on the same commit must not suppress the + # dispatch. + if [ "$wf_file" = "test.yml" ]; then + count=$(gh api \ + "/repos/$REPO/actions/workflows/$wf_file/runs?head_sha=$SHA&per_page=20" \ + --jq '[.workflow_runs[] | select(.event == "workflow_dispatch" or .event == "schedule" or .event == "push" and (.head_branch | startswith("v")))] | length') + else + count=$(gh api \ + "/repos/$REPO/actions/workflows/$wf_file/runs?head_sha=$SHA&per_page=1" \ + --jq '.total_count') + fi if [ "$count" = "0" ]; then # workflow_dispatch runs on the ref TIP — if the ref moved past # our SHA, a dispatched run would test the wrong commit and the @@ -174,7 +184,13 @@ jobs: exit 1 fi echo "No $wf_file run on $SHA yet — dispatching on $REF_NAME." - gh workflow run "$wf_file" --ref "$REF_NAME" -R "$REPO" + if [ "$wf_file" = "test.yml" ]; then + # The tiered CI workflow: `tier=full` is its dispatch default, + # but say so explicitly -- this is the release gate. + gh workflow run "$wf_file" --ref "$REF_NAME" -R "$REPO" -f tier=full + else + gh workflow run "$wf_file" --ref "$REF_NAME" -R "$REPO" + fi fi done fi @@ -231,6 +247,24 @@ jobs: retry_on_error=1 # Look for ANY successful run on this SHA (order-independent). success_url=$(echo "$api_out" | jq -r '.workflow_runs[] | select(.status == "completed" and .conclusion == "success") | .html_url' | head -1) + # test.yml is tiered (scripts/ci_plan.py): a push-to-main SWEEP + # or a PR-tier run on this SHA is green too, but only a FULL-tier + # run carries the release-grade suites (parity, compile-smoke, + # doc-tests, ...). Its fan-in job is NAMED `full-suite-gate` for + # exactly this check: keep only successful runs that have it. + if [ "$wf_file" = "test.yml" ] && [ -n "$success_url" ]; then + success_url="" + for run_id in $(echo "$api_out" | jq -r '.workflow_runs[] | select(.status == "completed" and .conclusion == "success") | .id'); do + if gh api "/repos/$REPO/actions/runs/$run_id/jobs?per_page=100" \ + --jq '.jobs[] | select(.name == "full-suite-gate" and .conclusion == "success") | .name' 2>/dev/null | grep -q full-suite-gate; then + success_url="https://github.com/$REPO/actions/runs/$run_id" + break + fi + done + if [ -z "$success_url" ]; then + echo " test.yml has green run(s) on $SHA but none is a full-tier run (no successful full-suite-gate job) — waiting for the dispatched one" + fi + fi # Look for any failed/cancelled/timed-out completed run as a # FAIL signal — but only if there's NO successful run for # this SHA. This way a flaky run that someone reran to green diff --git a/.github/workflows/security-audit.yml b/.github/workflows/security-audit.yml index b492494247..05a4206d8b 100644 --- a/.github/workflows/security-audit.yml +++ b/.github/workflows/security-audit.yml @@ -1,9 +1,14 @@ name: Security Audit on: - pull_request: - push: - branches: [main] + # Called from .github/workflows/test.yml: in the PR tier only when the diff + # touches a lockfile / manifest / policy file (scripts/ci_plan.py DEPS_GLOBS), + # and unconditionally in the post-merge sweep and the full tier. That is why + # there is no `pull_request` / `push` trigger here any more -- running these + # five jobs on every PR push was ~5 of the 48 runner slots each push burned + # on a 20-slot org (see the header of test.yml). + workflow_call: + # Weekly on its own: the advisory database moves without any commit. schedule: - cron: "0 12 * * 1" workflow_dispatch: diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 42e2a02d20..fd2cae8d49 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -1,57 +1,80 @@ -name: Tests +name: CI + +# --------------------------------------------------------------------------- +# ONE workflow, THREE tiers. Which jobs run is decided by `scripts/ci_plan.py` +# in the `plan` job below; every other job is `needs: plan` and gated on +# `fromJSON(needs.plan.outputs.plan).jobs.`. Read that script's docstring +# for the policy and `python3 scripts/ci_plan.py --table` for the job x tier +# matrix. docs/src/testing/ci-tiers.md is the long-form page. +# +# pr pull_request the required gate. Small, fast, must be green +# on main. Fan-in job `pr-gate` is the ONLY +# required status context. +# sweep push: main coalesced post-merge sweep (at most one running +# + one pending, so a burst of merges is tested at +# its tip). PR tier unscoped + medium-weight jobs. +# full schedule / tags / everything, incl. parity, compile-smoke, +# workflow_dispatch doc-tests, package smokes, the 8-shard +# / label auto-optimize gap suite. `full-suite-gate` is +# what release-packages.yml waits for. +# +# WHY (2026-08-16): the org runs on 20 concurrent hosted runners (5 macOS). +# The previous shape fanned every PR push out to 14 workflows / 48 jobs / +# ~650 runner-minutes; at ~66 pushes/day that is 1.5-2x total capacity, so the +# queue never drained (job queue waits of 3-7 h), 0 of 66 PR runs of this +# workflow reached a conclusion in the sample window, and every merge was an +# admin bypass. Two required contexts (`parity`, `compile-smoke`) could not +# even report on a PR. See changelog.d/ for the measured numbers. +# --------------------------------------------------------------------------- on: - # Run on version tags so `release-packages.yml` can gate publish on a green - # Tests workflow for the exact commit being released. Direct pushes to main - # do NOT trigger tests — the gates that matter are PRs (pre-merge) and tags - # (pre-release). - push: - tags: ['v*'] pull_request: branches: [main] - # `labeled` is here so the optional opt-in jobs (parity, compile-smoke, - # doc-tests) re-fire when a maintainer or the PR author applies the - # `run-extended-tests` label. Without it, applying the label on an - # existing PR wouldn't re-trigger the workflow. + # `labeled` re-fires the run when a maintainer applies `run-extended-tests` + # (which promotes the PR to the `full` tier -- see ci_plan.py) or + # `skip-changelog`. NO `paths-ignore` here: a docs-only PR must still + # produce a `pr-gate` status (the plan turns everything but `lint` off for + # it), otherwise the required context never reports and the PR is stuck + # behind an admin bypass. types: [opened, synchronize, reopened, labeled] - paths-ignore: - - 'docs/src/**' - - '*.md' - - '!CLAUDE.md' - - '!CHANGELOG.md' + push: + branches: [main] + tags: ['v*'] schedule: - # Nightly full-workspace cargo-test safety net (04:00 UTC). The per-PR - # cargo-test gate only exercises crates affected by the diff - # (scripts/ci_test_scope.py); a cross-crate regression that slips a scoped - # PR is caught here within a day. `schedule` is not `pull_request`, so the - # cargo-test job runs the full workspace. + # Nightly full tier at 04:00 UTC. The sweep already runs on every merge; + # this is the daily arm for the slow suites (parity, compile-smoke, + # doc-tests, package smokes, the auto-optimize gap shards). - cron: '0 4 * * *' - # Manual escape hatch for the opt-in jobs. Maintainers (write access) - # can dispatch the workflow against any ref with `run_extended_tests=true` - # to run parity / compile-smoke / package smokes / doc-tests on demand - # without tagging a release. workflow_dispatch: inputs: - run_extended_tests: - description: 'Run extended tests (parity, compile-smoke, package smokes, doc-tests)' + tier: + description: 'Which tier to run (release-packages.yml dispatches `full`)' + type: choice + options: [pr, sweep, full] + default: full + update_gap_snapshot: + description: >- + Re-baseline test-parity/gap_snapshot.json: run the whole gap suite as + ONE shard with UPDATE_SNAPSHOT=1 and upload the resulting snapshot as + an artifact (`gap-snapshot-update`) for you to commit. Nothing else + about the run changes. type: boolean default: false -# Superseded PR pushes cancel their in-flight run (the real minute saver); the -# nightly cron and release-tag runs never do (#5960). Keying the group on the -# event as well as the ref means a manual `workflow_dispatch` on `main` — the -# only other run that shares `refs/heads/main` with the cron, since pushes to -# main don't trigger this workflow — can't cancel a nightly mid-flight, and -# `cancel-in-progress` is off for those events regardless. The nightly is the -# only backstop for integration-suite regressions a scoped PR run can't see, so -# it must always reach a conclusion. concurrency: - # #7966: keyed per RUN for non-PR events. A group that is constant across - # scheduled runs lets only the first one execute -- GitHub keeps at most one - # PENDING run per group and cancels the rest with `jobs: 0`, regardless of - # `cancel-in-progress`. PR runs keep the shared per-ref group so superseded - # pushes still coalesce. Enforced by scripts/gc_gate_wiring_check.py. - group: test-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} + # pull_request: superseded pushes cancel their in-flight run (per PR). + # push to main: ONE constant group with cancel-in-progress OFF -- GitHub + # keeps at most one running + one pending run per group and replaces the + # pending one with the newest, so merges coalesce onto the tip instead of + # queueing 58 sweeps a day (#7205 / #7856 -- read + # docs/src/testing/ci-gate-scheduling.md before "fixing" this). + # everything else (schedule, tags, dispatch): keyed per RUN, because a group + # that is constant across scheduled runs lets only the first one execute + # (#7966). Enforced by scripts/gc_gate_wiring_check.py. + group: >- + ${{ github.event_name == 'pull_request' && format('ci-pr-{0}', github.event.pull_request.number) + || (github.event_name == 'push' && github.ref == 'refs/heads/main' && 'ci-main-sweep') + || format('ci-{0}', github.run_id) }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: @@ -60,37 +83,73 @@ env: jobs: # --------------------------------------------------------------------------- - # Changeset gate: PRs touching crates/ must ship changelog.d/-.md - # (see changelog.d/README.md; fragments fold into GitHub Release notes at - # tag time via scripts/cut_release_notes.sh — CHANGELOG.md is frozen). - # Standalone job, NOT a step inside `lint`, so it can be its own required - # status check: lint carries unrelated red debt that gets admin-bypassed, - # and this gate must not ride along with that. The `skip-changelog` label - # skips it (the `labeled` trigger above refires the workflow, and a skipped - # required check counts as satisfied). + # plan: the one job that decides what this run does. ~20 s, no toolchain. + # Everything below is `needs: plan` + `if: fromJSON(...).jobs.`. If THIS + # job fails, every other job is skipped and the fan-in gate fails on + # `needs.plan.result != 'success'` -- a broken plan cannot silently pass. # --------------------------------------------------------------------------- - changeset-gate: - if: github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'skip-changelog') + plan: runs-on: ubuntu-latest timeout-minutes: 5 + outputs: + plan: ${{ steps.plan.outputs.plan }} + tier: ${{ steps.plan.outputs.tier }} steps: - - name: Require a changelog.d/ fragment for crates/ changes + - uses: actions/checkout@v7 + with: + persist-credentials: false + + - name: Self-test the planner + run: python3 scripts/ci_plan.py --self-test + + - name: List changed files (pull_request only) + if: github.event_name == 'pull_request' env: GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + PR_NUMBER: ${{ github.event.pull_request.number }} run: | - gh api "repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files" --paginate > files.json - jq -r '.[].filename' files.json | grep -q '^crates/' || { echo "No crates/ changes — gate not applicable."; exit 0; } - # The fragment must be ADDED in this PR (editing a leftover file - # doesn't count) and match the root-level -.md shape - # cut_release_notes.sh folds at release time. - jq -r '.[] | select(.status == "added") | .filename' files.json | grep -qE '^changelog\.d/[0-9]+-[^/]+\.md$' && exit 0 - echo "::error::This PR changes crates/ but adds no changelog.d/ fragment. Add changelog.d/-.md (see changelog.d/README.md) or apply the 'skip-changelog' label." - exit 1 + set -euo pipefail + # An API failure aborts here (set -e), which fails `plan` and with it + # the gate. It must NOT fall through to an empty list: ci_plan.py + # treats an empty listing as `core` for the same reason, but a + # loud failure is better than a silent full run. + gh api "repos/$REPOSITORY/pulls/$PR_NUMBER/files" --paginate --jq '.[].filename' > changed-files.txt + echo "Changed files ($(wc -l < changed-files.txt)):" + sed 's/^/ /' changed-files.txt + + - name: Compute the plan + id: plan + env: + EVENT_NAME: ${{ github.event_name }} + REF: ${{ github.ref }} + LABELS: ${{ github.event_name == 'pull_request' && join(github.event.pull_request.labels.*.name, ',') || '' }} + TIER_INPUT: ${{ inputs.tier }} + UPDATE_GAP_SNAPSHOT: ${{ inputs.update_gap_snapshot }} + run: | + set -euo pipefail + args=(--event "$EVENT_NAME" --ref "$REF" --labels "$LABELS") + if [ "$EVENT_NAME" = "pull_request" ]; then + args+=(--changed-files changed-files.txt) + fi + if [ "$EVENT_NAME" = "workflow_dispatch" ]; then + args+=(--tier "${TIER_INPUT:-full}") + if [ "$UPDATE_GAP_SNAPSHOT" = "true" ]; then + args+=(--update-gap-snapshot) + fi + fi + python3 scripts/ci_plan.py "${args[@]}" + # --------------------------------------------------------------------------- - # Lint: cargo fmt --check (formatting gate for every PR) + # Lint: cargo fmt --check + every no-compile audit script. Runs in EVERY + # tier, including docs-only PRs (it is the one job the plan never turns + # off). ~4 min. Add the ci_plan self-test here too so a policy edit that + # breaks its own invariants is red before it can plan anything. # --------------------------------------------------------------------------- lint: + needs: plan + if: fromJSON(needs.plan.outputs.plan).jobs.lint # Was macos-14 — moved to ubuntu-latest in v0.5.428 since `cargo fmt # --check` is portable. The 6 multiplier-min cut is small in absolute # terms (lint runs in ~30s) but it's free. @@ -99,6 +158,28 @@ jobs: steps: - uses: actions/checkout@v7 + # Changeset gate (was its own job): PRs touching crates/ must ship + # changelog.d/-.md (see changelog.d/README.md; fragments fold + # into GitHub Release notes at tag time via scripts/cut_release_notes.sh + # -- CHANGELOG.md is frozen). The `skip-changelog` label skips it (the + # `labeled` trigger refires the workflow). Lives in `lint` now because + # `pr-gate` is the single required context, so a red step here is as + # blocking as a red job was -- and one fewer job is one fewer runner + # slot on a 20-slot org. + - name: Require a changelog.d/ fragment for crates/ changes + if: github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'skip-changelog') + env: + GH_TOKEN: ${{ github.token }} + run: | + gh api "repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files" --paginate > files.json + jq -r '.[].filename' files.json | grep -q '^crates/' || { echo "No crates/ changes — gate not applicable."; exit 0; } + # The fragment must be ADDED in this PR (editing a leftover file + # doesn't count) and match the root-level -.md shape + # cut_release_notes.sh folds at release time. + jq -r '.[] | select(.status == "added") | .filename' files.json | grep -qE '^changelog\.d/[0-9]+-[^/]+\.md$' && exit 0 + echo "::error::This PR changes crates/ but adds no changelog.d/ fragment. Add changelog.d/-.md (see changelog.d/README.md) or apply the 'skip-changelog' label." + exit 1 + - name: Setup Node.js for benchmark harness tests uses: actions/setup-node@v7 with: @@ -136,6 +217,30 @@ jobs: if: ${{ !cancelled() }} run: cargo fmt --all -- --check + # The CI tier policy (scripts/ci_plan.py) is code; its self-test asserts + # the invariants CLAUDE.md cares about (gc-stress main-line reachable, + # docs-only PRs skip the compile tier, an empty file listing is treated + # as core, ...). The table in docs/src/testing/ci-tiers.md is generated + # from it and must not drift. + - name: CI plan policy self-test + docs table freshness + if: ${{ !cancelled() }} + run: | + python3 scripts/ci_plan.py --self-test + python3 scripts/ci_plan.py --table > /tmp/ci-plan-table.md + if ! grep -qF -- "$(head -1 /tmp/ci-plan-table.md)" docs/src/testing/ci-tiers.md; then + echo "::error::docs/src/testing/ci-tiers.md is missing the tier table header"; exit 1 + fi + python3 - <<'PY' + import re, sys + want = open('/tmp/ci-plan-table.md').read().strip() + doc = open('docs/src/testing/ci-tiers.md').read() + if want not in doc: + print("::error::docs/src/testing/ci-tiers.md tier table is stale. Regenerate with:") + print(" python3 scripts/ci_plan.py --table # and paste over the table") + sys.exit(1) + print("ci-tiers.md tier table matches ci_plan.py") + PY + - name: Audit workspace architecture if: ${{ !cancelled() }} run: | @@ -531,26 +636,23 @@ jobs: python3 scripts/global_sink_isolation.py # --------------------------------------------------------------------------- - # Clippy — enforces the deny-level lints in [workspace.lints] (root + # check: clippy (both scopes) + API-docs drift, in ONE job. Was three jobs + # (Clippy x2 matrix, api-docs-drift); merged because each job is a runner + # slot on a 20-slot org and all three share the same toolchain, cache and + # `cargo check` metadata. Steps are `if: !cancelled()` so a clippy failure + # still reports the docs verdict. + # + # Clippy enforces the deny-level lints in [workspace.lints] (root # Cargo.toml). `cargo clippy` exits nonzero only on `deny` lints, so # warn-level output is informational and never blocks a PR. The product leg # gives fast feedback for the CLI; the host-compatible leg names every Linux # package explicitly. Neither scope depends on Cargo default-members. # --------------------------------------------------------------------------- - clippy: - name: Clippy (${{ matrix.scope }}) + check: + needs: plan + if: fromJSON(needs.plan.outputs.plan).jobs.check runs-on: ubuntu-latest timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - scope: [product, host-compatible] - env: - RUSTC_WRAPPER: sccache - SCCACHE_GHA_ENABLED: "false" - SCCACHE_DIR: ${{ github.workspace }}/.sccache - SCCACHE_CACHE_SIZE: "12G" - CARGO_INCREMENTAL: "0" steps: - uses: actions/checkout@v7 @@ -560,44 +662,61 @@ jobs: components: clippy - uses: ./.github/actions/setup-llvm22 - - name: Install sccache - uses: mozilla-actions/sccache-action@v0.0.11 - - # Same tarball'd-disk-cache pattern as api-docs-drift (see the comment - # there for why SCCACHE_GHA_ENABLED is off). - - name: Cache sccache objects - uses: actions/cache@v6 - with: - path: ${{ github.workspace }}/.sccache - key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} - restore-keys: | - sccache-${{ runner.os }}-perry- - - uses: Swatinem/rust-cache@v2 with: shared-key: "${{ runner.os }}-perry" save-if: ${{ github.ref == 'refs/heads/main' }} - - name: Run clippy for explicit scope + - name: Clippy (product) + if: ${{ !cancelled() }} + run: cargo clippy -p perry --bins + + - name: Clippy (host-compatible) + if: ${{ !cancelled() }} run: | - if [[ "${{ matrix.scope }}" == "product" ]]; then - cargo clippy -p perry --bins - else - mapfile -t excluded < <(python3 scripts/workspace_architecture.py \ - --print-excluded-scope host-compatible) - cargo_args=(--workspace) - for package in "${excluded[@]}"; do - cargo_args+=(--exclude "$package") - done - cargo clippy "${cargo_args[@]}" + mapfile -t excluded < <(python3 scripts/workspace_architecture.py \ + --print-excluded-scope host-compatible) + cargo_args=(--workspace) + for package in "${excluded[@]}"; do + cargo_args+=(--exclude "$package") + done + cargo clippy "${cargo_args[@]}" + + - name: Regenerate API docs + if: ${{ !cancelled() }} + run: ./scripts/regen_api_docs.sh + + - name: Check for API docs drift + if: ${{ !cancelled() }} + run: | + if ! git diff --quiet -- docs/src/api/reference.md docs/api/perry.d.ts; then + echo "" + echo "::error::API docs drift detected. The compile-time manifest in" + echo "::error::crates/perry-api-manifest/src/entries.rs changed but the" + echo "::error::generated artifacts under docs/ weren't regenerated." + echo "" + echo "Fix by running:" + echo " ./scripts/regen_api_docs.sh" + echo " git add docs/src/api/reference.md docs/api/perry.d.ts" + echo " git commit -m 'docs: regenerate API reference + .d.ts'" + echo "" + echo "Diff:" + git --no-pager diff --stat -- docs/src/api/reference.md docs/api/perry.d.ts + echo "" + git --no-pager diff -- docs/src/api/reference.md docs/api/perry.d.ts | head -200 + exit 1 fi + echo "✅ API docs match the manifest." # --------------------------------------------------------------------------- # rustc warnings gate # # `cargo check` with `-D warnings`, so a PR cannot add a rustc warning. This - # is deliberately separate from the clippy job above: clippy's own warn-level - # lints are informational here, while rustc's are not. + # is deliberately separate from the `check` job above: clippy's own + # warn-level lints are informational there, while rustc's are not -- and + # `-D warnings` on the clippy invocation would promote every clippy warning + # too. Both scopes run in ONE job (was a 2-way matrix): they share the + # toolchain and cache, and each job is a runner slot. # # Both legs are needed because they compile different code. `perry` depends on # perry-runtime with `default-features = false`, so the product leg sees a @@ -608,14 +727,11 @@ jobs: # perry-ui-macos is in the excluded scope (this runs on ubuntu), so its # warnings are not gated here. # --------------------------------------------------------------------------- - rustc-warnings: - name: Warnings (${{ matrix.scope }}) + warnings: + needs: plan + if: fromJSON(needs.plan.outputs.plan).jobs.warnings runs-on: ubuntu-latest timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - scope: [product, host-compatible] env: RUSTFLAGS: -D warnings RUSTC_WRAPPER: sccache @@ -633,8 +749,8 @@ jobs: - name: Install sccache uses: mozilla-actions/sccache-action@v0.0.11 - - name: Cache sccache objects - uses: actions/cache@v6 + - name: Restore sccache objects + uses: actions/cache/restore@v6 with: path: ${{ github.workspace }}/.sccache key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} @@ -646,103 +762,34 @@ jobs: shared-key: "${{ runner.os }}-perry" save-if: ${{ github.ref == 'refs/heads/main' }} - - name: Check for rustc warnings - run: | - if [[ "${{ matrix.scope }}" == "product" ]]; then - cargo check -p perry --bins - else - mapfile -t excluded < <(python3 scripts/workspace_architecture.py \ - --print-excluded-scope host-compatible) - cargo_args=(--workspace --all-targets) - for package in "${excluded[@]}"; do - cargo_args+=(--exclude "$package") - done - cargo check "${cargo_args[@]}" - fi - - # --------------------------------------------------------------------------- - # API docs drift gate (#465) - # - # Regenerates `docs/src/api/reference.md` and `docs/api/perry.d.ts` from - # the compile-time manifest in `crates/perry-api-manifest/src/entries.rs`, - # then `git diff --exit-code`s the result. Fails when a code change updated - # the manifest without re-committing the artifacts. Closes the "Release - # workflow regenerates docs automatically (no drift from code)" criterion - # — committing the diff is the easiest way to keep the docs in sync, - # since the diff is reviewable in the PR that introduces it. - # --------------------------------------------------------------------------- - api-docs-drift: - runs-on: ubuntu-latest - timeout-minutes: 30 - env: - RUSTC_WRAPPER: sccache - # sccache on a LOCAL DISK cache, persisted as a single tarball via - # actions/cache (see the "Cache sccache objects" step) — NOT the GitHub - # Actions cache backend (SCCACHE_GHA_ENABLED). The GHA backend stores one - # cache object per compilation unit; GitHub's cache service throttled / - # LRU-evicted the thousands of tiny entries, so a full build wrote ~3.3k - # objects (≈35 min of write time) yet the next run got ~0% Rust hits - # (measured: 3 hits / 3209 misses, 613 write errors) — i.e. every run - # recompiled the dependency graph cold. A single tarball'd disk cache - # restores in one step and gives real cross-run hit rates. Note - # SCCACHE_CACHE_SIZE is honoured by the disk backend (it was a silent - # no-op under the GHA backend, which is why the old "2G" never mattered). - SCCACHE_GHA_ENABLED: "false" - SCCACHE_DIR: ${{ github.workspace }}/.sccache - SCCACHE_CACHE_SIZE: "12G" - CARGO_INCREMENTAL: "0" - steps: - - uses: actions/checkout@v7 - - - name: Install Rust toolchain - uses: dtolnay/rust-toolchain@stable - - uses: ./.github/actions/setup-llvm22 - - - name: Install sccache - uses: mozilla-actions/sccache-action@v0.0.11 + - name: rustc warnings (product) + if: ${{ !cancelled() }} + run: cargo check -p perry --bins - # Persist the sccache disk cache across runs. The github.job + - # github.run_id key makes every run (including PRs) save its own fresh - # entry — distinct per job so the three sccache jobs don't collide on - # save — while the shared prefix restore-keys pull the most recent prior - # cache from ANY of them. The object cache therefore warms continuously - # and cross-pollinates instead of starting cold each run. - - name: Cache sccache objects - uses: actions/cache@v6 + - name: rustc warnings (host-compatible, all targets) + if: ${{ !cancelled() }} + run: | + mapfile -t excluded < <(python3 scripts/workspace_architecture.py \ + --print-excluded-scope host-compatible) + cargo_args=(--workspace --all-targets) + for package in "${excluded[@]}"; do + cargo_args+=(--exclude "$package") + done + cargo check "${cargo_args[@]}" + + # sccache SAVE is main-line only (restore above is unconditional). PR + # runs used to write a fresh ~0.5-1.3 GB tarball per job per push -- + # ~200 GB/day into a 10 GB repo cache budget -- which evicted every + # useful entry (including rust-cache's) within the hour. Now only sweep / + # nightly / release runs write, and PRs restore the newest main-line blob + # via the restore-keys prefix, i.e. a cache built from the tip they + # branched from. + - name: Save sccache objects (main-line runs only) + if: always() && github.event_name != 'pull_request' + uses: actions/cache/save@v6 with: path: ${{ github.workspace }}/.sccache key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} - restore-keys: | - sccache-${{ runner.os }}-perry- - - - uses: Swatinem/rust-cache@v2 - with: - shared-key: "${{ runner.os }}-perry" - save-if: ${{ github.ref == 'refs/heads/main' }} - - - name: Regenerate API docs - run: ./scripts/regen_api_docs.sh - - - name: Check for drift - run: | - if ! git diff --quiet -- docs/src/api/reference.md docs/api/perry.d.ts; then - echo "" - echo "::error::API docs drift detected. The compile-time manifest in" - echo "::error::crates/perry-api-manifest/src/entries.rs changed but the" - echo "::error::generated artifacts under docs/ weren't regenerated." - echo "" - echo "Fix by running:" - echo " ./scripts/regen_api_docs.sh" - echo " git add docs/src/api/reference.md docs/api/perry.d.ts" - echo " git commit -m 'docs: regenerate API reference + .d.ts'" - echo "" - echo "Diff:" - git --no-pager diff --stat -- docs/src/api/reference.md docs/api/perry.d.ts - echo "" - git --no-pager diff -- docs/src/api/reference.md docs/api/perry.d.ts | head -200 - exit 1 - fi - echo "✅ API docs match the manifest." # --------------------------------------------------------------------------- # Rust unit tests (266+ tests across all crates) @@ -757,6 +804,8 @@ jobs: # required the manual simulator-runtime wipe + cache=registry-only # workaround). Each downstream job below builds in parallel directly. cargo-test: + needs: plan + if: fromJSON(needs.plan.outputs.plan).jobs.cargo_test # Was macos-14 — moved to ubuntu-latest in v0.5.392 to drop the 10× # billing weight. The centralized Linux test scope already filters out # platform-specific UI crates, so @@ -818,8 +867,8 @@ jobs: # save — while the shared prefix restore-keys pull the most recent prior # cache from ANY of them. The object cache therefore warms continuously # and cross-pollinates instead of starting cold each run. - - name: Cache sccache objects - uses: actions/cache@v6 + - name: Restore sccache objects + uses: actions/cache/restore@v6 with: path: ${{ github.workspace }}/.sccache key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} @@ -883,7 +932,7 @@ jobs: # whole workspace (~90 min). Release tags, the nightly cron, and # workflow_dispatch run the FULL workspace as the safety net. See # scripts/ci_test_scope.py for the rules. - if [ "${{ github.event_name }}" = "pull_request" ]; then + if [ "${{ fromJSON(needs.plan.outputs.plan).cargo_test_scope }}" = "pr" ]; then changed_files="$(gh pr view "${{ github.event.pull_request.number }}" \ --json files --jq '.files[].path')" echo "Changed files in PR:"; printf '%s\n' "$changed_files" @@ -1002,6 +1051,20 @@ jobs: CARGO_PROFILE_TEST_DEBUG: "0" run: cargo test -p perry-ffi --features runtime-link --lib + # sccache SAVE is main-line only (restore above is unconditional). PR + # runs used to write a fresh ~0.5-1.3 GB tarball per job per push -- + # ~200 GB/day into a 10 GB repo cache budget -- which evicted every + # useful entry (including rust-cache's) within the hour. Now only sweep / + # nightly / release runs write, and PRs restore the newest main-line blob + # via the restore-keys prefix, i.e. a cache built from the tip they + # branched from. + - name: Save sccache objects (main-line runs only) + if: always() && github.event_name != 'pull_request' + uses: actions/cache/save@v6 + with: + path: ${{ github.workspace }}/.sccache + key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} + # --------------------------------------------------------------------------- # Scoped e2e: run the integration suites NAMED BY THE DIFF (#5960) # @@ -1032,9 +1095,10 @@ jobs: # for that class — hence the concurrency carve-out above. # --------------------------------------------------------------------------- e2e-scoped: + needs: plan + if: fromJSON(needs.plan.outputs.plan).jobs.e2e_scoped # PR-only: tags / nightly / workflow_dispatch already run every suite in the # full `cargo-test` path. - if: github.event_name == 'pull_request' runs-on: ubuntu-latest # MUST cover the worst case the per-suite bound allows, or the job wall # clock kills legitimately-running suites and we get an uninformative @@ -1105,9 +1169,9 @@ jobs: if: steps.scope.outputs.suites != '' uses: mozilla-actions/sccache-action@v0.0.11 - - name: Cache sccache objects + - name: Restore sccache objects if: steps.scope.outputs.suites != '' - uses: actions/cache@v6 + uses: actions/cache/restore@v6 with: path: ${{ github.workspace }}/.sccache key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} @@ -1194,6 +1258,20 @@ jobs: fi exit "$status" + # sccache SAVE is main-line only (restore above is unconditional). PR + # runs used to write a fresh ~0.5-1.3 GB tarball per job per push -- + # ~200 GB/day into a 10 GB repo cache budget -- which evicted every + # useful entry (including rust-cache's) within the hour. Now only sweep / + # nightly / release runs write, and PRs restore the newest main-line blob + # via the restore-keys prefix, i.e. a cache built from the tip they + # branched from. + - name: Save sccache objects (main-line runs only) + if: always() && github.event_name != 'pull_request' + uses: actions/cache/save@v6 + with: + path: ${{ github.workspace }}/.sccache + key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} + # --------------------------------------------------------------------------- # Windows build gate # @@ -1209,6 +1287,11 @@ jobs: # --release would be far too slow for per-PR CI. # --------------------------------------------------------------------------- windows-build: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.windows_build runs-on: windows-latest # Cold Windows builds are slow, and this workflow never runs on pushes to # main, so rust-cache only saves on the nightly cron (github.ref is @@ -1338,6 +1421,11 @@ jobs: # execute an ARM binary, so keep a focused native job that proves the # compiler, runtime ABI, final linker, and advertised target all agree. windows-arm64-build: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.windows_arm64_build runs-on: windows-11-arm timeout-minutes: 75 steps: @@ -1448,11 +1536,8 @@ jobs: # silently come back. # --------------------------------------------------------------------------- gc-stress: - if: >- - github.event_name == 'push' || - github.event_name == 'pull_request' || - github.event_name == 'schedule' || - github.event_name == 'workflow_dispatch' + needs: plan + if: fromJSON(needs.plan.outputs.plan).jobs.gc_stress runs-on: ubuntu-latest timeout-minutes: 90 steps: @@ -1495,11 +1580,11 @@ jobs: # #6946, #6950) — the script asserts liveness from the collector's own # PERRY_GC_TRACE / PERRY_GC_DIAG output rather than trusting the env var. - name: GC x representation-selection matrix (PR subset) - if: github.event_name == 'pull_request' + if: fromJSON(needs.plan.outputs.plan).gc_stress_mode == 'pr' run: ./scripts/gc_repsel_matrix.sh --no-build --arms pr --json gc-repsel-matrix.json - name: GC x representation-selection matrix (full) - if: github.event_name != 'pull_request' + if: fromJSON(needs.plan.outputs.plan).gc_stress_mode == 'full' run: ./scripts/gc_repsel_matrix.sh --no-build --arms all --json gc-repsel-matrix.json # GATING, and deliberately so. CLAUDE.md's GC knob kill-policy requires @@ -1550,6 +1635,11 @@ jobs: # Fails when hot-loop structural contracts regress. # --------------------------------------------------------------------------- compiler-output-regression: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.compiler_output_regression runs-on: ubuntu-latest timeout-minutes: 45 env: @@ -1585,8 +1675,8 @@ jobs: # save — while the shared prefix restore-keys pull the most recent prior # cache from ANY of them. The object cache therefore warms continuously # and cross-pollinates instead of starting cold each run. - - name: Cache sccache objects - uses: actions/cache@v6 + - name: Restore sccache objects + uses: actions/cache/restore@v6 with: path: ${{ github.workspace }}/.sccache key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} @@ -1694,6 +1784,20 @@ jobs: name: compiler-output-regression path: target/compiler-output-regression/ + # sccache SAVE is main-line only (restore above is unconditional). PR + # runs used to write a fresh ~0.5-1.3 GB tarball per job per push -- + # ~200 GB/day into a 10 GB repo cache budget -- which evicted every + # useful entry (including rust-cache's) within the hour. Now only sweep / + # nightly / release runs write, and PRs restore the newest main-line blob + # via the restore-keys prefix, i.e. a cache built from the tip they + # branched from. + - name: Save sccache objects (main-line runs only) + if: always() && github.event_name != 'pull_request' + uses: actions/cache/save@v6 + with: + path: ${{ github.workspace }}/.sccache + key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} + # --------------------------------------------------------------------------- # Representation-selection promotion census (#7106) # @@ -1734,6 +1838,11 @@ jobs: # follow-up is skipped. # --------------------------------------------------------------------------- repsel-census: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.repsel_census runs-on: ubuntu-latest timeout-minutes: 45 env: @@ -1756,8 +1865,8 @@ jobs: - name: Install sccache uses: mozilla-actions/sccache-action@v0.0.11 - - name: Cache sccache objects - uses: actions/cache@v6 + - name: Restore sccache objects + uses: actions/cache/restore@v6 with: path: ${{ github.workspace }}/.sccache key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} @@ -1881,6 +1990,20 @@ jobs: name: repsel-census path: target/repsel-census/ + # sccache SAVE is main-line only (restore above is unconditional). PR + # runs used to write a fresh ~0.5-1.3 GB tarball per job per push -- + # ~200 GB/day into a 10 GB repo cache budget -- which evicted every + # useful entry (including rust-cache's) within the hour. Now only sweep / + # nightly / release runs write, and PRs restore the newest main-line blob + # via the restore-keys prefix, i.e. a cache built from the tip they + # branched from. + - name: Save sccache objects (main-line runs only) + if: always() && github.event_name != 'pull_request' + uses: actions/cache/save@v6 + with: + path: ${{ github.workspace }}/.sccache + key: sccache-${{ runner.os }}-perry-${{ github.job }}-${{ github.run_id }} + # --------------------------------------------------------------------------- # Native ABI evidence packet # @@ -1891,10 +2014,11 @@ jobs: # the lighter report/unit and compiler-output structural gates above. # --------------------------------------------------------------------------- native-abi-evidence-packet: - if: >- - github.event_name == 'push' || - (github.event_name == 'workflow_dispatch' && inputs.run_extended_tests) || - (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.native_abi_evidence_packet runs-on: ubuntu-latest timeout-minutes: 90 env: @@ -1913,8 +2037,8 @@ jobs: - name: Install sccache uses: mozilla-actions/sccache-action@v0.0.11 - - name: Cache sccache objects - uses: actions/cache@v6 + - name: Restore sccache objects + uses: actions/cache/restore@v6 with: path: ${{ github.workspace }}/.sccache key: sccache-${{ runner.os }}-perry-native-abi-evidence-${{ github.run_id }} @@ -1946,10 +2070,26 @@ jobs: name: native-abi-evidence-packet path: target/native-abi-evidence-packet/ + # sccache SAVE is main-line only (restore above is unconditional). PR + # runs used to write a fresh ~0.5-1.3 GB tarball per job per push -- + # ~200 GB/day into a 10 GB repo cache budget -- which evicted every + # useful entry (including rust-cache's) within the hour. Now only sweep / + # nightly / release runs write, and PRs restore the newest main-line blob + # via the restore-keys prefix, i.e. a cache built from the tip they + # branched from. + - name: Save sccache objects (main-line runs only) + if: always() && github.event_name != 'pull_request' + uses: actions/cache/save@v6 + with: + path: ${{ github.workspace }}/.sccache + key: sccache-${{ runner.os }}-perry-native-abi-evidence-${{ github.run_id }} + # --------------------------------------------------------------------------- # Gap-suite conformance smoke (every PR + main push) # --------------------------------------------------------------------------- - conformance-smoke: + gap-suite: + needs: plan + if: fromJSON(needs.plan.outputs.plan).jobs.gap_suite # 2026-07-02 audit §11: nothing on the default PR path exercised # TypeScript SEMANTICS — lint/cargo-test/api-docs-drift build and unit- # test the compiler but never diff a compiled program against node, so @@ -1980,8 +2120,7 @@ jobs: strategy: fail-fast: false matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8] - if: github.event_name == 'pull_request' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' + shard: ${{ fromJSON(needs.plan.outputs.plan).gap.shards }} runs-on: ubuntu-latest # Per-shard wall time has crept up as the gap suite grew: the original # sharded run was 19-37 min/shard, but 2026-07-16 measured 33-44 min for @@ -2016,45 +2155,70 @@ jobs: # inline here. See CLAUDE.md ("TypeScript Parity Status"). node-version-file: .node-version - - name: Run gap suite (shard ${{ matrix.shard }}/8) - run: ./scripts/run_gap_tests.sh --shard ${{ matrix.shard }}/8 + # Two harness modes, chosen by the plan (see ci_plan.py GAP_SUITE): + # fast: build the release compiler + runtime archives ONCE here, then + # PERRY_SKIP_BUILD=1 so every test links the prebuilt archives + # (~1.5 s/test). Only ext-routed tests (http/net/ws/zlib/events) + # still take the per-test auto-optimize path, because no single + # prebuilt stdlib can serve them (#7629). PR + sweep tiers. + # full: the harness's default -- every test compiles through + # auto-optimize, which rebuilds a feature-stripped runtime per + # distinct feature set (~200 s each; measured 96% of a shard's + # wall time). It is the arm that sees auto-optimize-only bugs, + # so it stays in the nightly/release tier at 8 shards. + # Both compare against the SAME committed Linux snapshot; a divergence + # between them is a real auto-optimize-specific finding, not noise. + - name: Build compiler + runtime archives (fast mode) + if: fromJSON(needs.plan.outputs.plan).gap.mode == 'fast' + # Release profile, codegen-units=1: the parity harness's own build + # command. cu=16 miscompiles the release runtime (see CLAUDE.md). + run: cargo build --release -p perry -p perry-runtime -p perry-stdlib -p perry-runtime-static -p perry-stdlib-static + + - name: Run gap suite (shard ${{ matrix.shard }}/${{ fromJSON(needs.plan.outputs.plan).gap.total }}, ${{ fromJSON(needs.plan.outputs.plan).gap.mode }} mode) + env: + GAP_MODE: ${{ fromJSON(needs.plan.outputs.plan).gap.mode }} + GAP_TOTAL: ${{ fromJSON(needs.plan.outputs.plan).gap.total }} + GAP_SHARD: ${{ matrix.shard }} + UPDATE_SNAPSHOT: ${{ fromJSON(needs.plan.outputs.plan).gap.update_snapshot && '1' || '0' }} + run: | + set -euo pipefail + if [ "$GAP_MODE" = "fast" ]; then + export PERRY_SKIP_BUILD=1 + export PERRY_BIN="$PWD/target/release/perry" + export PERRY_RUNTIME_DIR="$PWD/target/release" + fi + if [ "$GAP_TOTAL" = "1" ]; then + ./scripts/run_gap_tests.sh + else + ./scripts/run_gap_tests.sh --shard "$GAP_SHARD/$GAP_TOTAL" + fi + + # Only produced by a `workflow_dispatch` with update_gap_snapshot=true + # (one shard, whole suite). Download it and commit test-parity/ + # gap_snapshot.json -- see docs/src/testing/ci-tiers.md "Re-baselining". + - name: Upload re-baselined gap snapshot + if: fromJSON(needs.plan.outputs.plan).gap.update_snapshot + uses: actions/upload-artifact@v7 + with: + name: gap-snapshot-update + path: test-parity/gap_snapshot.json - name: Upload gap report if: always() uses: actions/upload-artifact@v7 with: - name: gap-suite-report-shard-${{ matrix.shard }} + name: gap-suite-report-${{ fromJSON(needs.plan.outputs.plan).gap.mode }}-shard-${{ matrix.shard }} path: test-parity/reports/ - # --------------------------------------------------------------------------- - # Fan-in for the sharded gap suite. `needs.conformance-smoke.result` is - # "success" only when EVERY shard succeeded, so this single context is a - # faithful all-shards-green gate — the one branch protection requires. - # `if: always()` guarantees it runs and reports even when a shard fails - # (a required check that got *skipped* would otherwise wedge the PR on - # "Expected"); the event guard mirrors the shards so it no-ops off the PR - # path just like they do. - # --------------------------------------------------------------------------- - conformance-smoke-complete: - needs: conformance-smoke - if: always() && (github.event_name == 'pull_request' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') - runs-on: ubuntu-latest - timeout-minutes: 5 - steps: - - name: Require every gap-suite shard to pass - run: | - result='${{ needs.conformance-smoke.result }}' - echo "conformance-smoke (all 8 shards) => $result" - if [[ "$result" != "success" ]]; then - echo "::error::gap suite failed, was cancelled, or was skipped in one or more shards" - exit 1 - fi - echo "All gap-suite shards passed." - # --------------------------------------------------------------------------- # Parity tests (Perry output vs Node.js) # --------------------------------------------------------------------------- parity: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.parity # Release-publish decoupling: aspirational extended suite. Per maintainer # decision it no longer BLOCKS package publishing — release-packages.yml's # await-tests gate keys on this workflow's run conclusion, and job-level @@ -2062,7 +2226,6 @@ jobs: # conclusion. The job still runs on every tag + shows its own pass/fail as # an informational signal (and core jobs — cargo-test/lint/api-docs-drift/ # compiler-output-regression — still gate publish). - continue-on-error: true # Was macos-14 — moved to ubuntu-latest in v0.5.392. Parity tests # just compare Perry's stdout against `node --experimental-strip-types`'s # stdout per test file; both run cleanly on Linux. `gtimeout` on @@ -2081,10 +2244,6 @@ jobs: # Opt-in: apply the `run-extended-tests` label to a PR, or dispatch # the workflow manually with `run_extended_tests=true`, to run this # job on demand. PR authors and maintainers can both apply labels. - if: >- - github.event_name == 'push' || - (github.event_name == 'workflow_dispatch' && inputs.run_extended_tests) || - (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 @@ -2173,9 +2332,13 @@ jobs: # Compile smoke test (all 130+ test files must compile) # --------------------------------------------------------------------------- compile-smoke: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.compile_smoke # Release-publish decoupling (see `parity` above): aspirational extended # suite, informational only — does not block package publishing. - continue-on-error: true # Was macos-14 — moved to ubuntu-latest in v0.5.392. The smoke # compiles every `test-files/*.ts` with the bare `perry foo.ts -o # out` path; the auto-optimize cache + clang link steps work @@ -2193,10 +2356,6 @@ jobs: # # Opt-in: `run-extended-tests` PR label or `workflow_dispatch` with # `run_extended_tests=true` runs this job on demand. - if: >- - github.event_name == 'push' || - (github.event_name == 'workflow_dispatch' && inputs.run_extended_tests) || - (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 @@ -2569,12 +2728,16 @@ jobs: # entries for the workflow). # --------------------------------------------------------------------------- harmonyos-smoke: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.harmonyos_smoke # Aspirational smoke (informational) — must not block package publish. # release-packages await-tests keys on this workflow's run conclusion; # continue-on-error keeps a red result here from failing it (same as # parity/compile-smoke/doc-tests/drizzle/effect-basic-smoke). Core jobs # (cargo-test/lint/api-docs-drift/compiler-output-regression) still gate. - continue-on-error: true runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 @@ -2617,13 +2780,13 @@ jobs: # default. Opt-in via the `run-extended-tests` label. # --------------------------------------------------------------------------- drizzle-mysql-smoke: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.drizzle_mysql_smoke # Release-publish decoupling (see `parity` above): aspirational extended # suite, informational only — does not block package publishing. - continue-on-error: true - if: >- - github.event_name == 'push' || - (github.event_name == 'workflow_dispatch' && inputs.run_extended_tests) || - (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) runs-on: ubuntu-latest services: mysql: @@ -2698,12 +2861,12 @@ jobs: # `run-extended-tests` label or workflow dispatch. # --------------------------------------------------------------------------- ink-link-smoke: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.ink_link_smoke # Aspirational smoke (informational) — see harmonyos-smoke. Does not block publish. - continue-on-error: true - if: >- - github.event_name == 'push' || - (github.event_name == 'workflow_dispatch' && inputs.run_extended_tests) || - (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 @@ -2753,11 +2916,11 @@ jobs: # progress. Gated to tag pushes + opt-in, matching the named package smokes. # --------------------------------------------------------------------------- effect-basic-smoke: - continue-on-error: true - if: >- - github.event_name == 'push' || - (github.event_name == 'workflow_dispatch' && inputs.run_extended_tests) || - (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.effect_basic_smoke runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 @@ -2817,12 +2980,13 @@ jobs: # demand. PR authors and maintainers can both apply labels. # --------------------------------------------------------------------------- doc-tests: + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.doc_tests # Blocking host runs and the portable cross-compile subset gate the job. # Known platform-tail failures remain advisory at their individual steps. - if: >- - github.event_name == 'push' || - (github.event_name == 'workflow_dispatch' && inputs.run_extended_tests) || - (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-extended-tests')) strategy: fail-fast: false matrix: @@ -3058,7 +3222,11 @@ jobs: # Binary size tracking (main branch only) # --------------------------------------------------------------------------- binary-size: - if: github.ref == 'refs/heads/main' + # Sweep/full-tier only. Chained behind `check` so a sweep's fan-out does + # not take every runner slot the moment a merge lands -- PR gates share + # the same 20-slot pool. + needs: [plan, check] + if: fromJSON(needs.plan.outputs.plan).jobs.binary_size runs-on: macos-14 steps: - uses: actions/checkout@v7 @@ -3088,3 +3256,93 @@ jobs: with: name: binary-sizes path: /tmp/sizes.md + + + # --------------------------------------------------------------------------- + # security-audit: cargo-audit / cargo-deny / supply-chain soak gate / agent + + # skills scans, from the reusable .github/workflows/security-audit.yml. In + # the PR tier the plan turns it on only when the diff touches a lockfile, + # manifest or policy file (ci_plan.py DEPS_GLOBS) -- a code-only PR cannot + # introduce a new advisory, and the weekly schedule in that workflow catches + # advisory-database updates on their own. Sweep and full tiers always run it. + # --------------------------------------------------------------------------- + security-audit: + needs: plan + if: fromJSON(needs.plan.outputs.plan).jobs.security_audit + uses: ./.github/workflows/security-audit.yml + permissions: + contents: read + secrets: inherit + + # --------------------------------------------------------------------------- + # THE fan-in. One job, one status, named by tier so the outside world can + # key on it without knowing the job list: + # + # pr-gate pull_request runs. THE ONLY REQUIRED STATUS CONTEXT. + # Adding or removing a job above never needs a + # branch-protection edit again. + # main-gate push-to-main sweeps. + # full-suite-gate full-tier runs (nightly, tags, dispatch, labelled PRs + # -- a labelled PR is still `pr-gate`, the event wins). + # release-packages.yml's await-tests polls for THIS job + # on the release SHA, so a sweep-level green can never + # be mistaken for a release-grade one. + # + # `if: always()` so it reports even when something upstream failed or was + # cancelled -- a fan-in that is skipped when a dependency fails would leave + # the required context in "expected" limbo, which is exactly the state that + # forces admin bypasses. Verdict rules: + # * `plan` itself must be `success` (a broken planner must not turn into + # "everything skipped, therefore green"); + # * every other needed job must be `success` or `skipped` (skipped == the + # plan turned it off, which is fine); `failure` and `cancelled` fail. + # --------------------------------------------------------------------------- + gate: + name: ${{ github.event_name == 'pull_request' && 'pr-gate' || (needs.plan.outputs.tier == 'full' && 'full-suite-gate' || 'main-gate') }} + if: always() + needs: + - plan + - lint + - check + - warnings + - cargo-test + - e2e-scoped + - windows-build + - windows-arm64-build + - gc-stress + - compiler-output-regression + - repsel-census + - native-abi-evidence-packet + - gap-suite + - parity + - compile-smoke + - harmonyos-smoke + - drizzle-mysql-smoke + - ink-link-smoke + - effect-basic-smoke + - doc-tests + - binary-size + - security-audit + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Require the plan and every planned job to have passed + env: + NEEDS: ${{ toJSON(needs) }} + TIER: ${{ needs.plan.outputs.tier }} + run: | + set -euo pipefail + echo "tier: ${TIER:-}" + echo "$NEEDS" | jq -r 'to_entries[] | " \(.key): \(.value.result)"' + plan_result="$(echo "$NEEDS" | jq -r '.plan.result')" + if [ "$plan_result" != "success" ]; then + echo "::error::the plan job did not succeed ($plan_result); nothing below it ran, so this is a red run, not a green one." + exit 1 + fi + bad="$(echo "$NEEDS" | jq -r 'to_entries[] | select(.value.result == "failure" or .value.result == "cancelled") | .key')" + if [ -n "$bad" ]; then + echo "::error::failed or cancelled: $(echo "$bad" | tr '\n' ' ')" + exit 1 + fi + ran="$(echo "$NEEDS" | jq -r 'to_entries[] | select(.value.result == "success") | .key' | wc -l)" + echo "OK: $ran job(s) passed, the rest were turned off by the plan." diff --git a/.github/workflows/tls-budget.yml b/.github/workflows/tls-budget.yml index 482223890c..69ab3f76a4 100644 --- a/.github/workflows/tls-budget.yml +++ b/.github/workflows/tls-budget.yml @@ -54,6 +54,12 @@ name: TLS Budget on: pull_request: + # PR arm is OPT-IN via the `run-extended-tests` label (see the header of + # test.yml and docs/src/testing/ci-tiers.md): an unlabelled PR still gets + # a run, but every job in it is skipped, which costs no runner slot. The + # main-line arm (schedule / tags) is unchanged. `labeled` re-fires the run + # when the label lands. + types: [opened, synchronize, reopened, labeled] # POST-MERGE ARM: staggered six-hourly sweep of `main`, NOT one run per merge. # `push: branches: [main]` starved this gate and nine others for two days # (#7856): ~29 jobs enqueued per merge at 58 merges/day against a ~9-job @@ -112,6 +118,8 @@ jobs: run: python3 scripts/check_thread_locals.py tls-budget: + # PR arm is opt-in (label `run-extended-tests`); see the `on:` block. + if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-extended-tests') # macos-14 is arm64. `_tlv_get_addr` is a Mach-O TLS artefact and the # direct thread-specific-data path in tls_hot.rs is Apple-aarch64 only, so # this measurement does not exist on any other platform -- the gate script diff --git a/CLAUDE.md b/CLAUDE.md index 455cc7db48..64102465f4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -53,7 +53,7 @@ A `--module` selector scopes `--check`/`--update-baseline` to just that slice (a ## Workflow Requirements -**Default flow is PR-based.** `main` is protected: pushes require a pull request, CI must pass (`lint`, `cargo-test`, `api-docs-drift`, `security-audit`), and only squash or rebase merges are allowed (no merge commits, linear history enforced). `parity` and `compile-smoke` are gated to tag pushes only (v0.5.1018) — they no longer run on PRs but still gate the release-packages.yml publish step. Admins can bypass for hotfixes/version bumps, but the standard path is: +**Default flow is PR-based.** `main` is protected: pushes require a pull request, CI must pass, and only squash or rebase merges are allowed (no merge commits, linear history enforced). **The single required status context is `pr-gate`** — the fan-in of `test.yml`'s PR tier (`lint`, `check`, `warnings`, scoped `cargo-test`, the 4-shard fast-mode gap suite, `gc-stress`, `e2e-scoped`, and `security-audit` when a lockfile/manifest changed). What runs in which tier is decided by `scripts/ci_plan.py` (`--table`), documented in `docs/src/testing/ci-tiers.md`: **pr** (every PR push, ~11 jobs, must be green on `main`), **sweep** (every push to `main`, coalesced), **full** (nightly / tags / dispatch / `run-extended-tests` label — parity, compile-smoke, doc-tests, package smokes, the auto-optimize gap shards). Releases wait for a `full-suite-gate` on the release SHA. The satellite GC/perf gates run on PRs only with the `run-extended-tests` label; their six-hourly `main` sweeps are unchanged. Admins can bypass for hotfixes/version bumps, but the standard path is: 1. Branch from `main`, push, open a PR. 2. Wait for required checks to go green. @@ -237,8 +237,8 @@ Build outputs are invisible to `git status`, so a clean tree tells you nothing a ### CI gates that surprise people - **2000-line-per-file cap** (`scripts/check_file_size.sh`) — run it before pushing; adding a long doc comment can trip it. - **addr-class ratchet** (`scripts/addr_class_inventory.py`) — a file gaining a bare-address site fails `lint`. -- **`conformance-smoke` shards are flaky.** Before believing a red shard, re-run it and A/B the named tests against a pristine `main` build; several are already in `test-parity/known_failures.json`. -- **Integration suites under `crates/*/tests/*.rs` do not run per-PR** (nightly/tag only) — a regression there can land green and sit red for days. Prefer putting acceptance coverage in `cargo-test`-visible unit tests (#5960). +- **`gap-suite` shards (formerly `conformance-smoke`)**: the PR/sweep tiers run the harness in `fast` mode (`PERRY_SKIP_BUILD=1`, one prebuilt release compiler, ~1.5 s/test); the full tier runs the 8-shard auto-optimize mode. Both compare against the same `test-parity/gap_snapshot.json`, so a divergence between them is an auto-optimize-specific finding. Before believing a red shard, A/B the named tests against a pristine `main` build; re-baseline only via the documented CI dispatch (`docs/src/testing/ci-tiers.md`). +- **Integration suites under `crates/*/tests/*.rs` run per-PR only when the diff names them** (`e2e-scoped`); the sweep/full tiers' `cargo test --workspace` is the backstop — a regression there lands on `main` and is attributed by sweep window. Prefer putting acceptance coverage in `cargo-test`-visible unit tests (#5960). - **`perry-runtime`'s tests are not parallel-safe — run them `RUST_TEST_THREADS=1`.** They share process-global side tables (#1444), and ~180 readers are not required to take the clearing lock (see `gc::tests::global_sink_isolation`'s header; #7672 is converting them to `per_test_global!` one at a time). Every CI path already pins `RUST_TEST_THREADS=1`; a local `cargo test --workspace` does NOT, which is the whole gap. Measured on the default pool: 600 full-suite runs at `--test-threads=16` (load ~90) were clean, but at `--test-threads=64` (load ~115) **2 of 320 runs failed** — in `proxy::…numeric_write_guard…` and `array::element_shape::matrix_tests::matrix_delete_revokes`, neither related to the change under test. The tell is the message: these fail on their own *fixture precondition* ("fixture must start proven, or every verdict below is vacuous"), i.e. another thread wiped the global, not a real defect in the code under test. Chasing such a failure as if it were a regression is wasted days. ### ★ Four ways a gate can be unable to fail diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 296f64436b..22dad3ae3f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -115,10 +115,17 @@ Behavioral changes (HIR, codegen, runtime) also need the conformance suite, which diffs compiled programs byte-for-byte against Node: ```bash -./scripts/run_gap_tests.sh # 401 tests; ~1h serially +./scripts/run_gap_tests.sh # ~560 tests; the PR gate runs this in 4 shards ./scripts/run_doc_tests.sh # Compile + run every docs/examples/*.ts ``` +What CI runs on a PR versus after a merge versus before a release is one +policy file, `scripts/ci_plan.py` (`python3 scripts/ci_plan.py --table`); +[docs/src/testing/ci-tiers.md](docs/src/testing/ci-tiers.md) explains the +three tiers. The single required status check is `pr-gate`. Apply the +`run-extended-tests` label to a PR to run the full tier and the satellite GC / +perf gates on it before merge. + **Use the Node version pinned in `.node-version`, not whatever you have.** Node is the oracle the suite diffs against, so its version is a correctness input: on a Node too old for a test, *node* fails, the harness classifies the diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 8f7128c8d4..349c43dc0b 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -151,6 +151,7 @@ - [Test Registration (dark tests)](testing/test-registration.md) - [Geisterhand (UI Fuzzer)](testing/geisterhand.md) - [Node Compatibility Matrix](testing/node-compat-matrix.md) +- [CI Tiers (PR gate / sweep / full)](testing/ci-tiers.md) - [CI Gate Scheduling](testing/ci-gate-scheduling.md) # CLI Reference diff --git a/docs/src/contributing/releasing.md b/docs/src/contributing/releasing.md index abfe43c948..790e565372 100644 --- a/docs/src/contributing/releasing.md +++ b/docs/src/contributing/releasing.md @@ -119,13 +119,36 @@ amending the existing one. ## 4. Release gates (what blocks a release) -- Parity tests must clear the threshold in `test-parity/threshold.json` -- `cargo test --workspace` (macOS excluded list as above) must be green -- `compile-smoke` must compile every file under `test-files/` -- `doc-tests` must compile + run every example under `docs/examples/` +`release-packages.yml`'s `await-tests` job dispatches `test.yml` with `tier=full` +on the pinned release branch and waits for a run whose **`full-suite-gate`** job +succeeded (a green PR-tier or push-to-main sweep run on the same SHA does *not* +count — only the full tier carries the release-grade suites; see +[CI tiers](../testing/ci-tiers.md)). It also waits for `simctl-tests.yml`. The +full tier is: + +- everything the PR gate and the post-merge sweep run (`lint`, `check`, `warnings`, + `cargo test --workspace`, the gap suite, `gc-stress`, Windows x64 + ARM64 builds, + compiler-output gates, `repsel-census`, `harmonyos-smoke`, `binary-size`, + `security-audit`), plus +- `parity` — must clear the threshold in `test-parity/threshold.json` and add no + new / stale known-failure entries +- `compile-smoke` — must compile every file under `test-files/`, plus the UI + styling matrix, Fastify integration and memory-stability tests +- the gap suite in its 8-shard **auto-optimize** mode (the PR/sweep tiers use the + prebuilt-runtime `fast` mode) +- `doc-tests` (macOS + Windows) — must compile + run every example under + `docs/examples/` +- the package smokes (`drizzle-mysql-smoke`, `ink-link-smoke`, `effect-basic-smoke`) + and `native-abi-evidence-packet` - Benchmark regressions in `benchmark.yml` hard-fail on release tags (warn only on main-branch pushes) +None of these carries `continue-on-error` any more: a red suite in the full tier +blocks the release. If a suite is red for a reason that is not the release +candidate's fault, fix it on `main` first (or open an issue and consciously +re-add a job-level `continue-on-error: true` with that issue number) — do not +publish past it. + ## 4a. What tells you a release is overdue Nothing in the sections above fires if a release simply never happens. That is diff --git a/docs/src/testing/ci-gate-scheduling.md b/docs/src/testing/ci-gate-scheduling.md index b7404df9e7..c91b24bcee 100644 --- a/docs/src/testing/ci-gate-scheduling.md +++ b/docs/src/testing/ci-gate-scheduling.md @@ -161,6 +161,14 @@ on this slice**, which is what lets the remaining queue drain. reports *no* status rather than a passing one, which can wedge a required context. Not worth the risk here. +> **Superseded for the PR arm (2026-08-16, CI tiers).** The PR arm of every gate +> in the table above is now **opt-in via the `run-extended-tests` label** — an +> unlabelled PR still gets a run, but every job in it is skipped at the job level +> (`if:`), which costs no runner slot and cannot wedge anything because none of +> these is a required context (the only required context is `test.yml`'s +> `pr-gate`). The six-hourly `main` sweeps, the tag arms and `gate-freshness` are +> unchanged. Rationale and the measured numbers: [CI tiers](ci-tiers.md). + ### The cost, stated plainly **Attribution latency.** A regression that slips past the PR arm used to be pinned diff --git a/docs/src/testing/ci-tiers.md b/docs/src/testing/ci-tiers.md new file mode 100644 index 0000000000..d7d1c830c9 --- /dev/null +++ b/docs/src/testing/ci-tiers.md @@ -0,0 +1,158 @@ +# CI tiers: what runs on a PR, on a merge, and before a release + +Perry's CI is one workflow — `.github/workflows/test.yml` — with **three tiers**. +The tier a run belongs to, and the exact set of jobs it executes, is decided by +one script, `scripts/ci_plan.py`, in the workflow's first job (`plan`). Every other +job is `needs: plan` and gated on `fromJSON(needs.plan.outputs.plan).jobs.`. + +``` +python3 scripts/ci_plan.py --table # the job x tier matrix below +python3 scripts/ci_plan.py --self-test # the policy's own invariants +``` + +| tier | trigger | what it is for | fan-in job | +|---|---|---|---| +| **pr** | every `pull_request` push | the required gate. Small, fast, must be green on `main`. | `pr-gate` — **the only required status context** | +| **sweep** | every `push` to `main`, coalesced | post-merge truth for `main`: the PR tier unscoped plus the medium-weight jobs that do not fit the PR budget | `main-gate` | +| **full** | nightly `schedule`, `v*` tags, `workflow_dispatch`, PRs labelled `run-extended-tests` | everything, incl. parity, compile-smoke, doc-tests, package smokes, the 8-shard auto-optimize gap suite | `full-suite-gate` — what `release-packages.yml` waits for | + +## The job × tier matrix + +Generated by `python3 scripts/ci_plan.py --table`; the `lint` job checks that this +copy is current. + +| job | pr | sweep | full | +|---|:-:|:-:|:-:| +| `lint` | always | yes | yes | +| `check` | yes | yes | yes | +| `warnings` | yes | yes | yes | +| `cargo-test` | yes | yes | yes | +| `gap-suite` | 4x fast | 2x fast | 8x full | +| `gc-stress` | yes | yes | yes | +| `e2e-scoped` | yes | | | +| `security-audit` | deps only | yes | yes | +| `windows-build` | | yes | yes | +| `windows-arm64-build` | | yes | yes | +| `compiler-output-regression` | | yes | yes | +| `repsel-census` | | yes | yes | +| `harmonyos-smoke` | | yes | yes | +| `binary-size` | | yes | yes | +| `parity` | | | yes | +| `compile-smoke` | | | yes | +| `native-abi-evidence-packet` | | | yes | +| `drizzle-mysql-smoke` | | | yes | +| `ink-link-smoke` | | | yes | +| `effect-basic-smoke` | | | yes | +| `doc-tests` | | | yes | + +Within the **pr** tier the changed-file list narrows the plan further: + +- **docs-only** (only `docs/`, `*.md`, `benchmarks/`, `npm/`, `packaging/`, + `.claude/`, … — see `NON_CORE_GLOBS` in `ci_plan.py`) → only `lint` runs. +- **core** (anything that can change the compiler, the runtime, or a test outcome) + → the whole PR tier. +- **deps** (a lockfile, manifest, `deny.toml`, `package.json`, `.claude/`, `skills/`, + …) → additionally the `security-audit` reusable workflow. + +An empty or failed file listing is treated as **core** and a failed `plan` job fails +the gate outright — a broken planner must never turn into "everything skipped, +therefore green". + +## Why it looks like this (measured 2026-08-16) + +The organisation runs on GitHub's Free plan: **20 concurrent hosted jobs, 5 of them +macOS**, for the whole org. Before this shape, every PR push fanned out to **14 +workflows / 48 jobs / ~650 runner-minutes**, and `main` saw ~66 PR pushes and ~58 +merges a day. That is 1.5–2× total capacity, so: + +- job queue waits were 3–7 h (`conformance-smoke` shards: median 4 h); +- **0 of 66** PR runs of the `Tests` workflow reached a conclusion in the sample + window — 8 failed, 56 were cancelled by the next push; +- the required contexts included two jobs (`parity`, `compile-smoke`) that never ran + on a PR and one (`conformance-smoke-complete`) that had been red on `main` for + days, so **every one of the last 12 merges was an admin bypass** with `lint` and + `cargo-test` still queued. + +Two specific costs dominated: + +- `conformance-smoke` was 8 shards × ~60 min = **480 job-minutes per push**, and 96 % + of each shard's test time was ~10 tests at ~200 s each — the auto-optimize path + rebuilding a feature-stripped runtime per distinct feature set, redundantly in + every shard. The harness's `PERRY_SKIP_BUILD=1` mode runs the same tests against + one prebuilt release compiler at ~1.5 s each; that is the `fast` gap mode the PR + and sweep tiers use. The 8-shard auto-optimize mode is kept in the full tier + because it is the only arm that sees auto-optimize-only link bugs. +- Every job saved a fresh ~0.5–1.3 GB sccache tarball on every PR push (~200 GB/day + into a 10 GB repo cache budget), evicting every useful entry within the hour — + and PR-scoped caches are not even readable by other PRs. sccache now saves only + from main-line runs; PRs restore the newest main-line blob. `cache-warm.yml` is + gone: the sweep is the cache-producing build on `main`. + +The **satellite gates** (`gc-ratchet`, `gc-root-dominance`, `gc-native-roots`, +`gc-moving-witnesses`, `gc-parse-churn-gate`, `gc-ptr-shape-off-witness`, +`tls-budget`, `auto-opt-app-patterns`, `eh-transport`, `llvm-inprocess`, `ext-link`, +`container-tests`) keep their six-hourly / nightly main-line sweeps and their +release-tag arms (see [CI gate scheduling](ci-gate-scheduling.md) and +`scripts/gate_freshness.json`), but their **PR arm is opt-in**: apply the +`run-extended-tests` label to a PR and they run on it, along with the `full` tier +of `test.yml`. An unlabelled PR still gets a run of each — with every job skipped, +which costs no runner slot. + +## Budgets + +| tier | typical jobs | typical runner-minutes | target wall clock | +|---|--:|--:|--:| +| pr (core) | ~11 (4 gap shards) | ~140 | ≤ 30 min once queued | +| pr (docs-only) | 2 | ~5 | ≤ 5 min | +| sweep | ~19 | ~240 | not a target — it coalesces | +| full | ~30 | ~1500 | not a target | + +`push` to `main` uses ONE constant concurrency group with `cancel-in-progress: false`: +GitHub keeps at most one running + one pending run per group and replaces the pending +run with the newest, so a burst of merges is tested at its tip instead of queueing 58 +sweeps. Sweep-only jobs are chained behind `check` so a sweep's fan-out does not take +every runner slot the moment a merge lands. Attribution of a sweep failure is by +window (`previous sweep SHA .. this sweep SHA`), exactly as for the six-hourly gates. + +## Opting a PR into more + +- **`run-extended-tests` label** — promotes the PR's `test.yml` run to the `full` + tier AND enables the PR arm of every satellite gate. Use it for GC / codegen + changes that should be measured before merge, and for anything touching a + full-tier-only suite. Applying the label re-fires the runs (`labeled` trigger). +- **`skip-changelog` label** — skips the changeset step in `lint` (a `crates/` + change must otherwise add a `changelog.d/-.md` fragment). +- **`workflow_dispatch`** on any branch: `tier` (`pr` / `sweep` / `full`) and + `update_gap_snapshot`. + +## Re-baselining the gap snapshot + +`test-parity/gap_snapshot.json` is a both-direction snapshot: any divergence — a +test that started failing OR started passing — is red. When `main` has legitimately +moved (a triaged regression, an oracle change), regenerate it **from CI**, not from +a laptop (the required baseline is Linux, `fast` mode): + +```bash +gh workflow run test.yml --ref -f tier=pr -f update_gap_snapshot=true +# wait for the run, then: +gh run download -n gap-snapshot-update -D /tmp/snap +cp /tmp/snap/gap_snapshot.json test-parity/gap_snapshot.json +# fill in `issue` / `reason` for every new entry, then commit +``` + +New entries land as `category: untriaged`; give each an issue and a reason before +merging. A crash may never be parked in the snapshot (`run_gap_tests.sh` refuses). + +## Branch protection + +Required status checks on `main`: **`pr-gate`** only. Adding, removing or renaming a +job in `test.yml` never needs a branch-protection edit again; the fan-in job carries +the verdict. `gate-freshness.yml` (`scripts/gate_freshness.json`) still watches +that each main-line sweep produces a *successful* run within its budget. + +## Release + +`release-packages.yml`'s `await-tests` dispatches `test.yml` with `tier=full` on the +pinned release branch and polls the SHA for a run whose **`full-suite-gate`** job +succeeded — a green sweep or PR-tier run on the same SHA does not count. See +[Releasing](../contributing/releasing.md). diff --git a/scripts/ci_plan.py b/scripts/ci_plan.py new file mode 100755 index 0000000000..8f7a4b2db0 --- /dev/null +++ b/scripts/ci_plan.py @@ -0,0 +1,446 @@ +#!/usr/bin/env python3 +"""Decide which CI jobs a run of `.github/workflows/test.yml` executes. + +WHY THIS EXISTS +--------------- +`test.yml` used to answer "does job X run in this event?" with a different +`if:` on each of 25 jobs — event names, labels, refs and a dispatch input, all +spelled slightly differently. Two required contexts (`parity`, `compile-smoke`) +never ran on a pull request at all, a third (`conformance-smoke-complete`) had +been red on `main` for days, and every PR fanned out to 48 jobs against an +org-wide cap of 20 concurrent runners. Measured 2026-08-16: 0 of 66 PR runs of +`Tests` reached a conclusion; the last 12 merges all bypassed branch protection +with `lint`/`cargo-test` still queued. + +This script is the ONE place that policy lives. The workflow's `plan` job runs +it and every other job is `needs: plan` + `if: fromJSON(plan).jobs.`. The +policy is therefore testable (`--self-test`) and printable (`--table`), and a +change to "what gates a PR" is a diff to this file, not an archaeology of +`if:` blocks. + +THE THREE TIERS +--------------- + pr every pull_request push. Small, fast, deterministic. Everything in it + must be green on `main`, because its fan-in (`pr-gate`) is the ONLY + required status context. Budget: <= ~12 jobs, <= ~30 min wall. + sweep every push to `main`, coalesced (at most one running + one pending, so + a burst of merges is tested at its tip). The PR tier unscoped, plus the + medium-weight jobs that do not fit the PR budget (Windows builds, + compiler-output gates, the full GC x repsel matrix, ...). + full nightly, release tags, `workflow_dispatch` (the release pipeline's + `await-tests` dispatches this and waits for the `full-suite-gate` job), + and PRs carrying the `run-extended-tests` label. The sweep plus the + slow/opt-in suites (parity, compile-smoke, doc-tests, package smokes, + the gap suite in its 8-shard auto-optimize mode). + +PR SCOPE +-------- +Within the `pr` tier the changed-file list further narrows the plan: + docs-only -> lint only (fmt/markdown/audits still run; nothing compiles) + core -> anything that can change the compiler, runtime, or the tests + themselves; runs the whole PR tier + deps -> a lockfile / manifest / policy change; additionally runs the + security-audit workflow (cargo-audit, cargo-deny, soak gate, + agent + skills scans) + +A job that is OFF in the plan is `skipped` in Actions. A skipped job counts as +satisfied for a required status check, and `pr-gate` (the fan-in) checks +`needs.plan.result == 'success'` explicitly, so a broken plan step fails the PR +instead of silently skipping everything. + +Usage: + scripts/ci_plan.py --event pull_request --ref refs/pull/1/merge \ + --changed-files files.txt [--labels a,b] # -> JSON on stdout + scripts/ci_plan.py --event push --ref refs/heads/main + scripts/ci_plan.py --event workflow_dispatch --ref refs/heads/x --tier full + scripts/ci_plan.py --table # markdown tier table + scripts/ci_plan.py --self-test +When GITHUB_OUTPUT is set the JSON is also written there as `plan=`. +""" + +from __future__ import annotations + +import argparse +import fnmatch +import json +import os +import sys + +# --------------------------------------------------------------------------- +# Jobs, in workflow order. Each entry: (job id in test.yml, tiers it runs in). +# `pr` here means "runs in the PR tier when the PR is in `core` scope"; +# `lint` is the one job that also runs for docs-only PRs (see plan()). +# --------------------------------------------------------------------------- +JOBS: dict[str, tuple[str, ...]] = { + "lint": ("pr", "sweep", "full"), + "check": ("pr", "sweep", "full"), + "warnings": ("pr", "sweep", "full"), + "cargo_test": ("pr", "sweep", "full"), + "gap_suite": ("pr", "sweep", "full"), + "gc_stress": ("pr", "sweep", "full"), + "e2e_scoped": ("pr",), # scoped to the PR's diff; meaningless without one + "security_audit": ("pr", "sweep", "full"), # pr: only when `deps` changed + "windows_build": ("sweep", "full"), + "windows_arm64_build": ("sweep", "full"), + "compiler_output_regression": ("sweep", "full"), + "repsel_census": ("sweep", "full"), + "harmonyos_smoke": ("sweep", "full"), + "binary_size": ("sweep", "full"), + "parity": ("full",), + "compile_smoke": ("full",), + "native_abi_evidence_packet": ("full",), + "drizzle_mysql_smoke": ("full",), + "ink_link_smoke": ("full",), + "effect_basic_smoke": ("full",), + "doc_tests": ("full",), +} + +TIERS = ("pr", "sweep", "full") + +# The gap suite: shard count and harness mode per tier. +# fast = PERRY_SKIP_BUILD=1 against one prebuilt release compiler + runtime +# archives; ~1.5 s/test. Only ext-routed tests (http/net/ws/zlib/ +# events) still take the per-test auto-optimize path. +# full = the harness's default: every test compiles through auto-optimize, +# which rebuilds a feature-stripped runtime per distinct feature set +# (~200 s each, redundantly per shard). ~40 min/shard. This is the +# arm that sees auto-optimize-only link bugs, so it stays in the +# nightly/release tier. +GAP_SUITE = { + "pr": {"mode": "fast", "total": 4}, + "sweep": {"mode": "fast", "total": 2}, + "full": {"mode": "full", "total": 8}, +} + +EXTENDED_LABEL = "run-extended-tests" + +# --------------------------------------------------------------------------- +# PR scope classification. +# --------------------------------------------------------------------------- +# Paths that cannot change the compiler, the runtime, or any test outcome. +# A PR touching ONLY these is docs-only. Anchored globs; `**` crosses `/`. +NON_CORE_GLOBS = ( + "docs/**", + "*.md", + "**/*.md", + "LICENSE*", + ".gitignore", + ".gitattributes", + ".editorconfig", + "gc-handoff/**", + "benchmarks/**", + "homebrew/**", + "packaging/**", + "npm/**", + ".claude/**", + "skills/**", + ".github/**", # re-included below: the gate's own file and shared actions + "test-compat/**", # node-core corpus, driven by its own scheduled workflow + "web/**", + "www/**", +) +# Exceptions to NON_CORE_GLOBS: these ARE core even though the glob above +# would exclude them. +CORE_OVERRIDES = ( + ".github/workflows/test.yml", + ".github/workflows/security-audit.yml", + ".github/actions/**", + "docs/api/**", # generated API docs are checked for drift by `check` + "docs/src/api/**", + "CLAUDE.md", # lint's doc-claim audits read it +) + +# A change here can alter the dependency graph or the supply-chain policy, +# so the security-audit workflow (cargo-audit / cargo-deny / soak gate / +# agent + skills scans) joins the PR tier. +DEPS_GLOBS = ( + "Cargo.lock", + "Cargo.toml", + "**/Cargo.toml", + "deny.toml", + "rust-toolchain", + "rust-toolchain.toml", + "package.json", + "package-lock.json", + ".npmrc", + "external-tools.json", + ".github/dependabot.yml", + ".github/workflows/security-audit.yml", + "scripts/soak/**", + "npm/**", + ".claude/**", + "skills/**", +) + + +def _match(path: str, globs: tuple[str, ...]) -> bool: + # fnmatch's `*` matches `/` too, so `docs/**` covers every depth and + # `*.md` already matches `crates/x/README.md`; the `**/` spellings are + # kept for readers used to gitignore semantics. + return any(fnmatch.fnmatchcase(path, g) for g in globs) + + +def is_core(path: str) -> bool: + if _match(path, CORE_OVERRIDES): + return True + return not _match(path, NON_CORE_GLOBS) + + +def classify(changed: list[str]) -> dict[str, bool]: + changed = [p.strip() for p in changed if p.strip()] + core = any(is_core(p) for p in changed) + deps = any(_match(p, DEPS_GLOBS) for p in changed) + return { + "docs_only": bool(changed) and not core, + "core": core, + "deps": deps, + # An empty list means the file listing failed or the PR is empty. + # Treat it as core: silently skipping the whole tier is the failure + # mode this file exists to prevent. + "unknown": not changed, + } + + +# --------------------------------------------------------------------------- +# Tier derivation. +# --------------------------------------------------------------------------- +def derive_tier(event: str, ref: str, labels: list[str], tier_input: str | None) -> str: + if event == "pull_request": + return "full" if EXTENDED_LABEL in labels else "pr" + if event == "push": + if ref == "refs/heads/main": + return "sweep" + if ref.startswith("refs/tags/"): + return "full" + # A push to any other branch does not trigger the workflow (see + # `on.push.branches`), but be explicit if one ever does. + return "sweep" + if event == "schedule": + return "full" + if event == "workflow_dispatch": + return tier_input or "full" + raise SystemExit(f"ci_plan: unsupported event {event!r}") + + +def plan( + event: str, + ref: str, + labels: list[str] | None = None, + changed: list[str] | None = None, + tier_input: str | None = None, + update_gap_snapshot: bool = False, +) -> dict: + labels = labels or [] + tier = derive_tier(event, ref, labels, tier_input) + if event == "pull_request": + scope = classify(changed or []) + if scope["unknown"]: + scope["core"] = True + else: + scope = {"docs_only": False, "core": True, "deps": True, "unknown": False} + + jobs: dict[str, bool] = {} + for job, tiers in JOBS.items(): + on = tier in tiers + if tier == "pr": + if job == "lint": + on = True + elif job == "security_audit": + on = scope["deps"] + else: + on = on and scope["core"] + jobs[job] = on + + gap = dict(GAP_SUITE[tier]) + if update_gap_snapshot: + # Re-baselining needs the WHOLE suite in one report so the snapshot + # is written from a single consistent run. Fast mode: that is the arm + # the PR gate measures against. + gap = {"mode": "fast", "total": 1} + gap["shards"] = list(range(1, gap["total"] + 1)) + gap["update_snapshot"] = bool(update_gap_snapshot) + + return { + "tier": tier, + "event": event, + "scope": scope, + "jobs": jobs, + "gap": gap, + # cargo-test: the PR tier scopes to the diff via ci_test_scope.py; the + # other tiers run the full workspace. + "cargo_test_scope": "pr" if tier == "pr" else "full", + # gc-stress: the PR subset of the GC x repsel matrix vs the full one. + "gc_stress_mode": "pr" if tier == "pr" else "full", + } + + +# --------------------------------------------------------------------------- +# Presentation. +# --------------------------------------------------------------------------- +def table() -> str: + lines = ["| job | pr | sweep | full |", "|---|:-:|:-:|:-:|"] + for job, tiers in JOBS.items(): + cells = [] + for t in TIERS: + mark = "yes" if t in tiers else "" + if t == "pr" and job == "security_audit" and mark: + mark = "deps only" + if t == "pr" and job == "lint": + mark = "always" + if job == "gap_suite" and mark: + g = GAP_SUITE[t] + mark = f"{g['total']}x {g['mode']}" + cells.append(mark) + lines.append(f"| `{job.replace('_', '-')}` | " + " | ".join(cells) + " |") + return "\n".join(lines) + + +# --------------------------------------------------------------------------- +# Self-test: the policy must be able to fail, and the invariants CLAUDE.md +# cares about must hold. +# --------------------------------------------------------------------------- +def _self_test() -> int: + failures: list[str] = [] + + def check(name: str, cond: bool): + if not cond: + failures.append(name) + + # 1. Tiers. + check("PR is pr", derive_tier("pull_request", "refs/pull/1/merge", [], None) == "pr") + check("labelled PR is full", derive_tier("pull_request", "refs/pull/1/merge", [EXTENDED_LABEL], None) == "full") + check("push main is sweep", derive_tier("push", "refs/heads/main", [], None) == "sweep") + check("tag is full", derive_tier("push", "refs/tags/v0.5.9999", [], None) == "full") + check("schedule is full", derive_tier("schedule", "refs/heads/main", [], None) == "full") + check("dispatch default is full", derive_tier("workflow_dispatch", "refs/heads/x", [], None) == "full") + check("dispatch tier honoured", derive_tier("workflow_dispatch", "refs/heads/x", [], "sweep") == "sweep") + + # 2. PR scope. + docs = plan("pull_request", "refs/pull/1/merge", changed=["docs/src/foo.md", "README.md"]) + check("docs-only PR: lint on", docs["jobs"]["lint"]) + check("docs-only PR: gap suite off", not docs["jobs"]["gap_suite"]) + check("docs-only PR: cargo-test off", not docs["jobs"]["cargo_test"]) + check("docs-only PR: security-audit off", not docs["jobs"]["security_audit"]) + check("docs-only PR flagged", docs["scope"]["docs_only"]) + + core = plan("pull_request", "refs/pull/1/merge", changed=["crates/perry-runtime/src/gc/mod.rs"]) + check("core PR: gap suite on", core["jobs"]["gap_suite"]) + check("core PR: gc-stress on", core["jobs"]["gc_stress"]) + check("core PR: e2e-scoped on", core["jobs"]["e2e_scoped"]) + check("core PR: windows off", not core["jobs"]["windows_build"]) + check("core PR: parity off", not core["jobs"]["parity"]) + check("core PR: security-audit off (no deps change)", not core["jobs"]["security_audit"]) + check("core PR: 4 fast gap shards", core["gap"] == {"mode": "fast", "total": 4, "shards": [1, 2, 3, 4], "update_snapshot": False}) + check("core PR: cargo-test scoped", core["cargo_test_scope"] == "pr") + + deps = plan("pull_request", "refs/pull/1/merge", changed=["Cargo.lock"]) + check("deps PR: security-audit on", deps["jobs"]["security_audit"]) + check("deps PR: core (lockfile changes the build)", deps["jobs"]["cargo_test"]) + + empty = plan("pull_request", "refs/pull/1/merge", changed=[]) + check("empty listing is treated as core", empty["jobs"]["gap_suite"] and empty["scope"]["unknown"]) + + # The gate's own wiring and the shared actions are core even though the + # rest of .github/ is not. + check("test.yml change is core", is_core(".github/workflows/test.yml")) + check("shared action change is core", is_core(".github/actions/setup-llvm22/action.yml")) + check("other workflow change is not core", not is_core(".github/workflows/benchmark.yml")) + check("nested .md is not core", not is_core("crates/perry/README.md")) + check("CLAUDE.md is core (lint doc-claim audits)", is_core("CLAUDE.md")) + check("a .ts test file is core", is_core("test-files/test_gap_x.ts")) + check("scripts are core", is_core("scripts/run_gap_tests.sh")) + check("generated api docs are core", is_core("docs/api/perry.d.ts")) + + # 3. Sweep and full. + sweep = plan("push", "refs/heads/main") + check("sweep: windows on", sweep["jobs"]["windows_build"] and sweep["jobs"]["windows_arm64_build"]) + check("sweep: parity off", not sweep["jobs"]["parity"]) + check("sweep: e2e-scoped off", not sweep["jobs"]["e2e_scoped"]) + check("sweep: 2 fast gap shards", sweep["gap"]["total"] == 2 and sweep["gap"]["mode"] == "fast") + check("sweep: cargo-test full", sweep["cargo_test_scope"] == "full") + check("sweep: security-audit on", sweep["jobs"]["security_audit"]) + + full = plan("schedule", "refs/heads/main") + check("full: every job except e2e-scoped", all(v for k, v in full["jobs"].items() if k != "e2e_scoped")) + check("full: 8 auto-optimize gap shards", full["gap"]["total"] == 8 and full["gap"]["mode"] == "full") + + labelled = plan("pull_request", "refs/pull/1/merge", labels=[EXTENDED_LABEL], changed=["README.md"]) + check("labelled PR runs the full tier regardless of scope", labelled["jobs"]["parity"] and labelled["jobs"]["gap_suite"]) + + snap = plan("workflow_dispatch", "refs/heads/x", tier_input="pr", update_gap_snapshot=True) + check("snapshot update: one fast shard", snap["gap"] == {"mode": "fast", "total": 1, "shards": [1], "update_snapshot": True}) + + # 4. The gc_gate_wiring_check contract: `gc-stress` is a registered + # moving-GC gate and MUST be main-line reachable (push:main or + # schedule). The checker cannot see through fromJSON(plan), so this is + # where that guarantee lives. + check("gc-stress reachable on push:main", sweep["jobs"]["gc_stress"] and sweep["gc_stress_mode"] == "full") + check("gc-stress reachable on schedule", full["jobs"]["gc_stress"] and full["gc_stress_mode"] == "full") + + # 5. Sabotage: the checker can fail. + check("sabotage: a job cannot be in no tier", all(JOBS.values())) + check("sabotage: unknown event raises", _raises(lambda: derive_tier("issue_comment", "x", [], None))) + + if failures: + print("ci_plan --self-test FAILED:", file=sys.stderr) + for f in failures: + print(f" - {f}", file=sys.stderr) + return 1 + print(f"ci_plan --self-test: {len(JOBS)} jobs, 3 tiers, all invariants hold") + return 0 + + +def _raises(fn) -> bool: + try: + fn() + except SystemExit: + return True + return False + + +def main(argv: list[str]) -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--event") + ap.add_argument("--ref", default="") + ap.add_argument("--labels", default="", help="comma-separated PR label names") + ap.add_argument("--changed-files", help="file with one changed path per line (PR only)") + ap.add_argument("--tier", choices=TIERS, help="workflow_dispatch tier input") + ap.add_argument("--update-gap-snapshot", action="store_true") + ap.add_argument("--table", action="store_true", help="print the tier table as markdown") + ap.add_argument("--self-test", action="store_true") + args = ap.parse_args(argv) + + if args.self_test: + return _self_test() + if args.table: + print(table()) + return 0 + if not args.event: + ap.error("--event is required") + + changed: list[str] = [] + if args.changed_files: + with open(args.changed_files, encoding="utf-8") as fh: + changed = [ln.rstrip("\n") for ln in fh if ln.strip()] + labels = [s.strip() for s in args.labels.split(",") if s.strip()] + + p = plan( + args.event, + args.ref, + labels=labels, + changed=changed, + tier_input=args.tier, + update_gap_snapshot=args.update_gap_snapshot, + ) + out = json.dumps(p, separators=(",", ":"), sort_keys=True) + print(json.dumps(p, indent=2, sort_keys=True)) + gh_out = os.environ.get("GITHUB_OUTPUT") + if gh_out: + with open(gh_out, "a", encoding="utf-8") as fh: + fh.write(f"plan={out}\n") + fh.write(f"tier={p['tier']}\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) From 263d6408b161d327114cfb01e1f9f07c2a77ef9d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sun, 16 Aug 2026 07:56:50 +0200 Subject: [PATCH 2/8] changelog: fragment for #8187 (CI tiers) --- changelog.d/8187-ci-tiers.md | 52 ++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 changelog.d/8187-ci-tiers.md diff --git a/changelog.d/8187-ci-tiers.md b/changelog.d/8187-ci-tiers.md new file mode 100644 index 0000000000..1c070dc881 --- /dev/null +++ b/changelog.d/8187-ci-tiers.md @@ -0,0 +1,52 @@ +### CI restructured into three tiers: PR gate / main sweep / full suite (#8187) + +**Why.** CI could not gate a merge. Measured 2026-08-16 over ~24 h of runs: the org +runs on 20 concurrent hosted jobs (5 macOS) and every PR push fanned out to 14 +workflows / 48 jobs / ~650 runner-minutes; at ~66 pushes and ~58 merges a day that +is 1.5–2× total capacity. Job queue waits were 3–7 h, **0 of 66** PR runs of `Tests` +reached a conclusion, and the last 12 merges all bypassed branch protection with +`lint`/`cargo-test` still queued, `conformance-smoke-complete` red on `main` +(#8117) and two required contexts (`parity`, `compile-smoke`) that never ran on a PR +at all. `conformance-smoke` alone was 480 job-minutes per push, 96 % of it the +auto-optimize path rebuilding a feature-stripped runtime per feature set, +redundantly in every shard; sccache wrote ~200 GB/day of PR-scoped tarballs (which +other PRs cannot even read) into a 10 GB cache budget; `cache-warm.yml` had not +completed since 07-28. + +**What.** One workflow (`test.yml`), one policy file (`scripts/ci_plan.py`, +self-tested; `--table` prints the job × tier matrix and `lint` keeps the docs copy +current), one fan-in status per tier: + +- **pr** (every PR push): `lint` (now also the changeset step), `check` (clippy ×2 + + api-docs-drift), `warnings`, scoped `cargo-test`, the gap suite in **4 shards of + the harness's `fast` mode** (`PERRY_SKIP_BUILD=1`, one prebuilt release compiler, + ~1.5 s/test), `gc-stress` (PR subset), `e2e-scoped`, and `security-audit` only when + a lockfile/manifest/policy file changed. Docs-only PRs run `lint` only. Fan-in + **`pr-gate` is the single required status context** — adding/removing a job never + needs a branch-protection edit again, and a docs-only PR still gets a verdict (no + more `paths-ignore` wedge). +- **sweep** (every push to `main`, coalesced via a constant concurrency group with + `cancel-in-progress: false`): the PR tier unscoped plus Windows x64/ARM64 builds, + full `gc-stress`, `compiler-output-regression`, `repsel-census`, `harmonyos-smoke`, + `binary-size`, `security-audit`. Sweep-only jobs chain behind `check` so a merge + does not take every runner slot. Fan-in `main-gate`. This is also the + cache-producing build on `main` (rust-cache + sccache save here), replacing + `cache-warm.yml`. +- **full** (nightly, `v*` tags, `workflow_dispatch`, PRs labelled `run-extended-tests`): + the sweep plus `parity`, `compile-smoke`, the 8-shard auto-optimize gap suite, + `doc-tests`, the drizzle/ink/effect smokes and `native-abi-evidence-packet` — + **all without `continue-on-error` now**. Fan-in `full-suite-gate`; + `release-packages.yml` dispatches `tier=full` and accepts only a run whose + `full-suite-gate` succeeded on the release SHA. + +The twelve satellite gates (`gc-*`, `tls-budget`, `auto-opt-app-patterns`, +`eh-transport`, `llvm-inprocess`, `ext-link`, `container-tests`) keep their +six-hourly / nightly `main` sweeps and tag arms; their PR arm is now **opt-in via +the `run-extended-tests` label** (job-level `if:`, so an unlabelled PR costs no +runner slot). `ext-link` gains a nightly arm (it had no main-line arm); `container- +tests` drops its `push: main` arm. `security-audit.yml` becomes `workflow_call` + +weekly. sccache saves only from main-line runs; PRs restore the newest main-line +blob. + +**Follow-up (admin, after merge):** set the required status checks on `main` to +`pr-gate` only. Full page: `docs/src/testing/ci-tiers.md`. From ca9d92d18c39b91f170d2806cf12e94b72d15e19 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sun, 16 Aug 2026 07:57:53 +0200 Subject: [PATCH 3/8] =?UTF-8?q?ci:=20binary-size=20is=20report-only=20on?= =?UTF-8?q?=20macOS=20=E2=80=94=20full=20tier=20only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- changelog.d/8187-ci-tiers.md | 5 +++-- docs/src/contributing/releasing.md | 4 ++-- docs/src/testing/ci-tiers.md | 2 +- scripts/ci_plan.py | 3 ++- 4 files changed, 8 insertions(+), 6 deletions(-) diff --git a/changelog.d/8187-ci-tiers.md b/changelog.d/8187-ci-tiers.md index 1c070dc881..a717ed6213 100644 --- a/changelog.d/8187-ci-tiers.md +++ b/changelog.d/8187-ci-tiers.md @@ -28,13 +28,14 @@ current), one fan-in status per tier: - **sweep** (every push to `main`, coalesced via a constant concurrency group with `cancel-in-progress: false`): the PR tier unscoped plus Windows x64/ARM64 builds, full `gc-stress`, `compiler-output-regression`, `repsel-census`, `harmonyos-smoke`, - `binary-size`, `security-audit`. Sweep-only jobs chain behind `check` so a merge + `security-audit`. Sweep-only jobs chain behind `check` so a merge does not take every runner slot. Fan-in `main-gate`. This is also the cache-producing build on `main` (rust-cache + sccache save here), replacing `cache-warm.yml`. - **full** (nightly, `v*` tags, `workflow_dispatch`, PRs labelled `run-extended-tests`): the sweep plus `parity`, `compile-smoke`, the 8-shard auto-optimize gap suite, - `doc-tests`, the drizzle/ink/effect smokes and `native-abi-evidence-packet` — + `doc-tests`, `binary-size`, the drizzle/ink/effect smokes and + `native-abi-evidence-packet` — **all without `continue-on-error` now**. Fan-in `full-suite-gate`; `release-packages.yml` dispatches `tier=full` and accepts only a run whose `full-suite-gate` succeeded on the release SHA. diff --git a/docs/src/contributing/releasing.md b/docs/src/contributing/releasing.md index 790e565372..d9e8162767 100644 --- a/docs/src/contributing/releasing.md +++ b/docs/src/contributing/releasing.md @@ -128,8 +128,8 @@ full tier is: - everything the PR gate and the post-merge sweep run (`lint`, `check`, `warnings`, `cargo test --workspace`, the gap suite, `gc-stress`, Windows x64 + ARM64 builds, - compiler-output gates, `repsel-census`, `harmonyos-smoke`, `binary-size`, - `security-audit`), plus + compiler-output gates, `repsel-census`, `harmonyos-smoke`, `security-audit`), + plus `binary-size` and - `parity` — must clear the threshold in `test-parity/threshold.json` and add no new / stale known-failure entries - `compile-smoke` — must compile every file under `test-files/`, plus the UI diff --git a/docs/src/testing/ci-tiers.md b/docs/src/testing/ci-tiers.md index d7d1c830c9..2656f0eaa6 100644 --- a/docs/src/testing/ci-tiers.md +++ b/docs/src/testing/ci-tiers.md @@ -36,7 +36,7 @@ copy is current. | `compiler-output-regression` | | yes | yes | | `repsel-census` | | yes | yes | | `harmonyos-smoke` | | yes | yes | -| `binary-size` | | yes | yes | +| `binary-size` | | | yes | | `parity` | | | yes | | `compile-smoke` | | | yes | | `native-abi-evidence-packet` | | | yes | diff --git a/scripts/ci_plan.py b/scripts/ci_plan.py index 8f7a4b2db0..a26cf79596 100755 --- a/scripts/ci_plan.py +++ b/scripts/ci_plan.py @@ -85,7 +85,7 @@ "compiler_output_regression": ("sweep", "full"), "repsel_census": ("sweep", "full"), "harmonyos_smoke": ("sweep", "full"), - "binary_size": ("sweep", "full"), + "binary_size": ("full",), # report-only, macOS: not worth a scarce mac slot per merge "parity": ("full",), "compile_smoke": ("full",), "native_abi_evidence_packet": ("full",), @@ -354,6 +354,7 @@ def check(name: str, cond: bool): # 3. Sweep and full. sweep = plan("push", "refs/heads/main") check("sweep: windows on", sweep["jobs"]["windows_build"] and sweep["jobs"]["windows_arm64_build"]) + check("sweep: binary-size off (macOS report-only, nightly is enough)", not sweep["jobs"]["binary_size"]) check("sweep: parity off", not sweep["jobs"]["parity"]) check("sweep: e2e-scoped off", not sweep["jobs"]["e2e_scoped"]) check("sweep: 2 fast gap shards", sweep["gap"]["total"] == 2 and sweep["gap"]["mode"] == "fast") From e9116afe72acac91b3141fb4c557509f4a455fa2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sun, 16 Aug 2026 07:59:19 +0200 Subject: [PATCH 4/8] docs: point CI-authority references at the tiered gate --- .github/workflows/test.yml | 24 ++++++++++++++---------- docs/src/internals/garbage-collector.md | 17 +++++++++-------- test-parity/README.md | 4 +++- 3 files changed, 26 insertions(+), 19 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index fd2cae8d49..6e0459c386 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -2085,7 +2085,10 @@ jobs: key: sccache-${{ runner.os }}-perry-native-abi-evidence-${{ github.run_id }} # --------------------------------------------------------------------------- - # Gap-suite conformance smoke (every PR + main push) + # gap-suite (was `conformance-smoke`): the gap suite, sharded. Runs in every + # tier -- 4 fast-mode shards on a PR, 2 in the sweep, 8 auto-optimize shards + # in the full tier (scripts/ci_plan.py GAP_SUITE). The `gate` fan-in below is + # what branch protection requires; a single shard's red bubbles up through it. # --------------------------------------------------------------------------- gap-suite: needs: plan @@ -2110,13 +2113,14 @@ jobs: # in .node-version so it can't drift out from under the suite again; raise # it deliberately, and re-measure the delta when you do (see #6364). # - # Sharded across 8 parallel runners (--shard N/8) so the ~258-test suite - # runs in ~1/8 the wall-time — the 258 sequential Perry compiles dominate - # the cost and split evenly, taking the single-job ~57 min down to ~15 min - # per shard. The fan-in job `conformance-smoke-complete` below is the ONE - # status branch protection requires; a single shard's red bubbles up - # through it. Each shard's no-new-untriaged gate (run_gap_tests.sh) covers - # only its own slice, which is exactly the right per-shard semantics. + # Sharded (--shard N/M, M from the plan). Each shard's snapshot gate + # (run_gap_tests.sh) covers only its own slice, which is exactly the right + # per-shard semantics; the `gate` fan-in job is the one status branch + # protection requires. 2026-08-16: in the harness's default (auto-optimize) + # mode 96% of a shard's wall time was ~10 tests at ~200 s each -- the + # feature-stripped runtime rebuild per distinct feature set, redone in + # every shard. That mode is now the full tier's 8-shard arm; PR and sweep + # tiers use `fast` mode against one prebuilt release build (~1.5 s/test). strategy: fail-fast: false matrix: @@ -2127,8 +2131,8 @@ jobs: # shards 2-8 and shard 1 (its slice is the heaviest) hit 55:18 — the exact # 55-min cap — and CANCELLED on every run, incl. `gh run rerun --failed`. # On slow-runner days a second shard (observed: shard 3) also grazed 55. - # Because `conformance-smoke-complete` is a required context, that - # deterministic timeout flaky-red-blocked every PR (#6456). Bump to 75 for + # Because the fan-in was a required context, that deterministic timeout + # flaky-red-blocked every PR (#6456). Bump to 75 for # comfortable headroom (~35 min over the normal-day slowest, ~20 over the # heavy shard) while still bounding a genuine hang. Durable fix — raising # the shard count 8->12 so no single slice approaches the cap — tracked as diff --git a/docs/src/internals/garbage-collector.md b/docs/src/internals/garbage-collector.md index a9a39a43b7..95234ed6c9 100644 --- a/docs/src/internals/garbage-collector.md +++ b/docs/src/internals/garbage-collector.md @@ -223,17 +223,18 @@ neither can go quiet while this page keeps claiming the old thing. ## Validation and CI authority -As of 2026-08-11, branch protection requires `lint`, `cargo-test`, `parity`, -`compile-smoke`, `api-docs-drift`, `security-audit`, and -`conformance-smoke-complete`. The GC-specific coverage is split deliberately: +As of 2026-08-16, branch protection requires exactly one status, `pr-gate` — the +fan-in of `test.yml`'s PR tier (see `docs/src/testing/ci-tiers.md`; the tier +policy is `scripts/ci_plan.py`). The GC-specific coverage is split deliberately: | check | where it runs | required status | |---|---|---| -| root-holder custody, GC-knob drift, and this page's path/number claims | `test.yml` → `lint` | yes (`lint`) | -| runtime unit suite and `run_memory_stability_tests.sh` four-mode matrix | `test.yml` → `cargo-test` | yes (`cargo-test`) | -| emitted root dominance, including native statepoint IR | `gc-root-dominance.yml` | not currently branch-required | -| pinned collector counters/RSS/wall matrix | `gc-ratchet.yml` | not currently branch-required | -| thread-local mechanism/policy budget | `tls-budget.yml` | not currently branch-required | +| root-holder custody, GC-knob drift, and this page's path/number claims | `test.yml` → `lint` | yes (via `pr-gate`) | +| runtime unit suite and `run_memory_stability_tests.sh` four-mode matrix | `test.yml` → `cargo-test` | yes (via `pr-gate`; the PR tier is diff-scoped, the sweep/full tiers run the workspace) | +| GC × representation-selection matrix, rooting-bug instruments, write-barrier stress | `test.yml` → `gc-stress` | yes (via `pr-gate`; PR subset on PRs, full matrix in the sweep) | +| emitted root dominance, including native statepoint IR | `gc-root-dominance.yml` | not branch-required; PR arm opt-in via `run-extended-tests`, six-hourly on `main` | +| pinned collector counters/RSS/wall matrix | `gc-ratchet.yml` | not branch-required; PR arm opt-in via `run-extended-tests`, six-hourly on `main` | +| thread-local mechanism/policy budget | `tls-budget.yml` | not branch-required; PR arm opt-in via `run-extended-tests`, six-hourly on `main` | Useful local preflight commands: diff --git a/test-parity/README.md b/test-parity/README.md index 57ea605f35..a393cd5d79 100644 --- a/test-parity/README.md +++ b/test-parity/README.md @@ -6,7 +6,9 @@ differently; the difference is the point. ## `gap_snapshot.json` — the gap suite (per-PR gate) Generated by `scripts/gap_snapshot.py`, consumed by `scripts/run_gap_tests.sh`, -which is what the required `conformance-smoke` check runs on every PR. +which is what the `gap-suite` job of `test.yml` runs on every PR (4 shards, `fast` +mode), on every push to `main` and nightly (8 shards, auto-optimize mode) — see +`docs/src/testing/ci-tiers.md`, including how to re-baseline this file from CI. It lists every `test-files/test_gap_*.ts` that is **not passing**. A test absent from the file is expected to pass. The check is **bidirectional**: a new failure From dff0c99ce2ed4f28291d46205077e42d6a745346 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sun, 16 Aug 2026 07:59:47 +0200 Subject: [PATCH 5/8] ci_plan: e2e-scoped needs a pull_request event, not just the pr tier --- scripts/ci_plan.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/scripts/ci_plan.py b/scripts/ci_plan.py index a26cf79596..85023187c2 100755 --- a/scripts/ci_plan.py +++ b/scripts/ci_plan.py @@ -249,6 +249,11 @@ def plan( on = scope["deps"] else: on = on and scope["core"] + if job == "e2e_scoped": + # Reads the PR's file list via `gh pr view`; there is no PR on a + # `workflow_dispatch --tier pr`, so the job would fail on a + # missing PR number rather than skip. + on = on and event == "pull_request" jobs[job] = on gap = dict(GAP_SUITE[tier]) @@ -370,6 +375,7 @@ def check(name: str, cond: bool): snap = plan("workflow_dispatch", "refs/heads/x", tier_input="pr", update_gap_snapshot=True) check("snapshot update: one fast shard", snap["gap"] == {"mode": "fast", "total": 1, "shards": [1], "update_snapshot": True}) + check("dispatch --tier pr has no PR to scope e2e against", not snap["jobs"]["e2e_scoped"]) # 4. The gc_gate_wiring_check contract: `gc-stress` is a registered # moving-GC gate and MUST be main-line reachable (push:main or From 5c37d3e7d920bc5bbdfedab505941c5075807c3f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sun, 16 Aug 2026 08:42:57 +0200 Subject: [PATCH 6/8] ci: cargo-test target selection keys on the plan; scoped mode needs a pull_request event --- .github/workflows/test.yml | 2 +- scripts/ci_plan.py | 9 ++++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 6e0459c386..a78060501a 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -954,7 +954,7 @@ jobs: # flakes) and races the GC/threading tests into intermittent SIGSEGV. # Run perry-runtime single-threaded so the tests can't interfere. - if [ "${{ github.event_name }}" != "pull_request" ]; then + if [ "${{ fromJSON(needs.plan.outputs.plan).cargo_test_scope }}" != "pr" ]; then # ---- FULL run: release tags / nightly cron / workflow_dispatch ---- # Every target, including the slow auto-optimize integration tests. if printf '%s\n' "$scope" | grep -qx 'perry-runtime'; then diff --git a/scripts/ci_plan.py b/scripts/ci_plan.py index 85023187c2..a83521f2df 100755 --- a/scripts/ci_plan.py +++ b/scripts/ci_plan.py @@ -271,9 +271,11 @@ def plan( "scope": scope, "jobs": jobs, "gap": gap, - # cargo-test: the PR tier scopes to the diff via ci_test_scope.py; the - # other tiers run the full workspace. - "cargo_test_scope": "pr" if tier == "pr" else "full", + # cargo-test: a pull_request run scopes to the diff via ci_test_scope.py + # (`--lib --bins` of the affected crates); everything else -- including a + # `workflow_dispatch --tier pr`, which has no PR to read -- runs the full + # workspace with integration suites. + "cargo_test_scope": "pr" if event == "pull_request" else "full", # gc-stress: the PR subset of the GC x repsel matrix vs the full one. "gc_stress_mode": "pr" if tier == "pr" else "full", } @@ -376,6 +378,7 @@ def check(name: str, cond: bool): snap = plan("workflow_dispatch", "refs/heads/x", tier_input="pr", update_gap_snapshot=True) check("snapshot update: one fast shard", snap["gap"] == {"mode": "fast", "total": 1, "shards": [1], "update_snapshot": True}) check("dispatch --tier pr has no PR to scope e2e against", not snap["jobs"]["e2e_scoped"]) + check("dispatch --tier pr runs cargo-test unscoped (no PR to read)", snap["cargo_test_scope"] == "full") # 4. The gc_gate_wiring_check contract: `gc-stress` is a registered # moving-GC gate and MUST be main-line reachable (push:main or From fb3710dff6148aeda603b907584b9f124351f1f3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sun, 16 Aug 2026 10:09:08 +0200 Subject: [PATCH 7/8] ci: 6 PR gap shards (measured), 3 in the sweep Measured on the 4-shard PR run: the slowest shard took 39.5 min because it happened to receive 6 ext-routed tests, each of which triggers a ~4-5 min auto-optimize runtime rebuild for a feature set the shard had not seen. Six shards levels that at ~28 min, in line with gc-stress, for ~170 job-minutes (was 480 for 8 auto-optimize shards). The gap snapshot itself is main's (#8198 fixed the ten node_fail entries at the source -- fixtures + npm devDependencies -- and regenerated it; the CI re-baseline dispatch on this branch had reached the same 5 standing entries plus those ten, so nothing else moved). --- .github/workflows/test.yml | 2 +- CLAUDE.md | 2 +- CONTRIBUTING.md | 2 +- changelog.d/8187-ci-tiers.md | 2 +- docs/src/testing/ci-tiers.md | 6 +++--- scripts/ci_plan.py | 16 ++++++++++++---- test-parity/README.md | 2 +- 7 files changed, 20 insertions(+), 12 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index a78060501a..54f5824753 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -2086,7 +2086,7 @@ jobs: # --------------------------------------------------------------------------- # gap-suite (was `conformance-smoke`): the gap suite, sharded. Runs in every - # tier -- 4 fast-mode shards on a PR, 2 in the sweep, 8 auto-optimize shards + # tier -- 6 fast-mode shards on a PR, 3 in the sweep, 8 auto-optimize shards # in the full tier (scripts/ci_plan.py GAP_SUITE). The `gate` fan-in below is # what branch protection requires; a single shard's red bubbles up through it. # --------------------------------------------------------------------------- diff --git a/CLAUDE.md b/CLAUDE.md index 64102465f4..5e23d062ab 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -53,7 +53,7 @@ A `--module` selector scopes `--check`/`--update-baseline` to just that slice (a ## Workflow Requirements -**Default flow is PR-based.** `main` is protected: pushes require a pull request, CI must pass, and only squash or rebase merges are allowed (no merge commits, linear history enforced). **The single required status context is `pr-gate`** — the fan-in of `test.yml`'s PR tier (`lint`, `check`, `warnings`, scoped `cargo-test`, the 4-shard fast-mode gap suite, `gc-stress`, `e2e-scoped`, and `security-audit` when a lockfile/manifest changed). What runs in which tier is decided by `scripts/ci_plan.py` (`--table`), documented in `docs/src/testing/ci-tiers.md`: **pr** (every PR push, ~11 jobs, must be green on `main`), **sweep** (every push to `main`, coalesced), **full** (nightly / tags / dispatch / `run-extended-tests` label — parity, compile-smoke, doc-tests, package smokes, the auto-optimize gap shards). Releases wait for a `full-suite-gate` on the release SHA. The satellite GC/perf gates run on PRs only with the `run-extended-tests` label; their six-hourly `main` sweeps are unchanged. Admins can bypass for hotfixes/version bumps, but the standard path is: +**Default flow is PR-based.** `main` is protected: pushes require a pull request, CI must pass, and only squash or rebase merges are allowed (no merge commits, linear history enforced). **The single required status context is `pr-gate`** — the fan-in of `test.yml`'s PR tier (`lint`, `check`, `warnings`, scoped `cargo-test`, the 6-shard fast-mode gap suite, `gc-stress`, `e2e-scoped`, and `security-audit` when a lockfile/manifest changed). What runs in which tier is decided by `scripts/ci_plan.py` (`--table`), documented in `docs/src/testing/ci-tiers.md`: **pr** (every PR push, ~11 jobs, must be green on `main`), **sweep** (every push to `main`, coalesced), **full** (nightly / tags / dispatch / `run-extended-tests` label — parity, compile-smoke, doc-tests, package smokes, the auto-optimize gap shards). Releases wait for a `full-suite-gate` on the release SHA. The satellite GC/perf gates run on PRs only with the `run-extended-tests` label; their six-hourly `main` sweeps are unchanged. Admins can bypass for hotfixes/version bumps, but the standard path is: 1. Branch from `main`, push, open a PR. 2. Wait for required checks to go green. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 22dad3ae3f..19507a19eb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -115,7 +115,7 @@ Behavioral changes (HIR, codegen, runtime) also need the conformance suite, which diffs compiled programs byte-for-byte against Node: ```bash -./scripts/run_gap_tests.sh # ~560 tests; the PR gate runs this in 4 shards +./scripts/run_gap_tests.sh # ~560 tests; the PR gate runs this in 6 shards ./scripts/run_doc_tests.sh # Compile + run every docs/examples/*.ts ``` diff --git a/changelog.d/8187-ci-tiers.md b/changelog.d/8187-ci-tiers.md index a717ed6213..85e8b84a5e 100644 --- a/changelog.d/8187-ci-tiers.md +++ b/changelog.d/8187-ci-tiers.md @@ -18,7 +18,7 @@ self-tested; `--table` prints the job × tier matrix and `lint` keeps the docs c current), one fan-in status per tier: - **pr** (every PR push): `lint` (now also the changeset step), `check` (clippy ×2 + - api-docs-drift), `warnings`, scoped `cargo-test`, the gap suite in **4 shards of + api-docs-drift), `warnings`, scoped `cargo-test`, the gap suite in **6 shards of the harness's `fast` mode** (`PERRY_SKIP_BUILD=1`, one prebuilt release compiler, ~1.5 s/test), `gc-stress` (PR subset), `e2e-scoped`, and `security-audit` only when a lockfile/manifest/policy file changed. Docs-only PRs run `lint` only. Fan-in diff --git a/docs/src/testing/ci-tiers.md b/docs/src/testing/ci-tiers.md index 2656f0eaa6..263fce8162 100644 --- a/docs/src/testing/ci-tiers.md +++ b/docs/src/testing/ci-tiers.md @@ -27,7 +27,7 @@ copy is current. | `check` | yes | yes | yes | | `warnings` | yes | yes | yes | | `cargo-test` | yes | yes | yes | -| `gap-suite` | 4x fast | 2x fast | 8x full | +| `gap-suite` | 6x fast | 3x fast | 8x full | | `gc-stress` | yes | yes | yes | | `e2e-scoped` | yes | | | | `security-audit` | deps only | yes | yes | @@ -102,9 +102,9 @@ which costs no runner slot. | tier | typical jobs | typical runner-minutes | target wall clock | |---|--:|--:|--:| -| pr (core) | ~11 (4 gap shards) | ~140 | ≤ 30 min once queued | +| pr (core) | ~13 (6 gap shards) | ~200 | ≤ 30 min once queued | | pr (docs-only) | 2 | ~5 | ≤ 5 min | -| sweep | ~19 | ~240 | not a target — it coalesces | +| sweep | ~19 | ~300 | not a target — it coalesces | | full | ~30 | ~1500 | not a target | `push` to `main` uses ONE constant concurrency group with `cancel-in-progress: false`: diff --git a/scripts/ci_plan.py b/scripts/ci_plan.py index a83521f2df..984c80a340 100755 --- a/scripts/ci_plan.py +++ b/scripts/ci_plan.py @@ -106,9 +106,17 @@ # (~200 s each, redundantly per shard). ~40 min/shard. This is the # arm that sees auto-optimize-only link bugs, so it stays in the # nightly/release tier. +# +# Shard counts, measured 2026-08-16 on ubuntu-latest in fast mode: the release +# build is ~8 min per shard, non-ext tests are ~2 s each (~18 min for the whole +# suite), and every ext-routed test whose feature set the shard has not seen +# yet costs a ~4-5 min auto-optimize runtime rebuild (23 such tests, spread +# round-robin). At 4 shards the slowest shard was 39.5 min (6 rebuilds); at 6 +# it is ~28 min, level with gc-stress, for ~170 job-minutes -- against 480 for +# the old 8 x auto-optimize shards. GAP_SUITE = { - "pr": {"mode": "fast", "total": 4}, - "sweep": {"mode": "fast", "total": 2}, + "pr": {"mode": "fast", "total": 6}, + "sweep": {"mode": "fast", "total": 3}, "full": {"mode": "full", "total": 8}, } @@ -337,7 +345,7 @@ def check(name: str, cond: bool): check("core PR: windows off", not core["jobs"]["windows_build"]) check("core PR: parity off", not core["jobs"]["parity"]) check("core PR: security-audit off (no deps change)", not core["jobs"]["security_audit"]) - check("core PR: 4 fast gap shards", core["gap"] == {"mode": "fast", "total": 4, "shards": [1, 2, 3, 4], "update_snapshot": False}) + check("core PR: 6 fast gap shards", core["gap"] == {"mode": "fast", "total": 6, "shards": [1, 2, 3, 4, 5, 6], "update_snapshot": False}) check("core PR: cargo-test scoped", core["cargo_test_scope"] == "pr") deps = plan("pull_request", "refs/pull/1/merge", changed=["Cargo.lock"]) @@ -364,7 +372,7 @@ def check(name: str, cond: bool): check("sweep: binary-size off (macOS report-only, nightly is enough)", not sweep["jobs"]["binary_size"]) check("sweep: parity off", not sweep["jobs"]["parity"]) check("sweep: e2e-scoped off", not sweep["jobs"]["e2e_scoped"]) - check("sweep: 2 fast gap shards", sweep["gap"]["total"] == 2 and sweep["gap"]["mode"] == "fast") + check("sweep: 3 fast gap shards", sweep["gap"]["total"] == 3 and sweep["gap"]["mode"] == "fast") check("sweep: cargo-test full", sweep["cargo_test_scope"] == "full") check("sweep: security-audit on", sweep["jobs"]["security_audit"]) diff --git a/test-parity/README.md b/test-parity/README.md index a393cd5d79..1570330e82 100644 --- a/test-parity/README.md +++ b/test-parity/README.md @@ -6,7 +6,7 @@ differently; the difference is the point. ## `gap_snapshot.json` — the gap suite (per-PR gate) Generated by `scripts/gap_snapshot.py`, consumed by `scripts/run_gap_tests.sh`, -which is what the `gap-suite` job of `test.yml` runs on every PR (4 shards, `fast` +which is what the `gap-suite` job of `test.yml` runs on every PR (6 shards, `fast` mode), on every push to `main` and nightly (8 shards, auto-optimize mode) — see `docs/src/testing/ci-tiers.md`, including how to re-baseline this file from CI. From 603699ee5d8cf7c2ca4ec89f89b8c3be0db99ca5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sun, 16 Aug 2026 11:57:47 +0200 Subject: [PATCH 8/8] ci: npm ci before the gap suite (#8198 fixture deps); pipe-free changeset step * gap-suite: `npm ci --ignore-scripts --no-audit --no-fund` after setup-node so the six npm-fixture gap tests #8198 moved to root devDependencies have their oracle imports in CI. * lint/changeset step: `jq -s -e` instead of `jq | grep -q`. `lint` is now a registered moving-GC gate (main registered check_gc_env_knobs there), and gc_gate_wiring_check rejects an unguarded pipe in a gating step; `-s` because `gh api --paginate` emits one array per page. --- .github/workflows/test.yml | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 54f5824753..231ba4b619 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -171,12 +171,22 @@ jobs: env: GH_TOKEN: ${{ github.token }} run: | + set -euo pipefail gh api "repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files" --paginate > files.json - jq -r '.[].filename' files.json | grep -q '^crates/' || { echo "No crates/ changes — gate not applicable."; exit 0; } + # jq -e, no pipes: `lint` is a registered gate (gc_gate_wiring_check) + # and `jq | grep -q` under pipefail can fail on SIGPIPE when grep + # closes early on the first match. + # `-s` + `[.[][]]`: --paginate emits one JSON array per page, and + # `jq -e` reports only the LAST array's verdict without the slurp. + if ! jq -s -e '[.[][]] | any(.filename | startswith("crates/"))' files.json > /dev/null; then + echo "No crates/ changes — gate not applicable."; exit 0 + fi # The fragment must be ADDED in this PR (editing a leftover file # doesn't count) and match the root-level -.md shape # cut_release_notes.sh folds at release time. - jq -r '.[] | select(.status == "added") | .filename' files.json | grep -qE '^changelog\.d/[0-9]+-[^/]+\.md$' && exit 0 + if jq -s -e '[.[][]] | any(.status == "added" and (.filename | test("^changelog\\.d/[0-9]+-[^/]+\\.md$")))' files.json > /dev/null; then + exit 0 + fi echo "::error::This PR changes crates/ but adds no changelog.d/ fragment. Add changelog.d/-.md (see changelog.d/README.md) or apply the 'skip-changelog' label." exit 1 @@ -2159,6 +2169,13 @@ jobs: # inline here. See CLAUDE.md ("TypeScript Parity Status"). node-version-file: .node-version + # #8198: six gap tests import npm packages that are now root + # devDependencies (`.npmrc`: the root install "materializes the + # parity-test fixture deps"). Without this the oracle cannot resolve + # them, node exits 1, and the tests read as parity failures. + - name: Install the npm packages the gap tests' oracle imports + run: npm ci --ignore-scripts --no-audit --no-fund + # Two harness modes, chosen by the plan (see ci_plan.py GAP_SUITE): # fast: build the release compiler + runtime archives ONCE here, then # PERRY_SKIP_BUILD=1 so every test links the prebuilt archives