ci(test): fail on dark tests — a test file no registry runs

.github/workflows/test.yml

@@ -219,6 +219,44 @@ jobs:
           python3 scripts/gc_matrix_liveness_check.py --self-test
           python3 scripts/gc_matrix_liveness_check.py --check-registry
 
+      # Dark-test gate. Four of this repo's suites are driven by an explicit
+      # registry rather than a glob, and a test file added without its registry
+      # line runs NOWHERE while its PR stays green — #7192, #7216, #7252 and
+      # #7270/#7271 all shipped that way against test-parity/gc_repsel_corpus.txt.
+      #
+      # Registration checks already existed for two of those prefixes, but both
+      # live behind a 90-minute compiler build, behind a changed-paths relevance
+      # filter, and in workflows that are NOT in branch protection — so the check
+      # could not run on the pull request that needed it. This is the pure
+      # filesystem-and-text half (~0.2s, no compiler, no Node), placed in `lint`
+      # BECAUSE `lint` is already a required context: hazard 2 is the step people
+      # forget, so this gate is put where that step does not exist.
+      #
+      # The self-test plants an unregistered file into each mechanism and asserts
+      # the gate names it, then removes it and asserts green — over the real
+      # registries, through an in-memory overlay, so the checkout is never
+      # mutated. Each mechanism also floors its candidate set, so a stale glob
+      # fails loudly instead of making every future run vacuously green.
+      #
+      # `!cancelled()` is hazard 4 in a costume nobody has named yet: `lint` is a
+      # SEQUENCE of independent gates, and a step that fails takes every later
+      # step in the job to `skipped`. That is not hypothetical here — `Public
+      # benchmark evidence freshness` has failed on `main` on every run from
+      # 2026-07-29 onward, so `File size limit`, `GC store-site inventory`,
+      # `Address-classification audit`, `Gap snapshot checker self-test` and
+      # `Platform-aware parity allowlist self-test` have all been skipped for
+      # days while the job dutifully reported red for an unrelated reason. A gate
+      # that never executes cannot fail on its own subject. This step costs 0.2s
+      # and shares no state with anything above it, so it always speaks.
+      # (`!cancelled()` rather than `always()`: a cancelled run should stay
+      # cancelled.) The five steps above deserve the same treatment; that is a
+      # separate change from this one.
+      - name: Test registration (dark tests)
+        if: ${{ !cancelled() }}
+        run: |
+          python3 scripts/check_test_registration.py --self-test
+          python3 scripts/check_test_registration.py
+
   # ---------------------------------------------------------------------------
   # Clippy — enforces the deny-level lints in [workspace.lints] (root
   # Cargo.toml). `cargo clippy` exits nonzero only on `deny` lints, so

CONTRIBUTING.md

@@ -77,6 +77,7 @@ match the checkout.
 
 - One logical change. Small PRs land faster. If you find yourself writing "also, I fixed …", split.
 - Tests. New behavior needs a test; a bug fix needs a regression test. For compiler changes, drop a `.ts` file under [`test-files/`](test-files/) exercising the path. For runtime/stdlib, `#[test]` in the relevant crate.
+- **A new test file must be registered in its suite's registry, or it will not run.** Most suites glob their inputs, but four read an explicit list — a `test_gap_gc_*` witness needs a line in [`test-parity/gc_repsel_corpus.txt`](test-parity/gc_repsel_corpus.txt), a feature probe needs an entry in `test-features/feature_matrix.toml`, a compiler-output fixture needs an entry in `benchmarks/compiler_output/workloads.toml`, and a Rust test file below a suite root needs a `mod` declaration. An unregistered file is not a failing test, it is *no test at all*: your PR goes green having run nothing. `python3 scripts/check_test_registration.py` catches this in `lint` in under a second; `--list` names every registry. Full page: [`docs/src/testing/test-registration.md`](docs/src/testing/test-registration.md).
 - Docs where user-visible. New CLI flags, new perry.toml fields, new stdlib APIs → update [`docs/src/`](docs/src/).
 
 ### What does NOT go in a PR (maintainer handles these at merge)

benchmarks/compiler_output/workloads.toml

@@ -1,3 +1,15 @@
+# Registry for the compiler-output regression workloads.
+#
+# ***A NEW FIXTURE MUST BE REGISTERED HERE OR IT WILL NOT RUN.***
+# scripts/compiler_output_regression.py reads this file, not the fixtures
+# directory, so a .ts under fixtures/ with no `source =` entry has no IR checks
+# and no runtime budgets attached to it — it is compiled by nothing.
+# scripts/check_test_registration.py fails `lint` on an unregistered fixture,
+# and on a `source` here that no longer exists. A fixture driven by a DIFFERENT
+# registry (raw_numeric_layout_smoke.ts is a target-collector workload in
+# scripts/run_memory_stability_tests.sh) belongs in that script's
+# `compiler-output-workloads` exclusions, with a reason.
+
 schema_version = 1
 
 [workloads.image_convolution]

changelog.d/7278-dark-test-registration-gate.md

@@ -0,0 +1,59 @@
+Four PRs in a row shipped a test file that ran nowhere. #7192 and #7216 each
+added a `test_gap_gc_*` stale-root witness and no `test-parity/gc_repsel_corpus.txt`
+line; #7252 added a third; #7270/#7271 added two more, caught by hand at merge.
+An unregistered file is not a failing test, it is no test at all — the PR goes
+green having run it zero times.
+
+Registration checks existed for two of the three prefixes and neither could
+catch the pull request that needed it: `gc_repsel_matrix.sh` and
+`gc-moving-witnesses.yml` both sit behind a 90-minute release build, behind a
+changed-paths relevance filter, and in workflows that are not in branch
+protection.
+
+`scripts/check_test_registration.py` is the cheap half, pulled out to where it
+can block and generalised past that one corpus. Pure filesystem and text (no
+compiler, no Node, ~0.2s) over four mechanisms: the GC/repsel corpus,
+`test-features/feature_matrix.toml`, `benchmarks/compiler_output/workloads.toml`,
+and Rust test files below a suite root, which compile only if a `mod`
+declaration names them (rustc never parses an undeclared one — not dead code,
+not code, no warning).
+
+It runs in `lint`, which is ALREADY a required context, so no branch-protection
+change is needed. That placement is the point: forgetting to promote a new job
+is CLAUDE.md hazard 2, and it is what left `gc-root-dominance` red and blocking
+nothing for days.
+
+The step carries `if: ${{ !cancelled() }}`, which is hazard 4 wearing a costume
+this repo has not named yet: `lint` is a SEQUENCE of unrelated gates, and one
+failing step takes every later step to `skipped`. Not hypothetical — `Public
+benchmark evidence freshness` has failed on `main` on every run from 2026-07-29
+onward, so `File size limit`, `GC store-site inventory`,
+`Address-classification audit`, `Gap snapshot checker self-test` and
+`Platform-aware parity allowlist self-test` have all been skipped for days while
+the job reported red for an unrelated reason. The five steps above deserve the
+same treatment; that is a separate change from this one, and the stale public
+benchmark artifact needs regenerating either way.
+
+Built so it cannot pass vacuously: each mechanism floors its candidate set and
+fails if the glob stops matching, and every run prints
+`checked N files against M registries`. Exclusions are named with reasons rather
+than counted (a threshold cannot tell a new dark file from an old one), and a
+stale exclusion or a registry entry whose file is gone both fail. `--self-test`
+(39 cases, also run in `lint`) plants an unregistered file into each mechanism
+over the real registries, asserts the gate names it, then removes it and asserts
+green. It also asserts that excluding the planted file clears it and that a
+deleted or renamed registry fails by name instead of crashing with a raw
+`FileNotFoundError`.
+
+Zero dark files today across all four mechanisms, so it is green on `main` from
+the first run. Five candidates are excluded with reasons: four helper modules
+imported by a registered test, and `raw_numeric_layout_smoke.ts`, which is
+registered in a different registry (the `raw_numeric_layouts` target-collector
+workload in `scripts/run_memory_stability_tests.sh`).
+
+Out of scope and said out loud in `--list`: `tests/*.sh|py|ts`, where 143 of 171
+files are referenced by nothing in the tree. That has no registry to diff
+against and needs per-file triage, not a gate.
+
+The rule is documented in a new `docs/src/testing/test-registration.md`, in
+CONTRIBUTING.md, and in each of the three registry files' own headers.

docs/src/SUMMARY.md

@@ -147,6 +147,7 @@
 
 # Testing
 
+- [Test Registration (dark tests)](testing/test-registration.md)
 - [Geisterhand (UI Fuzzer)](testing/geisterhand.md)
 - [Node Compatibility Matrix](testing/node-compat-matrix.md)
 

docs/src/testing/test-registration.md

@@ -0,0 +1,122 @@
+# Test Registration (dark tests)
+
+> **A new test file must be registered in its suite's registry, or it will not
+> run.** Most of Perry's suites glob their inputs, but four do not — they read an
+> explicit list. A file added to one of those without its registry line is not a
+> failing test, it is *no test at all*.
+
+`scripts/check_test_registration.py` enforces this. It runs in the `lint` job on
+every pull request, takes about a fifth of a second, and needs no compiler, no
+Node and no build.
+
+```bash
+python3 scripts/check_test_registration.py             # the gate
+python3 scripts/check_test_registration.py --list      # what is in scope, and what is not
+python3 scripts/check_test_registration.py --self-test # prove the gate can still fail
+```
+
+## Why this exists
+
+A dark test is invisible in exactly the way that matters. The PR is green. The
+reviewer sees a witness in the diff and a passing CI run next to it and reads
+the two together as "covered". Nothing says otherwise, because nothing ran.
+
+It happened four times against `test-parity/gc_repsel_corpus.txt` alone:
+
+| PR | What went dark |
+|----|----------------|
+| #7192 | a `test_gap_gc_*` stale-root witness, dark from merge |
+| #7216 | a second one, same shape |
+| #7252 | `test_gap_gc_call_argument_rooting`, caught only once #7192/#7216's own registration assert reached `main` |
+| #7270 / #7271 | two more (rest-argument and same-module call-argument rooting), caught by the maintainer at merge |
+
+Two partial gates already existed and neither could catch the pull request that
+needed it. `scripts/gc_repsel_matrix.sh` auto-detects unregistered
+`test_gap_repsel_*` / `test_gap_specabi_*` files, and `gc-moving-witnesses.yml`
+adds `test_gap_gc_*` — but both sit behind a full release build of the compiler,
+behind a changed-paths relevance filter, and in workflows that are not in branch
+protection's required contexts. This script is the cheap half of those checks,
+pulled out to somewhere it can block a merge, and generalised to the other three
+places in the tree with the same shape.
+
+## Registry-driven suites
+
+Run `--list` for the authoritative version with every exclusion and its reason.
+
+| Registry | Candidate files | Runner |
+|----------|-----------------|--------|
+| `test-parity/gc_repsel_corpus.txt` | `test-files/test_gap_{gc,repsel,specabi}_*.ts` | `scripts/gc_repsel_matrix.sh` (`gc-stress`, `gc-moving-witnesses`) |
+| `test-features/feature_matrix.toml` | `test-features/probes/**/*.ts` | `scripts/gen_feature_matrix.py` (`feature-matrix`) |
+| `benchmarks/compiler_output/workloads.toml` | `benchmarks/compiler_output/fixtures/**/*.ts` | `scripts/compiler_output_regression.py` (`compiler-output-regression`) |
+| a `mod` declaration in the parent module | `crates/*/**/tests/**/*.rs` below a suite root | `cargo test` |
+
+The last one is the Rust analogue and it is worth spelling out: cargo
+auto-discovers `crates/<crate>/tests/<suite>.rs`, but a file one level deeper —
+a suite's module directory, or a `#[cfg(test)]` submodule under `src/` — only
+compiles if a `mod` declaration names it. Without one, rustc never parses the
+file. It is not dead code; it is not code. No warning fires.
+
+Everything else is glob-driven and cannot go dark. `--list` names those too, so
+"considered and safe" is distinguishable from "never looked at".
+
+## What the gate does, and what it refuses to do
+
+Per CLAUDE.md's *four ways a gate can be unable to fail*:
+
+- **It cannot pass vacuously.** Every mechanism declares a floor on its
+  candidate set and fails if the glob stops matching. "0 dark files over 0
+  candidates" and "0 dark files over 157 candidates" print the same verdict and
+  mean opposite things, so the summary always names the counts:
+  `checked 157 files against 4 registries`.
+- **It is proven able to fail.** `--self-test` plants an unregistered file into
+  each of the four mechanisms — over the real registries, via an in-memory
+  overlay, so nothing touches your working tree — asserts the gate names it,
+  then removes it and asserts the gate goes green again. It also asserts a
+  collapsed candidate set fails, a stale exclusion fails, and a registry entry
+  whose file is gone fails.
+- **Exclusions are named, not counted.** A numeric threshold cannot tell a new
+  dark file from an old one: fix one, add one, and the tally is unchanged. Every
+  non-registered candidate is listed in the script with a reason. A stale
+  exclusion — one that matches no file on disk — is itself a failure, so an
+  excuse cannot outlive the file it excuses.
+- **It runs where it blocks.** It is a step in `lint`, which is already a
+  required context. That placement is deliberate: forgetting to add a new job to
+  branch protection is hazard 2, and `gc-root-dominance` sat red and blocking
+  nothing for days because of it. This gate adds no new job, so there is no
+  branch-protection step left to forget.
+
+## When it fires
+
+You will see something like:
+
+```
+TEST REGISTRATION: a test file exists that nothing runs.
+
+  - DARK TEST test-files/test_gap_gc_rest_argument_rooting.ts
+      exists on disk but is not registered in test-parity/gc_repsel_corpus.txt, so
+      scripts/gc_repsel_matrix.sh (gc-stress, gc-moving-witnesses) never runs it.
+      Register it there, or add it to this script's `gc-repsel-corpus` exclusions
+      with a reason.
+```
+
+Two ways out, and only two:
+
+1. **Register it.** Add the line to the named registry. This is almost always
+   the right answer — you wrote the file to run.
+2. **Exclude it, with a reason.** If the file is genuinely a helper (a fixture
+   imported by a registered test, a vendored dependency, a workload driven by a
+   *different* registry), add it to that mechanism's `exclusions` dict in
+   `scripts/check_test_registration.py` and say why in prose. Reviewers read
+   that text; "excluded" on its own is not an answer.
+
+There is deliberately no third way. No threshold to bump, no `--allow-dark`, no
+environment variable.
+
+## Not covered
+
+`tests/*.sh`, `tests/*.py` and `tests/*.ts` have no registry to diff against —
+143 of the 171 files there are referenced by nothing in the tree. That is a
+separate archaeology problem (triage each one: wire it up, or delete it), not an
+unregistered-file problem, and inventing a registry for it retroactively would
+make this gate red on day one for reasons that have nothing to do with the four
+dark witnesses it was written for.