test(parity): make known_failures.json a ratchet, not a suppression list (#7582) - #7599
Merged
Conversation
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThe parity allowlist now validates platform-specific results, entry provenance, test coverage, and gap-snapshot consistency. It reports unallowed failures and stale entries, adds offline auditing, updates CI integration, and revises allowlist records and documentation. ChangesParity allowlist ratchet
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CI
participant parity_known_failures.py
participant known_failures.json
participant gap_snapshot.json
CI->>parity_known_failures.py: run parity check or --audit
parity_known_failures.py->>known_failures.json: validate and evaluate entries
parity_known_failures.py->>gap_snapshot.json: cross-check gap entries
parity_known_failures.py-->>CI: return failures, stale entries, and audit status
Possibly related issues
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/parity_known_failures.py`:
- Around line 300-303: Update print_stale and its run_audit call path so missing
test-file findings are not described as passing; use an audit-specific heading
such as entries that must be removed, while preserving the existing
passing-tests heading for genuine stale test results.
- Around line 181-183: Update the added-field validation to parse the string
with datetime.date.fromisoformat() after the existing DATE_RE format check, and
append the same validation problem for values that are not real calendar dates
while preserving the current message and handling parse failures safely.
- Around line 455-462: Update the audit summary around the known-failures entry
counting to report only Linux-applicable test_gap_* entries as cross-checked. In
test-parity/README.md lines 78-81 and test-parity/known_failures.json line 11,
revise the documentation/schema description to state that snapshot
cross-checking applies to Linux-applicable gap entries; update the Python
message and both documentation sites consistently.
- Around line 437-441: Update the --audit flow around audit() to require
args.gap_snapshot: load it unconditionally, reject a missing snapshot file, and
reject any root document lacking a tests object before calling audit(). Pass the
validated tests mapping to audit() so bidirectional gap checks always run.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a7043064-b32b-4a98-8daa-49d595238c08
📒 Files selected for processing (4)
.github/workflows/test.ymlscripts/parity_known_failures.pytest-parity/README.mdtest-parity/known_failures.json
Comment on lines
+181
to
+183
| added = record.get("added") | ||
| if not isinstance(added, str) or not DATE_RE.match(added): | ||
| problems.append(f"{test_id}: added must be an ISO date YYYY-MM-DD; got {added!r}") |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Validate calendar dates, not only the date format.
DATE_RE accepts invalid values such as 2026-02-31. This permits invalid provenance data even though added must be an ISO date. Parse the value with datetime.date.fromisoformat() after the format check.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/parity_known_failures.py` around lines 181 - 183, Update the
added-field validation to parse the string with datetime.date.fromisoformat()
after the existing DATE_RE format check, and append the same validation problem
for values that are not real calendar dates while preserving the current message
and handling parse failures safely.
Comment on lines
+300
to
+303
| def print_stale(stale: list[str], platform: str | None = None) -> None: | ||
| where = f" on {platform}" if platform else "" | ||
| print( | ||
| f"\nSTALE known_failures.json entries — these tests PASS{where}:", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not state that every offline-audit stale entry passes.
run_audit() passes missing test-file findings to print_stale(). Those tests cannot have passed because they no longer exist. Use an audit-specific heading such as “entries that must be removed,” or pass the reason into the formatter.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/parity_known_failures.py` around lines 300 - 303, Update print_stale
and its run_audit call path so missing test-file findings are not described as
passing; use an audit-specific heading such as entries that must be removed,
while preserving the existing passing-tests heading for genuine stale test
results.
Comment on lines
+437
to
+441
| known = load_json(args.known) if args.known.exists() else {} | ||
| snapshot_tests = None | ||
| if args.gap_snapshot.exists(): | ||
| snapshot_tests = load_json(args.gap_snapshot).get("tests", {}) | ||
| problems, stale = audit(known, snapshot_tests) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Require a valid gap snapshot in --audit mode.
If args.gap_snapshot is absent, this code sets snapshot_tests to None and audit() skips the bidirectional gap check. The command can then exit successfully while a stale test_gap_* suppression remains. This contradicts the required per-PR audit contract.
Load the snapshot unconditionally. Reject a missing file and reject a root document without a tests object.
Proposed fix
- snapshot_tests = None
- if args.gap_snapshot.exists():
- snapshot_tests = load_json(args.gap_snapshot).get("tests", {})
+ snapshot = load_json(args.gap_snapshot)
+ snapshot_tests = snapshot.get("tests") if isinstance(snapshot, dict) else None
+ if not isinstance(snapshot_tests, dict):
+ raise ValueError("gap snapshot must contain a tests object")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| known = load_json(args.known) if args.known.exists() else {} | |
| snapshot_tests = None | |
| if args.gap_snapshot.exists(): | |
| snapshot_tests = load_json(args.gap_snapshot).get("tests", {}) | |
| problems, stale = audit(known, snapshot_tests) | |
| known = load_json(args.known) if args.known.exists() else {} | |
| snapshot = load_json(args.gap_snapshot) | |
| snapshot_tests = snapshot.get("tests") if isinstance(snapshot, dict) else None | |
| if not isinstance(snapshot_tests, dict): | |
| raise ValueError("gap snapshot must contain a tests object") | |
| problems, stale = audit(known, snapshot_tests) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/parity_known_failures.py` around lines 437 - 441, Update the --audit
flow around audit() to require args.gap_snapshot: load it unconditionally,
reject a missing snapshot file, and reject any root document lacking a tests
object before calling audit(). Pass the validated tests mapping to audit() so
bidirectional gap checks always run.
Comment on lines
+455
to
+462
| entries = sum(1 for key in known if key != "_schema") | ||
| scope = ( | ||
| f" and cross-checked {sum(1 for k in known if k.startswith(GAP_PREFIX))} " | ||
| f"gap entries against {args.gap_snapshot.name}" | ||
| if snapshot_tests is not None | ||
| else " (no gap snapshot found — cross-check skipped)" | ||
| ) | ||
| print(f"known_failures.json audit OK — {entries} entries carry provenance{scope}.") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Report and document Linux-scoped snapshot coverage accurately.
The audit skips the snapshot check for test_gap_* entries scoped outside Linux. The current message and documentation claim that every gap entry is cross-checked.
scripts/parity_known_failures.py#L455-L462: count only Linux-applicable gap entries in the “cross-checked” message.test-parity/README.md#L78-L81: state that the snapshot cross-check applies to Linux-applicable gap entries.test-parity/known_failures.json#L11-L11: state the same platform scope in the schema description.
📍 Affects 3 files
scripts/parity_known_failures.py#L455-L462(this comment)test-parity/README.md#L78-L81test-parity/known_failures.json#L11-L11
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/parity_known_failures.py` around lines 455 - 462, Update the audit
summary around the known-failures entry counting to report only Linux-applicable
test_gap_* entries as cross-checked. In test-parity/README.md lines 78-81 and
test-parity/known_failures.json line 11, revise the documentation/schema
description to state that snapshot cross-checking applies to Linux-applicable
gap entries; update the Python message and both documentation sites
consistently.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test-parity/known_failures.json`:
- Around line 17-53: Update each affected known-failure entry, including
test_parity_stream_web, test_sock_write_map, test_ramda_sum,
test_gap_2159_defineproperty_class_prototype, test_gap_2514_settracesigint, and
test_gap_perfhooks_3088_3008_3010_3011, to reference a newly created or assigned
live tracker issue. Revise each entry’s issue, category, and reason to reflect
the live issue and current failure context, removing stale closed-issue
references while preserving actionable provenance.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: f48f4a27-db1b-4d41-a064-a70a9a1c556c
📒 Files selected for processing (1)
test-parity/known_failures.json
Comment on lines
+17
to
+53
| "reason": "Tracking issue #793 (roadmap umbrella) is CLOSED — this needs a live surface tracker (audited 2026-08-07, #7582). Node.js module inventory — `node:stream` surface not fully implemented. Tracker for surface coverage; flips to PASS as each API lands. Not a regression." | ||
| }, | ||
| "test_parity_stream_web": { | ||
| "issue": "793", | ||
| "added": "2026-05-15", | ||
| "category": "module-inventory", | ||
| "reason": "Node.js module inventory — `node:stream/web` (WHATWG streams) surface not fully implemented. Tracker for surface coverage; flips to PASS as each API lands. Not a regression." | ||
| "reason": "Tracking issue #793 (roadmap umbrella) is CLOSED — this needs a live surface tracker (audited 2026-08-07, #7582). Node.js module inventory — `node:stream/web` (WHATWG streams) surface not fully implemented. Tracker for surface coverage; flips to PASS as each API lands. Not a regression." | ||
| }, | ||
| "test_sock_write_map": { | ||
| "issue": "1634", | ||
| "added": "2026-05-15", | ||
| "category": "ci-env", | ||
| "reason": "Tracked in #1634 (parity CI environment fixtures). Net test passes locally with an echo server running, fails on Linux CI (no echo fixture server). Environmental — issue #91 dispatch fix landed in v0.5.581 and is no longer the cause; needs a CI-side fixture." | ||
| "reason": "Tracking issue #1634 is CLOSED — this needs a live CI-fixture tracker (audited 2026-08-07, #7582; not adjudicated locally, the echo-server port was already bound). Tracked in #1634 (parity CI environment fixtures). Net test passes locally with an echo server running, fails on Linux CI (no echo fixture server). Environmental — issue #91 dispatch fix landed in v0.5.581 and is no longer the cause; needs a CI-side fixture." | ||
| }, | ||
| "test_ramda_sum": { | ||
| "issue": "1634", | ||
| "added": "2026-05-18", | ||
| "category": "ci-env", | ||
| "reason": "Tracked in #1634 (parity CI environment fixtures). Ramda npm-package fixture failure on Linux CI; observed on #1038's CI run pre-merge. Companion to the existing test_ramda_user_import skip in the compile-smoke list — the parity harness can't npm-install ramda. File a CI-side fixture issue if/when this matters for sweep coverage." | ||
| "reason": "Tracking issue #1634 is CLOSED — this needs a live CI-fixture tracker (audited 2026-08-07, #7582). Tracked in #1634 (parity CI environment fixtures). Ramda npm-package fixture failure on Linux CI; observed on #1038's CI run pre-merge. Companion to the existing test_ramda_user_import skip in the compile-smoke list — the parity harness can't npm-install ramda. File a CI-side fixture issue if/when this matters for sweep coverage." | ||
| }, | ||
| "test_gap_2159_defineproperty_class_prototype": { | ||
| "issue": "2159", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "defineProperty on class prototypes — standing gap (issue encoded in name); confirmed standing (pre-dates v0.5.1205) in the #5917 parity diff." | ||
| "category": "bug-stale", | ||
| "reason": "RE-TRIAGE: tracking issue #2159 is CLOSED but this still fails (audited 2026-08-07, #7582) — needs a new issue. defineProperty on class prototypes; standing gap, confirmed pre-dating v0.5.1205 in the #5917 parity diff." | ||
| }, | ||
| "test_gap_2514_settracesigint": { | ||
| "issue": "2514", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "process SIGINT trace hook gap; standing per #5917 diff." | ||
| }, | ||
| "test_gap_2754_2907_2908_bigint_semantics": { | ||
| "issue": "2754", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "BigInt semantics cluster (#2754/#2907/#2908); standing per #5917 diff." | ||
| }, | ||
| "test_gap_3828_function_method_values": { | ||
| "issue": "3828", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "Function method value forms (#3828); standing per #5917 diff." | ||
| }, | ||
| "test_gap_class_expr_extends_static_call_this": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "gap-bisect", | ||
| "reason": "class-expression extends + static call `this` — no dedicated issue; tracked via the #5917 standing-tail worklist (watch for newness: flagged there as one of two to eyeball)." | ||
| }, | ||
| "test_gap_class_expr_static_this": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "gap-bisect", | ||
| "reason": "class-expression static `this` — tracked via #5917 standing-tail worklist (eyeball-for-newness pair)." | ||
| }, | ||
| "test_gap_console_bare_global": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "gap-categorical", | ||
| "reason": "console formatting/global surface — CLAUDE.md categorical gap (console.dir/group formatting); standing per #5917." | ||
| }, | ||
| "test_gap_console_validate_write": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "gap-categorical", | ||
| "reason": "console write validation — categorical console gap; standing per #5917." | ||
| }, | ||
| "test_gap_dyn_index_get_denormal_safe": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "dynamic index-get denormal safety; standing per #5917 diff." | ||
| }, | ||
| "test_gap_fetch_instanceof_5433": { | ||
| "issue": "5433", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "fetch Response/Request instanceof (#5433); standing per #5917." | ||
| }, | ||
| "test_gap_fetch_response_json_init": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "Response.json()/init surface; standing per #5917 diff." | ||
| }, | ||
| "test_gap_global_apis": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "module-inventory", | ||
| "reason": "global API surface inventory; standing per #5917." | ||
| }, | ||
| "test_gap_module_const_local_shadow": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "module-const local shadowing; standing per #5917 diff." | ||
| }, | ||
| "test_gap_node_v8_3137plus": { | ||
| "issue": "3137", | ||
| "added": "2026-07-04", | ||
| "category": "module-inventory", | ||
| "reason": "node:v8 surface (#3137+); standing per #5917." | ||
| }, | ||
| "test_gap_object_methods": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "Object static-method tail; standing per #5917 (cf. #5588 lineage)." | ||
| "category": "bug-stale", | ||
| "reason": "RE-TRIAGE: tracking issue #2514 is CLOSED but this still fails (audited 2026-08-07, #7582) — needs a new issue. process SIGINT trace hook gap; standing per the #5917 diff." | ||
| }, | ||
| "test_gap_perfhooks_3088_3008_3010_3011": { | ||
| "issue": "3088", | ||
| "added": "2026-07-04", | ||
| "category": "module-inventory", | ||
| "reason": "perf_hooks cluster (#3088/#3008/#3010/#3011); standing per #5917." | ||
| }, | ||
| "test_gap_sqlite_3183plus": { | ||
| "issue": "3183", | ||
| "added": "2026-07-04", | ||
| "category": "module-inventory", | ||
| "reason": "node:sqlite surface (#3183+); standing per #5917." | ||
| }, | ||
| "test_gap_static_member_call": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "static member call form; standing per #5917 diff." | ||
| }, | ||
| "test_gap_string_coercion_tostring": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "string coercion/toString edge; standing per #5917 diff." | ||
| }, | ||
| "test_gap_string_locale_2781_2845_2897": { | ||
| "issue": "2781", | ||
| "added": "2026-07-04", | ||
| "category": "bug-open", | ||
| "reason": "locale string cluster (#2781/#2845/#2897); standing per #5917." | ||
| }, | ||
| "test_gap_symbols": { | ||
| "issue": "5917", | ||
| "added": "2026-07-04", | ||
| "category": "gap-categorical", | ||
| "reason": "Symbol surface tail; standing per #5917." | ||
| "category": "bug-stale", | ||
| "reason": "RE-TRIAGE: tracking issue #3088 is CLOSED but this still fails (audited 2026-08-07, #7582) — needs a new issue. node:perf_hooks cluster (#3088/#3008/#3010/#3011) module-inventory gap." |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Replace closed issue references with live tracker issues.
These entries state that their current issue is closed and that a new tracker is needed. The issue field still points to the closed issue. Create or assign a live issue for each remaining failure, then update issue, category, and reason. This gives each suppression actionable provenance.
Also applies to: 61-65
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test-parity/known_failures.json` around lines 17 - 53, Update each affected
known-failure entry, including test_parity_stream_web, test_sock_write_map,
test_ramda_sum, test_gap_2159_defineproperty_class_prototype,
test_gap_2514_settracesigint, and test_gap_perfhooks_3088_3008_3010_3011, to
reference a newly created or assigned live tracker issue. Revise each entry’s
issue, category, and reason to reflect the live issue and current failure
context, removing stale closed-issue references while preserving actionable
provenance.
added 7 commits
August 8, 2026 00:28
…ist (#7582) `parity_known_failures.py` computed `failures - allowed` and stopped, so an entry whose test had started passing was inert forever — it never failed, never reported, and never asked to be removed, while silently converting that test's next regression into a non-event. That is a fifth way a gate can be unable to fail, and the worst of them: the job is genuinely green AND genuinely running. It cost a real bug. `test_gap_diagchannel_3082_3084_3085_3086` was listed on 2026-07-04 for a shipped feature cluster; when #7105 broke the test again for an unrelated reason the entry absorbed it, and the underlying defect — every `let`/`const` in an ES module's top-level bare block reading back stale — sat six days until #7580. The check now runs both ways, like gap_snapshot.json and gc_root_dominance_allowlist.json already do here: * fails and is not allowed here -> regression * allowed here and now PASSES -> STALE, delete the entry, this PR "Now passes" needs positive evidence and is per-platform: the test must appear in the report's `results[]` with status `pass` on the platform the run executed on. Filtered out, not in this shard, node_fail, skipped, or scoped elsewhere via `platforms` is never flagged — absence of evidence is not a pass (#6364). `results[]` is now mandatory rather than best-effort, so the gate cannot silently degrade back to suppression-only, and a passing run prints how many entries it actually adjudicated. Because the `parity` job that consumes this file is TAG-gated, the live half fires after the merges it was meant to judge. `--audit` is the offline half and runs on `lint`, a required per-PR context: it enforces provenance (#797 — issue + date + a category from the documented enum), rejects an entry naming a test file that no longer exists, and cross-checks every `test_gap_*` entry against `gap_snapshot.json`, which is generated and bidirectional. An entry absent from that snapshot is one the snapshot asserts passes. It would have named the diagchannel entry on the day it was added. Entry retirement follows in the next commit — the gate cannot land green until the entries it names are gone.
…i-env sharp edge validate_entry() now returns (problems, platforms_ok) instead of having the caller sniff 'platforms' out of an error string — a test id containing that word would have silently changed how the entry was scoped. README: a ci-env entry passing on a dev box that HAS the fixture CI lacks will be reported stale. That is the check working; the CI run is authoritative, and a genuinely OS-confined failure should carry a platforms scope instead.
…names (#7582) The new gate has never been green, so it cannot be flipped blind (CLAUDE.md's corollary). This is the flip: every entry the audit names is deleted here, in the same PR, so `--audit` lands green and the ratchet can only go red on a NEW attempt to park a passing test. 37 entries -> 9. All 28 retired are gap-suite entries that the generated Linux `gap_snapshot.json` — the baseline the required conformance-smoke job enforces BIDIRECTIONALLY on every PR, regenerated 2026-08-06 — asserts pass. Linux is the platform this file's consumer (`parity`, ubuntu-latest) actually runs on, so that is the authoritative verdict, not a local macOS one. Five of the 28 (`test_gap_3662_node_argvalidation`, `test_gap_constants_tail_3683plus`, `test_gap_handle_band_object_ops`, `test_gap_zlib_4917_level`, `test_gap_zlib_fs_assert_2935_2752_2971`) were the auto-opt cold-object-cache zlib link failure. #6847 was closed COMPLETED on 2026-07-30, verified against a genuinely cold cache, fixed by #7021 — so those entries outlived their bug by a week. They also carried `category: "toolchain"`, which is not in the documented enum and which nothing validated; the checker now rejects it, and no entry uses it any more. Provenance refresh on the 9 survivors (#797). Every tracking issue in this file except #6477 is CLOSED. The four survivors that are bug claims with a closed tracker are re-categorised `bug-stale` — the category the schema already defines for exactly this ("the tracking issue is closed but the test still fails; needs re-triage and a new issue") — with the closed-tracker fact leading the reason. The module-inventory and ci-env survivors keep their categories, because `bug-stale` would assert something false about them, and carry the same actionable note. `issue` still points at the original tracker: that is its provenance, not a claim that it is open. Survivors, all re-checked: test_gap_2159_defineproperty_class_prototype (#2159, closed), test_gap_2514_settracesigint (#2514, closed), test_gap_perfhooks_3088_3008_3010_3011 (#3088, closed), test_gap_v8_2_3680plus (#3680, closed), test_gap_stream_tee_tick_parity (#6477, OPEN), test_parity_stream + test_parity_stream_web (#793, closed umbrella), test_sock_write_map + test_ramda_sum (#1634, closed).
…still reproduces there The audit's own instrument caught this. The entry looked stale from the Linux snapshot, so it was retired with the other 27. Re-running it live in auto-optimize mode on macOS/arm64 reproduced #6847's exact signature — `Undefined symbols: _js_zlib_deflate_raw_sync, _js_zlib_inflate_raw_sync`, auto-opt pairing the ext-zlib provider archive with a feature-stripped stdlib rebuild — 3 times out of 3, so not one of this suite's host-local flakes. Its two zlib siblings (test_gap_zlib_3285_params, test_gap_zlib_fs_assert_...) both PASS in the same mode, so it is specific to the raw-sync entry points. #6847 was closed COMPLETED on 2026-07-30, fixed by #7021 and verified against a genuinely cold cache. It needs reopening for macOS, or a new issue. Scoped rather than global, which is the whole point of the per-platform rule: Linux passes this test, so an unscoped entry would go on suppressing a passing test on the platform the gate runs on — the exact failure mode this PR exists to remove. Scoped, it suppresses nothing in CI, records something true, and the ratchet still governs it: if macOS starts passing, the gate says delete it.
…ift (26.5.0 -> 26.5.1)
proggeramlug
force-pushed
the
test/7582-known-failures-ratchet
branch
from
August 7, 2026 22:29
6a8a3da to
c1b3ffa
Compare
Contributor
Author
Audit before merge — verified, merged as v0.5.1345Sabotage re-verified with the historically-loaded case: re-adding the exact The design detail that earns the merge: the offline Also folded into the merge commit: CLAUDE.md's oracle prose said Node Two findings from the audit actioned separately: #6847 reopened (live macOS The 37 → 10 retirement with per-entry provenance and pass-reason ("substantive" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7582. Carries out the audit #797 asked for on this file, and puts its provenance rules in code so the next one is mechanical.
The problem
test-parity/known_failures.jsonwas a pure suppression list.scripts/parity_known_failures.pycomputedfailures - allowedand stopped, so an entry whose test had started passing was inert forever — it never failed, never reported, and never asked to be removed, while silently converting that test's next regression into a non-event.That is a fifth way a gate can be unable to fail, alongside CLAUDE.md's four, and the worst of them: the job is genuinely green and genuinely running.
test_gap_diagchannel_3082_3084_3085_3086was listed on 2026-07-04 for a feature cluster that had shipped. When #7105 broke the test again for an unrelated reason (PreallocateBoxesshadowing a module-level global) the entry absorbed it. The defect emptied everylet/constin an ES module's top-level bare block that a siblingfunctiondeclaration read — six days, found by accident, fixed in #7580.The mechanism
Data flow, traced rather than assumed. The file has exactly two code readers:
scripts/parity_known_failures.py(the gate, run from the tag-gatedparityjob and--self-testfromlint) andscripts/parity_matrix_trend.py(importsnormalize_platform, re-derives platform selection to mark recordsknown; same tag-gated job).scripts/run_gap_tests.shdoes not read it — the gap gate isgap_snapshot.json, which has been bidirectional since #6755.Live half.
check()now also reads the report'sresults[]and fails on any allowlist entry selected for this platform whose test ran and returnedpass, naming the entry to delete.results[]is mandatory rather than best-effort, so the gate cannot silently degrade back to suppression-only, and a passing run prints how many entries it actually adjudicated (5/10above) — a gate must assert its subject was live."Now passes" needs positive evidence and is per platform. Not in
results[]at all (filtered out, not in this shard),node_fail,skipped, or scoped elsewhere viaplatformsis never flagged. Absence of evidence is not a pass — that is the hole a Node-22 pin used to hide 14 tests in (#6364).Offline half —
--audit, new, onlint. Theparityjob is tag-gated, so the live half fires after the merges it was meant to judge.--auditneeds no parity run: it enforces provenance (issue + date + a category from the documented enum), rejects an entry naming a test file that no longer exists, and cross-checks everytest_gap_*entry againstgap_snapshot.json— generated, bidirectional, regenerated 2026-08-06, enforced by required CI on every PR. An entry absent from that snapshot is one the snapshot asserts passes. This check would have named the diagchannel entry on 2026-07-04, the day it was added. It runs in ~0.1s.Schema tightening in the same pass:
issueandaddedwere previously unvalidated, andcategorywas only checked for non-emptiness — which is howcategory: "toolchain"(five entries, undocumented) slipped in unnoticed.The audit
37 entries → 10. Every tracking issue in the file except #6477 is closed.
27 retired. All are gap-suite entries the Linux
gap_snapshot.jsonasserts pass — Linux is the platform this file's consumer runs on. Corroborated live on macOS: a full 500-test gap run, plus a second auto-optimize pass over the ten whose first verdict came from a run mode that could not link them.A read of all 28 candidate test bodies says 26 are substantive (real values, real round-trips, spec-mandated throw messages) — they pass because the feature works. Two are narrow:
test_gap_console_bare_globalandtest_gap_readline_3698plusassert mostlytypeof/shape. That is on-target for their specific defects (consoleas a bare identifier resolving to the0sentinel;createInterface/connectnamed exports resolving toundefined) but shallow beyond them.The audit found a live reproduction of a closed issue.
test_gap_zlib_4917_levellooked stale from the Linux snapshot and was retired with the rest — then reproduced #6847's exact signature on macOS/arm64 in auto-optimize mode, 3 runs out of 3:Undefined symbols: _js_zlib_deflate_raw_sync, _js_zlib_inflate_raw_sync, auto-opt pairing the ext-zlib provider archive with a feature-stripped stdlib rebuild. Its two zlib siblings pass in the same mode, so it is specific to the raw-sync entry points. #6847 was closed COMPLETED on 2026-07-30 (fixed by #7021, verified on a cold cache) and needs reopening for macOS.It is kept with
platforms: ["macos"], not globally. Linux passes the test, so an unscoped entry would go on suppressing a passing test on the platform the gate runs on — the exact failure mode this PR removes. Scoped, it suppresses nothing in CI, records something true, and the ratchet still governs it.10 survivors, provenance refreshed. The four that are bug claims with a closed tracker (#2159, #2514, #3088, #3680) plus the macOS zlib entry (#6847) are re-categorised
bug-stale— the category the schema already defines for exactly this — with the closed-tracker fact leading the reason.test_gap_stream_tee_tick_paritykeepsbug-open(#6477 is the one open tracker). Themodule-inventorysurvivors (test_parity_stream,test_parity_stream_web, #793) andci-envsurvivors (test_sock_write_map,test_ramda_sum, #1634) keep their categories, becausebug-stalewould assert something false about them, and carry the same note.issuestill points at the original tracker — that is its provenance, not a claim that it is open. All five need new tracking issues, now visible instead of buried.The four survivors adjudicated by the live run all still fail on macOS, three of them with the same status the Linux snapshot records. The four non-gap survivors did not run under a
test_gap_filter and were correctly left un-adjudicated.Sabotage evidence
The gate must be able to fail. Both halves, real exit codes, no pipe (a wrapper's status is not the harness's — that has bitten this repo three times this week).
Offline half, against the committed files:
Live half, against the real macOS gap report projected onto the audit set:
ARM4 is the one that matters for #7568's per-platform rule: a passing test whose entry is scoped elsewhere must not be flagged.
--self-testgained arms for all of it: stale detection,node_fail/skipped/compile_failnot counting as passes, a test that did not run not counting as a pass, a platform-scoped entry not being judged, a report with noresults[]being rejected outright, and each provenance field failing on its own.Validation
Local, and local is what counts here — CI has a deep backlog and none of this has run there yet.
parity_known_failures.py --self-test→ 0;--audit→ 0;gap_snapshot.py --self-test→ 0 (unchanged neighbour);parity_matrix_trend.pystill imports cleanly (it consumesnormalize_platform, which is untouched).node v26.5.1matching.node-version. (Worth flagging: CLAUDE.md's prose still says 26.5.0 while.node-version— which CLAUDE.md itself names as the single source of truth — says 26.5.1. The file is right; the prose is stale.)scripts/check_file_size.sh→ 0.scripts/raw_handle_debt.py→ 998, baseline 998. No Rust touched, so nocargo fmt. Workflow YAML parses.Two things seen in passing, not addressed here:
test_gap_gc_rest_argument_rootingandtest_gap_gc_same_module_call_argument_rootingcrashed on macOS underPERRY_NO_AUTO_OPTIMIZE=1while the Linux snapshot has them passing, and that mode produces ~58 spurious ext-linkcompile_fails on macOS despiterun_parity_tests.sh's comment claiming its ext-package build compensates.Summary by CodeRabbit
Bug Fixes
Documentation
Chores