Summary
conformance-smoke is a required status context and it fails 8 of 8 shards on every recent PR. I pulled all eight shard logs from #8090's completed run and tallied the actual gate output. It is not flake, and it is not one cause — there are 23 status changes in four distinct categories.
CLAUDE.md warns that these shards are flaky and that a red shard should be re-run and A/B'd before being believed. Eight of eight is not the flake signature, which is why I went looking.
The tally
10 node_fail -> parity_fail
8 pass -> parity_fail
3 pass -> crash
1 pass -> compile_fail
1 parity_fail -> pass
10 × node_fail -> parity_fail — previously invisible, now exposed
test_gap_4510_enum_forward_ref test_gap_moment_methods
test_gap_backoff_options test_gap_prop_plan_cache_invalidation
test_gap_cron_cronjob test_gap_ratelimiter_memory
test_gap_dayjs_factory_arg test_gap_slugify_options
test_gap_derived_param_props test_gap_enum_in_function_body
This is exactly the #6364 pattern CLAUDE.md documents: when the oracle cannot run a test it exits non-zero, the harness classifies it node_fail, and the test is silently dropped from the gate rather than going red. These ten were in that hidden state and are now running — so these are not new breakage, they are newly visible breakage. The shard header says so itself: "STATUS CHANGES — including tests the oracle stopped covering".
That makes them the least urgent of the four groups and arguably good news, but they must be triaged and recorded rather than left to hold the gate red.
12 × genuine regressions from pass
pass -> crash test_gap_6386_dataview_concat_regex_fastpaths
test_gap_7238_i64_specialization_exactness
test_gap_buffer_own_props
pass -> compile_fail test_gap_zlib_4917_level
pass -> parity_fail test_gap_array_methods
test_gap_builtin_alias_construct_7524
test_gap_collection_foreach_member_receiver_thisarg
test_gap_dynamic_builtin_construct_dispatch
test_gap_new_globalthis_builtin_6726
test_gap_set_map_foreach_fused_receiver
test_gap_specabi_reassign
test_gap_sso_concat_string_index
These are the ones that matter. Three pass -> crash are the worst of them. test_gap_specabi_reassign is #8100, fixed by #8109 — so at least one of the twelve is already resolved, which suggests the rest are individually tractable.
I have not attributed these to a commit. They are red with similar counts on every PR I checked, so they are almost certainly main-side rather than any one PR's doing, but that is an inference and someone should bisect rather than trust it.
1 × parity_fail -> pass — a fix that fails the gate
test_gap_iterator_helpers_2874 now passes and the snapshot still lists it as failing. The snapshot gates divergence in either direction by design, so an improvement turns the gate red until the snapshot is regenerated. Working as intended, but worth naming: part of why this context is red is that something got better.
Why this matters beyond the tests themselves
conformance-smoke-complete is one of seven required contexts. With it red, no PR can merge without admin bypass, which is the "required and always red trains bypass" failure mode CLAUDE.md calls out — and it is how #8100 (a silent wrong-answer miscompile in the shipped default) sat unnoticed. Its shard was cancelled early on recent runs, so nothing surfaced it.
Tracking the required-context picture in #8092. The others: lint is fixed as of #8087; cargo-test fails on four ELF-only byte-identity tests; parity/compile-smoke report SKIPPED by design and are continue-on-error.
Suggested split
- Triage the 10
node_fail -> parity_fail and record them in test-parity/gap_snapshot.json with issue links — they were always broken and are cheap to classify.
- Bisect the 12 genuine regressions. Three crashes first.
- Regenerate the snapshot for
test_gap_iterator_helpers_2874.
- Only then is the gate meaningful enough to keep required.
Reproduce with ./scripts/run_gap_tests.sh --shard N/8.
Summary
conformance-smokeis a required status context and it fails 8 of 8 shards on every recent PR. I pulled all eight shard logs from #8090's completed run and tallied the actual gate output. It is not flake, and it is not one cause — there are 23 status changes in four distinct categories.CLAUDE.md warns that these shards are flaky and that a red shard should be re-run and A/B'd before being believed. Eight of eight is not the flake signature, which is why I went looking.
The tally
10 ×
node_fail -> parity_fail— previously invisible, now exposedThis is exactly the #6364 pattern CLAUDE.md documents: when the oracle cannot run a test it exits non-zero, the harness classifies it
node_fail, and the test is silently dropped from the gate rather than going red. These ten were in that hidden state and are now running — so these are not new breakage, they are newly visible breakage. The shard header says so itself: "STATUS CHANGES — including tests the oracle stopped covering".That makes them the least urgent of the four groups and arguably good news, but they must be triaged and recorded rather than left to hold the gate red.
12 × genuine regressions from
passThese are the ones that matter. Three
pass -> crashare the worst of them.test_gap_specabi_reassignis #8100, fixed by #8109 — so at least one of the twelve is already resolved, which suggests the rest are individually tractable.I have not attributed these to a commit. They are red with similar counts on every PR I checked, so they are almost certainly main-side rather than any one PR's doing, but that is an inference and someone should bisect rather than trust it.
1 ×
parity_fail -> pass— a fix that fails the gatetest_gap_iterator_helpers_2874now passes and the snapshot still lists it as failing. The snapshot gates divergence in either direction by design, so an improvement turns the gate red until the snapshot is regenerated. Working as intended, but worth naming: part of why this context is red is that something got better.Why this matters beyond the tests themselves
conformance-smoke-completeis one of seven required contexts. With it red, no PR can merge without admin bypass, which is the "required and always red trains bypass" failure mode CLAUDE.md calls out — and it is how #8100 (a silent wrong-answer miscompile in the shipped default) sat unnoticed. Its shard was cancelled early on recent runs, so nothing surfaced it.Tracking the required-context picture in #8092. The others:
lintis fixed as of #8087;cargo-testfails on four ELF-only byte-identity tests;parity/compile-smokereportSKIPPEDby design and arecontinue-on-error.Suggested split
node_fail -> parity_failand record them intest-parity/gap_snapshot.jsonwith issue links — they were always broken and are cheap to classify.test_gap_iterator_helpers_2874.Reproduce with
./scripts/run_gap_tests.sh --shard N/8.