Skip to content

conformance-smoke fails 8/8 shards: 12 real regressions, 10 newly-visible node_fail transitions, 1 fix the snapshot hasn't accepted #8117

Description

@proggeramlug

Summary

conformance-smoke is a required status context and it fails 8 of 8 shards on every recent PR. I pulled all eight shard logs from #8090's completed run and tallied the actual gate output. It is not flake, and it is not one cause — there are 23 status changes in four distinct categories.

CLAUDE.md warns that these shards are flaky and that a red shard should be re-run and A/B'd before being believed. Eight of eight is not the flake signature, which is why I went looking.

The tally

 10  node_fail  -> parity_fail
  8  pass       -> parity_fail
  3  pass       -> crash
  1  pass       -> compile_fail
  1  parity_fail -> pass

10 × node_fail -> parity_fail — previously invisible, now exposed

test_gap_4510_enum_forward_ref          test_gap_moment_methods
test_gap_backoff_options                test_gap_prop_plan_cache_invalidation
test_gap_cron_cronjob                   test_gap_ratelimiter_memory
test_gap_dayjs_factory_arg              test_gap_slugify_options
test_gap_derived_param_props            test_gap_enum_in_function_body

This is exactly the #6364 pattern CLAUDE.md documents: when the oracle cannot run a test it exits non-zero, the harness classifies it node_fail, and the test is silently dropped from the gate rather than going red. These ten were in that hidden state and are now running — so these are not new breakage, they are newly visible breakage. The shard header says so itself: "STATUS CHANGES — including tests the oracle stopped covering".

That makes them the least urgent of the four groups and arguably good news, but they must be triaged and recorded rather than left to hold the gate red.

12 × genuine regressions from pass

pass -> crash          test_gap_6386_dataview_concat_regex_fastpaths
                       test_gap_7238_i64_specialization_exactness
                       test_gap_buffer_own_props
pass -> compile_fail   test_gap_zlib_4917_level
pass -> parity_fail    test_gap_array_methods
                       test_gap_builtin_alias_construct_7524
                       test_gap_collection_foreach_member_receiver_thisarg
                       test_gap_dynamic_builtin_construct_dispatch
                       test_gap_new_globalthis_builtin_6726
                       test_gap_set_map_foreach_fused_receiver
                       test_gap_specabi_reassign
                       test_gap_sso_concat_string_index

These are the ones that matter. Three pass -> crash are the worst of them. test_gap_specabi_reassign is #8100, fixed by #8109 — so at least one of the twelve is already resolved, which suggests the rest are individually tractable.

I have not attributed these to a commit. They are red with similar counts on every PR I checked, so they are almost certainly main-side rather than any one PR's doing, but that is an inference and someone should bisect rather than trust it.

1 × parity_fail -> pass — a fix that fails the gate

test_gap_iterator_helpers_2874 now passes and the snapshot still lists it as failing. The snapshot gates divergence in either direction by design, so an improvement turns the gate red until the snapshot is regenerated. Working as intended, but worth naming: part of why this context is red is that something got better.

Why this matters beyond the tests themselves

conformance-smoke-complete is one of seven required contexts. With it red, no PR can merge without admin bypass, which is the "required and always red trains bypass" failure mode CLAUDE.md calls out — and it is how #8100 (a silent wrong-answer miscompile in the shipped default) sat unnoticed. Its shard was cancelled early on recent runs, so nothing surfaced it.

Tracking the required-context picture in #8092. The others: lint is fixed as of #8087; cargo-test fails on four ELF-only byte-identity tests; parity/compile-smoke report SKIPPED by design and are continue-on-error.

Suggested split

  1. Triage the 10 node_fail -> parity_fail and record them in test-parity/gap_snapshot.json with issue links — they were always broken and are cheap to classify.
  2. Bisect the 12 genuine regressions. Three crashes first.
  3. Regenerate the snapshot for test_gap_iterator_helpers_2874.
  4. Only then is the gate meaningful enough to keep required.

Reproduce with ./scripts/run_gap_tests.sh --shard N/8.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions