Skip to content

lint: the public-baseline freshness gate has been failing for 40+ commits, making every later lint step unreachable #7257

Description

@proggeramlug

lint is a required status check and has been red on main for weeks. The most common cause is step 3, "Public benchmark evidence freshness":

public baseline error: public artifact benchmark inputs changed; regenerate it with ./benchmarks/run_public_baseline.sh
##[error]Process completed with exit code 2.

Seen in nightly runs 30735945650 (08-02) and 30519299737 (07-30); 30687607155 (08-01) and 30610234977 (07-31) failed one step earlier on workspace_architecture, which has since been fixed.

Why this is the urgent one

The steps in the lint job run in order and the job stops at the first failure. This step is #3 of 10, so while it is red, every later gate is unreachable and therefore inert:

# step status at cdb3934
1 cargo fmt --all -- --check PASS
2 workspace_architecture.py --check PASS
3 ci_public_baseline_check.py FAIL (2)
4 check_file_size.sh FAIL (fixed by #7256)
5 binding_pins.mjs --check PASS
6 gc_store_site_inventory.py FAIL
7 addr_class_inventory.py FAIL
8 gap_snapshot.py --self-test PASS
9 parity_known_failures.py --self-test PASS
10 gc_gate_wiring_check.py PASS

This is CLAUDE.md's hazard (2) in its most expensive form: a required job that is always red blocks nothing, because every merge bypasses it with --admin. Notably #7253 wired the new gc_gate_wiring_check.py into step 10 — a correctly-built gate that cannot currently fail a merge, because the job never reaches it.

Diagnosis

Both fingerprints are stale, not just one:

recorded source_fingerprint : 9713a2e1131dbc407e2f6e7b056b9e2e1092ed45be7ecd5bf88b3d5a9910405f
computed source_fingerprint : ca3101d49a02fdd17c3955c9d26d87206fe2d9e063ee5978b62181d59d5885d5
recorded harness_fingerprint: 437f64a8020aebd4d677405b4e5c8d915b2c76b10977df397ed324eef7b3ae23
computed harness_fingerprint: 8c6d17228131f215878846a970d7f97156bb9e8de73bdbee5537ee055b389aaf

benchmarks/results/public-node-bun-v1.json records generated_at: 2026-07-13T14:10:04Z / perry 0.5.1258. Recomputing tracked_fingerprint(SOURCE_PATHS) and tracked_fingerprint(HARNESS_PATHS) at each of the last 40 commits on main shows both have been stale for every one of them — this is not a single recent regression.

MAX_AGE_DAYS = 45 is not the problem (the artifact is ~20 days old); it is validate_public's fingerprint equality checks at benchmarks/public_baseline.py:520 and :524.

The last commit to touch the artifact is 7beb3a5 (#7165, 08-01, "scrub committed home-directory paths, stop the harnesses re-emitting them"). That PR edited harness files under HARNESS_PATHS and the artifact, but the recorded fingerprints do not match the tree at any point since.

Why it can't be fixed by hand

This is by design — harness_fingerprint changing means "the measurement apparatus changed, so the numbers are not comparable". Clearing it requires actually regenerating the artifact:

./benchmarks/run_public_baseline.sh

which needs the pinned Node v22.23.1 + Bun 1.3.14, a dist-profile Perry, the quiet-CPU gates, and the documented host. Editing the JSON to paste in new hashes would defeat the gate's entire purpose.

Suggested resolution

  1. Regenerate the artifact on the pinned host and land it.
  2. Consider whether SOURCE_PATHS including the root Cargo.toml is too broad — it already special-cases the version field (fix(bench): version-invariant Cargo.toml fingerprint in the public-baseline gate #6528), but any dependency bump also invalidates it and forces a full benchmark re-run to keep lint green. That coupling is why this goes stale so easily.
  3. Once green, verify that steps 4–10 actually execute, and re-check that gc_gate_wiring_check.py can fail.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions