ci: "Test default on Linux" flakes on unrelated PRs, dying without logs (likely runner OOM)

[pacu]

Symptom: the required CI job "Test default on Linux" fails intermittently on PRs whose diffs cannot plausibly affect it, and passes on re-run of the same commit. Because it feeds the "Required status checks have passed" gate, every flake blocks a PR until someone re-runs it.

Observed occurrences (all within 2026-07-15 → 2026-07-16, all docs-only or docs+snapshot diffs that pass the full suite locally and on macOS/Windows):

PR	Run	Outcome
#612	29460281803	"Test default on Linux" failed with no failing step recorded; passed on re-run of failed jobs, same commit
#627	29508225415	same job failed with no failing step and its log absent from the run's log archive; separately, "Latest build on ubuntu-latest" failed at "Build crates (zebra-state backend)" with collect2: fatal error: ld terminated with signal 7 [Bus error], core dumped
#629	29513543191 (attempt 1)	same job failed; in the attempt-1 log archive every other job's log is present, this job's is missing
#631	29523742999	same job failed (docs-only diff); re-run triggered

Why this looks like runner resource exhaustion rather than a flaky test:

• A test failure records a failing step and uploads a log; these runs have neither — the job appears to die at the runner level before log upload.
• The one adjacent failure that did leave a log is a linker Bus error (signal 7) on the same ubuntu-latest runner class, a classic symptom of memory/disk pressure while linking large binaries.
• The zebra/zaino dependency trees produce very large test binaries; cargo test links several of them in parallel on a 7 GB standard runner.

Possible directions (for whoever picks this up to evaluate, not prescriptions):

• Limit parallel linking in CI (e.g. -j for the final test build, or building test binaries with cargo test --no-run in a staged step).
• Larger runner for the Linux test legs, or splitting the job.
• A retry policy for infrastructure-class failures so the required gate doesn't hold PRs hostage.

Logs for the failed attempts expire quickly (several were already unavailable while investigating); the run links above may need GitHub support/api access to recover more detail.

---

Filed with AI assistance (Claude) while working on #600-series documentation PRs, which surfaced the pattern.

[linear]

COR-1584

[pacu]

Follow-up with a sharper hypothesis and concrete mitigations.

Refined diagnosis: disk (or memory) exhaustion from dev-profile debuginfo, during linking.

The one flake that left a usable log failed with collect2: fatal error: ld terminated with signal 7 [Bus error], core dumped. A SIGBUS while ld is writing its (mmap'd) output file is the classic signature of the underlying filesystem being unable to grow the file — i.e. the runner ran out of disk, not a test failure. The jobs that died with no step recorded and no log uploaded fit the same picture: the runner hit a resource wall before it could finalize/upload logs.

The mechanism is in our build config:

• The CI test step (ci.yml, "Run tests (zallet-core)") builds with the dev/test profile, then repeats for both backends, then builds three release-less binaries — all with the profile default debug = true (full DWARF).
• Meanwhile Cargo.toml already sets [profile.release] debug = "line-tables-only" — the dev/test builds get no such trim.
• Full debuginfo across the zebra-* + zaino-* dependency trees produces very large objects and test binaries; --workspace links ~30 test binaries, and three backends are built in one job. That is a lot of large-file writes competing for the runner's ~14 GB of free disk.

Concrete mitigations (ranked; I can implement whichever the team prefers — flagging up front that I cannot verify any of these against an intermittent failure, only reason about the mechanism):

1. Trim dev/test debuginfo — the direct analogue of the existing release setting. Add to Cargo.toml:

   [profile.dev]
   debug = "line-tables-only"   # or debug = 0

   or scope it to CI via CARGO_PROFILE_DEV_DEBUG=line-tables-only. Lowest-risk (only reduces backtrace detail on test failures), directly attacks the disk/memory pressure.
2. Add a diagnostic df -h / free -m before and after the heavy steps. Zero behavioral change; would confirm the exhaustion hypothesis on the next flake and tell us whether it's disk or RAM. Good first step if the team wants evidence before changing build config.
3. Free runner disk up front (rm -rf the preinstalled toolchains, or the jlumbroso/free-disk-space action) — buys headroom without touching debuginfo.
4. Retry policy for these infra-class failures so the required gate stops holding unrelated PRs hostage (orthogonal to the root cause).

Occurrence count is now up to ~5 across docs-only PRs (#612, #627, #629, #631, plus the zizmor tool-download flake on #626 was a different infra issue). My recommendation: (2) then (1) — confirm, then trim.

---

Investigated with AI assistance (Claude); mechanism verified against .github/workflows/ci.yml and Cargo.toml.