ci: cut test wall time without losing coverage#3267
Merged
Conversation
The slowest matrix cells (windows-latest, macos-15-intel) spend ~12 of their ~20 minutes in nextest, and the log timings show most of the tail is serialized: - The slowest tests (pgo, conda, uniffi, uv multi-python) were scheduled last, so the run drains into a single-test tail. Schedule them first via nextest priority overrides (nextest >= 0.9.91). - threads-required = num-cpus made every uniffi test run exclusively, ~4 min of serialized wall time per job. Relax it to 2 threads. - test_integration_conda created five conda envs (3.10-3.14, 152s on windows); oldest + newest keeps the coverage for half the cost. - Enable the existing faster-tests feature in CI so test wheels are stored uncompressed and stripped; the deflate path stays covered by unit tests and the release-binary jobs (auditwheel, docker, cross). - Disable debuginfo on all platforms instead of only Windows: smaller artifacts mean faster links, wheel packing and pip installs. - Run the zig cross-compile step once per host OS instead of twice (it was running for both 3.9 and pypy); zig mostly exercises host-OS-specific plumbing. macos-latest keeps its pypy run since it has no 3.9 cell. - Skip miniconda setup on the -dev cells; the conda tests create their own envs independent of the matrix python, so once per OS is enough and the -dev cells skip the ~2 min setup. See PyO3#2189 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNNLg7XZnxM8Aiqrzo6c5q
The faster-tests feature was coupled to strip=true in the test harness, and enabling it in CI broke uniffi proc-macro tests: bindings are generated by reading uniffi metadata from the compiled cdylib, which -C strip=symbols removes, producing an empty Python module. pyo3 stub generation introspects artifacts the same way (nextest's fail-fast just cancelled the run before those could fail). Decouple strip from faster-tests: tests always build with strip disabled, matching what CI did before faster-tests was enabled, and faster-tests keeps only its uncompressed-wheel behavior which is where the speedup comes from. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNNLg7XZnxM8Aiqrzo6c5q
messense
added a commit
that referenced
this pull request
Jul 19, 2026
## Summary Follow-up to #3267 (see #2189): the remaining big lever for test time. Isolated per-case target dirs meant every test case recompiled its whole dependency tree — ~3100 rustc invocations per CI job, 55% of which sccache cannot cache (cdylib links, proc-macros, build scripts). ### Design - Cases with compatible build configurations share a per-family target dir (`shared-pyo3` / `shared-cffi` / `shared-uniffi` / `shared-misc`), so dependency trees compile once per family. Cargo's own build lock makes concurrent builds of *different* crates in one dir safe, and artifacts for different feature sets coexist keyed by cargo's fingerprint. - Concurrent maturin builds of the **same** crate are *not* safe in a shared dir: maturin stages built artifacts through `target/maturin/` (renaming them out of cargo's output path and back), so two same-crate builds race on the staged files — reproduced locally with the develop + integration uniffi-proc-macro cases running concurrently. Tests therefore hold a per-fixture `fs4` file lock for the duration of the build (the same pattern the harness already uses for the cffi-provider venv). Different fixtures still build in parallel. - Cases keep an isolated dir when they change compile flags (pgo, zig, explicit `--target`), build against unusual interpreters (conda, uv multi-python), or assert on target dir contents (pep517 profile checks, stable-ABI selection which deletes the dir). - One small `src/` hardening: the pyo3 config file maturin writes into `target/maturin/` is now written via temp file + rename so concurrent builds sharing a target dir never observe a partial config. - The develop test timeouts go from 120s to 600s since a test can now legitimately wait on the fixture lock; the timeouts still catch hangs. ### Local measurements (full suite, cold state, no sccache, Apple Silicon) | | main | this branch | |---|---|---| | Wall time | 50.7s | 47.5–63.8s | | User CPU | **511s** | **287s** | | Sys CPU | 85s | 52s | | Involuntary context switches | 899k | 467k | | `test-crates/targets` size after run | 3.6 GB | 2.1 GB | | Result | 345 passed | 345 passed (3 runs) + warm rerun 32.8s | Total CPU work drops ~44% with no wall-time regression and no lock starvation (the slowest single test got *faster*, 38.5s → 27.2s). A many-core dev machine hides main's redundant compiles behind parallelism; CI's 4-core runners are CPU-bound, so the CPU cut is what should translate to CI wall time there. CI won't see the full 44% since sccache already absorbs ~60% of compiles — this targets exactly the uncacheable remainder.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses part of #2189. Measured where the time goes in a recent
Testrun (job/step timings + the full nextest log of the slowest cell): the slowest matrix cells (windows-latest,macos-15-intel) take 19–22 min, of which ~12 min is nextest — and the last ~5 minutes of that run nearly serially. All changes below are scheduling/config-level; no test is removed.nextest scheduling (
.config/nextest.toml)priority = 100override (supported since nextest 0.9.91). Previously the pgo tests (77s, 115s) andintegration_pyo3_mixed_conda(152s) started last and drained the run into a single-test tail.threads-required = "num-cpus"to2. Claiming the whole machine made all 9 uniffi tests run one at a time — ~4 min of serialized wall time per job on Windows.Test changes
test_integration_condabuilds against 2 conda envs (3.10 + 3.14) instead of 5 (3.10–3.14). Conda env creation dominates this test (152s on Windows); oldest + newest keeps the multi-env interpreter discovery coverage.Workflow changes
faster-testsfeature in the nextest runs (matrix + Alpine): test wheels are stored uncompressed and stripped, which is exactly what the feature was added for. The deflate path stays covered by the compression unit tests and by the jobs that use a release maturin without the feature (auditwheel, docker, cross-compile).macos-latestkeeps its pypy run since it has no 3.9 cell, so every OS retains zig coverage. This shaves ~2–3 min off two of the three slowest jobs (windows/pypy,macos-intel/pypy).setup-minicondaon the-devcells. The conda tests create their own envs independent of the matrix python, so once per OS (the 3.9/pypy cells) is enough, and the-devcells drop the ~2 min miniconda setup.Expected effect
The slowest cells should go from ~20 min to roughly 12–14 min. Follow-up (separate PR): sharing cargo target dirs per fixture would cut the ~3100 rustc invocations per job (55% uncacheable by sccache: cdylibs, proc-macros, build scripts), which is the next big lever.
🤖 Generated with Claude Code
https://claude.ai/code/session_01NNNLg7XZnxM8Aiqrzo6c5q