The cache key in scripts/make_git_package.sh is <package>-<git_describe>; source identity only. It doesn't include any build-input identity (RUSTFLAGS, GOFLAGS, cargo profile env vars, or the apt-installed toolchain version that comes from Snapshot= in images/*.conf). When the build env changes but source doesn't, the cache silently reuses binaries linked under different conditions.
What we hit
In our seismic-images fork of flashbots-images, Snapshot= was bumped to a value where Debian trixie's rustc had rolled back from 1.86 → 1.85 (trixie release-freeze churn). Per-package caches silently kept serving binaries linked against rustc 1.86 so we didn't cache the regression. Only a few PRs later an unrelated source-pin bump forced one package to recompile, which failed an MSRV check (alloy 1.0.24 requires rustc ≥1.86), which really confused me because none of the dependencies had changed in that bump.
The class of bug: any change to the build environment that doesn't touch the source pin can produce stale-but-cached binaries that are only exposed when an unrelated bump forces a fresh compile. If this was ever used by the person generating and publishing measurements it could really cause havoc.
Two ways to fix
- Delete the cache layer; delegate to cargo's
target/
Persist target/ across mkosi builds via host volume mount. Cargo's cache is correctness-aware (invalidates per-crate on rustc / RUSTFLAGS / source / feature changes). No second cache to keep in sync.
Cost: ~10–30 GB persistent state per project, more chroot/host volume coupling, no zero-time happy path (cargo runs every build even when nothing changed). Bigger change.
- Hash build inputs into the cache key
Change the key to <package>-<git_describe>-sha256(build_cmd || $EXTRA). $build_cmd is always hashed (catches RUSTFLAGS / cargo profile / command shape changes).
I'm assuming you guys already opted against 1, so I'll send a PR for 2.
The cache key in
scripts/make_git_package.shis<package>-<git_describe>; source identity only. It doesn't include any build-input identity (RUSTFLAGS, GOFLAGS, cargo profile env vars, or the apt-installed toolchain version that comes fromSnapshot=inimages/*.conf). When the build env changes but source doesn't, the cache silently reuses binaries linked under different conditions.What we hit
In our seismic-images fork of flashbots-images,
Snapshot=was bumped to a value where Debian trixie'srustchad rolled back from 1.86 → 1.85 (trixie release-freeze churn). Per-package caches silently kept serving binaries linked against rustc 1.86 so we didn't cache the regression. Only a few PRs later an unrelated source-pin bump forced one package to recompile, which failed an MSRV check (alloy 1.0.24 requires rustc ≥1.86), which really confused me because none of the dependencies had changed in that bump.The class of bug: any change to the build environment that doesn't touch the source pin can produce stale-but-cached binaries that are only exposed when an unrelated bump forces a fresh compile. If this was ever used by the person generating and publishing measurements it could really cause havoc.
Two ways to fix
target/Persist
target/across mkosi builds via host volume mount. Cargo's cache is correctness-aware (invalidates per-crate on rustc / RUSTFLAGS / source / feature changes). No second cache to keep in sync.Cost: ~10–30 GB persistent state per project, more chroot/host volume coupling, no zero-time happy path (cargo runs every build even when nothing changed). Bigger change.
Change the key to
<package>-<git_describe>-sha256(build_cmd || $EXTRA).$build_cmdis always hashed (catches RUSTFLAGS / cargo profile / command shape changes).I'm assuming you guys already opted against 1, so I'll send a PR for 2.