fix(ci): give each runner its own uv tool cache — the shared one is destructive#18
Conversation
…estructive ~80 Spartan runners inherited RUNNER_TOOL_CACHE=~/.runner-toolcache from the supervisor, sharing ONE mutable cache. @actions/tool-cache's writer rmRF's the version directory and its .complete marker before repopulating, while its reader treats dir+.complete as a hit and adds it to PATH without checking the binary is present. A new uv release makes every concurrent job a miss at once, so jobs delete the tree the others are using or filling. Two observed faces, one cause: ENOENT copying into the toolcache (colliding writers), and 'Added .../uv/<ver>/x86_64 to the path' followed by '.../uv: No such file or directory' -> exit 127 (a reader's hit deleted under it; figrecipe main run 29945387939, all three matrix legs on three runners). Scoping per runner removes the sharing. Note this is why pre-seeding uv into the image does NOT fix it — _createToolPath deletes a pre-seeded tree too. /tmp is node-local and may be cleared: that degrades to a cache miss, never a dangling path. Deliberately not on GPFS — punim0264 is at 98.2% of its 8M inode limit.
|
Reviewed — recording sign-off here (shared bot identity blocks a formal approve). The mechanism is right: the destructive On your own flagged caveat — One note from my side: the three figrecipe fork PRs (#318/#319/#320) are running the CURRENT pytest-matrix (without this env) right now and going green off a freshly-populated cache — so a green there is NOT evidence for #18. The test for #18 is the log path, not a passing run. Good to merge from my side. |
The bug
Every runner on the Spartan lease inherits
RUNNER_TOOL_CACHE=~/.runner-toolcachefrom the supervisor process (verified by reading/proc/<listener-pid>/environon a live listener — no runner.envsets it; 0 of 80). So ~80 runners share one mutable cache on GPFS.@actions/tool-cache's writer is destructive, not additive (read out of the pinnedsetup-uv@v7bundle,dist/setup/index.cjs):and the reader treats dir +
.completeas a hit, adding it to PATH without checking the binary is inside:A new uv release makes every concurrent job a cache miss at once, so each one deletes the tree the others are using or filling.
Two observed faces, one cause
ENOENT: copyfile ... -> ~/.runner-toolcache/uv/<ver>/x86_64/uvAdded .../uv/0.11.31/x86_64 to the paththen.../uv: No such file or directory→ exit 127The second is figrecipe
main, run 29945387939 (2026-07-22): all three matrix legs, three different runners, same version dir. Four dated samples across three repos since 07-16.Why the previously-agreed fix would not have worked
The standing suggestion on the tracking card was to pre-seed uv into the image so
setup-uvis always a cache hit — "a pre-seeded tool cannot lose a populate race." It can:_createToolPathrmRFs a pre-seeded tree exactly like any other. Sample 4 is a cache hit losing the race. Recording it because the reasoning was sound and still wrong.The change
One step-level
envscoping the cache per runner. No sharing → no job can destroy another's tree.runnercontext is not available to job-levelenv.setup-uvwrites the resolved path to$GITHUB_PATH, so later steps resolveuvwithout needing the variable./tmp, not GPFS: node-local and may be cleared — that degrades to a cache miss (re-download, slower), never a dangling path, sincemkdirPrecreates it. Deliberately not per-runner on GPFS: punim0264 is at 98.2% of its 8M inode limit and ~80 per-runner caches there would multiply inode use.Verification
src/, the shippeddist).envblock lands on theastral-sh/setup-uv@v7step and the job structure is unchanged.Not yet proven end-to-end: that
runner.nameinterpolates as expected in step-levelenvon our self-hosted runners. First run on this repo's self-test will show it — if it renders empty the path collapses to one shared dir and we are no worse than today, but I would rather see it green before we call the flake closed.Blast radius is org-wide (every repo's pytest-matrix), which is why it is a PR. @figrecipe reviewing — this changes how your tests are wired.
🤖 Generated with Claude Code