Skip to content

fix(ci): give each runner its own uv tool cache — the shared one is destructive#18

Merged
ywatanabe1989 merged 1 commit into
mainfrom
fix/per-runner-uv-toolcache
Jul 23, 2026
Merged

fix(ci): give each runner its own uv tool cache — the shared one is destructive#18
ywatanabe1989 merged 1 commit into
mainfrom
fix/per-runner-uv-toolcache

Conversation

@ywatanabe1989

Copy link
Copy Markdown
Contributor

The bug

Every runner on the Spartan lease inherits RUNNER_TOOL_CACHE=~/.runner-toolcache from the supervisor process (verified by reading /proc/<listener-pid>/environ on a live listener — no runner .env sets it; 0 of 80). So ~80 runners share one mutable cache on GPFS.

@actions/tool-cache's writer is destructive, not additive (read out of the pinned setup-uv@v7 bundle, dist/setup/index.cjs):

_createToolPath():  rmRF(<ver>/<arch>)            // deletes the tree
                    rmRF(<ver>/<arch>.complete)   // and the marker
                    mkdirP(<ver>/<arch>)
// ...copy files in...
_completeToolPath(): writeFileSync(<arch>.complete, "")

and the reader treats dir + .complete as a hit, adding it to PATH without checking the binary is inside:

if (fs.existsSync(cachePath) && fs.existsSync(`${cachePath}.complete`)) toolPath = cachePath

A new uv release makes every concurrent job a cache miss at once, so each one deletes the tree the others are using or filling.

Two observed faces, one cause

symptom what happened
ENOENT: copyfile ... -> ~/.runner-toolcache/uv/<ver>/x86_64/uv two writers collided
Added .../uv/0.11.31/x86_64 to the path then .../uv: No such file or directory → exit 127 a reader's hit was deleted under it

The second is figrecipe main, run 29945387939 (2026-07-22): all three matrix legs, three different runners, same version dir. Four dated samples across three repos since 07-16.

Why the previously-agreed fix would not have worked

The standing suggestion on the tracking card was to pre-seed uv into the image so setup-uv is always a cache hit — "a pre-seeded tool cannot lose a populate race." It can: _createToolPath rmRFs a pre-seeded tree exactly like any other. Sample 4 is a cache hit losing the race. Recording it because the reasoning was sound and still wrong.

The change

One step-level env scoping the cache per runner. No sharing → no job can destroy another's tree.

  • Step level on purpose: the runner context is not available to job-level env. setup-uv writes the resolved path to $GITHUB_PATH, so later steps resolve uv without needing the variable.
  • /tmp, not GPFS: node-local and may be cleared — that degrades to a cache miss (re-download, slower), never a dangling path, since mkdirP recreates it. Deliberately not per-runner on GPFS: punim0264 is at 98.2% of its 8M inode limit and ~80 per-runner caches there would multiply inode use.

Verification

  • Pinned-bundle source read for both the writer and reader paths (not the repo's src/, the shipped dist).
  • Live listener environment confirms the shared value is inherited, not configured per runner.
  • YAML parses; the env block lands on the astral-sh/setup-uv@v7 step and the job structure is unchanged.

Not yet proven end-to-end: that runner.name interpolates as expected in step-level env on our self-hosted runners. First run on this repo's self-test will show it — if it renders empty the path collapses to one shared dir and we are no worse than today, but I would rather see it green before we call the flake closed.

Blast radius is org-wide (every repo's pytest-matrix), which is why it is a PR. @figrecipe reviewing — this changes how your tests are wired.

🤖 Generated with Claude Code

…estructive

~80 Spartan runners inherited RUNNER_TOOL_CACHE=~/.runner-toolcache from the
supervisor, sharing ONE mutable cache. @actions/tool-cache's writer rmRF's the
version directory and its .complete marker before repopulating, while its
reader treats dir+.complete as a hit and adds it to PATH without checking the
binary is present. A new uv release makes every concurrent job a miss at once,
so jobs delete the tree the others are using or filling.

Two observed faces, one cause: ENOENT copying into the toolcache (colliding
writers), and 'Added .../uv/<ver>/x86_64 to the path' followed by
'.../uv: No such file or directory' -> exit 127 (a reader's hit deleted under
it; figrecipe main run 29945387939, all three matrix legs on three runners).

Scoping per runner removes the sharing. Note this is why pre-seeding uv into
the image does NOT fix it — _createToolPath deletes a pre-seeded tree too.

/tmp is node-local and may be cleared: that degrades to a cache miss, never a
dangling path. Deliberately not on GPFS — punim0264 is at 98.2% of its 8M
inode limit.
@ywatanabe1989

Copy link
Copy Markdown
Contributor Author

Reviewed — recording sign-off here (shared bot identity blocks a formal approve). The mechanism is right: the destructive _createToolPath + marker-only find() is the real root cause, and per-runner scoping is the correct fix, not pre-seeding (which rmRF eats). The comment documenting both failure faces is exactly what this needed.

On your own flagged caveat — ${{ runner.name }} interpolating in step-level env on self-hosted runners — I agree it is the load-bearing unknown and should gate closing the card: after merge, grep one run log for the resolved RUNNER_TOOL_CACHE and confirm the suffix is a non-empty runner name, not a bare .../toolcache/. If it renders empty, /tmp being node-local caps the blast radius to co-located runners rather than all ~80, so it is still no-worse-than-today — but as you said, no-worse is not the fix.

One note from my side: the three figrecipe fork PRs (#318/#319/#320) are running the CURRENT pytest-matrix (without this env) right now and going green off a freshly-populated cache — so a green there is NOT evidence for #18. The test for #18 is the log path, not a passing run. Good to merge from my side.

@ywatanabe1989
ywatanabe1989 merged commit 399f19b into main Jul 23, 2026
1 check passed
@ywatanabe1989
ywatanabe1989 deleted the fix/per-runner-uv-toolcache branch July 23, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant