Concurrent / high-fan-out checkout races in the restore path (Errno 2/17/39, orphaned temp dirs)

[tbroadley]

✨ Opus 4.8 | Comment reviewed by tbroadley

Bug

When multiple pivot pull processes run concurrently against the same project (or, more rarely, within a single high-fan-out pull), the checkout/restore path races on shared parent directories and fails non-deterministically. Errors include:

Error: [Errno 2]  No such file or directory: 'data/<dir>/<output>'
Error: [Errno 17] File exists: '<cache>/files/xx/…' -> 'data/<dir>/<output>'
Error: [Errno 39] Directory not empty: 'data/<dir>/<output>'
Error: [Errno 39] Directory not empty: 'data/<dir>/.pivot_restore_xxxxxxxx' -> 'data/<dir>/<output>'

It can also leave orphaned .pivot_backup_* / .pivot_restore_* temp dirs in the workspace.

Environment

• pivot @ a recent metr-branch build (the restore path looks unchanged for a while, so likely pre-existing).
• Linux, project with many cached outputs that share parent directories (e.g. many sibling outputs under the same dir).

Reproduction

In any project with a populated cache and several outputs under shared parent dirs:

for c in 1 2 3 4 5 6; do
  pivot pull --force &   # all writing into the same workspace concurrently
done
wait

This reliably produces a handful of the errors above per batch. Single-process pulls are robust (0 failures observed across ~85 runs), so the trigger is concurrent access to shared parent directories — not any single output.

Root cause (analysis)

cli/checkout.py restores up to MAX_CONCURRENT_RESTORES = 32 outputs in parallel (asyncio.to_thread + TaskGroup, cli/checkout.py:144-179). _restore_path_sync in storage/cache.py takes a per-output-path lock (os.open(lock_path, …) + flock, ~cache.py:617), but the surrounding directory-level operations are not serialized across sibling outputs that share a parent:

• path.parent.mkdir(parents=True, exist_ok=True)
• _cleanup_stale_restore_temps(path.parent, …)
• temp-dir creation under path.parent and the final replace() / rename into place, plus _clear_path(path)

Two outputs under the same parent (or two processes restoring overlapping trees) can interleave these, yielding the File exists / Directory not empty / No such file races and leaving temp/backup dirs behind. The except* Exception handler (cli/checkout.py:181) then surfaces whatever was raised as Error: <message>.

Suggested directions

• Serialize directory-level mutations on a per-parent-directory basis (lock keyed by path.parent), or make the temp-create → clear → rename sequence fully idempotent/atomic against concurrent siblings.
• For cross-process safety, the parent-dir lock would need to be filesystem-based (like the existing per-path lock) rather than in-process.
• Ensure orphaned .pivot_restore_* / .pivot_backup_* dirs are always cleaned up on failure.