macOS: APFS clonefile fast path + concurrency cap + zero-byte fix for Bazel input materialization

[erneestoc]

Summary

Four small, atomic commits that make macOS RBE workers materialize Bazel
action input trees dramatically faster on APFS. Extracted from
TraceMachina/nativelink#2243
as the safest, most independently-verifiable subset.

Commit	Scope
b3b0cd3f	fs_util: APFS clonefile(2) fast path in hardlink_directory_tree (with set_readwrite_recursive so cloned trees stay writable). Falls through to per-file hard_link on Linux/Windows.
ab49f162	DirectoryCache: clonefile_hits / hardlink_hits AtomicU64 telemetry so production can confirm the fast path is actually firing.
1ddce0fc	running_actions_manager::download_to_directory: cap concurrent hard_link calls at 64 via stream::buffer_unordered (was unbounded FuturesUnordered).
8051ca9e	DirectoryCache: don't crash when the CAS has no entry for the zero-byte digest — short-circuit and create an empty file.

Why

Bazel exec-log analysis on our worker fleet shows that file fetching
dominates wall time on macOS RBE workers:

Metric	Value
Sum of action durations (process wall time)	4,947 s
Sum of file fetching durations	7,313 s
Critical path	1,074 s
Total wall time	1,213 s

File-fetching wall time exceeds all process wall time combined — the
worker is spending more time materializing input trees than running
actions. The dominant shape is SwiftCompile (635 input files / 183 MB
mean, 1,978 files / 466 MB at p95), exactly the shape APFS
clonefile(2) was designed for.

PR TraceMachina#2243 on upstream is large and changes many unrelated things. We're
pulling out just the four pieces that (a) directly attack the
file-fetching bottleneck on macOS and (b) can be A/B tested
independently with the telemetry in ab49f162.

Benchmark proof

See #1 — adds two criterion
benches in `nativelink-util` that reproduce the wins on a real APFS
volume.

Headline numbers on Apple M4 Max / APFS:

• `hardlink_directory_tree` (b3b0cd3) — 3.6×–4.0× faster across
  all five SwiftCompile-shaped trees. p95 shape (1,978 files / 466 MB)
  drops from 590 ms → 150 ms per action. Scaled by 814 such actions
  per CI build, that's a ~6 minute upper-bound saving per build from
  this single optimization.
• `download_to_directory` 64-cap (1ddce0f) — performance-neutral
  on a single-process microbench (1.01× vs unbounded at 1,978 files).
  Confirms the cap is not a regression; multi-action contention wins
  will show up in production telemetry.

Caveat: 4× not 10×

The handoff predicted ≥10× based on upstream PR TraceMachina#2243's reported wins.
We see a stable ~4×, bounded by the O(N) `set_readwrite_recursive`
chmod walk that runs after the O(1) `clonefile(2)`. Diagnosis and
follow-up (single top-level chmod + lazy per-file) documented in the
bench PR. 4× on the dominant shape already moves the needle hard —
shipping now, optimizing the walk later.

Telemetry & rollback

`ab49f162` adds `DirectoryCache::stats()` with `clonefile_hits` and
`hardlink_hits` counters so we can confirm in production:

• macOS workers should show `clonefile_hits` climbing and
  `hardlink_hits` near zero
• Linux workers should show the inverse (clonefile path is
  macOS-gated at compile time)

If anything looks wrong, the four commits revert independently — each
is its own atomic change with its own test coverage.

Security

10 `fs_util` tests cover the clonefile path (COW isolation, symlink
handling under `CLONE_NOFOLLOW`, error paths, permission transitions
from `0o555` → `0o755`). 2 `DirectoryCache` tests cover the
telemetry counters and the zero-byte short-circuit. All green on macOS
arm64 and Linux x86_64. Details in the bench PR.

What's deliberately NOT included

PR TraceMachina#2243 also touches `has_with_results` chunking and BFS-parallel
`mkdir` — those are deferred because they require their own bench
infrastructure and aren't on the file-fetching critical path. Easy to
pull in as a follow-up if we want them.

🤖 Generated with Claude Code