test(graph): make the fixture diagnostic see the object store, and git's own stderr

[arozumenko]

Seven occurrences of the packages/graph e2e fixture fault have produced no root cause. This PR fixes the reason why, and records what a day of local experiments eliminated.

The diagnostic was blind to the only thing that was ever wrong

describeRepoState checked HEAD, refs/heads and index.lock — all of which are intact in every observed failure — and never looked at the object store, which is what every observed failure is actually about. Its catch handler also did .split("\n")[0], keeping Node's Command failed: <argv> line (which we already know) and discarding git's stderr underneath it, where git says what it could not read.

The one occurrence that named its own cause got there through the separate two-line slice in step, not through the diagnostic.

Now added: fsck --connectivity-only, cat-file -t HEAD, the loose-object-directory count, the pack list, df, and the real stderr on every probe — enough to say which object is gone and whether the store was emptied or has one bad member.

What the experiments established

The message means exactly one thing. Deleting HEAD's tree object reproduces error: bad tree object HEAD byte-for-byte on git 2.39 (Linux, container) and git 2.50 (macOS). It is a missing tree, not a corrupt one and not a ref problem.

CI's damage is bigger than one object. In that simulation git rev-list --count --all survived — rev-list walks commits, not trees. On CI it failed. So CI lost commit objects as well: a chunk of the store went missing, which is why the "one transient hiccup" model never fit.

Disk exhaustion is not the mechanism. Forcing ENOSPC on a size-capped filesystem makes git fail loudly (fatal: … write error. Out of diskspace) and leaves the store healthy — fsck clean, rev-parse HEAD and cat-file -t HEAD both fine. That is the opposite of the flake. The prior note ruled disk pressure out by measuring the suite's ~5 MB footprint; this rules it out by showing the failure mode is wrong at any level of fullness.

The fixture's git sequence alone is decisively not sufficient. 3,100 fixture builds — 2,100 on macOS at 14 workers, 1,000 on Linux at 4 CPUs — ≈ 589,000 git invocations, zero failures. Each build is one test (g) equivalent, and the 1.9%/suite rate is concentrated in that test, so if the sequence alone carried the risk this should have produced ~59 failures. P(0) ≈ e⁻⁵⁹. Something about the CI environment or the surrounding suite is required.

No fixture is built at module or describe scope, so no fixture can be deleted out from under a later test by mkdtempClean's onTestFinished. Every suspicious-looking call site is inside a helper function called from a test body.

What it did NOT establish

Root cause. I did not find it, and I am not claiming it.

The 25 full-suite Linux runs were clean but underpowered: at 1.9%/run the chance of seeing zero failures is ≈62%, so they rule out nothing. Meaningful power needs ~150 runs (P(0) ≈ 5.6%). The 3,100-build figure above is the only statistically strong result here, and it is an elimination, not a cause.

Verification

pnpm --filter @octoshell/graph test — fixture-retry (2) and conventions (30) green against the changed file; 25 clean full-suite runs on Linux.

Knowledge note update follows once #95 merges (its seventh-observation section lives on that branch).

🤖 Generated with Claude Code