Skip to content

perf(covgate): pre-warm build cache before parallel coverage runs#34

Merged
ben-miru merged 2 commits into
mainfrom
perf/covgate-prewarm-build-cache
Jun 24, 2026
Merged

perf(covgate): pre-warm build cache before parallel coverage runs#34
ben-miru merged 2 commits into
mainfrom
perf/covgate-prewarm-build-cache

Conversation

@ben-miru

Copy link
Copy Markdown
Contributor

What

Add a build-cache pre-warm pass to the covgate service so its parallel per-package coverage runs don't stampede the Go build cache.

Why

covgate fans out N parallel go test -coverpkg=<pkg> processes (N = effective parallelism, default runtime.NumCPU()). When they start together, each independently begins compiling the same shared dependency packages before any finishes populating ~/.cache/go-build — and Go does not lock build actions across separate go test processes. The result is a compile stampede that roughly doubles per-package wall time on heavy dependency graphs.

Proven downstream (the backend repo): inserting a single coherent go test -run='^$' compile pass before covgate cut its wall time from ~188s → ~44s. That was a CI-side stopgap; the fix belongs here, because covgate is what creates the stampede — and fixing it in the tool also speeds local runs (e.g. preflight), which suffer the same contention.

Key insight: the stampede is on the shared non-instrumented dependency compiles, not the per-package instrumented target (each run instruments only its own small, unique -coverpkg target — never a shared contention point). So a single plain compile pass is sufficient; the warm pass does not replicate per-package instrumentation.

How

  • New prewarm func(testPaths []string) error seam on the runner struct (mirrors the existing goModule/goListPackages/measure seams), wired in Run() to a new gocover.PrewarmBuild, which runs one go test -run=^$ <paths...> (no -coverpkg/-coverprofile) and propagates any build error with combined output.
  • collectWarmPaths(pkgs, ctx) reuses the same gocover.RelPkg + gocover.BuildTestPaths(...) calls checkPackage already uses — unioned and de-duplicated — so the warmed set exactly matches what the real runs build (no duplicated path logic).
  • Inserted in (*runner).run after checkPackageCtx is built and before the start := time.Now() bracket, gated on parallelism > 1 && len(pkgs) > 1 (no concurrency ⇒ no stampede ⇒ no warm pass).
  • Runs no tests (-run=^$), needs no DB/external services, and is excluded from the reported "Total time" (still measured only around runPackages), so covgate's output stays byte-identical.

Tests

  • TestRun_Prewarm*: warm pass invoked once with the full package/test-path set when parallel + multi-package; skipped when serial or single-package; warm-pass build error propagated and measure never called.
  • TestCollectWarmPaths_DeduplicatesSharedPaths; TestPrewarmBuild_Empty/_Success/_BuildError.
  • Pre-existing parallel runner{} test literals given a no-op prewarm seam so the new gated call doesn't nil-panic (found all four, not just the obvious two).

Validation

  • go build ./..., go vet ./..., repo lint (./scripts/lint.sh, custom stricter ruleset), gofumpt, and gopls modernize — all clean.
  • go test ./internal/services/covgate/... ./internal/services/gocover/... — pass. covgate gate 100.0%, gocover gate 93.3% (.covgate floor bumped for the new code).

Out of scope / pre-existing: internal/services/lint shows a LOOSE covgate threshold (64.7% actual vs 63.6% floor). Verified present on main before this change and in a package this PR does not touch — left for a separate ratchet commit (./scripts/ratchet-covgates.sh).

Downstream follow-up

Once released, backend bumps its gotools dependency and removes its CI pre-warm stopgap (keeping only the unrelated cache-save-on-main fix). See backend PR #321.

🤖 Generated with Claude Code

ben-miru and others added 2 commits June 24, 2026 15:52
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
covgate fans out N parallel `go test -coverpkg=<pkg>` processes; without a
warm cache they each begin compiling the same shared dependency packages
simultaneously (Go does not lock build actions across processes), a stampede
that roughly doubles per-package wall time on heavy deps.

Run one coherent `go test -run=^$ <paths>` compile pass before fanning out,
so shared deps land in the build cache once and the per-package runs only
re-link. Gated on parallelism > 1 && len(pkgs) > 1 (no concurrency, no
stampede). The warm pass runs no tests, needs no DB, and is excluded from
the reported "Total time" so output stays byte-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ben-miru ben-miru merged commit 0c3c0af into main Jun 24, 2026
3 checks passed
@ben-miru ben-miru deleted the perf/covgate-prewarm-build-cache branch June 24, 2026 23:12
ben-miru added a commit that referenced this pull request Jun 24, 2026
## What

Make the covgate build-cache pre-warm pass **observable**: announce it
and report its wall time.

```
Pre-warming build cache (197 packages)...
Pre-warm complete in 1m46s

Running 197 packages with parallelism=8; progress:
...
Total time: 2m01s
```

## Why

The pre-warm pass (added in #34) is a large slice of a covgate run but
was **invisible**: it's deliberately excluded from "Total time" (which
times only the parallel coverage phase) and emitted no output. So you
couldn't see how long it took, or tell whether it was helping — which is
exactly the situation that prompted this: a downstream run where the
pre-warm appeared to underperform, with no way to confirm from the logs.

## Change

- Time the pre-warm and print `Pre-warming build cache (N packages)...`
before it and `Pre-warm complete in X` after, to the same writer as the
rest of the output.
- Reorder so the pre-warm log sits **above** the "Running N packages..."
progress header (clean ordering).
- Extract the warm block into `prewarmCache(...)` so `run()` stays
within the repo's function-length limit.

No behavior change to coverage measurement, thresholds, exclude
handling, gating (`parallelism > 1 && len(pkgs) > 1`), or "Total time" —
purely additive observability.

## Tests

- Extended `TestRun_Prewarm_InvokedOnce_WhenParallelAndMultiPkg` to
assert both new lines appear.
- Extended `TestRun_Prewarm_Skipped_WhenSinglePackageOrSerial` to assert
they do **not** appear when the pass is gated off.
- Build, vet, repo lint (custom ruleset), and the covgate self-gate
(100%) all pass.

## Context

This is the diagnostic prerequisite for an observed regression: when a
downstream repo (backend) switched from a CI-side pre-warm step to
covgate's internal self-warm, the combined warm + coverage time looked
worse than the external step. With this output we can measure the
pre-warm and coverage phases separately on a clean run and confirm the
cause before changing the warm logic.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant