controller: build benchmarks without debuginfo by default - #11
Merged
Conversation
Pods set CARGO_PROFILE_RELEASE_DEBUG=1 to keep DWARF symbols for gdb dumps
of hung jobs. That makes rustc and the linker use far more memory. On a live
wide_schema run I watched a single rustc climb to ~16 GB and the pod cgroup
reach its 65 GiB limit and OOM-kill — during compilation, before any
benchmark query executed. (The wide_schema queries themselves use ~1 GB.)
The wide_schema "run" step compiles via `cargo bench --bench sql`, so the
heavy compile lands inside the monitored window and is repeated per side.
Drop the default so builds are fast and lean. The behavior is still
available per-run: the trigger comment's shared `env:` block is inherited by
the build processes (spawn_command inherits the pod env), so a run can opt
back in with:
run benchmark <name>
env:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
(Per-side baseline:/changed: env only reaches the benchmark run, not the
build, so build flags belong in the shared block.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Benchmark pods set
CARGO_PROFILE_RELEASE_DEBUG=1(to keep DWARF symbols for gdb dumps of hung jobs). Building with debuginfo makesrustcand the linker use dramatically more memory.I debugged a live
wide_schemarun on the cluster (kubectl) end to end. The OOM is a compile-time OOM, not the benchmark:anon(real RSS)cargo bench --bench sqlcompile)The pod died in
./bench.sh run wide_schema→cargo bench --bench sql, i.e. while compiling the criterion SQL bench with debuginfo at-j12. The wide_schema queries themselves use ~1 GB and never ran.wide_schemais hit hardest because its "run" step compiles a bench binary inside the monitored window, once per side.Fix
Stop setting
CARGO_PROFILE_RELEASE_DEBUG=1by default — builds are then fast and lean (a single fast lever; without debuginfo,-j12fits comfortably).The behavior is still available per run: the trigger comment's shared
env:block is inherited by the build processes (spawn_commanddoesn't clear the env), so a run can opt back in:CARGO_BUILD_JOBS: "1"serializes codegen so peak ≈ one rustc (~16 GB) instead of 12×, staying within the 65 GiB limit. Per-sidebaseline:/changed:env only reaches the benchmark run (not the build), so build flags belong in the shared block — the help message is updated to say so.Follow-ups (not in this PR)
memory.current/memory.peak), so it includes the compiler — it currently attributes rustc's memory to the benchmark. Worth measuring the bench process RSS, or starting the monitor only after compilation.Testing
cargo fmt --check,cargo clippy -- -D warnings,cargo test(134 lib tests) all pass.🤖 Generated with Claude Code