release-26.2: roachtest: rebalance/by-load/replicas/mixed-version failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on release-26.2 @ cc34a3eefe9f9548beaa55ced23d3503332bac20:

Failed with:

(mixedversion.go:926).Run: mixed-version test failure while running step 51 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=74.9 tolerance=20.0% (±15.0) bounds=[59.9, 89.8]
	below  = []
	within = [s2: 65 (-12.5%), s3: 74 (-0.2%), s4: 72 (-3.8%), s5: 71 (-5.0%), s6: 69 (-7.6%)]
	above  = [s1: 96 (+29.1%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21273442-1773901322-273-n7cpu4-0001	35.229.29.146	10.142.0.154
teamcity-21273442-1773901322-273-n7cpu4-0002	34.24.188.146	10.142.0.166
teamcity-21273442-1773901322-273-n7cpu4-0003	35.185.17.15	10.142.0.167
teamcity-21273442-1773901322-273-n7cpu4-0004	34.139.78.29	10.142.0.108
teamcity-21273442-1773901322-273-n7cpu4-0005	104.196.24.191	10.142.0.94
teamcity-21273442-1773901322-273-n7cpu4-0006	34.138.242.131	10.142.0.82
teamcity-21273442-1773901322-273-n7cpu4-0007	34.26.232.208	10.142.0.114

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• mvtDeploymentMode=shared-process
• mvtVersions=v25.2.14 → v25.3.7 → v25.4.5 → v26.2.0-beta.1-dev-cc34a3eefe9f9548beaa55ced23d3503332bac20 (release-26.2)
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-61806

[pav-kv]

/investigate

[github-actions]

Investigation: rebalance/by-load/replicas/mixed-version

Investigated failure: #166218 (issue body)
Failure SHA: cc34a3eefe9f9548beaa55ced23d3503332bac20
Confidence: moderate

What This Test Does

The rebalance/by-load/replicas/mixed-version roachtest creates a 6-node CockroachDB cluster, runs a mixed-version upgrade sequence (v25.2.14 → v25.3.7 → v25.4.5 → v26.2.0-beta.1-dev), and during each upgrade stage runs a KV workload. The test asserts that the host CPU utilization across all nodes stays within ±20% of the mean — i.e., that the store rebalancer successfully distributes load. The test allows 15 minutes for CPU to balance and requires 1 minute of sustained balance to pass.

Where the Failure Occurs

The test fails at step 51, which is the "rebalance load run" step executed during the upgrade from v25.4.5 to v26.2.0-beta.1-dev. At this point, only node 1 has been upgraded to v26.2 (step 50) while nodes 2–6 remain on v25.4.5. The failure is in rebalance_load.go:332:

CPU not evenly balanced after timeout: outside bounds mean=74.9 tolerance=20.0% (±15.0) bounds=[59.9, 89.8]
  below  = []
  within = [s2: 65 (-12.5%), s3: 74 (-0.2%), s4: 72 (-3.8%), s5: 71 (-5.0%), s6: 69 (-7.6%)]
  above  = [s1: 96 (+29.1%)]

Node 1 (the sole v26.2 node) is at 96% host CPU while others are at 65–74%.

Analysis

Hypothesis 1 (likely cause, moderate confidence): The v26.2 runtime assertions build consumes significantly more host CPU than the v25.4.5 non-assertions builds, creating an imbalance the store rebalancer cannot resolve.

Evidence:

1. Runtime assertions build: This is a runtime assertions build (runtimeAssertionsBuild=true). Only the current version (v26.2 on n1) has assertions enabled; the predecessor binaries (v25.4.5 on n2–n6) are standard builds without assertions. Assertions add overhead (e.g., extra validation checks, debug.panic_on_failed_assertions) that inflate CPU on the upgraded node.

2. Store-level CPU is balanced, but host CPU is not: The store rebalancer logs show s1's store-level cpu-per-second settled to ~525ms, close to the mean of ~470ms and upper threshold of ~520ms. The rebalancer ran out of leases and replicas to transfer. Meanwhile, the host CPU (what the test asserts on) remained at 96% on n1 vs ~70% on other nodes. This ~25% gap is entirely non-store CPU — CPU consumed by background processes, assertions overhead, SQL execution overhead, or other system work that the store rebalancer does not account for.

3. Consistent pattern across all samples: For the entire 15-minute test duration, s1 never dropped below 93% host CPU. The imbalance persisted without any sign of convergence — the gap is structural, not transient.

4. Previous runs in the same test balanced quickly: Steps 10, 17, 24, 31, and 38 (earlier upgrade stages where both nodes ran similar-vintage binaries) all balanced in ~2 minutes. Step 51 is the first time a single node runs a much newer binary with assertions against older non-assertions binaries.

5. Known issue pattern: The code at rebalance_load.go:128-133 explicitly notes: "the CPU utilization attributed to SQL can vary significantly between versions, which can lead to flakiness in these tests since the StoreRebalancer, which operates at the store level, is unaware of such CPU use". The meanCPUTolerance of 20% (with a TODO at line 46-49 referencing roachtest: rebalance/by-load/replicas/ssds=2 failed #109768) was intended to buffer this, but 20% is insufficient when one node has assertions-inflated overhead against non-assertions predecessors.

Hypothesis 2 (possible, low confidence): The v26.2 binary inherently consumes more CPU than v25.4.5 even without assertions.

The v26.2-beta.1-dev binary may have inherently higher CPU usage due to new features, additional background tasks, or changes in SQL execution cost. This would compound with the assertions overhead. Without a non-assertions failure to compare against, this remains speculative.

Related Issues and PRs

• #161339 — Same test failure on master, 20+ occurrences. Same pattern: one node (always the one with the newest binary) has elevated host CPU. This is the canonical tracking issue.
• #163542 — Same failure on release-25.2, marked as dup of roachtest: rebalance/by-load/replicas/mixed-version failed #161339.
• #150603 — Referenced in test code; documents SQL CPU variance between versions causing flakiness.
• #109768 — Referenced in TODO; tracks unattributed CPU usage (SQL execution, stats collection, compactions) not visible to the store rebalancer.
• #161688, #161601, #160835, #158758, #154838, #147487 — Prior closed issues with the same test and similar failure mode.
• #161794 — Similar failure for rebalance/by-load/leases/mixed-version.

Timeline

• The test has been flaking intermittently across many branches since at least 2024, with the same "one node has higher CPU" pattern.
• The MinimumSupportedVersion("v25.1.0") constraint was added to reduce flakiness from large version gaps.
• In this specific run, node 1 was restarted with the v26.2 binary at 21:07:01 (step 50), the rebalance test started at 21:07:18 (step 51), and it ran until 21:22:35 — the full 15-minute timeout. At no point during this window did s1's host CPU drop below 93%.

Detailed evidence and investigation notes

Log Excerpts

Store rebalancer on s1 — initial state after upgrade (21:08):

considering load-based lease transfers for s1 with (cpu-per-second=1.1s) load, mean=(cpu-per-second=463ms)

s1 starts with ~2.4× the mean store CPU. The rebalancer transfers ~10 leases away.

Store rebalancer on s1 — steady state (21:19–21:22):

considering load-based lease transfers for s1 with (cpu-per-second=525ms) load, mean=(cpu-per-second=471ms), upperThreshold=(cpu-per-second=521ms)
ran out of leases worth transferring... still above desired threshold
ran out of replicas worth transferring... will check again soon

s1's store CPU converged to near the mean (~525ms vs 471ms mean, threshold 521ms), but it has no more leases to shed. The store rebalancer is stuck — it believes load is nearly balanced.

Meanwhile, host CPU tells a different story:

cpu outside bounds mean=74.9 tolerance=20.0% (±15.0) bounds=[59.9, 89.8]
  within = [s2: 65, s3: 74, s4: 72, s5: 71, s6: 69]
  above  = [s1: 96 (+29.1%)]

~25 percentage points of CPU on n1 are consumed by something outside the store rebalancer's view.

Mixed-version state at failure:

n1: v26.2.0-beta.1-dev (release-26.2) — runtime assertions enabled
n2-n6: v25.4.5 — standard builds (no assertions)
cluster version: 25.4

Code References

• rebalance_load.go:332 — Error originates from the cpu-balance goroutine
• rebalance_load.go:320 — isLoadEvenlyDistributed check using meanCPUTolerance (20%)
• rebalance_load.go:354 — cr.node.sys.cpu.host.combined.percent-normalized is the metric checked (host CPU, not store CPU)
• rebalance_load.go:128-133 — Comment about CPU utilization variance between versions

Things Ruled Out

• Store-level CPU imbalance: The store rebalancer successfully reduced s1's store CPU from 1.1s to ~525ms (near the 471ms mean). The issue is not that the rebalancer failed to act — it's that host CPU includes overhead the rebalancer cannot address.
• Lease/replica count imbalance: s1 ended with ~1940 QPS, roughly half the cluster mean of ~3877 QPS. It actually had fewer leases than other stores. Moving more leases away wouldn't help because the excess CPU isn't from store work.
• Transient spike: The imbalance persisted for the entire 15 minutes with no sign of convergence. s1 was consistently at 93–97% CPU.
• Infrastructure issue: The test ran on GCE with identical 4-vCPU machines. Earlier upgrade stages balanced within ~2 minutes, ruling out a bad machine.

Other Failure Occurrences

All failure occurrences in #161339 and related issues show the same pattern:

• One node (always the one with the newest binary) has ~20–30% higher host CPU than others
• The store rebalancer converges on store-level CPU but cannot address host-level overhead
• The 20% tolerance is consistently exceeded by ~5–10 percentage points
• Runtime assertions builds appear more frequently, suggesting assertions overhead is a contributing factor

Tooling Feedback

No significant limitations encountered. All artifacts were accessible via TeamCity guest authentication.

---

Workflow run

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[pav-kv]

Dup of #161339