roachtest: perturbation/full/partition failed

[cockroach-teamcity]

roachtest.perturbation/full/partition failed with artifacts on master @ 64a34e4b79b6956c8a2653139cddbf34239d5ae3:

Failed with:

(assertions.go:387).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/perturbation/framework.go:1044
	            				pkg/cmd/roachtest/test_runner.go:1561
	            				src/runtime/asm_amd64.s:1771
	Error:      	FAILURE: follower-read  : throughput ratio 1.73x exceeds limit 1.2x (baseline: 10276 ops/s, measured: 5955 ops/s)
	            	FAILURE: read           : throughput ratio 1.69x exceeds limit 1.2x (baseline: 10276 ops/s, measured: 6077 ops/s)
	            	FAILURE: write          : throughput ratio 1.69x exceeds limit 1.2x (baseline: 20648 ops/s, measured: 12235 ops/s)
	Test:       	perturbation/full/partition
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/partition/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21370099-1780466536-200-n13cpu16sm-0001	35.231.35.33	10.142.3.30
teamcity-21370099-1780466536-200-n13cpu16sm-0002	35.243.174.119	10.142.1.197
teamcity-21370099-1780466536-200-n13cpu16sm-0003	35.231.46.188	10.142.3.23
teamcity-21370099-1780466536-200-n13cpu16sm-0004	34.139.0.223	10.142.1.108
teamcity-21370099-1780466536-200-n13cpu16sm-0005	34.23.207.210	10.142.2.156
teamcity-21370099-1780466536-200-n13cpu16sm-0006	35.190.139.151	10.142.1.209
teamcity-21370099-1780466536-200-n13cpu16sm-0007	34.138.81.9	10.142.3.24
teamcity-21370099-1780466536-200-n13cpu16sm-0008	34.24.53.96	10.142.3.35
teamcity-21370099-1780466536-200-n13cpu16sm-0009	34.139.215.44	10.142.3.27
teamcity-21370099-1780466536-200-n13cpu16sm-0010	35.227.7.31	10.142.3.29
teamcity-21370099-1780466536-200-n13cpu16sm-0011	34.24.223.134	10.142.2.237
teamcity-21370099-1780466536-200-n13cpu16sm-0012	34.26.20.38	10.142.3.22
teamcity-21370099-1780466536-200-n13cpu16sm-0013	35.231.102.212	10.142.3.25

Parameters:

• acMode=defaultOption
• arch=amd64
• blockSize=4096
• cloud=gce
• coverageBuild=false
• cpu=16
• diskBandwidthLimit=0
• diskCount=2
• disks=2
• encrypted=false
• fillDuration=10m0s
• leaseType=expiration
• mem=standard
• numNodes=12
• numWorkloadNodes=1
• perturbationDuration=10m0s
• ratioOfMax=0.5
• runtimeAssertionsBuild=false
• seed=0
• splits=10000
• validationDuration=5m0s
• vcpu=16

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-64505

[cockroach-teamcity]

roachtest.perturbation/full/partition failed with artifacts on master @ d03408200b31b486a9d9ca7afa8488d0738b23a3:

Failed with:

(assertions.go:387).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/perturbation/framework.go:1044
	            				pkg/cmd/roachtest/test_runner.go:1561
	            				src/runtime/asm_amd64.s:1771
	Error:      	FAILURE: follower-read  : throughput ratio 1.52x exceeds limit 1.2x (baseline: 10335 ops/s, measured: 6799 ops/s)
	            	FAILURE: read           : throughput ratio 1.58x exceeds limit 1.2x (baseline: 10285 ops/s, measured: 6526 ops/s)
	            	FAILURE: write          : throughput ratio 1.51x exceeds limit 1.2x (baseline: 20765 ops/s, measured: 13734 ops/s)
	Test:       	perturbation/full/partition
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/partition/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21371966-1780552855-203-n13cpu16sm-0001	34.23.161.213	10.142.0.70
teamcity-21371966-1780552855-203-n13cpu16sm-0002	34.75.223.156	10.142.0.37
teamcity-21371966-1780552855-203-n13cpu16sm-0003	34.148.162.136	10.142.0.169
teamcity-21371966-1780552855-203-n13cpu16sm-0004	34.139.176.250	10.142.0.47
teamcity-21371966-1780552855-203-n13cpu16sm-0005	35.229.70.249	10.142.0.101
teamcity-21371966-1780552855-203-n13cpu16sm-0006	34.24.166.156	10.142.0.156
teamcity-21371966-1780552855-203-n13cpu16sm-0007	34.23.84.198	10.142.0.206
teamcity-21371966-1780552855-203-n13cpu16sm-0008	34.138.71.132	10.142.0.177
teamcity-21371966-1780552855-203-n13cpu16sm-0009	35.237.31.72	10.142.0.204
teamcity-21371966-1780552855-203-n13cpu16sm-0010	35.196.182.49	10.142.0.97
teamcity-21371966-1780552855-203-n13cpu16sm-0011	35.196.193.231	10.142.0.105
teamcity-21371966-1780552855-203-n13cpu16sm-0012	34.139.135.26	10.142.0.52
teamcity-21371966-1780552855-203-n13cpu16sm-0013	34.75.113.100	10.142.0.35

Parameters:

• acMode=defaultOption
• arch=amd64
• blockSize=4096
• cloud=gce
• coverageBuild=false
• cpu=16
• diskBandwidthLimit=0
• diskCount=2
• disks=2
• encrypted=false
• fillDuration=10m0s
• leaseType=expiration
• mem=standard
• numNodes=12
• numWorkloadNodes=1
• perturbationDuration=10m0s
• ratioOfMax=0.5
• runtimeAssertionsBuild=false
• seed=0
• splits=10000
• validationDuration=5m0s
• vcpu=16

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!

[stevendanna]

/investigate

[github-actions]

Investigation: perturbation/full/partition

Investigated failure: comment on Jun 4
Failure SHA: d03408200b31b486a9d9ca7afa8488d0738b23a3
Confidence: moderate

Warning: Could not check out failure SHA d03408200b31b486a9d9ca7afa8488d0738b23a3. Analysis is based on the default branch tip, which may differ from the code that produced the failure.

What This Test Does

The perturbation/full/partition test measures how well CockroachDB recovers from a region-wide network partition. It sets up a 12-node cluster across 3 regions (4 nodes each), runs a KV workload at 50% of max throughput, then uses iptables to partition the first region (nodes 1–4) from the target node (node 12) for 10 minutes. The test's pass/fail signal is whether throughput returns to within 1.25x of baseline during the 5-minute recovery window after the partition heals.

Where the Failure Occurs

The failure occurs at framework.go:1044 during validation of the recovery interval (T4→T5). The perturbation interval itself passes fine (uses noImpactThresholds()). The recovery interval uses defaultThresholds() which enforces a 1.25x throughput floor.

All three workload operation types fail the recovery throughput check:

Operation	Baseline p5(tput/s)	Recovery p5(tput/s)	Ratio	Limit
follower-read	10,335	6,799	1.52x	1.25x
read	10,285	6,526	1.58x	1.25x
write	20,765	13,734	1.51x	1.25x

The median throughput during recovery is nearly identical to baseline (~10,450 / ~20,900 ops/s). The problem is entirely in the p5 (worst 5%) of per-second throughput ticks, which drop to ~60-66% of baseline. This means ~15 seconds out of the 302-second recovery window have significantly degraded throughput—consistent with a brief but intense rebalancing storm immediately after the partition heals.

Analysis

Hypothesis 1: The 5-minute recovery window is too short for the cluster to fully rebalance after a region-wide partition (likely cause, moderate confidence)

The partition isolates 4 of 12 nodes for 10 minutes. When it heals, several things happen simultaneously:

• Expiration leases held by partitioned nodes have expired and been acquired by other nodes; the original leaseholders now need leases back.
• Raft leadership must be re-established for ranges where the partitioned nodes were leaders.
• Snapshot transfers may be needed to bring partitioned replicas up to date.

The recovery stats show very high p99 latencies (88ms follower-read, 44ms read, 402ms write p95-of-p99) alongside the throughput dip. The median throughput is fine, meaning recovery completes within seconds for most of the window, but the initial rebalancing storm causes a brief severe dip that is enough to push the p5 throughput metric below the 1.25x threshold.

This test was re-enabled on May 27 in d43059fe44c with recoveryImpact = defaultThresholds() (1.25x throughput floor). Both failures since re-enablement show the same pattern, suggesting the threshold may be too tight for a region-wide partition recovery. Previously, this test was skipped via #149662 specifically because such thresholds were being stabilized.

Possible fixes (ordered by simplicity):

1. Increase the recovery throughput threshold for the partition test to e.g. 1.75x or 2.0x, acknowledging that region-wide partition recovery produces a brief rebalancing storm.
2. Extend the recovery window beyond 5 minutes so the brief dip is diluted by more baseline-level seconds.
3. Exclude the first N seconds of recovery (similar to how the perturbation measurement already excludes the first 20 seconds in startPerturbation), to allow the rebalancing storm to settle before measuring.

Hypothesis 2: A regression in lease rebalancing or Raft recovery speed (low confidence)

No specific KV changes between May 27 (re-enable) and June 3 (first failure) appear to target lease transfer or partition recovery behavior. The recent allocator/MMA prototype changes don't affect the rebalancer in production configurations.

Related Issues and PRs

• roachtest: perturbation/full/partition failed #130772 — Prior perturbation/full/partition failure (closed, Sep 2024). Different failure mode: permission denied when creating artifacts directory, not a throughput regression.
• kv,roachtest: investigate and recalibrate perturbation tests #149662 — Skipped perturbation/full tests to stabilize one test at a time. Resolved by d43059fe44c which re-enabled them.
• gossip: partition recovery is slow enough to trigger false-positives in ranges.unavailable #133159 — gossip: partition recovery is slow enough to trigger false-positives in ranges.unavailable (closed). Addressed gossip recovery slowness after partitions.
• kv: partial/asymmetric partition that does not isolate node liveness does not force range lease revocation #49220 — kv: partial/asymmetric partition that does not isolate node liveness does not force range lease revocation (open). Describes partition scenarios where lease handling is suboptimal.
• kvserver: exclude rejoined nodes as lease transfer targets for a longer suspect duration than failed heartbeats #132796 — kvserver: exclude rejoined nodes as lease transfer targets for a longer suspect duration (closed). Addressed QoS impact when nodes rejoin after partitions.
• kvcoord: improve DistSender circuit breaker recovery time #121209 — kvcoord: improve DistSender circuit breaker recovery time (open). Addresses recovery time after failures.

Timeline

Date	Event
May 27, 2026	d43059fe44c re-enables perturbation/full/partition with recoveryImpact = defaultThresholds() (1.25x throughput floor)
Jun 3, 2026	First failure on SHA 64a34e4b79b (build 21370099): recovery throughput 1.69-1.73x
Jun 4, 2026	Second failure on SHA d03408200b3 (build 21371966): recovery throughput 1.51-1.58x

The test has failed every time it has run since re-enablement (2/2 runs), indicating this is not a flake but a systematic threshold mismatch.

Detailed evidence and investigation notes

Log Excerpts

Second failure (build 21371966) — recovery stats:

Recovery stats
follower-read  : median(p99/s): 1.7ms, median(tput/s): 10449 ops/s, p95(p99/s): 88.1ms, p95(p50/s): 1.2ms, p5(tput/s): 6799 ops/s (302 ticks)
read           : median(p99/s): 1.5ms, median(tput/s): 10450 ops/s, p95(p99/s): 44ms, p95(p50/s): 1.1ms, p5(tput/s): 6526 ops/s (302 ticks)
write          : median(p99/s): 7.9ms, median(tput/s): 20915 ops/s, p95(p99/s): 402.7ms, p95(p50/s): 2.8ms, p5(tput/s): 13734 ops/s (302 ticks)

First failure (build 21370099) — recovery stats:

Recovery stats
follower-read  : median(p99/s): 1.9ms, median(tput/s): 10421 ops/s, p95(p99/s): 46.1ms, p95(p50/s): 1.3ms, p5(tput/s): 5955 ops/s (302 ticks)
read           : median(p99/s): 1.7ms, median(tput/s): 10410 ops/s, p95(p99/s): 41.9ms, p95(p50/s): 1.2ms, p5(tput/s): 6077 ops/s (302 ticks)
write          : median(p99/s): 8.9ms, median(tput/s): 20812 ops/s, p95(p99/s): 251.7ms, p95(p50/s): 4.2ms, p5(tput/s): 12235 ops/s (302 ticks)

Key observations:

• Median throughput during recovery is nearly identical to baseline in both runs (~10,400-10,450 read; ~20,800-20,900 write).
• The p5 (worst 5%) throughput is where the failure is: ~5,955-6,799 for reads vs ~10,276-10,335 baseline.
• The p95 of p99 latency during recovery is extremely elevated (write: 251-402ms vs 10-11ms baseline), confirming a brief intense disruption.
• Both failures affect all three operation types uniformly.

Code References

The partition test configuration in network_partition.go:32-45:

• Sets partitionSite = true (partitions entire region, nodes 1-4)
• Uses noImpactThresholds() for the perturbation interval (no throughput gate while partition is active)
• Uses defaultThresholds() for recovery (1.25x throughput floor)
• Uses registry.ExpirationLeases (not leader leases)

The iptables rule in startPerturbation drops INPUT TCP traffic from the target node's IP on the first 4 nodes. The endPerturbation flushes iptables and waits validationDuration (5 minutes).

The recovery measurement window starts immediately when iptables is flushed and lasts 302 seconds (~5 minutes). The worst ~15 seconds (5% of 302) of throughput drive the p5 metric that exceeds the threshold.

Things Ruled Out

• Infrastructure/cloud issue: Both failures show the same pattern on different clusters with different IPs, ruling out a one-off cloud hiccup.
• Code regression since re-enablement: No KV or perturbation framework changes between the re-enable commit (May 27) and first failure (Jun 3) appear related to partition recovery.
• Perturbation interval failure: The perturbation interval passes cleanly in both runs (throughput ratio ~1.00x), confirming the workload correctly avoids the partitioned region.
• OOM or node crash: No fatal logs found. All nodes remain alive throughout.

Other Failure Occurrences

Both failures (Jun 3 and Jun 4) show the same failure mode: recovery throughput ratio 1.5-1.7x exceeding the 1.25x limit. The first failure has slightly worse ratios (1.69-1.73x) than the second (1.51-1.58x), but both are well above the threshold, suggesting this is a consistent behavior rather than a flake.

Minor Display Issue

The failure message shows "exceeds limit 1.2x" but the actual threshold is 1.25x. This is because the format string uses %.1f (one decimal place), which truncates 1.25 to 1.2. Not a correctness issue — the comparison uses the full value — but may confuse investigators.

Tooling Feedback

• Could not check out the failure SHA (d03408200b31b486a9d9ca7afa8488d0738b23a3) — the blobless clone could not fetch the ref. This is a common limitation for private repo SHAs accessed from a public repo clone.
• DuckDB installation and tsdump analysis were blocked by sandbox restrictions on writing to /tmp/claude/ and piping commands. Having duckdb pre-installed or relaxing sandbox for the tsdump workflow would have allowed timeseries analysis of throughput over time during recovery.

---

Workflow run

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[cockroach-teamcity]

roachtest.perturbation/full/partition failed with artifacts on master @ 04eaac162bc24e4e8a7454e2addd5a6f482b0c74:

Failed with:

(assertions.go:387).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/perturbation/framework.go:1044
	            				pkg/cmd/roachtest/test_runner.go:1561
	            				src/runtime/asm_amd64.s:1771
	Error:      	FAILURE: follower-read  : throughput ratio 1.73x exceeds limit 1.2x (baseline: 9359 ops/s, measured: 5400 ops/s)
	            	FAILURE: read           : throughput ratio 1.72x exceeds limit 1.2x (baseline: 9334 ops/s, measured: 5417 ops/s)
	            	FAILURE: write          : throughput ratio 1.75x exceeds limit 1.2x (baseline: 18769 ops/s, measured: 10731 ops/s)
	Test:       	perturbation/full/partition
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/partition/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21373976-1780638930-284-n13cpu16sm-0001	34.73.237.209	10.142.0.50
teamcity-21373976-1780638930-284-n13cpu16sm-0002	104.196.166.40	10.142.0.110
teamcity-21373976-1780638930-284-n13cpu16sm-0003	34.73.245.214	10.142.0.68
teamcity-21373976-1780638930-284-n13cpu16sm-0004	34.138.247.173	10.142.0.104
teamcity-21373976-1780638930-284-n13cpu16sm-0005	34.26.199.5	10.142.0.80
teamcity-21373976-1780638930-284-n13cpu16sm-0006	35.185.46.213	10.142.0.83
teamcity-21373976-1780638930-284-n13cpu16sm-0007	34.26.129.140	10.142.0.105
teamcity-21373976-1780638930-284-n13cpu16sm-0008	34.23.32.98	10.142.0.107
teamcity-21373976-1780638930-284-n13cpu16sm-0009	34.23.48.156	10.142.0.108
teamcity-21373976-1780638930-284-n13cpu16sm-0010	34.75.255.246	10.142.0.5
teamcity-21373976-1780638930-284-n13cpu16sm-0011	34.138.136.128	10.142.0.101
teamcity-21373976-1780638930-284-n13cpu16sm-0012	35.231.229.91	10.142.0.27
teamcity-21373976-1780638930-284-n13cpu16sm-0013	34.75.248.4	10.142.0.96

Parameters:

• acMode=defaultOption
• arch=amd64
• blockSize=4096
• cloud=gce
• coverageBuild=false
• cpu=16
• diskBandwidthLimit=0
• diskCount=2
• disks=2
• encrypted=false
• fillDuration=10m0s
• fs=ext4
• leaseType=expiration
• localSSD=true
• mem=standard
• numNodes=12
• numWorkloadNodes=1
• perturbationDuration=10m0s
• ratioOfMax=0.5
• runtimeAssertionsBuild=false
• seed=0
• splits=10000
• validationDuration=5m0s
• vcpu=16

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!