release-26.3: roachtest: perturbation/full/partition failed

[cockroach-teamcity]

roachtest.perturbation/full/partition failed with artifacts on release-26.3 @ 8a9a8e3305daa0890b7b083c13cb572f911bd6cc:

Failed with:

(assertions.go:387).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/perturbation/framework.go:1051
	            				pkg/cmd/roachtest/test_runner.go:1548
	            				src/runtime/asm_amd64.s:1771
	Error:      	FAILURE: follower-read  : throughput ratio 1.64x exceeds limit 1.2x (baseline: 10436 ops/s, measured: 6356 ops/s)
	            	FAILURE: read           : throughput ratio 1.62x exceeds limit 1.2x (baseline: 10397 ops/s, measured: 6413 ops/s)
	            	FAILURE: write          : throughput ratio 1.65x exceeds limit 1.2x (baseline: 20953 ops/s, measured: 12733 ops/s)
	Test:       	perturbation/full/partition
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/partition/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21392022-1781763104-317-n13cpu16sm-0001	34.74.123.170	10.142.0.191
teamcity-21392022-1781763104-317-n13cpu16sm-0002	35.196.195.136	10.142.0.185
teamcity-21392022-1781763104-317-n13cpu16sm-0003	34.23.214.156	10.142.0.106
teamcity-21392022-1781763104-317-n13cpu16sm-0004	34.75.188.172	10.142.0.190
teamcity-21392022-1781763104-317-n13cpu16sm-0005	35.196.60.41	10.142.0.27
teamcity-21392022-1781763104-317-n13cpu16sm-0006	34.23.160.170	10.142.0.62
teamcity-21392022-1781763104-317-n13cpu16sm-0007	34.138.18.43	10.142.0.101
teamcity-21392022-1781763104-317-n13cpu16sm-0008	34.148.11.128	10.142.0.127
teamcity-21392022-1781763104-317-n13cpu16sm-0009	34.73.230.145	10.142.0.118
teamcity-21392022-1781763104-317-n13cpu16sm-0010	35.243.184.20	10.142.0.178
teamcity-21392022-1781763104-317-n13cpu16sm-0011	35.196.139.56	10.142.0.192
teamcity-21392022-1781763104-317-n13cpu16sm-0012	35.196.156.238	10.142.0.189
teamcity-21392022-1781763104-317-n13cpu16sm-0013	34.26.164.63	10.142.0.111

Parameters:

• acMode=defaultOption
• arch=amd64
• blockSize=4096
• cloud=gce
• coverageBuild=false
• cpu=16
• diskBandwidthLimit=0
• diskCount=2
• disks=2
• encrypted=false
• fillDuration=10m0s
• leaseType=expiration
• mem=standard
• numNodes=12
• numWorkloadNodes=1
• perturbationDuration=10m0s
• ratioOfMax=0.5
• runtimeAssertionsBuild=false
• seed=0
• splits=10000
• validationDuration=5m0s
• vcpu=16

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-64933

[iskettaneh]

/investigate

[github-actions]

Investigation: perturbation/full/partition

Investigated failure: issue body
Failure SHA: 8a9a8e3305daa0890b7b083c13cb572f911bd6cc
Confidence: high

Warning: Could not check out failure SHA 8a9a8e3305daa0890b7b083c13cb572f911bd6cc. Analysis is based on the default branch tip, which may differ from the code that produced the failure.

What This Test Does

The perturbation/full/partition test measures CockroachDB's recovery from a region-wide network partition. It sets up a 12-node cluster across 3 regions (4 nodes each), runs a KV workload at 50% of max throughput, then uses iptables to partition the first region (nodes 1-4) from the target node (node 12) for 10 minutes. The pass/fail signal is whether throughput returns to within 1.25x of the baseline during the 5-minute recovery window after the partition heals.

Where the Failure Occurs

The failure occurs at framework.go:1051 during validation of the recovery interval. The perturbation interval passes cleanly (uses noImpactThresholds()). The recovery interval applies defaultThresholds() which enforces a 1.25x throughput floor via maxThroughputImpact.

The throughput ratio is computed as baseline.throughput / recovery.throughput where .throughput is the p5 (worst 5%) of per-second throughput ticks. All three operation types fail:

Operation	Baseline p5(tput/s)	Recovery p5(tput/s)	Ratio	Limit
follower-read	10,436	6,356	1.64x	1.25x
read	10,397	6,413	1.62x	1.25x
write	20,953	12,733	1.65x	1.25x

The median throughput during recovery is virtually identical to baseline (~10,545 / ~21,124 ops/s). The problem is entirely in the p5 tail — approximately 15 seconds out of the 302-second recovery window have significantly degraded throughput, consistent with a brief but intense rebalancing storm immediately after the partition heals.

Analysis

Hypothesis: The 1.25x recovery throughput threshold is systematically too tight for full-region partition recovery (likely cause, high confidence)

When the partition heals after 10 minutes:

• Expiration leases held by partitioned nodes have expired and been acquired by other nodes; original leaseholders need leases back.
• Raft leadership must be re-established for ranges where partitioned nodes were leaders.
• Snapshot transfers may be needed to bring partitioned replicas up to date.

This creates a brief, intense burst of rebalancing activity in the first ~15-30 seconds after partition healing. The recovery stats confirm this pattern:

• Median throughput is at baseline levels (cluster fully recovers)
• p95 of p99 latency is extremely elevated during recovery (54ms follower-read, 46ms read, 227ms write) vs baseline (2.4ms, 2.1ms, 11ms)
• p5 throughput drops to ~60-65% of baseline

This test has failed every single time it has run since being re-enabled on Jun 2 (commit 3c96eeb6571), both on master (#171399 — at least 4 consecutive failures with ratios 1.5x-2.65x) and on release-26.3 (this issue). This is not a flake — it's a systematic threshold mismatch.

The test was re-enabled with recoveryImpact = defaultThresholds() (1.25x floor) in commit d43059fe44c, merged to master via 3c96eeb6571. The test was previously skipped via #149662 specifically because thresholds were being stabilized.

Possible fixes (ordered by simplicity):

1. Increase maxThroughputImpact for the partition test's recoveryImpact to ~2.0x or higher, acknowledging the inevitable rebalancing storm.
2. Extend validationDuration beyond 5 minutes so the brief dip is diluted by more baseline-level seconds (the p5 calculation drops 5% of worst ticks — a longer window means the brief storm is a smaller fraction).
3. Add a warmup exclusion to the recovery measurement (similar to how startPerturbation already subtracts 20 seconds from its measurement window), allowing the rebalancing storm to settle before measuring.

Related Issues and PRs

• roachtest: perturbation/full/partition failed #171399 — Same test failing on master with identical failure mode. Open, 4+ consecutive failures since re-enablement. Prior /investigate analysis reached the same conclusion. This is the same underlying issue.
• kv,roachtest: investigate and recalibrate perturbation tests #149662 — "kv,roachtest: investigate and recalibrate perturbation tests" — the issue that originally skipped these tests. Closed when tests were re-enabled, but the partition test was not adequately calibrated.
• d43059f — The commit that re-enabled perturbation/full/partition with recoveryImpact = defaultThresholds() (1.25x floor).
• 3c96eeb — Merge of the re-enable commit to master (Jun 2, 2026).

Timeline

Date	Event
May 27, 2026	d43059fe44c re-enables perturbation/full/partition with recoveryImpact = defaultThresholds() (1.25x throughput floor)
Jun 2, 2026	Merged to master via 3c96eeb6571
Jun 3, 2026	First failure on master (#171399): ratio 1.69-1.73x
Jun 4, 2026	Second failure on master: ratio 1.51-1.58x
Jun 5, 2026	Third failure on master: ratio 1.72-1.75x
Jun 7, 2026	Fourth failure on master: ratio 2.54-2.65x
Jun 18, 2026	Failure on release-26.3 (this issue): ratio 1.62-1.65x

The test has never passed since re-enablement — 100% failure rate across all runs on both master and release-26.3.

Detailed evidence and investigation notes

Log Excerpts

This failure (build 21392022) — recovery stats from test.log:

Recovery stats
follower-read  : median(p99/s): 1.9ms, median(tput/s): 10545 ops/s, p95(p99/s): 54.5ms, p95(p50/s): 1.3ms, p5(tput/s): 6356 ops/s (302 ticks)
read           : median(p99/s): 1.7ms, median(tput/s): 10546 ops/s, p95(p99/s): 46.1ms, p95(p50/s): 1.2ms, p5(tput/s): 6413 ops/s (302 ticks)
write          : median(p99/s): 8.9ms, median(tput/s): 21124 ops/s, p95(p99/s): 226.5ms, p95(p50/s): 3.4ms, p5(tput/s): 12733 ops/s (302 ticks)

Baseline stats for comparison:

Baseline stats
follower-read  : median(p99/s): 1.7ms, median(tput/s): 10570 ops/s, p95(p99/s): 2.4ms, p95(p50/s): 1.2ms, p5(tput/s): 10436 ops/s (150 ticks)
read           : median(p99/s): 1.6ms, median(tput/s): 10554 ops/s, p95(p99/s): 2.1ms, p95(p50/s): 1.1ms, p5(tput/s): 10397 ops/s (150 ticks)
write          : median(p99/s): 7.6ms, median(tput/s): 21118 ops/s, p95(p99/s): 11ms, p95(p50/s): 2ms, p5(tput/s): 20953 ops/s (150 ticks)

Perturbation stats (during partition — passes as expected):

Perturbation stats
follower-read  : median(p99/s): 2ms, median(tput/s): 10562 ops/s, p95(p99/s): 2.6ms, p95(p50/s): 1.2ms, p5(tput/s): 10417 ops/s (580 ticks)
read           : median(p99/s): 2.2ms, median(tput/s): 10562 ops/s, p95(p99/s): 2.6ms, p95(p50/s): 1.1ms, p5(tput/s): 10416 ops/s (580 ticks)
write          : median(p99/s): 10ms, median(tput/s): 21118 ops/s, p95(p99/s): 13.6ms, p95(p50/s): 2ms, p5(tput/s): 20958 ops/s (580 ticks)

Key observations:

• The perturbation interval passes cleanly (ratios ~1.00x) — the workload correctly avoids the partitioned region.
• Median recovery throughput matches baseline almost exactly (~10,545 vs ~10,570 read; ~21,124 vs ~21,118 write).
• The failure is exclusively in the p5 tail: ~15 seconds of the 302-second window drag the p5 metric down.
• p99 latency during recovery spikes massively (p95 of p99: 54ms vs 2.4ms for reads), confirming a brief intense disruption.

Code References

The threshold configuration in network_partition.go:32-45:

• partitionSite = true — partitions entire region (nodes 1-4)
• impact = noImpactThresholds() — no throughput gate during partition
• recoveryImpact = defaultThresholds() — 1.25x throughput floor for recovery
• leaseType = ExpirationLeases

The isAcceptableChange function at framework.go:1107-1148 computes throughput ratio as baseline.throughput / stats.throughput (inverted so higher = worse), where .throughput is the p5 of per-second throughput ticks.

Things Ruled Out

• Infrastructure/cloud issue: Failures are reproducible across many different clusters on different dates and IPs. 100% failure rate rules out cloud flakes.
• Code regression: No KV, lease, or raft changes between the re-enable commit and the first failure. The issue is the threshold itself, not a behavioral change.
• OOM or node crash: All nodes remain alive throughout. No fatal logs or dmesg issues.
• Perturbation-interval failure: The perturbation interval passes cleanly in all runs with ~1.00x ratios.

Other Failure Occurrences

All occurrences (master #171399 and this release-26.3 issue) show the same failure mode:

Run	Branch	Throughput Ratio Range
Jun 3 (21370099)	master	1.69-1.73x
Jun 4 (21371966)	master	1.51-1.58x
Jun 5 (21373976)	master	1.72-1.75x
Jun 7 (21375650)	master	2.54-2.65x
Jun 18 (21392022)	release-26.3	1.62-1.65x

All share the same signature: p5 throughput ratio well above 1.25x, median throughput at baseline, elevated p95(p99) latency during recovery. The Jun 7 run shows a particularly severe ratio (2.65x), suggesting the rebalancing storm duration can vary significantly.

Tooling Feedback

• Could not check out the failure SHA (8a9a8e3305daa0890b7b083c13cb572f911bd6cc) — the blobless clone could not fetch the ref from release-26.3. The git fetch origin release-26.3 also failed. This meant the analysis relied on master branch code, though the perturbation framework code is identical between branches.

---

Workflow run

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[iskettaneh]

Some things that I noticed:

The test uses expiration leases.

This is fine, but given that leader leases is the default, I think we should set it to leader leases?

The name is full/partition. However, my understanding is that this is the current test setup:
We have 12 nodes, each 4 nodes are in a region and we partition the first 4 nodes from N12 only. This doesn't sound like a full partition to me. Like N1-N4 can still receive and replicate raft commands, just they cannot replicate to N12. Also, N12 can replicate raft commands, just not to N1-N4.
I ~believe the test intends to fully partition N1-N4 from the cluster, but I only saw this command: sudo iptables -A INPUT -p tcp -s 10.142.0.189 -j DROP  which ran on N1-N4 and partitions them from N12.

When the partition heals, N12 receives a TON of commands to replicate, but N12 is also the leaseholder to some ranges, so the latency suffers for sometime

[cockroach-teamcity]

roachtest.perturbation/full/partition failed with artifacts on release-26.3 @ 769f38f9d3492a70416a81f5ed38c5f09f4b4b83:

Failed with:

(assertions.go:387).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/perturbation/framework.go:1051
	            				pkg/cmd/roachtest/test_runner.go:1548
	            				src/runtime/asm_amd64.s:1771
	Error:      	FAILURE: follower-read  : throughput ratio 2.39x exceeds limit 1.2x (baseline: 11312 ops/s, measured: 4732 ops/s)
	            	FAILURE: read           : throughput ratio 2.38x exceeds limit 1.2x (baseline: 11371 ops/s, measured: 4778 ops/s)
	            	FAILURE: write          : throughput ratio 2.32x exceeds limit 1.2x (baseline: 22816 ops/s, measured: 9822 ops/s)
	Test:       	perturbation/full/partition
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/partition/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21393548-1781849444-329-n13cpu16sm-0001	34.75.117.170	10.142.0.43
teamcity-21393548-1781849444-329-n13cpu16sm-0002	104.196.67.117	10.142.0.10
teamcity-21393548-1781849444-329-n13cpu16sm-0003	34.148.54.26	10.142.0.171
teamcity-21393548-1781849444-329-n13cpu16sm-0004	34.24.103.104	10.142.0.172
teamcity-21393548-1781849444-329-n13cpu16sm-0005	34.75.31.143	10.142.0.37
teamcity-21393548-1781849444-329-n13cpu16sm-0006	104.196.51.49	10.142.0.112
teamcity-21393548-1781849444-329-n13cpu16sm-0007	34.73.149.62	10.142.0.9
teamcity-21393548-1781849444-329-n13cpu16sm-0008	104.196.112.122	10.142.0.173
teamcity-21393548-1781849444-329-n13cpu16sm-0009	34.24.59.78	10.142.0.68
teamcity-21393548-1781849444-329-n13cpu16sm-0010	34.24.97.129	10.142.0.19
teamcity-21393548-1781849444-329-n13cpu16sm-0011	35.231.120.236	10.142.0.170
teamcity-21393548-1781849444-329-n13cpu16sm-0012	34.73.54.54	10.142.0.174
teamcity-21393548-1781849444-329-n13cpu16sm-0013	34.74.137.234	10.142.0.16

Parameters:

• acMode=defaultOption
• arch=amd64
• blockSize=4096
• cloud=gce
• coverageBuild=false
• cpu=16
• diskBandwidthLimit=0
• diskCount=2
• disks=2
• encrypted=false
• fillDuration=10m0s
• fs=ext4
• leaseType=expiration
• localSSD=true
• mem=standard
• numNodes=12
• numWorkloadNodes=1
• perturbationDuration=10m0s
• ratioOfMax=0.5
• runtimeAssertionsBuild=false
• seed=0
• splits=10000
• validationDuration=5m0s
• vcpu=16

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.perturbation/full/partition failed with artifacts on release-26.3 @ 9caafebcafcd887c18107a3a718e5a07f38938d0:

Failed with:

(assertions.go:387).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/perturbation/framework.go:1051
	            				pkg/cmd/roachtest/test_runner.go:1548
	            				src/runtime/asm_amd64.s:1771
	Error:      	FAILURE: follower-read  : throughput ratio 1.89x exceeds limit 1.2x (baseline: 10774 ops/s, measured: 5707 ops/s)
	            	FAILURE: read           : throughput ratio 1.91x exceeds limit 1.2x (baseline: 10763 ops/s, measured: 5646 ops/s)
	            	FAILURE: write          : throughput ratio 1.96x exceeds limit 1.2x (baseline: 21708 ops/s, measured: 11060 ops/s)
	Test:       	perturbation/full/partition
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/partition/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21394403-1781935825-310-n13cpu16sm-0001	104.196.30.81	10.142.0.103
teamcity-21394403-1781935825-310-n13cpu16sm-0002	35.243.228.62	10.142.0.73
teamcity-21394403-1781935825-310-n13cpu16sm-0003	35.227.81.101	10.142.0.76
teamcity-21394403-1781935825-310-n13cpu16sm-0004	34.75.8.187	10.142.0.93
teamcity-21394403-1781935825-310-n13cpu16sm-0005	34.75.167.241	10.142.0.138
teamcity-21394403-1781935825-310-n13cpu16sm-0006	35.196.102.156	10.142.0.126
teamcity-21394403-1781935825-310-n13cpu16sm-0007	35.231.205.229	10.142.0.99
teamcity-21394403-1781935825-310-n13cpu16sm-0008	34.24.212.120	10.142.0.49
teamcity-21394403-1781935825-310-n13cpu16sm-0009	35.190.131.94	10.142.0.89
teamcity-21394403-1781935825-310-n13cpu16sm-0010	34.148.219.187	10.142.0.115
teamcity-21394403-1781935825-310-n13cpu16sm-0011	34.138.148.21	10.142.0.34
teamcity-21394403-1781935825-310-n13cpu16sm-0012	34.73.221.163	10.142.0.105
teamcity-21394403-1781935825-310-n13cpu16sm-0013	35.185.40.129	10.142.0.96

Parameters:

• acMode=defaultOption
• arch=amd64
• blockSize=4096
• cloud=gce
• coverageBuild=false
• cpu=16
• diskBandwidthLimit=0
• diskCount=2
• disks=2
• encrypted=false
• fillDuration=10m0s
• leaseType=expiration
• mem=standard
• numNodes=12
• numWorkloadNodes=1
• perturbationDuration=10m0s
• ratioOfMax=0.5
• runtimeAssertionsBuild=false
• seed=0
• splits=10000
• validationDuration=5m0s
• vcpu=16

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.perturbation/full/partition failed with artifacts on release-26.3 @ 8449e457346fdc20966c6f9acde86056a50dafd4:

Failed with:

(assertions.go:387).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/perturbation/framework.go:1051
	            				pkg/cmd/roachtest/test_runner.go:1548
	            				src/runtime/asm_amd64.s:1771
	Error:      	FAILURE: write          : throughput ratio 1.27x exceeds limit 1.2x (baseline: 22317 ops/s, measured: 17622 ops/s)
	Test:       	perturbation/full/partition
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/partition/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21395224-1782104923-308-n13cpu16sm-0001	35.196.60.194	10.142.0.16
teamcity-21395224-1782104923-308-n13cpu16sm-0002	34.26.50.127	10.142.0.43
teamcity-21395224-1782104923-308-n13cpu16sm-0003	34.75.35.119	10.142.0.29
teamcity-21395224-1782104923-308-n13cpu16sm-0004	34.24.10.230	10.142.0.74
teamcity-21395224-1782104923-308-n13cpu16sm-0005	34.75.76.239	10.142.0.18
teamcity-21395224-1782104923-308-n13cpu16sm-0006	35.237.124.122	10.142.0.37
teamcity-21395224-1782104923-308-n13cpu16sm-0007	34.148.184.27	10.142.0.33
teamcity-21395224-1782104923-308-n13cpu16sm-0008	34.148.16.159	10.142.0.72
teamcity-21395224-1782104923-308-n13cpu16sm-0009	35.196.98.206	10.142.0.35
teamcity-21395224-1782104923-308-n13cpu16sm-0010	34.24.34.16	10.142.0.31
teamcity-21395224-1782104923-308-n13cpu16sm-0011	35.229.21.177	10.142.0.34
teamcity-21395224-1782104923-308-n13cpu16sm-0012	35.237.1.254	10.142.0.39
teamcity-21395224-1782104923-308-n13cpu16sm-0013	35.243.203.125	10.142.0.15

Parameters:

• acMode=defaultOption
• arch=amd64
• blockSize=4096
• cloud=gce
• coverageBuild=false
• cpu=16
• diskBandwidthLimit=0
• diskCount=2
• disks=2
• encrypted=false
• fillDuration=10m0s
• fs=ext4
• leaseType=expiration
• localSSD=true
• mem=standard
• numNodes=12
• numWorkloadNodes=1
• perturbationDuration=10m0s
• ratioOfMax=0.5
• runtimeAssertionsBuild=false
• seed=0
• splits=10000
• validationDuration=5m0s
• vcpu=16

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.perturbation/full/partition failed with artifacts on release-26.3 @ 37e08669dc2879d4d006b6320244dd6aa918faad:

Failed with:

(assertions.go:387).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/perturbation/framework.go:1051
	            				pkg/cmd/roachtest/test_runner.go:1548
	            				src/runtime/asm_amd64.s:1771
	Error:      	FAILURE: follower-read  : throughput ratio 3.18x exceeds limit 1.2x (baseline: 11081 ops/s, measured: 3488 ops/s)
	            	FAILURE: read           : throughput ratio 3.20x exceeds limit 1.2x (baseline: 11108 ops/s, measured: 3470 ops/s)
	            	FAILURE: write          : throughput ratio 3.25x exceeds limit 1.2x (baseline: 22319 ops/s, measured: 6858 ops/s)
	Test:       	perturbation/full/partition
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/partition/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21396898-1782195300-308-n13cpu16sm-0001	34.26.201.76	10.142.0.13
teamcity-21396898-1782195300-308-n13cpu16sm-0002	35.231.18.104	10.142.0.92
teamcity-21396898-1782195300-308-n13cpu16sm-0003	35.237.148.248	10.142.0.194
teamcity-21396898-1782195300-308-n13cpu16sm-0004	34.75.151.96	10.142.0.100
teamcity-21396898-1782195300-308-n13cpu16sm-0005	35.185.47.94	10.142.0.77
teamcity-21396898-1782195300-308-n13cpu16sm-0006	35.243.209.147	10.142.0.195
teamcity-21396898-1782195300-308-n13cpu16sm-0007	35.229.57.203	10.142.0.22
teamcity-21396898-1782195300-308-n13cpu16sm-0008	35.243.149.6	10.142.0.33
teamcity-21396898-1782195300-308-n13cpu16sm-0009	104.196.35.199	10.142.0.58
teamcity-21396898-1782195300-308-n13cpu16sm-0010	35.190.134.13	10.142.0.63
teamcity-21396898-1782195300-308-n13cpu16sm-0011	34.23.106.208	10.142.0.73
teamcity-21396898-1782195300-308-n13cpu16sm-0012	35.227.43.127	10.142.0.56
teamcity-21396898-1782195300-308-n13cpu16sm-0013	34.139.2.64	10.142.0.29

Parameters:

• acMode=defaultOption
• arch=amd64
• blockSize=4096
• cloud=gce
• coverageBuild=false
• cpu=16
• diskBandwidthLimit=0
• diskCount=2
• disks=2
• encrypted=false
• fillDuration=10m0s
• leaseType=expiration
• mem=standard
• numNodes=12
• numWorkloadNodes=1
• perturbationDuration=10m0s
• ratioOfMax=0.5
• runtimeAssertionsBuild=false
• seed=0
• splits=10000
• validationDuration=5m0s
• vcpu=16

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!