roachtest: kv/contention/nodes=4 failed

[cockroach-teamcity]

roachtest.kv/contention/nodes=4 failed with artifacts on master @ 64e4fc9faa4e0ab19fe5ba78f053bc2b1390cb5e:

test artifacts and logs in: /artifacts/kv/contention/nodes=4/run_1
(cluster.go:1933).Run: output in run_075158.753543425_n5_workload_run_kv: ./workload run kv --init --secondary-index --duration=1h0m0s --cycle-length=512 --concurrency=128 --batch=4 --splits=4 {pgurl:1-4} returned: COMMAND_PROBLEM: exit status 1
(monitor.go:127).Wait: monitor failure: monitor task failed: t.Fatal() was called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-22877

Epic CRDB-18656

[cockroach-teamcity]

roachtest.kv/contention/nodes=4 failed with artifacts on master @ 2c3d75f1ce31024d7ffe530f91f22162c053abcd:

test artifacts and logs in: /artifacts/kv/contention/nodes=4/run_1
(cluster.go:1933).Run: output in run_074505.228160756_n5_workload_run_kv: ./workload run kv --init --secondary-index --duration=1h0m0s --cycle-length=512 --concurrency=128 --batch=4 --splits=4 {pgurl:1-4} returned: COMMAND_PROBLEM: exit status 1
(monitor.go:127).Wait: monitor failure: monitor task failed: t.Fatal() was called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[tbg]

I221229 07:59:58.946686 895 workload/pgx_helpers.go:79  [-] 5  pgx logger [error]: Exec logParams=map[args:[6119450608480346262 dd -5780470643412804901 b1 3307412084736161884 9f -2912259416495554981 60] err:ERROR: duplicate key value violates unique constraint "kv_v_idx" (SQLSTATE 23505) pid:3080458 sql:kv-2 time:38.379989ms]
Error: ERROR: duplicate key value violates unique constraint "kv_v_idx" (SQLSTATE 23505)
run_075158.753543425_n5_workload_run_kv: 07:59:58 cluster.go:1955: > result: ./workload run kv --init --secondary-index --duration=1h0m0s --cycle-length=512 --concurrency=128 --batch=4 --splits=4 {pgurl:1-4} returned: COMMAND_PROBLEM: exit status 1

[tbg]

I221229 07:59:58.946686 895 workload/pgx_helpers.go:79  [-] 5  pgx logger [error]: Exec logParams=map[args:[6119450608480346262 dd -5780470643412804901 b1 3307412084736161884 9f -2912259416495554981 60] err:ERROR: duplicate key value violates unique constraint "kv_v_idx" (SQLSTATE 23505) pid:3080458 sql:kv-2 time:38.379989ms]
Error: ERROR: duplicate key value violates unique constraint "kv_v_idx" (SQLSTATE 23505)
run_075158.753543425_n5_workload_run_kv: 07:59:58 cluster.go:1955: > result: ./workload run kv --init --secondary-index --duration=1h0m0s --cycle-length=512 --concurrency=128 --batch=4 --splits=4 {pgurl:1-4} returned: COMMAND_PROBLEM: exit status 1

[erikgrinaker]

This one seems legit. It's unclear to me whether the problem is in KV (txn isolation) or SQL (client misuse), but I'm tossing this one over to KV for immediate investigation. CC @nvanbenschoten.

[nvb]

Thanks @erikgrinaker. This has reproduced twice in the past week, so it appears amenable to bisecting. I'll do that.

[nvb]

This git bisect has been slow because of the high iteration time and variability of this failure. I've found that it dates back at least as far as 9d0ee35. I'll keep this thread updated with progress.

[nvb]

I've now seen the failure as far back as a2d9a0ec98. Notably, that commit does not include 0483f41901, which is a relief.