roachtest: sysbench/oltp_write_only/nodes=3/cpu=32/conc=256 failed [tocommit out of range]

[cockroach-teamcity]

roachtest.sysbench/oltp_write_only/nodes=3/cpu=32/conc=256 failed with artifacts on master @ 20e2adda3c76c7172dd986c871df0ae9a346918f:

test artifacts and logs in: /artifacts/sysbench/oltp_write_only/nodes=3/cpu=32/conc=256/run_1
(cluster.go:1969).Run: output in run_161729.508110704_n4_sysbench-dbdriverpgs: sysbench \
		--db-driver=pgsql \
		--pgsql-host={pghost:1} \
		--pgsql-port=26257 \
		--pgsql-user=root \
		--pgsql-password= \
		--pgsql-db=sysbench \
		--report-interval=1 \
		--time=600 \
		--threads=256 \
		--tables=10 \
		--table_size=10000000 \
		--auto_inc=false \
		oltp_write_only prepare returned: context canceled
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 7)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-24968

[renatolabs]

Quite a few liveness errors and eventually node 1 crashes with a panic from etcd/raft:

go.etcd.io/raft/v3/log.go:324 ⋮ [T1,n1,s1,r101/1:‹/Table/11{1/1/3781…-2}›] 739 
tocommit(489) is out of range [lastIndex(427)]. Was the raft log corrupted, truncated, or lost?

cc @cockroachdb/replication

[blathers-crl]

cc @cockroachdb/replication

[erikgrinaker]

@pavelkalinnikov This seems like a high-priority problem, would appreciate if you could have an initial look.

[pav-kv]

Node 1 got behind, and fails on an empty MsgApp in the probing state:

{
  "span": {
    "start_key": "/Table/111/1/3781091",
    "end_key": "/Table/112"
  },
  "raft_state": {
    "replica_id": 2,
    "hard_state": {
      "term": 6,
      "vote": 2,
      "commit": 821
    },
    "lead": 2,
    "state": "StateLeader",
    "applied": 821,
    "progress": {
      "1": {
        "match": 427,
        "next": 428,
        "state": "StateProbe",
        "paused": true
      },
      "2": {
        "match": 821,
        "next": 822,
        "state": "StateReplicate"
      },
      "3": {
        "match": 821,
        "next": 822,
        "state": "StateReplicate"
      }
    }
  },
  "state": {
    "state": {
      "raft_applied_index": 821,
      "lease_applied_index": 508,

[pav-kv]

Possibly this is the bug that @tbg mentioned in this comment (see below "Also, isn't there a (pre-existing) bug here?").