roachtest: restore/online-restore-chaos/tpcc-5k/gce/nodes=4/cpus=8 failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.restore/online-restore-chaos/tpcc-5k/gce/nodes=4/cpus=8 failed with artifacts on master @ d03408200b31b486a9d9ca7afa8488d0738b23a3:

Failed with:

(assertions.go:387).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:466
	            				pkg/cmd/roachtest/tests/online_restore.go:679
	            				pkg/cmd/roachtest/tests/online_restore.go:618
	            				pkg/cmd/roachtest/test_runner.go:1561
	            				src/runtime/asm_amd64.s:1771
	Error:      	Received unexpected error:
	            	only 0 consecutive intervals of balanced stores
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod.WaitForBalancedReplicas.func1
	            	  | 	pkg/roachprod/node_stability.go:287
	            	  | github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
	            	  | 	pkg/util/retry/retry.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod.WaitForBalancedReplicas
	            	  | 	pkg/roachprod/node_stability.go:271
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*GenericFailure).WaitForBalancedReplicas
	            	  | 	pkg/roachprod/failureinjection/failures/failure.go:359
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*GenericFailure).WaitForRestartedNodesToStabilize
	            	  | 	pkg/roachprod/failureinjection/failures/failure.go:397
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*ProcessKillFailure).WaitForFailureToRecover
	            	  | 	pkg/roachprod/failureinjection/failures/process_kill.go:126
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*Failer).WaitForFailureToRecover
	            	  | 	pkg/roachprod/failureinjection/failures/failer.go:195
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.injectAndRecoverFailure.func1
	            	  | 	pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:451
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func1
	            	  | 	pkg/cmd/roachtest/roachtestutil/task/manager.go:201
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func2
	            	  | 	pkg/cmd/roachtest/roachtestutil/task/manager.go:211
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:93
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1771
	            	Wraps: (2) only 0 consecutive intervals of balanced stores
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	restore/online-restore-chaos/tpcc-5k/gce/nodes=4/cpus=8
(require.go:1401).NoError: FailNow called
test artifacts and logs in: /artifacts/restore/online-restore-chaos/tpcc-5k/gce/nodes=4/cpus=8/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21371966-1780552855-192-n5cpu8-0001	34.73.170.72	10.142.0.175
teamcity-21371966-1780552855-192-n5cpu8-0002	34.139.19.191	10.142.0.141
teamcity-21371966-1780552855-192-n5cpu8-0003	35.237.131.7	10.142.0.146
teamcity-21371966-1780552855-192-n5cpu8-0004	35.237.184.149	10.142.0.43
teamcity-21371966-1780552855-192-n5cpu8-0005	34.73.191.189	10.142.0.3

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=8
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=false
• metamorphicDRPC=true
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-64546

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.restore/online-restore-chaos/tpcc-5k/gce/nodes=4/cpus=8 failed with artifacts on master @ 04eaac162bc24e4e8a7454e2addd5a6f482b0c74:

Failed with:

(test_runner.go:1592).runTest: test timed out (4h0m0s)
test artifacts and logs in: /artifacts/restore/online-restore-chaos/tpcc-5k/gce/nodes=4/cpus=8/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21373976-1780638930-208-n5cpu8-0001	35.196.2.116	10.142.0.191
teamcity-21373976-1780638930-208-n5cpu8-0002	35.231.172.57	10.142.0.184
teamcity-21373976-1780638930-208-n5cpu8-0003	34.26.243.70	10.142.0.173
teamcity-21373976-1780638930-208-n5cpu8-0004	34.138.169.188	10.142.0.203
teamcity-21373976-1780638930-208-n5cpu8-0005	104.196.4.203	10.142.0.84

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=8
• diskCount=0
• encrypted=true
• fs=ext4
• localSSD=false
• metamorphicDRPC=true
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

This test on roachdash | Improve this report!

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-2. Only P-0 and P-1 issues are considered release blockers.

[andrew-r-thomas]

Some notes from investigation:

Both of these are likely spurious issues, the test has been succeeding nightly after these two failures.

The first failure is due to the failure injection framework waiting for balanced stores post recovery, but not getting to balanced stores. The job itself did succeed.

The second failure hits an LOQ post failure injection, likely due to the failure injection itself, and subsequently times out