roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed

[cockroach-teamcity]

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on master @ 2a7edbeb0737b1309064c25c641a309c2980d9ba:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: full test specs: restore/tpce/8TB/inc-count=48/aws/nodes=10/cpus=8/volSize=2000GB: output in run_163914.880098375_n1_cockroach-sql-insecu: ./cockroach sql --insecure -e "RESTORE FROM LATEST IN 's3://cockroach-fixtures/backups/tpc-e/customers=500000/v22.2.1/inc-count=48?AUTH=implicit' AS OF SYSTEM TIME '2023-01-05 20:45:00'" returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_163914.888338965_n1_cockroach-sql-insecu.log: exit status 1

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-24470

[rhu713]

Error is

E230212 19:04:19.947035 2000 cloud/cloud_io.go:223 ⋮ [T1,n10,f‹7235878f›,job=839350871328948225,distsql.gateway=‹1›] 9061  Read err: read tcp ‹10.12.21.166:37394› -> ‹3.5.28.175:443›: read: connection timed out

I believe this is already part of the default retryable errors. We don't set custom timeouts or http clients anywhere in s3_storage.go, so I'm guessing the timeouts must come from one of the timeouts in http.DefaultTransport.

We can look to have a custom http client in s3_storage with longer timeouts, but I'm going to remove the release blocker for now.

[cockroach-teamcity]

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on master @ 64a867acd25c0a214209957eefb6483d1158b4f0:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(test_runner.go:989).runTest: test timed out (5h0m0s)

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[msbutler]

note that in last night's run, I accidentally changed the test to restore from 48 incremental backups instead of 11. It still may be interesting to investigate this timeout. I'm fixing that here:
#97217 (review)

[cockroach-teamcity]

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on master @ 3d054f37c7c87f53cb56fac4e5500f0d1130d09a:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 6: dead (exit status 137)

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on master @ dd2749ae4ab61eed2f99238acb74e8d3c6b4cb1d:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: read tcp 172.17.0.3:49252 -> 3.21.245.171:26257: read: connection reset by peer

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!