catalyst task lost (last update 1m58.390845005s ago) after retrying for an Acceleration fault

[yondonfu]

From looking at the catalyst-api logs it looks like after hitting an Acceleration fault, it does retry with acceleration disabled.

This can be reproduced by uploading using this URL ipfs://bafybeifapkmyphgrvresm76ocaq4wxin7plfvezhfz3ivehp43uj5iusge

[thomshutt]

This was on Staging:

Example task ID: eb3e3b2b-5145-4838-9c3e-6772fffbbf2c
Catalyst Request ID: gcchgdad

[yondonfu]

Does this reliably repro with the catalyst task lost error in task-runner every time with ipfs://bafybeifapkmyphgrvresm76ocaq4wxin7plfvezhfz3ivehp43uj5iusge ?

[yondonfu]

This happened in prod too with task ID 8c339f2a-be02-409c-89df-3f65232390d5.

Task handler processed task type="upload" id=8c339f2a-be02-409c-89df-3f65232390d5 output=<nil> error="catalyst task lost (last update 1m25.982483869s ago)" unretriable=false

https://eu-metrics-monitoring.livepeer.live/grafana/explore?orgId=1&left=%7B%22datasource%22:%22P8E80F9AEF21F6940%22,%22queries%22:%5B%7B%22exemplar%22:true,%22expr%22:%22%7Bapp%20%3D~%20%5C%22prod-task-runner%5C%22%7D%20%7C~%20%5C%228c339f2a-be02-409c-89df-3f65232390d5%5C%22%20%7C~%20%5C%22lost%5C%22%22,%22refId%22:%22A%22,%22editorMode%22:%22code%22,%22queryType%22:%22range%22,%22hide%22:false%7D%5D,%22range%22:%7B%22from%22:%22now-24h%22,%22to%22:%22now%22%7D%7D

The logs for the task ID show lots of:

API ended in error. method=POST url="/task-runner/webhook/catalyst/task/8c339f2a-be02-409c-89df-3f65232390d5?attemptId=2&nextStep=finalize" status=500, errors=[task 8c339f2a-be02-409c-89df-3f65232390d5 is not running]