fix(worker): harden BullMQ Redis connections against failover crash (#1291)

[saidai-bhuvanesh]

Summary

A transient Redis failure (socket disconnect, failover, ECONNREFUSED) crashed the BullMQ blockchain worker process instead of recovering. This PR hardens every BullMQ/ioredis connection so connection errors are logged and absorbed, and ioredis keeps retrying via its own retryStrategy — the worker stays alive and resumes processing once Redis returns.

Problem

backend/services/blockchainWorker.js created the worker's Redis connection via createQueueConnection() (backend/config/redis.js) and BullMQ's Worker/Queue/QueueEvents emit low-level ioredis connection failures as 'error' events on the connection object itself. In Node, an EventEmitter that emits 'error' with no listener throws // Unhandled 'error' event, which — when it escapes a promise — surfaces as an UnhandledPromiseRejectionError and terminates the process.

Reproducing the issue's steps (enqueue jobs, then docker restart redis): with no 'error' listener on the connection, the worker died with:

node:events:497
      throw er; // Unhandled 'error' event
Error: Connection is closed.
    at Redis.sendCommand (node_modules/ioredis/built/redis/index.js:636:24)
    at Worker.retry (node_modules/bullmq/dist/cjs/classes/worker.js:154:18)

All background blockchain transaction processing (batch creation minting, stage updates) halted for the whole server instance until a manual restart, and any in-flight jobs were not retried cleanly.

Fix

1. Connection-level error absorption (root cause) — config/redis.js
New attachConnectionHandlers(connection) attaches error, close, reconnecting, connect, and ready listeners to an ioredis connection. The error listener logs and absorbs the event so it can never become an unhandled exception; ioredis continues to retry per the existing retryStrategy (exponential backoff, capped). createQueueConnection() now hardens every connection it creates, and createPubSubClients() hardens both the pub and the duplicate()d sub client. Attachment is idempotent (a __cropchainResilienceAttached flag prevents stacking duplicate listeners on a duplicate() or re-call).

2. Process-level guards — blockchainWorker.js
New idempotent installProcessErrorGuards() (called at the start of initializeWorker()) installs process.on('unhandledRejection') and process.on('uncaughtException') handlers that detect connection-class errors (connection is closed, ECONNREFUSED, ECONNRESET, ETIMEDOUT, redis, bullmq, unhandled 'error' event) and log+suppress them, while still letting genuine non-connection fatal errors propagate. This is defense-in-depth for any error that escapes the connection listener (e.g. inside a BullMQ internal Worker.retry promise).

3. QueueEvents hardening — blockchainQueue.js
initializeQueue() previously created the QueueEvents connection inline without hardening; it now goes through createQueueConnection() + attachConnectionHandlers(), so a failover on the events stream no longer crashes the process either. The queueEvents.on("error", ...) handler is retained.

4. Removed a stray parse-breaking fragment — blockchainQueue.js
The module had a trailing .catch(err => console.error("Promise.all failed:", err)); at module scope (no preceding expression), which is a hard SyntaxError that made the entire module un-loadable via Babel/Node — directly breaking the worker's import of QUEUE_NAMES/JOB_TYPES. Removed so the module loads. (blockchainQueue.js is a target file of this issue.)

Verification

• New backend/tests/blockchainWorkerRedisResilience.test.js (4 tests, all passing): asserts createQueueConnection() attaches error/close/reconnecting listeners and that emitting error/close/reconnecting does not throw; idempotent attachment (no duplicate listeners on repeated calls); null/undefined safety; and pub/sub duplicate() hardening (a duplicate() has zero listeners until attachConnectionHandlers is applied, after which an emitted error no longer throws).
• npx jest tests/blockchainWorkerRedisResilience.test.js → 4/4 passing. The run's logs visibly show connection errors being logged ([Redis] Connection error (recovering): ...) rather than thrown.
• config/redis.js and blockchainQueue.js load cleanly under node -e require(...); services/blockchainWorker.js passes node --check.
• No regressions: the pre-existing failures in the suite are all SyntaxErrors from an unrelated stray-line corruption in models/Batch.js, controllers/batchController.js, and services/batchService.js (trailing .then(/.catch(...) fragments at EOF) — none caused by this change, and none in the files this PR touches. Those should be addressed in a separate PR.

Behaviour preserved

• ioredis still retries with exponential backoff via the existing retryStrategy; connection config is unchanged.
• maxRetriesPerRequest: null for BullMQ connections is preserved.
• worker.on("error"), worker.on("completed"), worker.on("failed"), worker.on("stalled") are unchanged.
• Genuine non-connection uncaught exceptions still surface (re-thrown) so real bugs are not silently swallowed.

Changes

• backend/config/redis.js — add attachConnectionHandlers(), apply in createQueueConnection() and createPubSubClients(); export it.
• backend/services/blockchainWorker.js — add installProcessErrorGuards(), call it from initializeWorker().
• backend/services/blockchainQueue.js — harden the QueueEvents connection; remove stray parse-breaking trailing fragment.
• backend/tests/blockchainWorkerRedisResilience.test.js — new regression tests (4 tests).

Closes #1291

[vercel]

@openhands-agent is attempting to deploy a commit to the Nitya Gosain's projects Team on Vercel.

A member of the Team first needs to authorize it.