fix(redis): handle BullMQ connection error events to prevent worker crash on Redis loss - #1352
Open
saidai-bhuvanesh wants to merge 1 commit into
Open
Conversation
…rash on Redis loss createQueueConnection returned a raw ioredis instance with no 'error' listener, and BullMQ does not attach one to the user-provided connection. On a Redis failover / socket disconnect, BullMQ's internal Worker.retry emits 'error' on the ioredis client with no listener, which Node treats as an unhandled error and crashes the process (UnhandledPromiseRejection Error), halting all background blockchain transaction processing. Attach error/close/reconnecting listeners so the failure is logged and ioredis recovers via retryStrategy / reconnectOnError with backoff instead of crashing. Covers the Worker, Queue, and QueueEvents connections. Closes Nitya-003#1291
|
@openhands-agent is attempting to deploy a commit to the Nitya Gosain's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1291
When Redis experiences a transient failover / socket disconnect, the BullMQ worker process crashes with an unhandled
errorevent (surfaced asUnhandledPromiseRejectionError), halting all background blockchain transaction processing:Root cause
createQueueConnection()(backend/config/redis.js) returns a rawnew Redis({...})with no'error'listener attached. BullMQ does not attach one to the user-provided connection either. In Node, anEventEmitterthat emits'error'with no listener throws and crashes the process. During a Redis drop, BullMQ's internalWorker.retrytriggers an'error'emission on the ioredis client → no listener → crash.Note:
getRedisConnection()(the shared app connection) already attaches an'error'handler — only the BullMQ-specificcreateQueueConnection()(used by the Worker, Queue, and QueueEvents) was missing it.Fix
Attach
'error'/'close'/'reconnecting'listeners to each BullMQ connection created bycreateQueueConnection(). The'error'listener logs the failure instead of crashing the process, and ioredis then recovers via the existingretryStrategy/reconnectOnErrorwith backoff.function createQueueConnection() { - return new Redis({ - ...connectionOptions, - maxRetriesPerRequest: null, - }); + const connection = new Redis({ + ...connectionOptions, + maxRetriesPerRequest: null, + }); + connection.on("error", (err) => { + logger.error("❌ BullMQ Redis connection error", { error: err.message }); + }); + connection.on("close", () => { logger.warn("⚠️ BullMQ Redis connection closed"); }); + connection.on("reconnecting", (delay) => { logger.warn(`🔄 BullMQ Redis reconnecting in ${delay}ms...`); }); + return connection; }This covers the Worker, Queue, and QueueEvents connections (all built via
createQueueConnection()).Behavior
UnhandledPromiseRejectionError)Files
backend/config/redis.jsThis PR was created by an AI agent (OpenHands) on behalf of @saidai-bhuvanesh.