Skip to content

fix(redis): handle BullMQ connection error events to prevent worker crash on Redis loss - #1352

Open
saidai-bhuvanesh wants to merge 1 commit into
Nitya-003:mainfrom
saidai-bhuvanesh:fix/1291-bullmq-redis-unhandled-error
Open

fix(redis): handle BullMQ connection error events to prevent worker crash on Redis loss#1352
saidai-bhuvanesh wants to merge 1 commit into
Nitya-003:mainfrom
saidai-bhuvanesh:fix/1291-bullmq-redis-unhandled-error

Conversation

@saidai-bhuvanesh

Copy link
Copy Markdown
Contributor

Summary

Fixes #1291

When Redis experiences a transient failover / socket disconnect, the BullMQ worker process crashes with an unhandled error event (surfaced as UnhandledPromiseRejectionError), halting all background blockchain transaction processing:

[INFO] [BlockchainWorker]: Worker initialized listening on queue 'blockchain-tx'
ERROR [ioredis]: Connection to 127.0.0.1:6379 failed - connect ECONNREFUSED
node:events:497
      throw er; // Unhandled 'error' event
Error: Connection is closed.
    at Redis.sendCommand (.../ioredis/built/redis/index.js:636:24)
    at Worker.retry (.../bullmq/dist/cjs/classes/worker.js:154:18)
Emitted 'error' event at:
    at Redis.emit (.../ioredis/built/redis/index.js:342:14)

Root cause

createQueueConnection() (backend/config/redis.js) returns a raw new Redis({...}) with no 'error' listener attached. BullMQ does not attach one to the user-provided connection either. In Node, an EventEmitter that emits 'error' with no listener throws and crashes the process. During a Redis drop, BullMQ's internal Worker.retry triggers an 'error' emission on the ioredis client → no listener → crash.

Note: getRedisConnection() (the shared app connection) already attaches an 'error' handler — only the BullMQ-specific createQueueConnection() (used by the Worker, Queue, and QueueEvents) was missing it.

Fix

Attach 'error' / 'close' / 'reconnecting' listeners to each BullMQ connection created by createQueueConnection(). The 'error' listener logs the failure instead of crashing the process, and ioredis then recovers via the existing retryStrategy / reconnectOnError with backoff.

function createQueueConnection() {
- return new Redis({
-   ...connectionOptions,
-   maxRetriesPerRequest: null,
- });
+ const connection = new Redis({
+   ...connectionOptions,
+   maxRetriesPerRequest: null,
+ });
+ connection.on("error", (err) => {
+   logger.error("❌ BullMQ Redis connection error", { error: err.message });
+ });
+ connection.on("close", () => { logger.warn("⚠️ BullMQ Redis connection closed"); });
+ connection.on("reconnecting", (delay) => { logger.warn(`🔄 BullMQ Redis reconnecting in ${delay}ms...`); });
+ return connection;
}

This covers the Worker, Queue, and QueueEvents connections (all built via createQueueConnection()).

Behavior

Event Before After
Redis drop / failover process crash (UnhandledPromiseRejectionError) logged + ioredis reconnects with backoff

Files

  • backend/config/redis.js

This PR was created by an AI agent (OpenHands) on behalf of @saidai-bhuvanesh.

…rash on Redis loss

createQueueConnection returned a raw ioredis instance with no 'error'
listener, and BullMQ does not attach one to the user-provided connection.
On a Redis failover / socket disconnect, BullMQ's internal Worker.retry
emits 'error' on the ioredis client with no listener, which Node treats
as an unhandled error and crashes the process (UnhandledPromiseRejection
Error), halting all background blockchain transaction processing. Attach
error/close/reconnecting listeners so the failure is logged and ioredis
recovers via retryStrategy / reconnectOnError with backoff instead of
crashing. Covers the Worker, Queue, and QueueEvents connections.

Closes Nitya-003#1291
@vercel

vercel Bot commented Aug 15, 2026

Copy link
Copy Markdown

@openhands-agent is attempting to deploy a commit to the Nitya Gosain's projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Unhandled Worker Rejection on Redis Connection Loss in BullMQ

2 participants