Distributed Task Scheduler

A robust, high-performance distributed task scheduler built with Node.js, TypeScript, and BullMQ. This system is designed to handle millions of tasks with high reliability, ensuring exactly-once execution and horizontal scalability.

🚀 Overview

The Distributed Task Scheduler allows you to offload time-consuming tasks or schedule delayed jobs across a cluster of workers. It uses Redis as a backing store and BullMQ for robust queue management, supporting features like job prioritization, delayed execution, and automatic retries.

Key Features

• Clean Architecture: Strictly decoupled layers (Domain, Application, Infrastructure, Interfaces).
• Horizontal Scaling: Easily scale workers to handle increased load.
• Distributed Locking (Redlock): Prevents duplicate job execution across parallel worker instances using the Redlock algorithm.
• Reliable Processing: Built-in support for retries, idempotency, and graceful shutdowns.
• Monitoring Dashboard: Real-time visibility into queue health via BullBoard.
• Dockerized Environment: Instant setup with Docker Compose.

---

🏗️ Architecture

The project follows the principles of Clean Architecture, ensuring that the business logic is independent of the underlying infrastructure and external frameworks.

graph TD
    subgraph Interfaces
        API[API Producer]
        Worker[Task Worker]
    end

    subgraph "Application Logic"
        UseCase[ProcessTaskUseCase]
    end

    subgraph Infrastructure
        Adapter[BullMQ Adapter]
        Config[Redis/BullMQ Config]
        Redlock[RedlockService]
    end

    subgraph Domain
        ILock[ILockService]
    end

    subgraph Data
        Redis[(Redis)]
    end

    API --> UseCase
    UseCase --> Adapter
    Adapter --> Config
    Config --> Redis
    Worker --> Adapter
    Worker --> ILock
    ILock --> Redlock
    Redlock --> Redis

Loading

---

🛠️ Tech Stack

• Backend: Node.js, TypeScript, Express
• Queue Engine: BullMQ
• Data Store: Redis
• Distributed Locking: Redlock (via the redlock npm package)
• Containerization: Docker, Docker Compose
• Monitoring: BullBoard

---

🚦 Getting Started

Prerequisites

• Docker & Docker Compose
• Node.js (v18+ recommended)

Quick Start with Docker

The easiest way to get the scheduler running is using Docker Compose, which spins up the API, three worker replicas, and a Redis instance.

# Clone the repository
git clone <your-repo-url>
cd Distributed_Task_Scheduler

# Start the cluster
docker-compose up -d --build

The API will be available at http://localhost:3000.

Local Development Setup

If you want to run the components individually:

1. Install dependencies:

   npm install

2. Ensure Redis is running (default: localhost:6379).
3. Start the API:

   npm run start:api

4. Start a Worker:

   npm run start:worker

---

📖 API Reference

Schedule a Task

Adds a new task to the distributed queue.

• URL: /api/tasks
• Method: POST
• Body:

  {
    "taskName": "sendEmail",
    "payload": { "userId": 123, "template": "welcome" },
    "delayMs": 5000,
    "priority": 1,
    "customJobId": "unique-id-123"
  }

• Success Response: 202 Accepted

---

📊 Monitoring

The project includes BullBoard, a visual dashboard for managing and monitoring your queues.

• Access URL: http://localhost:3000/admin/queues
• Features: View active/completed/failed jobs, retry failed jobs, and monitor throughput.

---

🔒 Distributed Locking with Redlock

To prevent duplicate job execution — especially when multiple worker instances run concurrently — each job is wrapped in a Redlock distributed lock before processing begins.

How It Works

Worker dequeues job
  └─► acquire("lock:task:{jobId}", 30 s TTL)
        ├─► Lock granted  → process job → release lock
        └─► Lock denied   → another worker is already handling it
                           (BullMQ retries the job automatically)

Lock Lifecycle (per job)

Step	Action
1	Worker picks up the job from the BullMQ queue
2	RedlockService.acquire("lock:task:{jobId}", 30_000) is called
3	If the lock is granted, job logic runs
4	Lock is always released in a finally block — even on failure
5	On graceful shutdown, the Redlock Redis client is disconnected cleanly

Configuration (src/infrastructure/lock/RedlockService.ts)

Setting	Value	Description
retryCount	3	Number of times to retry acquiring the lock
retryDelay	200 ms	Wait time between retries
retryJitter	100 ms	Random jitter added to avoid thundering-herd
Lock TTL	30,000 ms	Auto-expiry if a worker crashes mid-job

Note: If a job takes longer than the 30 s TTL, the lock may expire before the job finishes. Adjust LOCK_TTL in processor.ts to match your expected maximum job duration.

Files

• src/domain/services/ILockService.ts — Framework-agnostic domain interface
• src/infrastructure/lock/RedlockService.ts — Concrete Redlock implementation
• src/interfaces/workers/processor.ts — Lock acquired/released around each job
• src/interfaces/workers/index.ts — RedlockService instantiated and injected
• src/interfaces/workers/gracefulShutdown.ts — Redlock client disconnected on shutdown

---

📂 Project Structure

src/
├── domain/
│   ├── entities/           # Task entity
│   ├── repositories/       # ITaskQueue interface
│   └── services/           # ILockService interface  ← NEW
├── application/
│   └── use-cases/          # ProcessTaskUseCase
├── infrastructure/
│   ├── config/             # Redis / BullMQ config
│   ├── queue/              # BullMQ adapter
│   └── lock/               # RedlockService          ← NEW
└── interfaces/
    ├── http/               # Express API server
    └── workers/            # BullMQ worker, processor, graceful shutdown

---

📄 License

This project is licensed under the ISC License.