Skip to content

feat: Add distributed session coordination for horizontal worker scaling #216

@Xhristin3

Description

@Xhristin3

Problem Statement

The SessionRegistry (xstreamroll-processing/src/session-registry.ts) is a per-process Map<string, StreamSession>. If multiple worker instances run, each has its own independent registry. The same stream could have active sessions across multiple workers, leading to duplicate event processing.

Evidence

// xstreamroll-processing/src/session-registry.ts:26
private readonly sessions = new Map<string, StreamSession>()

Impact

Duplicate event processing across workers. No distributed coordination for session ownership. Cannot horizontally scale the worker without data duplication.

Proposed Solution

  1. Use a distributed lock (Redis Redlock, PostgreSQL advisory lock) to claim stream ownership
  2. Before creating a session, acquire a lock on the stream ID
  3. Release the lock when the session is stopped/errored
  4. Other workers skip polls for locked streams
  5. Implement lock TTL with heartbeat renewal

Acceptance Criteria

  • Two worker instances do not process the same stream simultaneously
  • Lock is released on worker crash (TTL expiry)
  • Existing single-worker behavior unchanged

File Map

  • xstreamroll-processing/src/session-registry.ts — add distributed locking
  • xstreamroll-processing/src/leader-election.ts — new coordination module

Labels: feature, infrastructure
Priority: Medium | Difficulty: Expert | Estimated Effort: 5d


Labels: feature,infrastructure
Priority: Medium | Difficulty: Expert | Estimated Effort: 5d
Backlog ID: REPO-033

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions