[S-01] Distributed Debates — RedisDebateOrchestrator, parallel round execution across workers

[techbuzzz]

Overview

Version: v10.5.0 Scale & Reliability
Effort: L · 7–10 days
Branch: feature/s01-distributed-debates

Enable Delibera to distribute debate round execution across multiple workers using Redis as the coordination backbone. Each council member turn becomes an independent work item that any available worker can pick up and execute, enabling true horizontal scaling for large councils and high-throughput deployments.

---

Architecture

┌───────────────────────────────┐
│  Delibera.Server (coordinator)  │
│  POST /api/v1/debates           │
└───────────────────────────────┘
         │ RedisDebateOrchestrator
         │ publishes MemberTurnJob
    ┌───┴───┐     ┌───────┐     ┌───────┐
    │Worker A│     │Worker B│     │Worker C│
    │Architect│    │TechLead│     │ DevOps │
    └────────┘     └───────┘     └───────┘
         │               │               │
         └───────┬───────┘
                │ Redis Streams
         ┌─────┴─────┐
         │  Aggregator   │
         │  (Chairman)   │
         └─────────────┘

---

Core Interfaces

// New interface — replaces direct CouncilExecutor call for distributed mode
public interface IDebateOrchestrator
{
    Task<DebateRecord> StartAsync(ScenarioRequest request, CancellationToken ct = default);
    Task<DebateRecord> GetStatusAsync(string debateId, CancellationToken ct = default);
    Task CancelAsync(string debateId, CancellationToken ct = default);
    IAsyncEnumerable<DebateRoundEvent> StreamAsync(string debateId, CancellationToken ct = default);
}

// Local (current) implementation — no change in behavior
public class LocalDebateOrchestrator : IDebateOrchestrator { ... }

// New distributed implementation
public class RedisDebateOrchestrator : IDebateOrchestrator
{
    // Uses Redis Streams for job queue + pub/sub for round events
    // Uses Redis Hash for debate state
    // Uses Redis sorted set for round ordering
}

---

Redis Data Model

Key Pattern	Type	Content
delibera:debate:{id}:state	Hash	DebateRecord fields
delibera:debate:{id}:rounds	Sorted Set	Round results, score = round index
delibera:debate:{id}:events	Stream	DebateRoundEvent, DebateCompletedEvent
delibera:jobs:{id}	Stream	MemberTurnJob items (consumer group per worker)
delibera:debates:active	Set	IDs of running debates

---

Worker Service

// New hosted service — runs in every Delibera.Server instance or standalone worker
public class DebateWorkerService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var job in _jobQueue.DequeueAsync(stoppingToken))
        {
            var result = await _memberExecutor.ExecuteTurnAsync(job, stoppingToken);
            await _resultPublisher.PublishAsync(job.DebateId, result, stoppingToken);
        }
    }
}

---

CouncilBuilder Integration

// Opt-in to distributed mode via builder
services.AddDelibera()
    .UseRedisOrchestrator(connectionString: "localhost:6379", 
                          workerCount: 4,
                          consumerGroup: "delibera-workers");

// OR per-debate via options
councilBuilder.WithDistributedExecution(orchestrator);

---

Tasks

• Define IDebateOrchestrator in Delibera.Core.Interfaces
• Implement LocalDebateOrchestrator wrapping current CouncilExecutor (no behavior change)
• Implement RedisDebateOrchestrator using StackExchange.Redis
• Implement MemberTurnJob record (debateId, round, member, context snapshot)
• Implement DebateWorkerService (BackgroundService) for consuming jobs
• Implement Redis Streams consumer group for job distribution
• Implement round aggregation + chairman synthesis after all members respond
• Add UseRedisOrchestrator(...) DI extension in Delibera.Server
• Update DebateEndpoints to use IDebateOrchestrator instead of direct executor
• Update SSE streaming to read from Redis Stream instead of in-process channel
• OpenTelemetry: MemberTurnExecute span with worker.id attribute
• Add Delibera.Worker standalone worker project (optional sidecar)
• Load test: 10 concurrent debates × 5 members each — validate ordering and correctness
• Docs: docs/distributed.md with architecture diagram and deployment guide

---

Acceptance Criteria

• LocalDebateOrchestrator is the default — no behavior change without Redis config
• RedisDebateOrchestrator correctly distributes member turns across workers
• Round ordering is preserved (chairman synthesis waits for all members in a round)
• SSE /stream endpoint works with Redis-backed events
• CancelAsync correctly stops all workers processing that debate
• Debate state survives a single worker restart (Redis persistence)
• Load test passes: 10 concurrent debates, all complete correctly
• OpenTelemetry spans emitted per worker turn

Labels

enhancement, v10.5.0, distributed, redis