[S-02] Rate-Limiting & Cost Gates — WithCostLimit, WithRateLimit, DebateResult.CostEstimate

[techbuzzz]

Overview

Version: v10.5.0 Scale & Reliability
Effort: M · 3–5 days
Branch: feature/s02-cost-gates

Add built-in guardrails to prevent runaway LLM costs and API rate-limit breaches. Two orthogonal mechanisms: cost gates (abort if estimated cost exceeds a cap) and rate limiters (throttle requests per provider per second/minute).

---

API Design

CouncilBuilder Extensions

// Hard cap: abort debate if estimated cost exceeds limit
councilBuilder.WithCostLimit(decimal maxUsd);
councilBuilder.WithCostLimit(decimal maxUsd, CostLimitBehavior behavior);
// behavior: Abort (default) | WarnAndContinue | PauseAndWait

// Rate limit: max N requests per window per provider
councilBuilder.WithRateLimit(int requestsPerMinute);
councilBuilder.WithRateLimit(int requestsPerMinute, TimeSpan window);
councilBuilder.WithRateLimit(RateLimitPolicy policy);

CostEstimate on DebateResult

public record CostEstimate
{
    public decimal TotalUsd { get; init; }
    public decimal InputUsd { get; init; }
    public decimal OutputUsd { get; init; }
    public IReadOnlyList<MemberCostBreakdown> ByMember { get; init; }
    public string Currency { get; init; } // always "USD"
    public bool IsEstimate { get; init; } // true when based on token prices, not actual billing
}

public record MemberCostBreakdown
{
    public string Role { get; init; }
    public string Model { get; init; }
    public int InputTokens { get; init; }
    public int OutputTokens { get; init; }
    public decimal CostUsd { get; init; }
}

// DebateResult gains:
public CostEstimate? CostEstimate { get; init; }

RateLimitPolicy

public record RateLimitPolicy
{
    public int RequestsPerWindow { get; init; }
    public TimeSpan Window { get; init; }          // default: 1 minute
    public RateLimitScope Scope { get; init; }     // PerProvider | PerModel | Global
    public RateLimitBehavior Behavior { get; init; } // Queue | Throw | Drop
}

public enum RateLimitScope   { PerProvider, PerModel, Global }
public enum RateLimitBehavior { Queue, Throw, Drop }
public enum CostLimitBehavior { Abort, WarnAndContinue, PauseAndWait }

---

Pricing Registry

public interface IModelPricingRegistry
{
    ModelPricing? GetPricing(string provider, string model);
    void Register(string provider, string model, decimal inputPer1kTokens, decimal outputPer1kTokens);
}

public record ModelPricing
{
    public string Provider { get; init; }
    public string Model { get; init; }
    public decimal InputPer1KTokens { get; init; }   // USD
    public decimal OutputPer1KTokens { get; init; }  // USD
    public DateTime EffectiveDate { get; init; }
}

Built-in pricing for common models (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Claude 3 Haiku). Pricing registry is editable via appsettings.json:

"Delibera": {
  "Pricing": {
    "openai/gpt-4o":       { "InputPer1K": 0.0025, "OutputPer1K": 0.01 },
    "openai/gpt-4o-mini":  { "InputPer1K": 0.00015, "OutputPer1K": 0.0006 },
    "anthropic/claude-3-5-sonnet": { "InputPer1K": 0.003, "OutputPer1K": 0.015 }
  }
}

---

Cost Gate Flow

Before each member turn:
  1. Estimate tokens for next turn (context length + expected output)
  2. Calculate incremental cost
  3. Add to running total
  4. If total > maxUsd:
     - Abort   → throw DebateCostLimitExceededException, save partial result
     - Warn    → log warning, continue
     - Pause   → checkpoint + raise DebatePausedEvent(reason: CostLimit)

---

Rate Limiter Flow

Before each LLM call:
  1. Acquire slot from RateLimiter for (provider, model)
  2. Queue  → wait until slot available (respects WithTimeout)
  3. Throw  → throw DebateRateLimitExceededException
  4. Drop   → skip member turn, log warning

Implemented using System.Threading.RateLimiting.TokenBucketRateLimiter (built-in .NET 7+).

---

Tasks

• Define IModelPricingRegistry + DefaultModelPricingRegistry
• Add built-in pricing for GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Claude 3 Haiku
• Implement CostEstimate + MemberCostBreakdown records
• Add CostEstimate to DebateResult
• Implement cost calculation in CouncilExecutor (after each turn)
• Add WithCostLimit(...) to ICouncilBuilder + CouncilBuilder
• Implement cost gate check before each member turn
• Define RateLimitPolicy record + enums
• Implement DebateRateLimiter using TokenBucketRateLimiter
• Add WithRateLimit(...) to ICouncilBuilder + CouncilBuilder
• Add pricing config section to appsettings.json in Delibera.Server
• Add CostEstimate to DebateResponse DTO in server
• OpenTelemetry: debate.cost_usd metric gauge
• Unit tests: cost calculation accuracy, gate behavior (Abort/Warn/Pause)
• Unit tests: rate limiter (Queue/Throw/Drop behaviors)
• Docs: docs/cost-gates.md

---

Acceptance Criteria

• WithCostLimit(0.01m) aborts a debate that would exceed $0.01
• DebateResult.CostEstimate is populated for all debates with known pricing
• WithRateLimit(60) throttles to 60 requests/minute per provider
• RateLimitBehavior.Queue correctly delays requests without dropping
• Pricing registry loads from appsettings.json and overrides built-in prices
• debate.cost_usd OpenTelemetry metric emitted after each debate
• No cost/rate behavior when neither WithCostLimit nor WithRateLimit is configured (zero overhead)

Labels

enhancement, v10.5.0, reliability, cost