Persist Dead Letter Topic messages to a quarantine table

[piush365]

Background

Messages that exhaust all retries are published to flash-sale-orders.DLT, but they are never inspected or replayed. In production, a silent DLT is a data-loss risk — orders that failed to persist disappear without any reconciliation path.

Proposed Change

Add a dedicated DltEventConsumer that consumes from flash-sale-orders.DLT and writes records to a quarantine_events table.

Schema

-- V2__create_quarantine_events_table.sql
CREATE TABLE quarantine_events (
    id           BIGSERIAL PRIMARY KEY,
    topic        VARCHAR(128) NOT NULL,
    partition    INT          NOT NULL,
    "offset"     BIGINT       NOT NULL,
    payload      JSONB        NOT NULL,
    error_class  VARCHAR(256),
    error_message TEXT,
    received_at  TIMESTAMPTZ  NOT NULL DEFAULT NOW()
);

Admin replay endpoint

POST /api/v1/admin/quarantine/{id}/replay

Re-publishes the stored payload to the main topic so the normal consumer flow retries it.

Acceptance Criteria

• Flyway migration V2 creates quarantine_events
• DltEventConsumer persists every DLT message with error metadata
• Admin replay endpoint re-enqueues the event
• Metric flashsale.dlt.received_total incremented per DLT message
• Integration test verifies a deliberately poisoned message lands in quarantine