feat(loops): enforce durable dispatch leases and retry backoff

docs/adr/0017-durable-dispatch-leases-and-retry-backoff.md

@@ -0,0 +1,90 @@
+# ADR 0017: Durable Dispatch Leases and Retry Backoff
+
+Status: Proposed
+Date: 2026-07-24
+Issue: [#96](https://github.com/ncolesummers/loopworks/issues/96)
+
+## Context
+
+Webhook idempotency prevents replaying one delivery, but it does not enforce a
+manifest concurrency cap across different deliveries or hosts. In-memory
+semaphores also lose authority on restart. Retry behavior needs the same durable
+boundary: delayed work cannot depend on sleeps, terminal evidence must remain
+inspectable, and attempt limits must mean the same thing in every runtime path.
+
+## Decision
+
+Use Postgres admission transactions and `idempotency_locks` as the durable
+dispatch authority. A persistent group-guard row is selected `FOR UPDATE`
+before capacity is counted. Acquired leases in the resolved manifest group
+consume capacity even after their expiry until their owner is finalized; this
+prevents an expired-but-running attempt from overlapping its successor.
+
+Every issue-backed run is protected by a partial unique index over repository
+and issue while its status is nonterminal. This applies across loop types.
+Delivery IDs remain replay keys, not concurrency identities. `{repo}` resolves
+to the canonical repository full name; other unresolved placeholders are
+rejected. A persistent repository/issue guard serializes development and
+research creation before the cross-loop uniqueness check, so contention is a
+typed outcome rather than a raw constraint error.
+
+Over-cap work is persisted as a queued run without a lease. Queue draining is
+ordered by `queuedAt`, then issue number, and acquires leases up to the manifest
+cap. Lease-less runs cannot enter a development stage. The winning terminal
+finalizer releases only a lease whose `run_id`, owner, and status still match;
+replay does not rewrite `releasedAt`, except to repair a legacy acquired leak
+using the already-persisted completion time. Terminal draining is scoped to the
+released repository group. Lease expiry is derived from
+`budgets.maxRunMinutes`.
+
+`retryPolicy.maxAttempts` counts the initial attempt. Fixed backoff is
+`min(initialSeconds, maxSeconds)`; exponential backoff is
+`min(initialSeconds * 2^(completedAttempt - 1), maxSeconds)`. Retryable stage
+failures keep the failed step evidence, release the lease, and receive a future
+eligibility time. The attempt increments only when a supervisor tick leases the
+due work. Stage promotions and linked runs consume the same dispatch-attempt
+budget. No hosted polling cadence is implied by this ADR.
+
+Reconciliation-authored `stalled` and `timed_out` outcomes preserve the source
+run as terminal and create a new trace-linked run from planning. The child
+inherits an immutable trigger snapshot, records `retryOfRunId`, and increments
+the dispatch attempt. Cancellation, success, and untyped terminal failure do
+not create linked retries.
+
+Admission primitives are loop-agnostic, while issue #96 integrates the complete
+lease and retry lifecycle only for the development loop. Research-loop durable
+transitions and enforcement require a separate M9 story.
+
+## Consequences
+
+- Restarts do not erase capacity ownership or retry eligibility.
+- Database row locks and uniqueness constraints, rather than process locality,
+  serialize competing dispatchers.
+- Terminal retry history is represented as linked immutable runs, increasing
+  row count while improving forensic clarity.
+- Operators must reconcile an expired owner before its capacity is reusable.
+- Missing lease evidence is unknown and fails open; released or expired lease
+  evidence is inactive. Queued acquired owners participate in reconciliation,
+  while queued deferred work does not.
+- Migration preflight fails closed when historical active duplicates exist.
+  Operators must inspect and terminalize the losing rows before retrying the
+  migration; the migration never guesses which forensic record to mutate.
+- A production scheduler may call the injected-clock supervisor, but its cadence
+  remains an explicit future hosting decision.
+
+## Validation
+
+1. PGlite integration tests cover deferral, contention, queue order, terminal
+   release, restart-equivalent reads, retry timing, exhaustion, and trace links.
+2. Migration tests apply and replay the schema, including the duplicate-active
+   preflight and partial unique index.
+3. Static observability tests keep dispatch and retry spans and metrics behind
+   central helpers with bounded attributes.
+
+## Follow-Ups
+
+- Draft an M9 story titled “Research-loop durable transitions, reconciliation,
+  lease lifecycle, and retry enforcement.” Do not create it without maintainer
+  authorization.
+- Add a real-Postgres multi-session admission lane if production lock-wait
+  behavior needs evidence stronger than embedded PGlite transactions.

docs/adr/README.md

@@ -37,6 +37,7 @@ Loopworks uses ADRs for durable technical and product architecture decisions. Gi
 | [0014](0014-guarded-github-pr-write-reconciliation.md) | Proposed | Guard GitHub PR writes with digest-bound approval, deterministic branches, and two-phase reconciliation. |
 | [0015](0015-stage-orchestrator-and-isolated-subagent-handoffs.md) | Proposed | Use a neutral root orchestrator, independent sibling subagents, and typed artifact handoffs between isolated sandboxes. |
 | [0016](0016-run-reconciliation-and-terminal-reasons.md) | Proposed | Reconcile active runs with typed terminal reasons, read-only tracker refresh, and injected execution liveness. |
+| [0017](0017-durable-dispatch-leases-and-retry-backoff.md) | Proposed | Serialize manifest admission with durable leases and preserve bounded retry evidence across restarts. |
 
 ## Template
 

docs/architecture.md

@@ -66,7 +66,9 @@ The initial schema should support these tables:
 7. Artifacts: run id, step id, type, title, URI, metadata, checksum where relevant.
 8. Approvals: run id, gate key, status, approver identity, rationale, evidence URI, timestamps.
 9. GitHub events: delivery id, event name, action, installation id, payload summary, processed status, error summary.
-10. Idempotency locks: key, scope, acquired/expired timestamps, owner, status.
+10. Idempotency locks: key, scope, run and trace correlation, acquired/expired
+    timestamps, owner, and status. Released group-guard rows serialize admission;
+    acquired run leases are the execution-liveness authority.
 11. Observability events or projections: correlation ids, metric counters, trace links, log summary links, and alert state.
 
 ## Observability
@@ -92,11 +94,23 @@ Logs are not the event store. The control plane must still persist durable run,
 5. Acquire a short-lived lock for repo plus issue plus trigger type.
 6. Normalize the event into Loopworks event tables.
 7. Evaluate loop triggers from the active manifest snapshot.
-8. Create a run or append a skipped/no-op decision with a durable reason.
-9. Write a summary or durable link back to GitHub only when useful to humans.
+8. Resolve the manifest concurrency group, lock its persistent guard row plus
+   the cross-loop repository/issue guard, and persist the run. Acquire a run
+   lease when capacity exists; otherwise retain the run as queued without a
+   lease. Stage transitions require that exact acquired lease.
+9. On terminal completion, release the exact owner/run lease and drain due work
+   for that repository group by oldest queue time then issue number. Retryable
+   failures keep their evidence and use manifest-computed future eligibility
+   rather than process sleeps.
+10. Write a summary or durable link back to GitHub only when useful to humans.
 
 Webhook processing must be repeat-safe. Re-delivery should not create duplicate runs, duplicate comments, or conflicting approval gates.
 
+One repository issue may have only one nonterminal run across loop types.
+Sequential runs are allowed after terminal completion. Expired but unreconciled
+leases continue consuming capacity so work cannot overlap before ownership is
+resolved.
+
 ## Loop Manifest
 
 The loop manifest should be versioned and validated. It must include:

docs/loop-manifest.md

@@ -103,6 +103,28 @@ terminal reasons are persisted separately from status: `stalled` and
 `timed_out` map to `failed`, while `canceled_by_reconciliation` maps to
 `canceled`. Execution liveness is supplied by the runtime boundary and an
 unknown signal fails open rather than terminating a potentially healthy run.
+For persistent development runs, the exact acquired run lease is the default
+liveness authority. Released or expired evidence is inactive; a missing lease
+is unknown and therefore fails open. Queued runs are reconciled only when they
+own an acquired lease, so deferred work is not mistaken for dead execution.
+
+Development-loop admission resolves `{repo}` in `concurrency.group` to the
+canonical repository full name and rejects any other unresolved placeholder.
+The group cap counts every acquired lease, including an expired lease whose
+owner has not yet been finalized. Over-cap triggers still create durable queued
+runs without leases, and stage transitions reject lease-less mutation. The
+persistent creation entry point reports `dispatched`, `deferred`, or
+`lease_contention` without flattening admission state. A terminal finalizer
+releases the matching run/owner lease and best-effort drains due work for the
+released repository group in `queuedAt`, then issue-number order.
+
+`retryPolicy.maxAttempts` is a total-attempt budget: attempt one is the initial
+execution. Fixed and exponential backoff are calculated from the completed
+attempt and capped by `maxSeconds`. Retryable stage failures remain inspectable
+until their due retry is leased, at which point the step attempt increments.
+`stalled` and `timed_out` create a new trace-linked run from planning when budget
+remains; `canceled_by_reconciliation`, success, and untyped terminal failure do
+not. The supervisor accepts an injected clock and defines no hosted cadence.
 
 The enabled `research-loop` is a parallel, fixture-backed generality probe. It
 requires both `spike` and `agent-ready`, uses a separate

drizzle/0007_remarkable_legion.sql

@@ -0,0 +1,21 @@
+ALTER TABLE "idempotency_locks" ADD COLUMN "run_id" uuid;--> statement-breakpoint
+ALTER TABLE "idempotency_locks" ADD COLUMN "trace_id" text;--> statement-breakpoint
+ALTER TABLE "idempotency_locks" ADD CONSTRAINT "idempotency_locks_run_id_loop_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."loop_runs"("id") ON DELETE set null ON UPDATE no action;--> statement-breakpoint
+CREATE INDEX "idempotency_locks_run_status_idx" ON "idempotency_locks" USING btree ("run_id","status");--> statement-breakpoint
+DO $$
+BEGIN
+  IF EXISTS (
+    SELECT 1
+    FROM "loop_runs"
+    WHERE "github_issue_number" IS NOT NULL
+      AND (
+        "status" IN ('queued', 'running', 'waiting_for_approval', 'blocked')
+        OR ("status" = 'failed' AND "completed_at" IS NULL)
+      )
+    GROUP BY "repository_id", "github_issue_number"
+    HAVING count(*) > 1
+  ) THEN
+    RAISE EXCEPTION 'Cannot enforce active issue uniqueness: duplicate nonterminal loop runs exist';
+  END IF;
+END $$;--> statement-breakpoint
+CREATE UNIQUE INDEX "loop_runs_active_repository_issue_idx" ON "loop_runs" USING btree ("repository_id","github_issue_number") WHERE "loop_runs"."github_issue_number" IS NOT NULL AND ("loop_runs"."status" IN ('queued', 'running', 'waiting_for_approval', 'blocked') OR ("loop_runs"."status" = 'failed' AND "loop_runs"."completed_at" IS NULL));