Enforce manifest concurrency, dispatch leases, and retry backoff

[ncolesummers]

Summary

The loop manifest declares dispatch policy that nothing enforces. src/lib/loops/manifest.ts:151 validates a concurrency block with a group key and maxInFlight, and emits a helpful validation error for it, but no runtime reads either field. retryPolicy is only partially consumed: development-run-transitions.ts:304 reads maxAttempts for the validation-review retry budget, while backoff is never applied and retries are only ever operator- or step-triggered.

The practical consequences are that any number of runs can be in flight for the same repository at once, and that a transient stage failure requires a human to notice and press retry.

This issue closes the gap between declared and enforced policy on the dispatch side. It depends on the terminal-reason contract and reconcile entry point from the reconciliation issue.

Design reference: OpenAI's Symphony spec (openai/symphony SPEC.md) gates dispatch on a claim set plus a global concurrency cap before launch, and applies delay = min(base * 2^(attempt-1), cap) to failure retries while using a short fixed delay for normal continuations. Symphony's claim set is in-memory and lost on restart. Loopworks should use a durable lease row instead, so a mid-deploy restart cannot double-dispatch.

Deliverables

• Enforce concurrency.maxInFlight per resolved concurrency group key before a run is dispatched; over-cap work is deferred rather than dropped or failed.
• Add durable dispatch leases so the same issue cannot be dispatched twice concurrently, including across host restarts and duplicate webhook deliveries. Extend the existing idempotency_locks table rather than introducing a parallel mechanism.
• Ensure a lease is always released on every terminal path, including the reconciliation-authored terminal reasons from the prerequisite issue.
• Apply retryPolicy.backoff to automatic retries of retryable stage failures, bounded by the already-consumed maxAttempts.
• Distinguish retryable from non-retryable terminal reasons: stalled and timed_out are eligible for automatic retry; canceled_by_reconciliation is not.
• Emit dispatch deferral and lease contention through existing src/lib/observability/ helpers, reusing loopworks.lock.contention where the semantics match rather than adding a near-duplicate metric.

Acceptance Criteria

• Focused tests are written first and demonstrate the red state before production code changes.
• A test asserts that dispatch is refused once maxInFlight is reached for a group, and resumes after an in-flight run reaches a terminal state.
• A test asserts that two concurrent dispatch attempts for the same issue result in exactly one run, with the second observing lease contention rather than erroring.
• A test asserts leases are released on succeeded, failed, and every reconciliation-authored terminal reason.
• Backoff delays are computed from manifest retryPolicy.backoff and asserted against an injected clock, with no wall-clock sleeps in tests.
• A test asserts canceled_by_reconciliation does not trigger an automatic retry.
• Automatic retries respect the existing maxAttempts budget and stop cleanly when exhausted, leaving the run inspectable.
• Concurrency and retry values come from the manifest; tests assert behavior changes when manifest values change.
• trace_id propagation stays aligned across dispatch, lease, and retry transitions.
• No instrumentation call site bypasses central observability helpers.
• Runtime changes include bun run build; broad changes finish with bun run validate.

Notes

• Ordering: this issue depends on the reconciliation issue for the terminal-reason contract and the reconcile entry point. Starting it first will force that contract to be invented twice.
• Prefer a Postgres-backed lease over an in-memory claim set. Vercel hosts restart frequently, and an in-memory claim would be lost exactly when double-dispatch is most likely.
• Do not add generic scheduling beyond what the manifest declares. If a policy is not in the manifest contract, it needs a manifest change and a docs update first.
• Keep observability attributes low-cardinality; avoid run IDs, issue payloads, and delivery IDs in metric attributes per the guidance in Instrument current webhook, approval, queue, lock, and run-start telemetry #64.

Refs #64
Refs ADR 0004
Refs ADR 0012

Refs #95
Blocked by #95