feat: reconcile running development loops

docs/adr/0016-run-reconciliation-and-terminal-reasons.md

@@ -0,0 +1,67 @@
+# ADR 0016: Run Reconciliation And Terminal Reasons
+
+Status: Proposed
+Date: 2026-07-22
+
+Driving issue: [#95](https://github.com/ncolesummers/loopworks/issues/95)
+
+## Context
+
+Development runs can remain `running` after agent silence, host termination, or
+backing-issue changes. Current persistence records only a broad run status and
+an optional untyped metadata reason. The database also has no durable execution
+lease yet; that dispatch-side contract belongs to issue #96.
+
+Reconciliation needs deterministic time, typed terminal evidence, read-only
+tracker refresh, and an execution-liveness seam that does not guess from queued
+or running step status.
+
+## Decision
+
+Each loop manifest declares a positive `reconciliation.silenceThresholdSeconds`
+value. A reconciliation pass reads active Postgres runs, refreshes their GitHub
+issues without mutation, and resolves cancellation policy before execution
+health. The precedence is tracker or manifest cancellation, inactive execution,
+silent active execution, then healthy.
+
+`loop_runs.terminal_reason` uses the typed values `succeeded`, `failed`,
+`timed_out`, `stalled`, and `canceled_by_reconciliation`. The terminal finalizer
+derives the broader run status, uses a compare-and-set transition, persists one
+correlated completion event, and emits the existing run completion and duration
+metrics only for the winning transition.
+
+Health-authored finalization revalidates the current stage, current step, and
+latest relevant activity immediately before the terminal transition. Queued
+future stages do not count as activity. A changed snapshot remains active so a
+reconcile pass cannot terminate work that progressed during tracker refresh.
+
+Execution liveness is injected as `active`, `inactive`, or `unknown`. Unknown
+liveness fails open. Reconciliation never infers host liveness from step status;
+issue #96 may later implement this boundary with durable dispatch leases.
+
+## Consequences
+
+Runs gain durable, queryable failure taxonomy and deterministic reconciliation
+without widening this issue into scheduling or dispatch. Replays and concurrent
+passes are safe, and GitHub remains a read-only source during reconciliation.
+
+Until issue #96 supplies durable lease-backed evidence, callers must provide a
+trustworthy liveness source. An unavailable source preserves the run and
+surfaces an explicit unknown outcome. Existing step rows remain unchanged as
+forensic evidence when the owning run is finalized.
+
+## Validation
+
+1. Manifest tests reject missing or non-positive silence thresholds.
+2. Migration replay and PGlite tests cover typed terminal reasons.
+3. Reconciliation tests independently cover healthy, stalled, timed-out,
+   canceled, continued-policy, tracker-failure, and unknown-liveness outcomes.
+4. Concurrent finalization produces one durable event and one metric emission.
+5. Race tests reject stale health evidence, and store tests exclude queued
+   future-stage timestamps and non-development loops.
+6. Static observability tests keep metric and span names behind shared helpers.
+
+## Follow-Ups
+
+1. Issue #96 provides durable dispatch leases and backs the liveness boundary.
+2. Issues #66 and #67 may project terminal reasons into dashboards and alerts.

docs/adr/README.md

@@ -36,6 +36,7 @@ Loopworks uses ADRs for durable technical and product architecture decisions. Gi
 | [0013](0013-planning-agent-contract.md) | Proposed | Define the planning agent as a planning-only runtime with guarded CLI inspection, explicit fixtures, eval coverage, and ADR 0012-aligned telemetry boundaries. |
 | [0014](0014-guarded-github-pr-write-reconciliation.md) | Proposed | Guard GitHub PR writes with digest-bound approval, deterministic branches, and two-phase reconciliation. |
 | [0015](0015-stage-orchestrator-and-isolated-subagent-handoffs.md) | Proposed | Use a neutral root orchestrator, independent sibling subagents, and typed artifact handoffs between isolated sandboxes. |
+| [0016](0016-run-reconciliation-and-terminal-reasons.md) | Proposed | Reconcile active runs with typed terminal reasons, read-only tracker refresh, and injected execution liveness. |
 
 ## Template
 

docs/loop-manifest.md

@@ -67,8 +67,12 @@ Each loop definition includes:
 9. `retryPolicy` with bounded attempts and backoff.
 10. `concurrency` with the group key, max in-flight runs, and cancellation
     behavior for overlapping work.
-11. `cancellation` for disabled or superseded work.
-12. `githubWriteback` for approved comments, labels, or status checks.
+11. `reconciliation` with a required positive integer
+    `silenceThresholdSeconds` value. A positively active execution is stalled
+    only when its latest persisted step activity is strictly older than this
+    threshold; activity exactly at the threshold remains healthy.
+12. `cancellation` for disabled or superseded work.
+13. `githubWriteback` for approved comments, labels, or status checks.
 
 The sample `development-loop` covers the `agent-ready` trigger and the first
 durable issue-backed implementation skeleton. The stage sequence is stable:
@@ -91,6 +95,15 @@ skipped/no-op reason such as `loop_disabled` so operators can explain why an
 `agent-ready` issue did not start. Research-loop disabled evidence is tracked
 separately from the development-loop skeleton.
 
+Active development runs are reconciled against this manifest contract without
+mutating GitHub. Closed issues and issues that lose required trigger labels use
+`cancellation.onSuperseded`; disabled loops use `cancellation.onDisabled`.
+Policies that continue existing work leave the run active. Reconcile-authored
+terminal reasons are persisted separately from status: `stalled` and
+`timed_out` map to `failed`, while `canceled_by_reconciliation` maps to
+`canceled`. Execution liveness is supplied by the runtime boundary and an
+unknown signal fails open rather than terminating a potentially healthy run.
+
 The enabled `research-loop` is a parallel, fixture-backed generality probe. It
 requires both `spike` and `agent-ready`, uses a separate
 `repo:{repo}:loop:research` concurrency group, and permits only repository read,

drizzle/0006_skinny_electro.sql

@@ -0,0 +1,2 @@
+CREATE TYPE "public"."run_terminal_reason" AS ENUM('succeeded', 'failed', 'timed_out', 'stalled', 'canceled_by_reconciliation');--> statement-breakpoint
+ALTER TABLE "loop_runs" ADD COLUMN "terminal_reason" "run_terminal_reason";