diff --git a/docs/adr/0016-run-reconciliation-and-terminal-reasons.md b/docs/adr/0016-run-reconciliation-and-terminal-reasons.md new file mode 100644 index 0000000..d5871e6 --- /dev/null +++ b/docs/adr/0016-run-reconciliation-and-terminal-reasons.md @@ -0,0 +1,67 @@ +# ADR 0016: Run Reconciliation And Terminal Reasons + +Status: Proposed +Date: 2026-07-22 + +Driving issue: [#95](https://github.com/ncolesummers/loopworks/issues/95) + +## Context + +Development runs can remain `running` after agent silence, host termination, or +backing-issue changes. Current persistence records only a broad run status and +an optional untyped metadata reason. The database also has no durable execution +lease yet; that dispatch-side contract belongs to issue #96. + +Reconciliation needs deterministic time, typed terminal evidence, read-only +tracker refresh, and an execution-liveness seam that does not guess from queued +or running step status. + +## Decision + +Each loop manifest declares a positive `reconciliation.silenceThresholdSeconds` +value. A reconciliation pass reads active Postgres runs, refreshes their GitHub +issues without mutation, and resolves cancellation policy before execution +health. The precedence is tracker or manifest cancellation, inactive execution, +silent active execution, then healthy. + +`loop_runs.terminal_reason` uses the typed values `succeeded`, `failed`, +`timed_out`, `stalled`, and `canceled_by_reconciliation`. The terminal finalizer +derives the broader run status, uses a compare-and-set transition, persists one +correlated completion event, and emits the existing run completion and duration +metrics only for the winning transition. + +Health-authored finalization revalidates the current stage, current step, and +latest relevant activity immediately before the terminal transition. Queued +future stages do not count as activity. A changed snapshot remains active so a +reconcile pass cannot terminate work that progressed during tracker refresh. + +Execution liveness is injected as `active`, `inactive`, or `unknown`. Unknown +liveness fails open. Reconciliation never infers host liveness from step status; +issue #96 may later implement this boundary with durable dispatch leases. + +## Consequences + +Runs gain durable, queryable failure taxonomy and deterministic reconciliation +without widening this issue into scheduling or dispatch. Replays and concurrent +passes are safe, and GitHub remains a read-only source during reconciliation. + +Until issue #96 supplies durable lease-backed evidence, callers must provide a +trustworthy liveness source. An unavailable source preserves the run and +surfaces an explicit unknown outcome. Existing step rows remain unchanged as +forensic evidence when the owning run is finalized. + +## Validation + +1. Manifest tests reject missing or non-positive silence thresholds. +2. Migration replay and PGlite tests cover typed terminal reasons. +3. Reconciliation tests independently cover healthy, stalled, timed-out, + canceled, continued-policy, tracker-failure, and unknown-liveness outcomes. +4. Concurrent finalization produces one durable event and one metric emission. +5. Race tests reject stale health evidence, and store tests exclude queued + future-stage timestamps and non-development loops. +6. Static observability tests keep metric and span names behind shared helpers. + +## Follow-Ups + +1. Issue #96 provides durable dispatch leases and backs the liveness boundary. +2. Issues #66 and #67 may project terminal reasons into dashboards and alerts. diff --git a/docs/adr/README.md b/docs/adr/README.md index 8be258d..a246260 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -36,6 +36,7 @@ Loopworks uses ADRs for durable technical and product architecture decisions. Gi | [0013](0013-planning-agent-contract.md) | Proposed | Define the planning agent as a planning-only runtime with guarded CLI inspection, explicit fixtures, eval coverage, and ADR 0012-aligned telemetry boundaries. | | [0014](0014-guarded-github-pr-write-reconciliation.md) | Proposed | Guard GitHub PR writes with digest-bound approval, deterministic branches, and two-phase reconciliation. | | [0015](0015-stage-orchestrator-and-isolated-subagent-handoffs.md) | Proposed | Use a neutral root orchestrator, independent sibling subagents, and typed artifact handoffs between isolated sandboxes. | +| [0016](0016-run-reconciliation-and-terminal-reasons.md) | Proposed | Reconcile active runs with typed terminal reasons, read-only tracker refresh, and injected execution liveness. | ## Template diff --git a/docs/loop-manifest.md b/docs/loop-manifest.md index 851907a..dfa083e 100644 --- a/docs/loop-manifest.md +++ b/docs/loop-manifest.md @@ -67,8 +67,12 @@ Each loop definition includes: 9. `retryPolicy` with bounded attempts and backoff. 10. `concurrency` with the group key, max in-flight runs, and cancellation behavior for overlapping work. -11. `cancellation` for disabled or superseded work. -12. `githubWriteback` for approved comments, labels, or status checks. +11. `reconciliation` with a required positive integer + `silenceThresholdSeconds` value. A positively active execution is stalled + only when its latest persisted step activity is strictly older than this + threshold; activity exactly at the threshold remains healthy. +12. `cancellation` for disabled or superseded work. +13. `githubWriteback` for approved comments, labels, or status checks. The sample `development-loop` covers the `agent-ready` trigger and the first durable issue-backed implementation skeleton. The stage sequence is stable: @@ -91,6 +95,15 @@ skipped/no-op reason such as `loop_disabled` so operators can explain why an `agent-ready` issue did not start. Research-loop disabled evidence is tracked separately from the development-loop skeleton. +Active development runs are reconciled against this manifest contract without +mutating GitHub. Closed issues and issues that lose required trigger labels use +`cancellation.onSuperseded`; disabled loops use `cancellation.onDisabled`. +Policies that continue existing work leave the run active. Reconcile-authored +terminal reasons are persisted separately from status: `stalled` and +`timed_out` map to `failed`, while `canceled_by_reconciliation` maps to +`canceled`. Execution liveness is supplied by the runtime boundary and an +unknown signal fails open rather than terminating a potentially healthy run. + The enabled `research-loop` is a parallel, fixture-backed generality probe. It requires both `spike` and `agent-ready`, uses a separate `repo:{repo}:loop:research` concurrency group, and permits only repository read, diff --git a/drizzle/0006_skinny_electro.sql b/drizzle/0006_skinny_electro.sql new file mode 100644 index 0000000..891fffa --- /dev/null +++ b/drizzle/0006_skinny_electro.sql @@ -0,0 +1,2 @@ +CREATE TYPE "public"."run_terminal_reason" AS ENUM('succeeded', 'failed', 'timed_out', 'stalled', 'canceled_by_reconciliation');--> statement-breakpoint +ALTER TABLE "loop_runs" ADD COLUMN "terminal_reason" "run_terminal_reason"; \ No newline at end of file diff --git a/drizzle/meta/0006_snapshot.json b/drizzle/meta/0006_snapshot.json new file mode 100644 index 0000000..3dc9148 --- /dev/null +++ b/drizzle/meta/0006_snapshot.json @@ -0,0 +1,2271 @@ +{ + "id": "7c56be06-d3f2-40e4-acc2-c1663d2e43fc", + "prevId": "c7100bf4-4ad0-484f-8b52-ca42b99de518", + "version": "7", + "dialect": "postgresql", + "tables": { + "public.accounts": { + "name": "accounts", + "schema": "", + "columns": { + "user_id": { + "name": "user_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "provider_account_id": { + "name": "provider_account_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "refresh_token": { + "name": "refresh_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "access_token": { + "name": "access_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "token_type": { + "name": "token_type", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "id_token": { + "name": "id_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "session_state": { + "name": "session_state", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": {}, + "foreignKeys": { + "accounts_user_id_users_id_fk": { + "name": "accounts_user_id_users_id_fk", + "tableFrom": "accounts", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "accounts_provider_provider_account_id_pk": { + "name": "accounts_provider_provider_account_id_pk", + "columns": ["provider", "provider_account_id"] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.agent_plans": { + "name": "agent_plans", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "loop_id": { + "name": "loop_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "issue_number": { + "name": "issue_number", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "agent_name": { + "name": "agent_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'planner'" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "input": { + "name": "input", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "plan": { + "name": "plan", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "agent_plans_loop_id_created_at_idx": { + "name": "agent_plans_loop_id_created_at_idx", + "columns": [ + { + "expression": "loop_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "agent_plans_run_id_created_at_idx": { + "name": "agent_plans_run_id_created_at_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "agent_plans_loop_id_loops_id_fk": { + "name": "agent_plans_loop_id_loops_id_fk", + "tableFrom": "agent_plans", + "tableTo": "loops", + "columnsFrom": ["loop_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "agent_plans_run_id_loop_runs_id_fk": { + "name": "agent_plans_run_id_loop_runs_id_fk", + "tableFrom": "agent_plans", + "tableTo": "loop_runs", + "columnsFrom": ["run_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.approval_transition_events": { + "name": "approval_transition_events", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "approval_id": { + "name": "approval_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "from_status": { + "name": "from_status", + "type": "approval_status", + "typeSchema": "public", + "primaryKey": false, + "notNull": true + }, + "to_status": { + "name": "to_status", + "type": "approval_status", + "typeSchema": "public", + "primaryKey": false, + "notNull": true + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "actor_id": { + "name": "actor_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "occurred_at": { + "name": "occurred_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "note": { + "name": "note", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "approval_transition_events_approval_created_at_idx": { + "name": "approval_transition_events_approval_created_at_idx", + "columns": [ + { + "expression": "approval_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "occurred_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "approval_transition_events_run_created_at_idx": { + "name": "approval_transition_events_run_created_at_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "occurred_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "approval_transition_events_approval_id_approvals_id_fk": { + "name": "approval_transition_events_approval_id_approvals_id_fk", + "tableFrom": "approval_transition_events", + "tableTo": "approvals", + "columnsFrom": ["approval_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "approval_transition_events_run_id_loop_runs_id_fk": { + "name": "approval_transition_events_run_id_loop_runs_id_fk", + "tableFrom": "approval_transition_events", + "tableTo": "loop_runs", + "columnsFrom": ["run_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.approvals": { + "name": "approvals", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "loop_id": { + "name": "loop_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "approval_status", + "typeSchema": "public", + "primaryKey": false, + "notNull": true, + "default": "'requested'" + }, + "requested_by": { + "name": "requested_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "requested_at": { + "name": "requested_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "resolved_by": { + "name": "resolved_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "resolved_at": { + "name": "resolved_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "note": { + "name": "note", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "approvals_loop_id_status_idx": { + "name": "approvals_loop_id_status_idx", + "columns": [ + { + "expression": "loop_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "approvals_run_id_status_idx": { + "name": "approvals_run_id_status_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "approvals_loop_id_loops_id_fk": { + "name": "approvals_loop_id_loops_id_fk", + "tableFrom": "approvals", + "tableTo": "loops", + "columnsFrom": ["loop_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "approvals_run_id_loop_runs_id_fk": { + "name": "approvals_run_id_loop_runs_id_fk", + "tableFrom": "approvals", + "tableTo": "loop_runs", + "columnsFrom": ["run_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.artifacts": { + "name": "artifacts", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "step_id": { + "name": "step_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "type": { + "name": "type", + "type": "artifact_type", + "typeSchema": "public", + "primaryKey": false, + "notNull": true, + "default": "'other'" + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "uri": { + "name": "uri", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "sha256": { + "name": "sha256", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "artifacts_run_type_idx": { + "name": "artifacts_run_type_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "type", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "artifacts_step_id_idx": { + "name": "artifacts_step_id_idx", + "columns": [ + { + "expression": "step_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "artifacts_run_id_loop_runs_id_fk": { + "name": "artifacts_run_id_loop_runs_id_fk", + "tableFrom": "artifacts", + "tableTo": "loop_runs", + "columnsFrom": ["run_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "artifacts_step_id_run_steps_id_fk": { + "name": "artifacts_step_id_run_steps_id_fk", + "tableFrom": "artifacts", + "tableTo": "run_steps", + "columnsFrom": ["step_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.deployments": { + "name": "deployments", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "repository_id": { + "name": "repository_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "loop_id": { + "name": "loop_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'vercel'" + }, + "external_id": { + "name": "external_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "project_name": { + "name": "project_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "deployment_status", + "typeSchema": "public", + "primaryKey": false, + "notNull": true + }, + "environment": { + "name": "environment", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "branch": { + "name": "branch", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "commit_sha": { + "name": "commit_sha", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "inspector_url": { + "name": "inspector_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "alias_urls": { + "name": "alias_urls", + "type": "text[]", + "primaryKey": false, + "notNull": true, + "default": "'{}'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "ready_at": { + "name": "ready_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "deployments_project_id_created_at_idx": { + "name": "deployments_project_id_created_at_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "deployments_run_id_created_at_idx": { + "name": "deployments_run_id_created_at_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "deployments_repository_id_repositories_id_fk": { + "name": "deployments_repository_id_repositories_id_fk", + "tableFrom": "deployments", + "tableTo": "repositories", + "columnsFrom": ["repository_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "deployments_loop_id_loops_id_fk": { + "name": "deployments_loop_id_loops_id_fk", + "tableFrom": "deployments", + "tableTo": "loops", + "columnsFrom": ["loop_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "deployments_run_id_loop_runs_id_fk": { + "name": "deployments_run_id_loop_runs_id_fk", + "tableFrom": "deployments", + "tableTo": "loop_runs", + "columnsFrom": ["run_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "deployments_external_id_unique": { + "name": "deployments_external_id_unique", + "nullsNotDistinct": false, + "columns": ["external_id"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.idempotency_locks": { + "name": "idempotency_locks", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "owner": { + "name": "owner", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "idempotency_lock_status", + "typeSchema": "public", + "primaryKey": false, + "notNull": true, + "default": "'acquired'" + }, + "acquired_at": { + "name": "acquired_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "released_at": { + "name": "released_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idempotency_locks_scope_status_idx": { + "name": "idempotency_locks_scope_status_idx", + "columns": [ + { + "expression": "scope", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idempotency_locks_expires_at_idx": { + "name": "idempotency_locks_expires_at_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "idempotency_locks_key_unique": { + "name": "idempotency_locks_key_unique", + "nullsNotDistinct": false, + "columns": ["key"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.loop_events": { + "name": "loop_events", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "loop_id": { + "name": "loop_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "actor_type": { + "name": "actor_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "actor_id": { + "name": "actor_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "event_type": { + "name": "event_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "from_state": { + "name": "from_state", + "type": "loop_state", + "typeSchema": "public", + "primaryKey": false, + "notNull": false + }, + "to_state": { + "name": "to_state", + "type": "loop_state", + "typeSchema": "public", + "primaryKey": false, + "notNull": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "payload": { + "name": "payload", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "loop_events_loop_id_created_at_idx": { + "name": "loop_events_loop_id_created_at_idx", + "columns": [ + { + "expression": "loop_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "loop_events_loop_id_loops_id_fk": { + "name": "loop_events_loop_id_loops_id_fk", + "tableFrom": "loop_events", + "tableTo": "loops", + "columnsFrom": ["loop_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.loop_runs": { + "name": "loop_runs", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "repository_id": { + "name": "repository_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "loop_key": { + "name": "loop_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "github_issue_number": { + "name": "github_issue_number", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "github_issue_url": { + "name": "github_issue_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "run_status", + "typeSchema": "public", + "primaryKey": false, + "notNull": true, + "default": "'queued'" + }, + "terminal_reason": { + "name": "terminal_reason", + "type": "run_terminal_reason", + "typeSchema": "public", + "primaryKey": false, + "notNull": false + }, + "current_stage": { + "name": "current_stage", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'planning'" + }, + "trace_id": { + "name": "trace_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "cost_cents": { + "name": "cost_cents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "queued_at": { + "name": "queued_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "started_at": { + "name": "started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "completed_at": { + "name": "completed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "canceled_at": { + "name": "canceled_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "loop_runs_repository_status_idx": { + "name": "loop_runs_repository_status_idx", + "columns": [ + { + "expression": "repository_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "loop_runs_repository_issue_idx": { + "name": "loop_runs_repository_issue_idx", + "columns": [ + { + "expression": "repository_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "github_issue_number", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "loop_runs_repository_id_repositories_id_fk": { + "name": "loop_runs_repository_id_repositories_id_fk", + "tableFrom": "loop_runs", + "tableTo": "repositories", + "columnsFrom": ["repository_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.loops": { + "name": "loops", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "repository_id": { + "name": "repository_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "github_issue_number": { + "name": "github_issue_number", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "loop_state", + "typeSchema": "public", + "primaryKey": false, + "notNull": true, + "default": "'intake'" + }, + "milestone": { + "name": "milestone", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "area_label": { + "name": "area_label", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "priority_label": { + "name": "priority_label", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "owner_github_login": { + "name": "owner_github_login", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "source_url": { + "name": "source_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "last_synced_at": { + "name": "last_synced_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "loops_repository_issue_number_idx": { + "name": "loops_repository_issue_number_idx", + "columns": [ + { + "expression": "repository_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "github_issue_number", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "loops_repository_id_repositories_id_fk": { + "name": "loops_repository_id_repositories_id_fk", + "tableFrom": "loops", + "tableTo": "repositories", + "columnsFrom": ["repository_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.observability_events": { + "name": "observability_events", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "repository_id": { + "name": "repository_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "step_id": { + "name": "step_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "event_type": { + "name": "event_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "severity": { + "name": "severity", + "type": "observability_severity", + "typeSchema": "public", + "primaryKey": false, + "notNull": true, + "default": "'info'" + }, + "correlation_id": { + "name": "correlation_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "trace_id": { + "name": "trace_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metric_name": { + "name": "metric_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metric_value": { + "name": "metric_value", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "payload": { + "name": "payload", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "observability_events_type_created_at_idx": { + "name": "observability_events_type_created_at_idx", + "columns": [ + { + "expression": "event_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "observability_events_run_created_at_idx": { + "name": "observability_events_run_created_at_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "observability_events_trace_id_idx": { + "name": "observability_events_trace_id_idx", + "columns": [ + { + "expression": "trace_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "observability_events_repository_id_repositories_id_fk": { + "name": "observability_events_repository_id_repositories_id_fk", + "tableFrom": "observability_events", + "tableTo": "repositories", + "columnsFrom": ["repository_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "observability_events_run_id_loop_runs_id_fk": { + "name": "observability_events_run_id_loop_runs_id_fk", + "tableFrom": "observability_events", + "tableTo": "loop_runs", + "columnsFrom": ["run_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "observability_events_step_id_run_steps_id_fk": { + "name": "observability_events_step_id_run_steps_id_fk", + "tableFrom": "observability_events", + "tableTo": "run_steps", + "columnsFrom": ["step_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.repositories": { + "name": "repositories", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "github_repo_id": { + "name": "github_repo_id", + "type": "bigint", + "primaryKey": false, + "notNull": true + }, + "owner": { + "name": "owner", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "full_name": { + "name": "full_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "bigint", + "primaryKey": false, + "notNull": false + }, + "is_active": { + "name": "is_active", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": true + }, + "health": { + "name": "health", + "type": "repo_health", + "typeSchema": "public", + "primaryKey": false, + "notNull": true, + "default": "'healthy'" + }, + "framework": { + "name": "framework", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'Unknown'" + }, + "default_branch": { + "name": "default_branch", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'main'" + }, + "ci_commands": { + "name": "ci_commands", + "type": "text[]", + "primaryKey": false, + "notNull": true, + "default": "'{}'" + }, + "docs_href": { + "name": "docs_href", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "observability_href": { + "name": "observability_href", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "design_system_href": { + "name": "design_system_href", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "enabled_loops": { + "name": "enabled_loops", + "type": "text[]", + "primaryKey": false, + "notNull": true, + "default": "'{}'" + }, + "validation_gates": { + "name": "validation_gates", + "type": "text[]", + "primaryKey": false, + "notNull": true, + "default": "'{}'" + }, + "last_synced_at": { + "name": "last_synced_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "repositories_github_repo_id_unique": { + "name": "repositories_github_repo_id_unique", + "nullsNotDistinct": false, + "columns": ["github_repo_id"] + }, + "repositories_full_name_unique": { + "name": "repositories_full_name_unique", + "nullsNotDistinct": false, + "columns": ["full_name"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.run_steps": { + "name": "run_steps", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "stage": { + "name": "stage", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "run_step_status", + "typeSchema": "public", + "primaryKey": false, + "notNull": true, + "default": "'queued'" + }, + "attempt": { + "name": "attempt", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "actor_type": { + "name": "actor_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "actor_id": { + "name": "actor_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "summary": { + "name": "summary", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "validation_command": { + "name": "validation_command", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "validation_status": { + "name": "validation_status", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "trace_id": { + "name": "trace_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "queued_at": { + "name": "queued_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "started_at": { + "name": "started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "completed_at": { + "name": "completed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "run_steps_run_stage_idx": { + "name": "run_steps_run_stage_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "stage", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "run_steps_run_status_idx": { + "name": "run_steps_run_status_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "run_steps_run_id_loop_runs_id_fk": { + "name": "run_steps_run_id_loop_runs_id_fk", + "tableFrom": "run_steps", + "tableTo": "loop_runs", + "columnsFrom": ["run_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.sessions": { + "name": "sessions", + "schema": "", + "columns": { + "session_token": { + "name": "session_token", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "expires": { + "name": "expires", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": {}, + "foreignKeys": { + "sessions_user_id_users_id_fk": { + "name": "sessions_user_id_users_id_fk", + "tableFrom": "sessions", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.users": { + "name": "users", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "email_verified": { + "name": "email_verified", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "image": { + "name": "image", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "github_login": { + "name": "github_login", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "users_email_unique": { + "name": "users_email_unique", + "nullsNotDistinct": false, + "columns": ["email"] + }, + "users_github_login_unique": { + "name": "users_github_login_unique", + "nullsNotDistinct": false, + "columns": ["github_login"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.vercel_projects": { + "name": "vercel_projects", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "repository_id": { + "name": "repository_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "project_name": { + "name": "project_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "team_id": { + "name": "team_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "team_slug": { + "name": "team_slug", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "production_url": { + "name": "production_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "dashboard_url": { + "name": "dashboard_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "vercel_projects_repository_id_idx": { + "name": "vercel_projects_repository_id_idx", + "columns": [ + { + "expression": "repository_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "vercel_projects_repository_id_repositories_id_fk": { + "name": "vercel_projects_repository_id_repositories_id_fk", + "tableFrom": "vercel_projects", + "tableTo": "repositories", + "columnsFrom": ["repository_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "vercel_projects_project_id_unique": { + "name": "vercel_projects_project_id_unique", + "nullsNotDistinct": false, + "columns": ["project_id"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.verification_tokens": { + "name": "verification_tokens", + "schema": "", + "columns": { + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "expires": { + "name": "expires", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": { + "verification_tokens_identifier_token_pk": { + "name": "verification_tokens_identifier_token_pk", + "columns": ["identifier", "token"] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.webhook_deliveries": { + "name": "webhook_deliveries", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'github'" + }, + "delivery_id": { + "name": "delivery_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "event": { + "name": "event", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "repository_full_name": { + "name": "repository_full_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "webhook_delivery_status", + "typeSchema": "public", + "primaryKey": false, + "notNull": true, + "default": "'received'" + }, + "payload": { + "name": "payload", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "received_at": { + "name": "received_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "processed_at": { + "name": "processed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "webhook_deliveries_delivery_id_unique": { + "name": "webhook_deliveries_delivery_id_unique", + "nullsNotDistinct": false, + "columns": ["delivery_id"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + } + }, + "enums": { + "public.approval_status": { + "name": "approval_status", + "schema": "public", + "values": ["requested", "approved", "rejected", "cancelled", "expired", "applied", "bypassed"] + }, + "public.artifact_type": { + "name": "artifact_type", + "schema": "public", + "values": [ + "plan", + "validation_report", + "test_plan", + "patch", + "pr_intent", + "deployment_summary", + "log_summary", + "trace", + "screenshot", + "other" + ] + }, + "public.deployment_status": { + "name": "deployment_status", + "schema": "public", + "values": ["queued", "building", "ready", "error", "canceled"] + }, + "public.idempotency_lock_status": { + "name": "idempotency_lock_status", + "schema": "public", + "values": ["acquired", "released", "expired"] + }, + "public.loop_state": { + "name": "loop_state", + "schema": "public", + "values": [ + "intake", + "triage", + "planned", + "in_progress", + "waiting_on_review", + "validating", + "blocked", + "done" + ] + }, + "public.observability_severity": { + "name": "observability_severity", + "schema": "public", + "values": ["debug", "info", "warn", "error"] + }, + "public.repo_health": { + "name": "repo_health", + "schema": "public", + "values": ["healthy", "watch", "blocked", "disconnected"] + }, + "public.run_status": { + "name": "run_status", + "schema": "public", + "values": [ + "queued", + "running", + "waiting_for_approval", + "blocked", + "failed", + "succeeded", + "canceled" + ] + }, + "public.run_step_status": { + "name": "run_step_status", + "schema": "public", + "values": ["queued", "running", "skipped", "failed", "succeeded"] + }, + "public.run_terminal_reason": { + "name": "run_terminal_reason", + "schema": "public", + "values": ["succeeded", "failed", "timed_out", "stalled", "canceled_by_reconciliation"] + }, + "public.webhook_delivery_status": { + "name": "webhook_delivery_status", + "schema": "public", + "values": ["received", "processed", "ignored", "failed"] + } + }, + "schemas": {}, + "sequences": {}, + "roles": {}, + "policies": {}, + "views": {}, + "_meta": { + "columns": {}, + "schemas": {}, + "tables": {} + } +} diff --git a/drizzle/meta/_journal.json b/drizzle/meta/_journal.json index aa0fb1e..9db3f3b 100644 --- a/drizzle/meta/_journal.json +++ b/drizzle/meta/_journal.json @@ -43,6 +43,13 @@ "when": 1784010988181, "tag": "0005_backfill_validation_screenshot_artifacts", "breakpoints": true + }, + { + "idx": 6, + "version": "7", + "when": 1784778985115, + "tag": "0006_skinny_electro", + "breakpoints": true } ] } diff --git a/schemas/loop-manifest.schema.json b/schemas/loop-manifest.schema.json index 260694d..beeaa58 100644 --- a/schemas/loop-manifest.schema.json +++ b/schemas/loop-manifest.schema.json @@ -116,6 +116,7 @@ "validationGates", "retryPolicy", "concurrency", + "reconciliation", "cancellation", "githubWriteback" ], @@ -173,6 +174,9 @@ "concurrency": { "$ref": "#/$defs/concurrency" }, + "reconciliation": { + "$ref": "#/$defs/reconciliation" + }, "cancellation": { "$ref": "#/$defs/cancellation" }, @@ -521,6 +525,17 @@ } } }, + "reconciliation": { + "type": "object", + "additionalProperties": false, + "required": ["silenceThresholdSeconds"], + "properties": { + "silenceThresholdSeconds": { + "type": "integer", + "exclusiveMinimum": 0 + } + } + }, "cancellation": { "type": "object", "additionalProperties": false, diff --git a/schemas/loop-manifest.ts b/schemas/loop-manifest.ts index 827f1fb..c5dde75 100644 --- a/schemas/loop-manifest.ts +++ b/schemas/loop-manifest.ts @@ -271,6 +271,12 @@ const concurrencySchema = z }) .strict(); +const reconciliationSchema = z + .object({ + silenceThresholdSeconds: z.number().int().positive(), + }) + .strict(); + const cancellationSchema = z .object({ onSuperseded: z.enum(cancellationOnSupersededValues), @@ -303,6 +309,7 @@ const loopDefinitionSchema = z validationGates: z.array(validationGateSchema).min(1), retryPolicy: retryPolicySchema, concurrency: concurrencySchema, + reconciliation: reconciliationSchema, cancellation: cancellationSchema, githubWriteback: githubWritebackSchema, }) diff --git a/src/db/schema.ts b/src/db/schema.ts index cc8527f..12aab6d 100644 --- a/src/db/schema.ts +++ b/src/db/schema.ts @@ -46,6 +46,13 @@ export const runStatusEnum = pgEnum("run_status", [ "succeeded", "canceled", ]); +export const runTerminalReasonEnum = pgEnum("run_terminal_reason", [ + "succeeded", + "failed", + "timed_out", + "stalled", + "canceled_by_reconciliation", +]); export const runStepStatusEnum = pgEnum("run_step_status", [ "queued", "running", @@ -216,6 +223,7 @@ export const loopRuns = pgTable( githubIssueNumber: integer("github_issue_number"), githubIssueUrl: text("github_issue_url"), status: runStatusEnum("status").default("queued").notNull(), + terminalReason: runTerminalReasonEnum("terminal_reason"), currentStage: text("current_stage").default("planning").notNull(), traceId: text("trace_id"), costCents: integer("cost_cents").default(0).notNull(), diff --git a/src/lib/github/issue-reader.ts b/src/lib/github/issue-reader.ts new file mode 100644 index 0000000..f828761 --- /dev/null +++ b/src/lib/github/issue-reader.ts @@ -0,0 +1,81 @@ +import { App } from "@octokit/app"; + +export type GitHubIssueSnapshot = { + labels: string[]; + state: "open" | "closed"; +}; + +export type GitHubIssueReaderInput = { + installationId: number; + issueNumber: number; + owner: string; + repo: string; +}; + +export type GitHubIssueReader = { + getIssue(input: GitHubIssueReaderInput): Promise; +}; + +type GitHubIssueReaderClient = { + rest: { + issues: { + get(input: { issue_number: number; owner: string; repo: string }): Promise<{ + data: { + labels: Array; + state: string; + }; + }>; + }; + }; +}; + +type GitHubIssueReaderDependencies = { + getInstallationClient?: (installationId: number) => Promise; +}; + +function requiredEnvironmentValue(name: "GITHUB_APP_ID" | "GITHUB_APP_PRIVATE_KEY"): string { + const value = process.env[name]?.trim(); + if (!value) throw new Error(`GitHub App configuration is missing ${name}.`); + return value; +} + +async function getDefaultInstallationClient( + installationId: number, +): Promise { + const app = new App({ + appId: requiredEnvironmentValue("GITHUB_APP_ID"), + privateKey: requiredEnvironmentValue("GITHUB_APP_PRIVATE_KEY").replaceAll("\\n", "\n"), + }); + return (await app.getInstallationOctokit(installationId)) as unknown as GitHubIssueReaderClient; +} + +function normalizeLabels(labels: Array): string[] { + return labels + .map((label) => (typeof label === "string" ? label : label.name)) + .map((label) => label?.trim().toLowerCase()) + .filter((label): label is string => Boolean(label)); +} + +export function createGitHubIssueReader( + dependencies: GitHubIssueReaderDependencies = {}, +): GitHubIssueReader { + const getInstallationClient = dependencies.getInstallationClient ?? getDefaultInstallationClient; + + return { + async getIssue(input) { + const client = await getInstallationClient(input.installationId); + const response = await client.rest.issues.get({ + issue_number: input.issueNumber, + owner: input.owner, + repo: input.repo, + }); + if (response.data.state !== "open" && response.data.state !== "closed") { + throw new Error(`Unsupported GitHub issue state: ${response.data.state}`); + } + return { + labels: normalizeLabels(response.data.labels), + state: response.data.state, + }; + }, + }; +} diff --git a/src/lib/loops/development-run-reconciliation-store.ts b/src/lib/loops/development-run-reconciliation-store.ts new file mode 100644 index 0000000..3fca3c6 --- /dev/null +++ b/src/lib/loops/development-run-reconciliation-store.ts @@ -0,0 +1,140 @@ +import { and, eq } from "drizzle-orm"; + +import type { db } from "@/db/client"; +import { loopRuns, repositories, runSteps } from "@/db/schema"; +import { + finalizeDevelopmentLoopRun, + DevelopmentLoopTransitionError, + type DevelopmentLoopTransitionDatabase, + type DevelopmentLoopTransitionMetrics, +} from "@/lib/loops/development-run-transitions"; +import type { + DevelopmentLoopActiveRun, + DevelopmentLoopExecutionLiveness, + DevelopmentLoopFinalizationSkipped, + DevelopmentLoopRunStore, +} from "@/lib/loops/development-run-reconciliation"; +import type { LoopworksLogger } from "@/lib/observability/logger"; + +export type DevelopmentLoopReconciliationDatabase = Pick; + +function latestActivity( + steps: Array, + currentStage: string, +): Date | null { + const timestamps = steps + .filter((step) => step.stage === currentStage || step.status !== "queued") + .flatMap((step) => + [step.queuedAt, step.startedAt, step.completedAt].filter((value): value is Date => + Boolean(value), + ), + ); + if (timestamps.length === 0) return null; + return new Date(Math.max(...timestamps.map((value) => value.getTime()))); +} + +export function createDevelopmentLoopRunStore(input: { + database: DevelopmentLoopReconciliationDatabase; + executionLiveness: (run: DevelopmentLoopActiveRun) => Promise; + logger?: LoopworksLogger; + metrics?: DevelopmentLoopTransitionMetrics; +}): DevelopmentLoopRunStore { + return { + async listActiveRuns() { + const rows = await input.database + .select({ + currentStage: loopRuns.currentStage, + githubIssueNumber: loopRuns.githubIssueNumber, + installationId: repositories.installationId, + loopKey: loopRuns.loopKey, + repositoryFullName: repositories.fullName, + repositoryName: repositories.name, + repositoryOwner: repositories.owner, + runId: loopRuns.id, + traceId: loopRuns.traceId, + }) + .from(loopRuns) + .innerJoin(repositories, eq(loopRuns.repositoryId, repositories.id)) + .where(and(eq(loopRuns.status, "running"), eq(loopRuns.loopKey, "development-loop"))); + + return Promise.all( + rows.map(async (row) => { + const steps = await input.database + .select() + .from(runSteps) + .where(eq(runSteps.runId, row.runId)); + const currentStep = steps.find((step) => step.stage === row.currentStage); + return { + currentStage: row.currentStage, + ...(currentStep ? { currentStepId: currentStep.id } : {}), + ...(currentStep?.traceId ? { currentStepTraceId: currentStep.traceId } : {}), + githubIssueNumber: row.githubIssueNumber, + installationId: row.installationId, + latestStepActivityAt: latestActivity(steps, row.currentStage), + loopKey: row.loopKey, + repositoryFullName: row.repositoryFullName, + repositoryName: row.repositoryName, + repositoryOwner: row.repositoryOwner, + runId: row.runId, + ...(row.traceId ? { traceId: row.traceId } : {}), + } satisfies DevelopmentLoopActiveRun; + }), + ); + }, + getExecutionLiveness(run) { + return input.executionLiveness(run); + }, + async finalizeRun(finalizeInput) { + if (finalizeInput.expected) { + const [freshRun] = await input.database + .select({ currentStage: loopRuns.currentStage, status: loopRuns.status }) + .from(loopRuns) + .where(eq(loopRuns.id, finalizeInput.runId)) + .limit(1); + const freshSteps = await input.database + .select() + .from(runSteps) + .where(eq(runSteps.runId, finalizeInput.runId)); + const freshCurrentStep = freshSteps.find((step) => step.stage === freshRun?.currentStage); + const freshActivity = freshRun ? latestActivity(freshSteps, freshRun.currentStage) : null; + const expectedActivityMs = finalizeInput.expected.latestStepActivityAt?.getTime() ?? null; + const freshActivityMs = freshActivity?.getTime() ?? null; + if ( + freshRun?.status !== "running" || + freshRun.currentStage !== finalizeInput.expected.currentStage || + freshCurrentStep?.id !== finalizeInput.expected.currentStepId || + freshActivityMs !== expectedActivityMs + ) { + return { + finalized: false, + reason: "state_changed", + runId: finalizeInput.runId, + } satisfies DevelopmentLoopFinalizationSkipped; + } + } + + try { + return await finalizeDevelopmentLoopRun({ + database: input.database as DevelopmentLoopTransitionDatabase, + ...(finalizeInput.expected + ? { expectedCurrentStage: finalizeInput.expected.currentStage } + : {}), + logger: input.logger, + metrics: input.metrics, + occurredAt: finalizeInput.occurredAt, + reason: finalizeInput.reason, + runId: finalizeInput.runId, + }); + } catch (error) { + if (finalizeInput.expected && error instanceof DevelopmentLoopTransitionError) { + return { + finalized: false, + reason: "state_changed", + runId: finalizeInput.runId, + } satisfies DevelopmentLoopFinalizationSkipped; + } + throw error; + } + }, + }; +} diff --git a/src/lib/loops/development-run-reconciliation.ts b/src/lib/loops/development-run-reconciliation.ts new file mode 100644 index 0000000..bebe615 --- /dev/null +++ b/src/lib/loops/development-run-reconciliation.ts @@ -0,0 +1,304 @@ +import type { GitHubIssueReader, GitHubIssueSnapshot } from "@/lib/github/issue-reader"; +import type { LoopManifest } from "../../../schemas/loop-manifest"; +import type { + DevelopmentLoopTerminalReason, + DevelopmentLoopTerminalStatus, +} from "@/lib/loops/development-run-transitions"; +import type { LoopworksLogger } from "@/lib/observability/logger"; +import { + markLoopworksSpanError, + markLoopworksSpanOk, + startDevelopmentLoopReconciliationSpan, +} from "@/lib/observability/trace-context"; + +export type DevelopmentLoopExecutionLiveness = "active" | "inactive" | "unknown"; + +export type DevelopmentLoopActiveRun = { + currentStage: string; + currentStepId?: string; + currentStepTraceId?: string; + githubIssueNumber: number | null; + installationId: number | null; + latestStepActivityAt: Date | null; + loopKey: string; + repositoryFullName: string; + repositoryName: string; + repositoryOwner: string; + runId: string; + traceId?: string; +}; + +export type DevelopmentLoopFinalizationResult = { + durationSeconds: number; + idempotent?: boolean; + reason: DevelopmentLoopTerminalReason; + runId: string; + status: DevelopmentLoopTerminalStatus; + traceId?: string; +}; + +export type DevelopmentLoopFinalizationSkipped = { + finalized: false; + reason: "state_changed"; + runId: string; +}; + +export type DevelopmentLoopRunStore = { + finalizeRun(input: { + expected?: { + currentStage: string; + currentStepId?: string; + latestStepActivityAt: Date | null; + }; + occurredAt: Date; + reason: DevelopmentLoopTerminalReason; + runId: string; + }): Promise; + getExecutionLiveness(run: DevelopmentLoopActiveRun): Promise; + listActiveRuns(): Promise; +}; + +function isSkippedFinalization( + result: DevelopmentLoopFinalizationResult | DevelopmentLoopFinalizationSkipped, +): result is DevelopmentLoopFinalizationSkipped { + return "finalized" in result && result.finalized === false; +} + +export type DevelopmentLoopResolvedCancellationPolicy = { + cause: "loop_disabled" | "issue_closed" | "trigger_removed"; + configuredValue: "skip_new_runs" | "cancel_running" | "mark_canceled" | "continue_existing"; + decision: "cancel" | "continue"; + source: "onDisabled" | "onSuperseded"; +}; + +export type DevelopmentLoopReconciliationOutcome = { + action: + | "finalized" + | "healthy" + | "policy_continued" + | "liveness_unknown" + | "issue_refresh_failed"; + policy?: DevelopmentLoopResolvedCancellationPolicy; + runId: string; + status?: DevelopmentLoopTerminalStatus; + terminalReason?: DevelopmentLoopTerminalReason; + traceId?: string; +}; + +function disabledPolicy( + loop: LoopManifest["loops"][number], +): DevelopmentLoopResolvedCancellationPolicy | undefined { + if (loop.enabled) return undefined; + return { + cause: "loop_disabled", + configuredValue: loop.cancellation.onDisabled, + decision: loop.cancellation.onDisabled === "cancel_running" ? "cancel" : "continue", + source: "onDisabled", + }; +} + +function supersededPolicy( + loop: LoopManifest["loops"][number], + issue: GitHubIssueSnapshot, +): DevelopmentLoopResolvedCancellationPolicy | undefined { + const triggerLabels = loop.triggers.issueLabels.map((label) => label.trim().toLowerCase()); + const issueLabels = new Set(issue.labels.map((label) => label.trim().toLowerCase())); + const cause = + issue.state === "closed" + ? "issue_closed" + : triggerLabels.every((label) => issueLabels.has(label)) + ? undefined + : "trigger_removed"; + if (!cause) return undefined; + return { + cause, + configuredValue: loop.cancellation.onSuperseded, + decision: loop.cancellation.onSuperseded === "mark_canceled" ? "cancel" : "continue", + source: "onSuperseded", + }; +} + +async function finalize(input: { + occurredAt: Date; + policy?: DevelopmentLoopResolvedCancellationPolicy; + reason: DevelopmentLoopTerminalReason; + run: DevelopmentLoopActiveRun; + runStore: DevelopmentLoopRunStore; +}): Promise { + const result = await input.runStore.finalizeRun({ + ...(input.reason === "stalled" || input.reason === "timed_out" + ? { + expected: { + currentStage: input.run.currentStage, + ...(input.run.currentStepId ? { currentStepId: input.run.currentStepId } : {}), + latestStepActivityAt: input.run.latestStepActivityAt, + }, + } + : {}), + occurredAt: input.occurredAt, + reason: input.reason, + runId: input.run.runId, + }); + if (isSkippedFinalization(result)) { + return { + action: "healthy", + ...(input.policy ? { policy: input.policy } : {}), + runId: input.run.runId, + ...(input.run.traceId ? { traceId: input.run.traceId } : {}), + }; + } + return { + action: "finalized", + ...(input.policy ? { policy: input.policy } : {}), + runId: input.run.runId, + status: result.status, + terminalReason: result.reason, + ...((result.traceId ?? input.run.traceId) + ? { traceId: result.traceId ?? input.run.traceId } + : {}), + }; +} + +export async function reconcileDevelopmentLoopRuns(input: { + clock: () => Date; + issueReader: GitHubIssueReader; + logger?: LoopworksLogger; + manifest: LoopManifest; + runStore: DevelopmentLoopRunStore; +}): Promise<{ outcomes: DevelopmentLoopReconciliationOutcome[]; reconciledAt: Date }> { + const reconciliationSpan = startDevelopmentLoopReconciliationSpan(); + const { span } = reconciliationSpan; + const reconciledAt = input.clock(); + + try { + const runs = await input.runStore.listActiveRuns(); + reconciliationSpan.setRunCount(runs.length); + const outcomes: DevelopmentLoopReconciliationOutcome[] = []; + + for (const run of runs) { + const loop = input.manifest.loops.find((candidate) => candidate.key === run.loopKey); + if (!loop) { + outcomes.push({ action: "issue_refresh_failed", runId: run.runId }); + continue; + } + + let policy = disabledPolicy(loop); + if (policy?.decision === "cancel") { + outcomes.push( + await finalize({ + occurredAt: reconciledAt, + policy, + reason: "canceled_by_reconciliation", + run, + runStore: input.runStore, + }), + ); + continue; + } + + let issueRefreshFailed = false; + if (run.installationId === null || run.githubIssueNumber === null) { + issueRefreshFailed = true; + } else { + try { + const issue = await input.issueReader.getIssue({ + installationId: run.installationId, + issueNumber: run.githubIssueNumber, + owner: run.repositoryOwner, + repo: run.repositoryName, + }); + const trackerPolicy = supersededPolicy(loop, issue); + if (trackerPolicy) policy = trackerPolicy; + } catch (error) { + issueRefreshFailed = true; + input.logger?.warn( + { + errorType: error instanceof Error ? error.name : "unknown", + runId: run.runId, + traceId: run.traceId, + }, + "development_loop_reconciliation_issue_refresh_failed", + ); + } + } + + if (policy?.decision === "cancel") { + outcomes.push( + await finalize({ + occurredAt: reconciledAt, + policy, + reason: "canceled_by_reconciliation", + run, + runStore: input.runStore, + }), + ); + continue; + } + + let liveness: DevelopmentLoopExecutionLiveness; + try { + liveness = await input.runStore.getExecutionLiveness(run); + } catch { + liveness = "unknown"; + input.logger?.warn( + { runId: run.runId, traceId: run.traceId }, + "development_loop_reconciliation_liveness_unavailable", + ); + } + if (liveness === "inactive") { + outcomes.push( + await finalize({ + occurredAt: reconciledAt, + ...(policy ? { policy } : {}), + reason: "timed_out", + run, + runStore: input.runStore, + }), + ); + continue; + } + if (liveness === "active" && run.latestStepActivityAt) { + const cutoff = reconciledAt.getTime() - loop.reconciliation.silenceThresholdSeconds * 1000; + if (run.latestStepActivityAt.getTime() < cutoff) { + outcomes.push( + await finalize({ + occurredAt: reconciledAt, + ...(policy ? { policy } : {}), + reason: "stalled", + run, + runStore: input.runStore, + }), + ); + continue; + } + } + + const action = + liveness === "unknown" + ? "liveness_unknown" + : policy?.decision === "continue" + ? "policy_continued" + : issueRefreshFailed + ? "issue_refresh_failed" + : "healthy"; + outcomes.push({ + action, + ...(policy ? { policy } : {}), + runId: run.runId, + ...(run.traceId ? { traceId: run.traceId } : {}), + }); + } + + input.logger?.info( + { outcomeCount: outcomes.length, reconciledAt: reconciledAt.toISOString() }, + "development_loop_runs_reconciled", + ); + markLoopworksSpanOk(span); + return { outcomes, reconciledAt }; + } catch (error) { + markLoopworksSpanError(span, error); + throw error; + } finally { + span.end(); + } +} diff --git a/src/lib/loops/development-run-transitions.ts b/src/lib/loops/development-run-transitions.ts index 3f8e3b7..2f66010 100644 --- a/src/lib/loops/development-run-transitions.ts +++ b/src/lib/loops/development-run-transitions.ts @@ -38,7 +38,7 @@ import { type ValidationReviewResult, validationReviewResultSchema, } from "@agent/validation-review-agent"; -import { and, eq, isNull, sql } from "drizzle-orm"; +import { and, eq, isNull, notInArray, sql } from "drizzle-orm"; import type { db } from "@/db/client"; import { agentPlans, @@ -84,6 +84,8 @@ import { type DevelopmentLoopStepRetryMetricInput, type DevelopmentLoopValidationDurationMetricInput, type DevelopmentLoopValidationOutcomeMetricInput, + developmentLoopRunCompletedEventType, + recordDevelopmentLoopRunCompletedObservability, recordDevelopmentLoopRunCompletedMetric, recordDevelopmentLoopRunDurationMetric, recordDevelopmentLoopStepDurationMetric, @@ -266,6 +268,12 @@ export async function recordDevelopmentLoopPlanArtifact( export type DevelopmentLoopValidationTransitionStatus = "advanced" | "blocked"; export type DevelopmentLoopTerminalStatus = "succeeded" | "failed" | "canceled"; +export type DevelopmentLoopTerminalReason = + | "succeeded" + | "failed" + | "timed_out" + | "stalled" + | "canceled_by_reconciliation"; export type DevelopmentLoopTransitionMetrics = { runCompleted?: (input: DevelopmentLoopRunCompletedMetricInput) => void; @@ -2566,17 +2574,34 @@ export async function executeDevelopmentLoopPrStage( }; } -export async function completeDevelopmentLoopRun(input: { +function terminalStatusForReason( + reason: DevelopmentLoopTerminalReason, +): DevelopmentLoopTerminalStatus { + if (reason === "succeeded") return "succeeded"; + if (reason === "canceled_by_reconciliation") return "canceled"; + return "failed"; +} + +function terminalReasonForStatus( + status: DevelopmentLoopTerminalStatus, +): DevelopmentLoopTerminalReason { + if (status === "succeeded") return "succeeded"; + if (status === "canceled") return "canceled_by_reconciliation"; + return "failed"; +} + +export async function finalizeDevelopmentLoopRun(input: { database: DevelopmentLoopTransitionDatabase; + expectedCurrentStage?: string; logger?: LoopworksLogger; metrics?: DevelopmentLoopTransitionMetrics; occurredAt?: Date; - reason?: string; + reason: DevelopmentLoopTerminalReason; runId: string; - status: DevelopmentLoopTerminalStatus; }): Promise<{ durationSeconds: number; idempotent?: boolean; + reason: DevelopmentLoopTerminalReason; runId: string; status: DevelopmentLoopTerminalStatus; traceId?: string; @@ -2584,7 +2609,7 @@ export async function completeDevelopmentLoopRun(input: { const occurredAt = input.occurredAt ?? new Date(); let runCompletedMetric: DevelopmentLoopRunCompletedMetricInput | undefined; let runDurationMetric: DevelopmentLoopRunDurationMetricInput | undefined; - const terminalReason = normalizeReasonCode(input.reason); + const status = terminalStatusForReason(input.reason); const result = await input.database.transaction(async (tx) => { const [run] = await tx @@ -2596,8 +2621,10 @@ export async function completeDevelopmentLoopRun(input: { metadata: loopRuns.metadata, queuedAt: loopRuns.queuedAt, repository: repositories.fullName, + repositoryId: loopRuns.repositoryId, startedAt: loopRuns.startedAt, status: loopRuns.status, + terminalReason: loopRuns.terminalReason, traceId: loopRuns.traceId, }) .from(loopRuns) @@ -2616,6 +2643,7 @@ export async function completeDevelopmentLoopRun(input: { return { durationSeconds: durationSecondsBetween(run.startedAt ?? run.queuedAt, run.completedAt), idempotent: true, + reason: run.terminalReason ?? terminalReasonForStatus(run.status), runId: input.runId, status: run.status as DevelopmentLoopTerminalStatus, ...(run.traceId ? { traceId: run.traceId } : {}), @@ -2623,35 +2651,95 @@ export async function completeDevelopmentLoopRun(input: { } const durationSeconds = durationSecondsBetween(run.startedAt ?? run.queuedAt, occurredAt); - await tx + const [currentStep] = await tx + .select({ id: runSteps.id, traceId: runSteps.traceId }) + .from(runSteps) + .where(and(eq(runSteps.runId, input.runId), eq(runSteps.stage, run.currentStage))) + .limit(1); + const updatePredicates = [ + eq(loopRuns.id, input.runId), + notInArray(loopRuns.status, ["succeeded", "failed", "canceled"]), + ...(input.expectedCurrentStage + ? [eq(loopRuns.currentStage, input.expectedCurrentStage)] + : []), + ]; + const [updated] = await tx .update(loopRuns) .set({ - ...(input.status === "canceled" ? { canceledAt: occurredAt } : {}), + ...(status === "canceled" ? { canceledAt: occurredAt } : {}), completedAt: occurredAt, - currentStage: input.status === "succeeded" ? "done" : run.currentStage, - metadata: { - ...(run.metadata ?? {}), - ...(terminalReason ? { terminalReason } : {}), - }, - status: input.status, + currentStage: status === "succeeded" ? "done" : run.currentStage, + status, + terminalReason: input.reason, }) - .where(eq(loopRuns.id, input.runId)); + .where(and(...updatePredicates)) + .returning({ id: loopRuns.id }); + + if (!updated) { + const [terminalRun] = await tx + .select({ + completedAt: loopRuns.completedAt, + queuedAt: loopRuns.queuedAt, + startedAt: loopRuns.startedAt, + status: loopRuns.status, + terminalReason: loopRuns.terminalReason, + traceId: loopRuns.traceId, + }) + .from(loopRuns) + .where(eq(loopRuns.id, input.runId)) + .limit(1); + if ( + !terminalRun?.completedAt || + !["succeeded", "failed", "canceled"].includes(terminalRun.status) + ) { + throw new DevelopmentLoopTransitionError( + `Run ${input.runId} could not be finalized because its state changed.`, + ); + } + return { + durationSeconds: durationSecondsBetween( + terminalRun.startedAt ?? terminalRun.queuedAt, + terminalRun.completedAt, + ), + idempotent: true, + reason: + terminalRun.terminalReason ?? + terminalReasonForStatus(terminalRun.status as DevelopmentLoopTerminalStatus), + runId: input.runId, + status: terminalRun.status as DevelopmentLoopTerminalStatus, + ...(terminalRun.traceId ? { traceId: terminalRun.traceId } : {}), + }; + } + + await recordDevelopmentLoopRunCompletedObservability({ + durationSeconds, + loopKey: run.loopKey, + repositoryFullName: run.repository, + repositoryId: run.repositoryId, + runId: input.runId, + status, + stepId: currentStep?.id, + terminalReason: input.reason, + traceId: currentStep?.traceId ?? run.traceId ?? undefined, + writer: tx, + }); runCompletedMetric = { loopKey: run.loopKey, repository: run.repository, - status: input.status, + status, }; runDurationMetric = { durationSeconds, loopKey: run.loopKey, - status: input.status, + status, }; return { durationSeconds, + reason: input.reason, runId: input.runId, - status: input.status, + status, ...(run.traceId ? { traceId: run.traceId } : {}), }; }); @@ -2673,16 +2761,47 @@ export async function completeDevelopmentLoopRun(input: { { durationSeconds: result.durationSeconds, idempotent: "idempotent" in result ? result.idempotent : undefined, + reason: result.reason, runId: result.runId, status: result.status, traceId: "traceId" in result ? result.traceId : undefined, }, - "development_loop_run_completed", + developmentLoopRunCompletedEventType, ); return result; } +export async function completeDevelopmentLoopRun(input: { + database: DevelopmentLoopTransitionDatabase; + logger?: LoopworksLogger; + metrics?: DevelopmentLoopTransitionMetrics; + occurredAt?: Date; + reason?: DevelopmentLoopTerminalReason; + runId: string; + status: DevelopmentLoopTerminalStatus; +}): ReturnType { + if (input.status === "canceled" && input.reason === undefined) { + throw new DevelopmentLoopTransitionError( + "Canceled runs require an explicit typed terminal reason.", + ); + } + const reason = input.reason ?? terminalReasonForStatus(input.status); + if (terminalStatusForReason(reason) !== input.status) { + throw new DevelopmentLoopTransitionError( + `Terminal reason ${reason} does not match status ${input.status}.`, + ); + } + return finalizeDevelopmentLoopRun({ + database: input.database, + logger: input.logger, + metrics: input.metrics, + occurredAt: input.occurredAt, + reason, + runId: input.runId, + }); +} + export async function retryDevelopmentLoopStep(input: { database: DevelopmentLoopTransitionDatabase; logger?: LoopworksLogger; diff --git a/src/lib/loops/manifest.ts b/src/lib/loops/manifest.ts index 118e48a..629552c 100644 --- a/src/lib/loops/manifest.ts +++ b/src/lib/loops/manifest.ts @@ -153,6 +153,9 @@ export const defaultLoopManifest: LoopManifest = loopManifestSchema.parse({ maxInFlight: 1, cancelInProgress: false, }, + reconciliation: { + silenceThresholdSeconds: 300, + }, cancellation: { onSuperseded: "mark_canceled", onDisabled: "skip_new_runs", @@ -260,6 +263,9 @@ export const defaultLoopManifest: LoopManifest = loopManifestSchema.parse({ maxInFlight: 1, cancelInProgress: false, }, + reconciliation: { + silenceThresholdSeconds: 300, + }, cancellation: { onSuperseded: "mark_canceled", onDisabled: "skip_new_runs", @@ -512,6 +518,10 @@ function getManifestValidationHint(path: string): string { return "Set a concurrency group and maxInFlight of at least 1."; } + if (path.includes("reconciliation")) { + return "Set a positive integer silence threshold in seconds for run reconciliation."; + } + if (path.includes("cancellation")) { return "Choose explicit cancellation behavior for superseded and disabled loop states."; } diff --git a/src/lib/observability/metrics.ts b/src/lib/observability/metrics.ts index c026c9a..a7d0292 100644 --- a/src/lib/observability/metrics.ts +++ b/src/lib/observability/metrics.ts @@ -141,6 +141,8 @@ export function resolveObservabilityMetricDefinition( export const developmentLoopRunCreatedEventType = "development_loop_run_created"; export const developmentLoopRunCreatedDurableMetricName = developmentLoopRunCreatedEventType; +export const developmentLoopRunCompletedEventType = "development_loop_run_completed"; +export const developmentLoopRunCompletedDurableMetricName = developmentLoopRunCompletedEventType; export const researchLoopRunCreatedEventType = "research_loop_run_created"; export const researchLoopRunCreatedDurableMetricName = researchLoopRunCreatedEventType; @@ -161,6 +163,11 @@ export const durableObservabilityEventContract = [ metricName: researchLoopRunCreatedDurableMetricName, otelMetricName: "loopworks.run.started", }, + { + eventType: developmentLoopRunCompletedEventType, + metricName: developmentLoopRunCompletedDurableMetricName, + otelMetricName: "loopworks.run.completed", + }, ] as const satisfies readonly DurableObservabilityEventDefinition[]; export type DurableObservabilityEventMetricName = @@ -816,6 +823,42 @@ export async function recordDevelopmentLoopRunCreatedObservability(input: { }; } +export async function recordDevelopmentLoopRunCompletedObservability(input: { + durationSeconds: number; + loopKey: string; + repositoryFullName: string; + repositoryId: string; + runId: string; + status: DevelopmentLoopRunMetricStatus; + stepId?: string; + terminalReason: string; + traceId?: string; + writer: ObservabilityEventsWriter; +}): Promise { + const event = resolveDurableObservabilityEventDefinition( + developmentLoopRunCompletedDurableMetricName, + ); + + await input.writer.insert(observabilityEvents).values({ + eventType: event.eventType, + message: "Development loop run reached a terminal state.", + metricName: event.metricName, + metricValue: 1, + payload: { + durationSeconds: input.durationSeconds, + loopKey: input.loopKey, + repositoryFullName: input.repositoryFullName, + status: normalizeRunMetricStatus(input.status), + terminalReason: input.terminalReason, + }, + repositoryId: input.repositoryId, + runId: input.runId, + severity: input.status === "succeeded" ? "info" : "warn", + stepId: input.stepId, + traceId: input.traceId, + }); +} + export async function recordResearchLoopRunCreatedObservability(input: { artifactCount: number; deliveryId?: string; diff --git a/src/lib/observability/trace-context.ts b/src/lib/observability/trace-context.ts index b55a165..a6a02be 100644 --- a/src/lib/observability/trace-context.ts +++ b/src/lib/observability/trace-context.ts @@ -22,6 +22,19 @@ export function startLoopworksSpan( return tracer.startSpan(name, options); } +export function startDevelopmentLoopReconciliationSpan(tracer = getLoopworksTracer()): { + setRunCount(count: number): void; + span: Span; +} { + const span = startLoopworksSpan("loopworks.run.reconcile", undefined, tracer); + return { + setRunCount(count) { + span.setAttribute("loopworks.run.count", count); + }, + span, + }; +} + export function markLoopworksSpanOk(span: Span): void { span.setStatus({ code: SpanStatusCode.OK }); } diff --git a/tests/unit/db/migrations.test.ts b/tests/unit/db/migrations.test.ts index 5e39d2b..37498e1 100644 --- a/tests/unit/db/migrations.test.ts +++ b/tests/unit/db/migrations.test.ts @@ -7,6 +7,7 @@ import { artifactTypeEnum, loopRuns, repositories, + runTerminalReasonEnum, } from "@/db/schema"; import { createPgliteTestDatabase } from "../../helpers/pglite"; @@ -83,6 +84,21 @@ describe("Drizzle migrations", () => { expect(migrationSql).toContain("'bypassed'"); }); + it("tracks typed run terminal reasons in schema and migrations", () => { + expect(runTerminalReasonEnum.enumValues).toEqual([ + "succeeded", + "failed", + "timed_out", + "stalled", + "canceled_by_reconciliation", + ]); + expect(Object.keys(loopRuns)).toContain("terminalReason"); + + const migrationSql = readMigrationSql(); + expect(migrationSql).toContain('CREATE TYPE "public"."run_terminal_reason"'); + expect(migrationSql).toContain('"terminal_reason" "run_terminal_reason"'); + }); + it( "tracks screenshot artifacts in the schema and generated migrations", async () => { diff --git a/tests/unit/github/issue-reader.test.ts b/tests/unit/github/issue-reader.test.ts new file mode 100644 index 0000000..378850a --- /dev/null +++ b/tests/unit/github/issue-reader.test.ts @@ -0,0 +1,34 @@ +/** @vitest-environment node */ +import { createGitHubIssueReader } from "@/lib/github/issue-reader"; + +describe("GitHub issue reader", () => { + it("reads and normalizes issue state without exposing mutation methods", async () => { + const get = vi.fn(async () => ({ + data: { + labels: [{ name: "Agent-Ready" }, { name: "AREA:LOOPS" }, "Priority:P1"], + state: "open" as const, + }, + })); + const reader = createGitHubIssueReader({ + getInstallationClient: vi.fn(async () => ({ rest: { issues: { get } } })), + }); + + await expect( + reader.getIssue({ + installationId: 95_001, + issueNumber: 95, + owner: "ncolesummers", + repo: "loopworks", + }), + ).resolves.toEqual({ + labels: ["agent-ready", "area:loops", "priority:p1"], + state: "open", + }); + expect(get).toHaveBeenCalledWith({ + issue_number: 95, + owner: "ncolesummers", + repo: "loopworks", + }); + expect(Object.keys(reader)).toEqual(["getIssue"]); + }); +}); diff --git a/tests/unit/loops/development-run-reconciliation-store.integration.test.ts b/tests/unit/loops/development-run-reconciliation-store.integration.test.ts new file mode 100644 index 0000000..b4e6f38 --- /dev/null +++ b/tests/unit/loops/development-run-reconciliation-store.integration.test.ts @@ -0,0 +1,171 @@ +/** @vitest-environment node */ +import { and, eq } from "drizzle-orm"; + +import { loopRuns, observabilityEvents, repositories, runSteps } from "@/db/schema"; +import { + createDevelopmentLoopRun, + type DevelopmentLoopRunDatabase, +} from "@/lib/loops/development-run"; +import { + createDevelopmentLoopRunStore, + type DevelopmentLoopReconciliationDatabase, +} from "@/lib/loops/development-run-reconciliation-store"; +import { createPgliteTestDatabase, type PgliteTestDatabase } from "../../helpers/pglite"; + +const trigger = { + body: "## Acceptance Criteria\n- Reconcile runs deterministically.", + deliveryId: "issue-95-reconcile", + issueNumber: 95, + issueUrl: "https://github.com/ncolesummers/loopworks/issues/95", + labels: ["agent-ready", "area:loops"], + milestone: "M3 Durable Loop MVP", + repositoryFullName: "ncolesummers/loopworks", + title: "Run reconciliation", +}; + +function runDatabase(context: PgliteTestDatabase): DevelopmentLoopRunDatabase { + return context.db as unknown as DevelopmentLoopRunDatabase; +} + +describe("development-loop reconciliation store", () => { + let context: PgliteTestDatabase; + + beforeEach(async () => { + context = await createPgliteTestDatabase(); + }); + + afterEach(async () => { + await context.close(); + }); + + it("lists active runs with latest step activity and delegates finalization", async () => { + const [repository] = await context.db + .insert(repositories) + .values({ + githubRepoId: 95_000_001, + installationId: 95_001, + owner: "ncolesummers", + name: "loopworks", + fullName: "ncolesummers/loopworks", + }) + .returning(); + if (!repository) throw new Error("Expected repository fixture."); + const created = await createDevelopmentLoopRun({ + database: runDatabase(context), + now: () => new Date("2026-07-22T16:00:00.000Z"), + traceId: "4bf92f3577b34da6a3ce929d0e0e4736", + trigger, + }); + if (created.mode !== "created") throw new Error("Expected a created run."); + await context.db + .update(loopRuns) + .set({ status: "running" }) + .where(eq(loopRuns.id, created.runId)); + await context.db.insert(loopRuns).values({ + githubIssueNumber: 43, + loopKey: "research-loop", + repositoryId: repository.id, + status: "running", + }); + + const metrics = { runCompleted: vi.fn(), runDuration: vi.fn() }; + const store = createDevelopmentLoopRunStore({ + database: context.db as unknown as DevelopmentLoopReconciliationDatabase, + executionLiveness: async () => "active", + metrics, + }); + const planning = await store.listActiveRuns(); + + expect(planning).toEqual([ + expect.objectContaining({ + currentStage: "planning", + latestStepActivityAt: new Date("2026-07-22T16:00:00.000Z"), + runId: created.runId, + }), + ]); + await context.db + .update(loopRuns) + .set({ currentStage: "development" }) + .where(eq(loopRuns.id, created.runId)); + await context.db + .update(runSteps) + .set({ startedAt: new Date("2026-07-22T16:07:00.000Z") }) + .where(and(eq(runSteps.runId, created.runId), eq(runSteps.stage, "development"))); + const active = await store.listActiveRuns(); + + expect(active).toEqual([ + expect.objectContaining({ + currentStepId: expect.any(String), + installationId: 95_001, + latestStepActivityAt: new Date("2026-07-22T16:07:00.000Z"), + runId: created.runId, + traceId: "4bf92f3577b34da6a3ce929d0e0e4736", + }), + ]); + await store.finalizeRun({ + occurredAt: new Date("2026-07-22T16:10:00.000Z"), + reason: "stalled", + runId: created.runId, + }); + + const [run] = await context.db.select().from(loopRuns).where(eq(loopRuns.id, created.runId)); + const completionEvents = await context.db + .select() + .from(observabilityEvents) + .where(eq(observabilityEvents.eventType, "development_loop_run_completed")); + expect(run).toMatchObject({ status: "failed", terminalReason: "stalled" }); + expect(completionEvents).toHaveLength(1); + expect(metrics.runCompleted).toHaveBeenCalledTimes(1); + expect(await store.listActiveRuns()).toEqual([]); + }); + + it("rejects finalization when current-step activity changed after the snapshot", async () => { + await context.db.insert(repositories).values({ + githubRepoId: 95_000_002, + installationId: 95_002, + owner: "ncolesummers", + name: "loopworks-race", + fullName: "ncolesummers/loopworks-race", + }); + const created = await createDevelopmentLoopRun({ + database: runDatabase(context), + now: () => new Date("2026-07-22T16:00:00.000Z"), + traceId: "4bf92f3577b34da6a3ce929d0e0e4736", + trigger: { + ...trigger, + deliveryId: "issue-95-race", + repositoryFullName: "ncolesummers/loopworks-race", + }, + }); + if (created.mode !== "created") throw new Error("Expected a created run."); + await context.db + .update(loopRuns) + .set({ status: "running" }) + .where(eq(loopRuns.id, created.runId)); + const store = createDevelopmentLoopRunStore({ + database: context.db as unknown as DevelopmentLoopReconciliationDatabase, + executionLiveness: async () => "active", + }); + const [snapshot] = await store.listActiveRuns(); + if (!snapshot) throw new Error("Expected active run snapshot."); + await context.db + .update(runSteps) + .set({ startedAt: new Date("2026-07-22T16:09:00.000Z") }) + .where(and(eq(runSteps.runId, created.runId), eq(runSteps.stage, "planning"))); + + await expect( + store.finalizeRun({ + expected: { + currentStage: snapshot.currentStage, + currentStepId: snapshot.currentStepId, + latestStepActivityAt: snapshot.latestStepActivityAt, + }, + occurredAt: new Date("2026-07-22T16:10:00.000Z"), + reason: "stalled", + runId: created.runId, + }), + ).resolves.toEqual({ finalized: false, reason: "state_changed", runId: created.runId }); + const [run] = await context.db.select().from(loopRuns).where(eq(loopRuns.id, created.runId)); + expect(run).toMatchObject({ status: "running", terminalReason: null }); + }); +}); diff --git a/tests/unit/loops/development-run-reconciliation.test.ts b/tests/unit/loops/development-run-reconciliation.test.ts new file mode 100644 index 0000000..9ae4d17 --- /dev/null +++ b/tests/unit/loops/development-run-reconciliation.test.ts @@ -0,0 +1,320 @@ +/** @vitest-environment node */ +import { defaultLoopManifest } from "@/lib/loops/manifest"; +import { + type DevelopmentLoopActiveRun, + type DevelopmentLoopRunStore, + reconcileDevelopmentLoopRuns, +} from "@/lib/loops/development-run-reconciliation"; + +const now = new Date("2026-07-22T16:10:00.000Z"); + +function activeRun(overrides: Partial = {}): DevelopmentLoopActiveRun { + return { + currentStage: "development", + currentStepId: "step-95", + githubIssueNumber: 95, + installationId: 95001, + latestStepActivityAt: new Date("2026-07-22T16:09:00.000Z"), + loopKey: "development-loop", + repositoryFullName: "ncolesummers/loopworks", + repositoryName: "loopworks", + repositoryOwner: "ncolesummers", + runId: "run-95", + traceId: "4bf92f3577b34da6a3ce929d0e0e4736", + ...overrides, + }; +} + +function manifest(overrides?: { + enabled?: boolean; + onDisabled?: "skip_new_runs" | "cancel_running"; + onSuperseded?: "mark_canceled" | "continue_existing"; + silenceThresholdSeconds?: number; +}) { + return { + ...defaultLoopManifest, + loops: defaultLoopManifest.loops.map((loop) => + loop.key === "development-loop" + ? { + ...loop, + enabled: overrides?.enabled ?? loop.enabled, + cancellation: { + ...loop.cancellation, + onDisabled: overrides?.onDisabled ?? loop.cancellation.onDisabled, + onSuperseded: overrides?.onSuperseded ?? loop.cancellation.onSuperseded, + }, + reconciliation: { + silenceThresholdSeconds: + overrides?.silenceThresholdSeconds ?? loop.reconciliation.silenceThresholdSeconds, + }, + } + : loop, + ), + }; +} + +function store(input: { + liveness?: "active" | "inactive" | "unknown"; + runs?: DevelopmentLoopActiveRun[]; +}) { + const finalizeRun = vi.fn(async ({ reason, runId }) => ({ + durationSeconds: 60, + reason, + runId, + status: + reason === "canceled_by_reconciliation" + ? ("canceled" as const) + : reason === "succeeded" + ? ("succeeded" as const) + : ("failed" as const), + })); + const value: DevelopmentLoopRunStore = { + finalizeRun, + getExecutionLiveness: vi.fn(async () => input.liveness ?? "active"), + listActiveRuns: vi.fn(async () => input.runs ?? [activeRun()]), + }; + return { finalizeRun, value }; +} + +function issueReader(input?: { labels?: string[]; state?: "open" | "closed"; throws?: boolean }) { + return { + getIssue: vi.fn(async () => { + if (input?.throws) throw new Error("github_unavailable"); + return { + labels: input?.labels ?? ["agent-ready", "area:loops"], + state: input?.state ?? ("open" as const), + }; + }), + }; +} + +describe("development-loop run reconciliation", () => { + it("leaves a healthy active run untouched and keeps the threshold boundary healthy", async () => { + const runStore = store({ + runs: [activeRun({ latestStepActivityAt: new Date("2026-07-22T16:05:00.000Z") })], + }); + + const result = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader(), + manifest: manifest(), + runStore: runStore.value, + }); + + expect(result.outcomes).toEqual([ + expect.objectContaining({ action: "healthy", runId: "run-95" }), + ]); + expect(runStore.finalizeRun).not.toHaveBeenCalled(); + }); + + it("changes stall detection when the manifest threshold changes", async () => { + const run = activeRun({ latestStepActivityAt: new Date("2026-07-22T16:08:00.000Z") }); + const shortStore = store({ runs: [run] }); + const longStore = store({ runs: [run] }); + + const short = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader(), + manifest: manifest({ silenceThresholdSeconds: 60 }), + runStore: shortStore.value, + }); + const long = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader(), + manifest: manifest({ silenceThresholdSeconds: 300 }), + runStore: longStore.value, + }); + + expect(short.outcomes).toEqual([ + expect.objectContaining({ action: "finalized", terminalReason: "stalled" }), + ]); + expect(shortStore.finalizeRun).toHaveBeenCalledWith( + expect.objectContaining({ + occurredAt: now, + reason: "stalled", + runId: "run-95", + }), + ); + expect(long.outcomes).toEqual([expect.objectContaining({ action: "healthy" })]); + }); + + it("terminates an orphaned run as timed out", async () => { + const runStore = store({ liveness: "inactive" }); + + const result = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader(), + manifest: manifest(), + runStore: runStore.value, + }); + + expect(result.outcomes).toEqual([ + expect.objectContaining({ action: "finalized", terminalReason: "timed_out" }), + ]); + }); + + it("fails open when execution liveness is unknown", async () => { + const runStore = store({ liveness: "unknown" }); + + const result = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader(), + manifest: manifest(), + runStore: runStore.value, + }); + + expect(result.outcomes).toEqual([ + expect.objectContaining({ action: "liveness_unknown", runId: "run-95" }), + ]); + expect(runStore.finalizeRun).not.toHaveBeenCalled(); + }); + + it("isolates a throwing liveness provider and continues reconciling later runs", async () => { + const runs = [activeRun({ runId: "first" }), activeRun({ runId: "second" })]; + const runStore: DevelopmentLoopRunStore = { + finalizeRun: vi.fn(), + getExecutionLiveness: vi + .fn() + .mockRejectedValueOnce(new Error("liveness unavailable")) + .mockResolvedValueOnce("active"), + listActiveRuns: vi.fn(async () => runs), + }; + + const result = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader(), + manifest: manifest(), + runStore, + }); + + expect(result.outcomes).toEqual([ + expect.objectContaining({ action: "liveness_unknown", runId: "first" }), + expect.objectContaining({ action: "healthy", runId: "second" }), + ]); + }); + + it.each([ + { labels: ["agent-ready"], name: "closed issue", state: "closed" as const }, + { labels: ["area:loops"], name: "removed trigger label", state: "open" as const }, + ])("cancels a run for a $name", async ({ labels, state }) => { + const runStore = store({ liveness: "inactive" }); + + const result = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader({ labels, state }), + manifest: manifest({ onSuperseded: "mark_canceled" }), + runStore: runStore.value, + }); + + expect(result.outcomes).toEqual([ + expect.objectContaining({ + action: "finalized", + policy: expect.objectContaining({ configuredValue: "mark_canceled" }), + terminalReason: "canceled_by_reconciliation", + }), + ]); + }); + + it("honors continue_existing for superseded work", async () => { + const runStore = store({}); + + const result = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader({ state: "closed" }), + manifest: manifest({ onSuperseded: "continue_existing" }), + runStore: runStore.value, + }); + + expect(result.outcomes).toEqual([ + expect.objectContaining({ + action: "policy_continued", + policy: expect.objectContaining({ configuredValue: "continue_existing" }), + }), + ]); + expect(runStore.finalizeRun).not.toHaveBeenCalled(); + }); + + it("surfaces continue_existing when local timeout evidence still finalizes the run", async () => { + const runStore = store({ liveness: "inactive" }); + + const result = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader({ state: "closed" }), + manifest: manifest({ onSuperseded: "continue_existing" }), + runStore: runStore.value, + }); + + expect(result.outcomes).toEqual([ + expect.objectContaining({ + action: "finalized", + policy: expect.objectContaining({ configuredValue: "continue_existing" }), + terminalReason: "timed_out", + }), + ]); + }); + + it("keeps a run active when the store rejects stale finalization evidence", async () => { + const runStore = store({ runs: [activeRun({ latestStepActivityAt: new Date(0) })] }); + runStore.finalizeRun.mockResolvedValueOnce({ + finalized: false, + reason: "state_changed", + runId: "run-95", + } as never); + + const result = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader(), + manifest: manifest(), + runStore: runStore.value, + }); + + expect(result.outcomes).toEqual([ + expect.objectContaining({ action: "healthy", runId: "run-95" }), + ]); + }); + + it.each([ + { action: "finalized", onDisabled: "cancel_running" as const }, + { action: "policy_continued", onDisabled: "skip_new_runs" as const }, + ])("honors disabled-loop policy $onDisabled", async ({ action, onDisabled }) => { + const runStore = store({}); + + const result = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: issueReader(), + manifest: manifest({ enabled: false, onDisabled }), + runStore: runStore.value, + }); + + expect(result.outcomes).toEqual([ + expect.objectContaining({ + action, + policy: expect.objectContaining({ configuredValue: onDisabled }), + }), + ]); + }); + + it("keeps tracker failures from canceling runs but still applies local timeout evidence", async () => { + const healthyStore = store({}); + const orphanStore = store({ liveness: "inactive" }); + const failingReader = issueReader({ throws: true }); + + const healthy = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: failingReader, + manifest: manifest(), + runStore: healthyStore.value, + }); + const orphan = await reconcileDevelopmentLoopRuns({ + clock: () => now, + issueReader: failingReader, + manifest: manifest(), + runStore: orphanStore.value, + }); + + expect(healthy.outcomes).toEqual([expect.objectContaining({ action: "issue_refresh_failed" })]); + expect(orphan.outcomes).toEqual([ + expect.objectContaining({ action: "finalized", terminalReason: "timed_out" }), + ]); + }); +}); diff --git a/tests/unit/loops/development-run-transitions.test.ts b/tests/unit/loops/development-run-transitions.test.ts index 4aa9519..3c4ced8 100644 --- a/tests/unit/loops/development-run-transitions.test.ts +++ b/tests/unit/loops/development-run-transitions.test.ts @@ -1,7 +1,7 @@ /** @vitest-environment node */ import { and, eq } from "drizzle-orm"; -import { artifacts, loopRuns, repositories, runSteps } from "@/db/schema"; +import { artifacts, loopRuns, observabilityEvents, repositories, runSteps } from "@/db/schema"; import { createDevelopmentLoopRun, type DevelopmentLoopRunDatabase, @@ -10,6 +10,7 @@ import { applyDevelopmentLoopValidationReport, completeDevelopmentLoopRun, type DevelopmentLoopTransitionDatabase, + finalizeDevelopmentLoopRun, retryDevelopmentLoopStep, } from "@/lib/loops/development-run-transitions"; import { @@ -605,6 +606,7 @@ describe("development-loop run transitions", () => { database: transitionDatabase(context), metrics, occurredAt: new Date("2026-07-08T16:10:00.000Z"), + ...(status === "canceled" ? { reason: "canceled_by_reconciliation" as const } : {}), runId, status, }); @@ -612,6 +614,7 @@ describe("development-loop run transitions", () => { database: transitionDatabase(context), metrics, occurredAt: new Date("2026-07-08T16:10:00.000Z"), + ...(status === "canceled" ? { reason: "canceled_by_reconciliation" as const } : {}), runId, status, }); @@ -640,6 +643,110 @@ describe("development-loop run transitions", () => { expect(expectedMetricStatus).toBe(status === "canceled" ? "cancelled" : status); }); + it("persists a typed terminal reason and emits completion observability once", async () => { + const runId = await createRun(context); + const metrics = createMetricRecorder(); + await context.db + .update(loopRuns) + .set({ + currentStage: "development", + startedAt: new Date("2026-07-08T16:00:00.000Z"), + status: "running", + }) + .where(eq(loopRuns.id, runId)); + + const first = await finalizeDevelopmentLoopRun({ + database: transitionDatabase(context), + metrics, + occurredAt: new Date("2026-07-08T16:10:00.000Z"), + reason: "stalled", + runId, + }); + const replay = await finalizeDevelopmentLoopRun({ + database: transitionDatabase(context), + metrics, + occurredAt: new Date("2026-07-08T16:11:00.000Z"), + reason: "timed_out", + runId, + }); + + const [run] = await context.db.select().from(loopRuns).where(eq(loopRuns.id, runId)); + const events = await context.db + .select() + .from(observabilityEvents) + .where(eq(observabilityEvents.runId, runId)); + + expect(first).toMatchObject({ reason: "stalled", status: "failed" }); + expect(replay).toMatchObject({ idempotent: true, reason: "stalled", status: "failed" }); + expect(run).toMatchObject({ + completedAt: new Date("2026-07-08T16:10:00.000Z"), + status: "failed", + terminalReason: "stalled", + traceId: "4bf92f3577b34da6a3ce929d0e0e4736", + }); + expect(events).toHaveLength(2); + expect(events[1]).toMatchObject({ + eventType: "development_loop_run_completed", + metricName: "development_loop_run_completed", + metricValue: 1, + payload: expect.objectContaining({ terminalReason: "stalled" }), + traceId: "4bf92f3577b34da6a3ce929d0e0e4736", + }); + expect(metrics.runCompleted).toHaveBeenCalledTimes(1); + expect(metrics.runDuration).toHaveBeenCalledTimes(1); + }); + + it("allows exactly one concurrent finalizer to own completion observability", async () => { + const runId = await createRun(context); + const metrics = createMetricRecorder(); + await context.db + .update(loopRuns) + .set({ startedAt: new Date("2026-07-08T16:00:00.000Z"), status: "running" }) + .where(eq(loopRuns.id, runId)); + + const results = await Promise.all([ + finalizeDevelopmentLoopRun({ + database: transitionDatabase(context), + metrics, + occurredAt: new Date("2026-07-08T16:10:00.000Z"), + reason: "stalled", + runId, + }), + finalizeDevelopmentLoopRun({ + database: transitionDatabase(context), + metrics, + occurredAt: new Date("2026-07-08T16:10:00.000Z"), + reason: "timed_out", + runId, + }), + ]); + const completionEvents = await context.db + .select() + .from(observabilityEvents) + .where(eq(observabilityEvents.eventType, "development_loop_run_completed")); + + expect(results.filter((result) => result.idempotent)).toHaveLength(1); + expect(completionEvents).toHaveLength(1); + expect(metrics.runCompleted).toHaveBeenCalledTimes(1); + expect(metrics.runDuration).toHaveBeenCalledTimes(1); + }); + + it("preserves a typed compatibility-wrapper reason", async () => { + const runId = await createRun(context); + await context.db.update(loopRuns).set({ status: "running" }).where(eq(loopRuns.id, runId)); + + await completeDevelopmentLoopRun({ + database: transitionDatabase(context), + occurredAt: new Date("2026-07-08T16:10:00.000Z"), + reason: "stalled", + runId, + status: "failed", + }); + + const [run] = await context.db.select().from(loopRuns).where(eq(loopRuns.id, runId)); + expect(run).toMatchObject({ status: "failed", terminalReason: "stalled" }); + }); + it("queues a retry for an implemented transition branch and records retry telemetry", async () => { const runId = await createRun(context); const metrics = createMetricRecorder(); diff --git a/tests/unit/loops/manifest.test.ts b/tests/unit/loops/manifest.test.ts index a4c3cfc..a44abe8 100644 --- a/tests/unit/loops/manifest.test.ts +++ b/tests/unit/loops/manifest.test.ts @@ -4,7 +4,7 @@ import loopManifestJsonSchema from "../../../schemas/loop-manifest.schema.json"; type JsonSchemaObject = { required?: string[]; - properties?: Record; + properties?: Record; $defs?: Record; }; @@ -138,6 +138,51 @@ describe("loop manifest schema", () => { group: "repo:{repo}:loop:development", maxInFlight: 1, }); + expect(developmentLoop.reconciliation).toEqual({ + silenceThresholdSeconds: 300, + }); + }); + + it("requires a positive reconciliation silence threshold for every loop", () => { + const developmentLoop = defaultLoopManifest.loops[0]; + const withoutReconciliation = { ...developmentLoop } as Record; + delete withoutReconciliation.reconciliation; + + const missing = validateLoopManifest({ + ...defaultLoopManifest, + loops: [withoutReconciliation], + }); + const zero = validateLoopManifest({ + ...defaultLoopManifest, + loops: [ + { + ...developmentLoop, + reconciliation: { silenceThresholdSeconds: 0 }, + }, + ], + }); + const fractional = validateLoopManifest({ + ...defaultLoopManifest, + loops: [ + { + ...developmentLoop, + reconciliation: { silenceThresholdSeconds: 0.5 }, + }, + ], + }); + + for (const result of [missing, zero, fractional]) { + expect(result.success).toBe(false); + if (result.success) throw new Error("Expected reconciliation validation to fail."); + expect(result.errors).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + path: expect.stringContaining("reconciliation"), + hint: expect.stringContaining("silence threshold"), + }), + ]), + ); + } }); it("returns actionable validation errors with paths and hints", () => { @@ -227,6 +272,8 @@ describe("loop manifest schema", () => { expect(definitions.modelPolicy?.required).toEqual(["defaultModel"]); expect(definitions.toolPolicy?.properties).not.toHaveProperty("forbiddenToolCategories"); expect(definitions.approvalGate?.properties?.required).toEqual({ const: true }); + expect(definitions.loopDefinition?.required).toContain("reconciliation"); + expect(definitions.reconciliation?.required).toEqual(["silenceThresholdSeconds"]); }); it("maps each MVP milestone and issue to persona-derived acceptance ids", () => { diff --git a/tests/unit/observability/metrics.test.ts b/tests/unit/observability/metrics.test.ts index be9cea0..88aad13 100644 --- a/tests/unit/observability/metrics.test.ts +++ b/tests/unit/observability/metrics.test.ts @@ -6,10 +6,12 @@ import ts from "typescript"; import { approvals, loopRuns, observabilityEvents, repositories } from "@/db/schema"; import { collectControlPlaneGaugeMeasurements, + developmentLoopRunCompletedDurableMetricName, developmentLoopRunCreatedDurableMetricName, observabilityMetricNames, recordApprovalWaitTimeMetric, recordDevelopmentLoopRunCompletedMetric, + recordDevelopmentLoopRunCompletedObservability, recordDevelopmentLoopRunCreatedObservability, recordDevelopmentLoopRunDurationMetric, recordDevelopmentLoopRunStartedMetric, @@ -549,6 +551,13 @@ describe("ADR 0012 observability metric contract", () => { metricName: "research_loop_run_created", otelMetricName: "loopworks.run.started", }); + expect( + resolveDurableObservabilityEventDefinition(developmentLoopRunCompletedDurableMetricName), + ).toMatchObject({ + eventType: "development_loop_run_completed", + metricName: "development_loop_run_completed", + otelMetricName: "loopworks.run.completed", + }); expect(() => resolveDurableObservabilityEventDefinition("development_loop_run_started"), @@ -708,6 +717,48 @@ describe("ADR 0012 observability metric contract", () => { ]); }); + it("records a correlated development-loop completion event through one helper", async () => { + const insertedRows: Record[] = []; + const writer = { + insert(table: unknown) { + expect(table).toBe(observabilityEvents); + return { + values(row: Record) { + insertedRows.push(row); + return Promise.resolve(); + }, + }; + }, + }; + + await recordDevelopmentLoopRunCompletedObservability({ + durationSeconds: 90, + loopKey: "development-loop", + repositoryFullName: "ncolesummers/loopworks", + repositoryId: "64f8ca7a-1b5d-4b3f-8c5e-2e2d814a18aa", + runId: "9a8d379f-1d65-4fb0-bd91-4f82306a3159", + status: "failed", + stepId: "62da31c0-cb98-4fe1-bdad-25295c03cf67", + terminalReason: "stalled", + traceId: "4bf92f3577b34da6a3ce929d0e0e4736", + writer, + }); + + expect(insertedRows).toEqual([ + expect.objectContaining({ + eventType: "development_loop_run_completed", + metricName: "development_loop_run_completed", + metricValue: 1, + payload: expect.objectContaining({ + status: "failed", + terminalReason: "stalled", + }), + stepId: "62da31c0-cb98-4fe1-bdad-25295c03cf67", + traceId: "4bf92f3577b34da6a3ce929d0e0e4736", + }), + ]); + }); + it("keeps the durable event insert authoritative when OTel metric emission fails", async () => { const insertedRows: Record[] = []; const writer = { @@ -760,6 +811,7 @@ describe("ADR 0012 observability metric contract", () => { /^loopworks\.(run|step|validation|webhook|deployment|approval|queue|lock|model)\./; const forbiddenMetricLiterals = new Set([ ...observabilityMetricNames, + developmentLoopRunCompletedDurableMetricName, developmentLoopRunCreatedDurableMetricName, researchLoopRunCreatedDurableMetricName, ]);