fix(sessions): present leaked coven-run 'running' rows as orphaned (cave-vor6)#3749
Merged
Conversation
…ave-vor6)
The Running-processes popover advertised sessions with no process behind
them: `coven run`-registered rows (board enrich-steps, workflows, chat
one-shots) stay 'running' forever when the CLI dies without reporting —
request abort SIGTERMs it, or the app/server quits mid-run. The daemon
only reconciles at its own restart; kill returns session_not_live,
sacrifice refuses ('still running'), and there is no HTTP delete route.
Clicking such a ghost opened a chat with no transcript ('Chat history
unavailable') and no live process to interact with. Live audit found 7 of
9 running-tone rows were ghosts.
Add a stale-running presentation sweep on the sessions/list read:
running-tone daemon rows stale past 30min on both timestamps, with no
in-process chat run and no process I/O events (output/input) in the
daemon event log, are presented as 'orphaned' — the daemon's own restart
verdict. The events probe is read-only and fails open; daemon-spawned
PTY children always emit output events, so genuinely-live sessions are
never touched. Verdicts cache per (id, updated_at) so the polled list
doesn't re-probe, and a daemon-side transition invalidates naturally.
Verified against the live daemon: classifies exactly the 7 dead ghosts,
spares 2 live PTY sessions and a fresh enrich run.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a read-only, best-effort “stale running” sweep on the sessions list path to stop leaked coven run daemon rows (stuck in running tone with no real process) from being presented as live sessions, by probing the daemon’s events log and presenting confirmed ghosts as orphaned before downstream merges/tones.
Changes:
- Introduces
stale-running-sweep.tsto identify stale running-tone daemon rows and classify “ghosts” via a read-only events probe with per-(id, updated_at) caching. - Wires the sweep into
src/app/api/sessions/list/route.tsso ghost rows are presented asorphanedprior tomergeSessionRows/ archive sweeps. - Adds a dedicated unit test file and registers it in the test runner suite + alias loader.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/lib/server/stale-running-sweep.ts | Implements candidate filtering, events-based ghost verdicting, caching, and status presentation rewrite. |
| src/lib/server/stale-running-sweep.test.ts | Adds unit coverage for candidate selection, verdict parsing, caching behavior, and route wiring pins. |
| src/app/api/sessions/list/route.ts | Applies the sweep on the daemon session list response before merging with local sessions. |
| scripts/run-tests.mjs | Registers the new server test in the appropriate suite and alias-loader list. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+117
to
+123
| async function probeSessionForGhost(sessionId: string): Promise<GhostVerdict> { | ||
| const res = await callDaemon<unknown>({ | ||
| path: `/api/v1/sessions/${encodeURIComponent(sessionId)}/events?limit=${EVENTS_PROBE_LIMIT}`, | ||
| }); | ||
| if (!res.ok) return null; | ||
| return ghostVerdictFromEventsResponse(res.data); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Autopilot objective: identify why some running processes open to non-existent chat sessions where you can't interact with a valid running process.
Root cause (live-audited, 3 compounding layers):
runningrows —coven run-registered sessions (board enrich-steps, workflows, chat one-shots) register a daemon row, but the CLI owns the harness process and reports terminal status itself. If that CLI dies without reporting (request abort → SIGTERM, or app/server quit mid-run), the row is stuckrunningforever: the daemon only reconciles at its own restart (marking rowsorphaned),killreturnssession_not_live,sacrificerefuses ("still running"), and there is no HTTP delete route. Live audit: 7 of 9 running-tone rows had no process behind them.What
Add
src/lib/server/stale-running-sweep.ts, applied on the sessions/list read (same piggyback pattern as the auto-archive sweeps): running-tone daemon rows stale ≥30 min on both timestamps, with no in-process chat run and no process I/O events in the daemon event log, are presented asorphaned— converging with the daemon's own restart verdict. They drop out of the Running-processes popover and show failed-tone everywhere else.Safety properties:
GET /api/v1/sessions/{id}/events; nothing is mutated daemon-side.outputevents; a metadata-only log (e.g. the real-worldpatch_metadatacodex leak) is still a ghost.(id, updated_at); a slow run that later reports completion transitions normally.Verification
node --experimental-strip-types --import ./scripts/test-alias-register.mjs src/lib/server/stale-running-sweep.test.ts✅pnpm run check:tests-wired✅ ·tsc --noEmit✅ · eslint touched files ✅Related beads