Skip to content

feat: stall watchdog — detect frozen-counter + prompt-stall sessions, nudge or alert #102

Description

@jakecelentano

Summary

Add a frozen-counter watchdog for dispatched deepseek sessions: detect sessions whose token counters haven't advanced in N minutes while not at a prompt/parked state, and auto-nudge (or surface to the dispatcher via the dispatch_failed/SLO alert path from tms#88).

Evidence

2026-07-19: 4 manual nudges required across 8 dispatched sessions (feat-tms#94 twice, feat-openagent#55, feat-pi-dotfiles#41). Signature is unmistakable: identical ↑/↓/R counters + $ cost across two polls ≥5 min apart, no AGENT-STATE marker, no interactive prompt visible. A nudge message revives the session every time (documented DeepSeek-v4-pro behavior in fleet memory).

Scope

  • lib/tms/ (new stall_watchdog.py or extend dispatch_monitor): poll live aoe sessions (tmux capture or aoe status), track per-session counter deltas between runs, classify: frozen (counters static + not marker-parked + no Allow? prompt) → nudge via aoe send (or Telegram alert if nudge fails twice).
  • Also detect the interactive-prompt stall (Allow? visible ≥N min) as a distinct class — alert, do NOT auto-send-keys (pd#40 class needs human or the approval endpoint, oa#52).
  • Cron wiring alongside the existing pollers; dry-run default like scan-reviews.

Out of scope

  • Root-causing the DeepSeek freeze itself (provider-side).
  • Auto-send-keys on prompts (deliberately human/endpoint territory).
  • Session kill/restart policy.

Acceptance criteria

  • Frozen session detected within 2 poll cycles and nudged, with a tms_review event written
  • Prompt-stall class detected and alerted (not auto-answered)
  • False-positive guard: marker-parked sessions (PLAN-REVIEW/MERGE-READY) never nudged
  • Tests: classification matrix (frozen/parked/prompt/active)

References

  • Jul-19 dogfood waves (4 manual nudges), fleet memory: deepseek silent stalls
  • tms#56 (stale-marker watchdog pattern), tms#88 (SLO alert path), pd#40, oa#52

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:dispatchtmq dispatch + worktree layerenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions