cc-watchdog: dead-man's switch against silent agent-session die-outs

.github/workflows/ci.yml

@@ -40,3 +40,18 @@ jobs:
         # Scan the working tree (not full history): a "don't add secrets" gate,
         # free of false positives from large base64 blobs in old commits.
         run: gitleaks dir --config .gitleaks.toml --redact --verbose .
+
+  watchdog:
+    name: cc-watchdog (shellcheck + unit tests)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: shellcheck
+        run: |
+          shellcheck onboarding/watchdog/cc-deadman \
+                     onboarding/watchdog/cc-stall-watchdog.sh \
+                     onboarding/watchdog/install.sh \
+                     onboarding/watchdog/test/run_tests.sh \
+                     onboarding/watchdog/test/shims/*
+      - name: unit tests
+        run: onboarding/watchdog/test/run_tests.sh

.gitignore

@@ -56,3 +56,5 @@ timer.dat
 # .vscode
 .vscode
 
+# Superpowers working files (brainstorming specs, plans) — local-only, never tracked
+docs/superpowers/

onboarding/CLAUDE.global.md

@@ -76,3 +76,18 @@ Before building on a premise — what a field actually contains at the read site
 Treat unverified premises as likely-wrong, not likely-right. Across long arcs of load-bearing checks the probes invert the premise more often than they confirm it (the moments weren't end-of-run; the "healthy" channel was dead; the MW didn't survive the data boundary; the core had zero reactions where the counts implied dozens; the "reference-state-paired" reaction wasn't, in the guard's eyes). A plan built on an unverified premise produces correct implementations of wrong designs — the most expensive failure class, because every downstream review validates against the same false premise.
 
 Corollary — don't build to pass the check. When a guard / test / tripwire refuses your construct, do not reshape the construct to slip under it; a thing built only to make a check go green validates nothing that transfers. Ask first whether the guard caught a real defect (then fix the defect) or exposed a false-positive in itself (then record it as a finding) — not how to get past it.
+
+# Silent-stall prevention (cc-watchdog contract)
+
+Long autonomous sessions must never wait indefinitely on work that could die
+silently:
+
+- Run final gates (full test suites, builds) in the mother session as tracked
+  background tasks — never owned by a subagent that will stop before the work
+  completes.
+- Before ending any turn while background/external work is still pending,
+  declare a dead-man deadline: `cc-deadman set <expected minutes + 50% margin>
+  "<reason>"`.
+- First action after resuming from a wait: `cc-deadman clear` (or re-arm for
+  the next wait).
+- Wrap any command without a natural bound in `timeout`.

onboarding/MAINTAINING.md

@@ -47,6 +47,7 @@ not by copying secrets or by being granted access to the PI's machines.
 | Statusline | the **simple context-% script**, not the PI's `~/agents` auto-handoff variant | Self-contained; no extra infra to stand up. |
 | Seeds | **sanitized**, real files under `vault-seeds/` | Easy `cp` into place; the PI's real remote-dev note (tailnet IPs, VPN endpoint, cluster/exit-node config) is excluded — a generic `Remote Dev — Pattern` replaces it. |
 | First pass | **Claude Code only** | Smaller surface; the deferred list below grows it later. |
+| Silent-stall detection | **cc-watchdog** (`onboarding/watchdog/`): timer-driven dead-man's switch, not hook-driven heuristics | CC-hook watchers only wake at turn ends, so they share the session's blind spot (a 36 h TA die-out in 2026-07 motivated this). Deadlines declared by the session itself make false positives ~zero (a 2.75 h quiet suite run is legitimate work); a 6 h notify-only backstop covers sessions that never declared. Unlike the PI-local auto-handoff/Phoenix stack, this ships to members: standalone (tmux + coreutils + systemd user timer + one Slack webhook). |
 
 ## Deferred — "later" (and how to un-defer)
 

onboarding/ONBOARDING.md

@@ -168,6 +168,41 @@ headroom perf                              # savings, once traffic has flowed
 > linger from step **c** — but only if no *other* systemd user service relies on it (linger keeps
 > all your user services alive across logout, not just Headroom's).
 
+### 13. cc-watchdog — silent-stall guard for long agent sessions
+
+Long autonomous Claude Code runs can die out silently: the session waits on
+background work (a test suite, a build) whose owner died, and nothing ever
+wakes it — it looks exactly like "done, waiting for you". cc-watchdog is a
+dead-man's switch that catches this.
+
+Before a session goes quiet while waiting on something, it declares a deadline
+(`cc-deadman set 40 "full test suite, ~25 min"`). A systemd user timer checks
+every 5 minutes; if the deadline passes you get a Slack ping, and if nothing
+recovers within an hour the watchdog nudges the session itself ("your awaited
+work likely died — check honest status and continue").
+
+Install:
+```bash
+cd ~/Code/DRGScripts/onboarding/watchdog
+./install.sh
+```
+Then put your Slack incoming-webhook URL in `~/.cc-watchdog/config`:
+```bash
+SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
+```
+(Create one: api.slack.com/apps → your app → Incoming Webhooks → add to your
+own DM or a private channel.)
+
+Works for agent sessions under tmux and under Herdr alike (inside a Herdr pane
+the deadline keys itself to the Herdr pane id and recovery uses `herdr pane
+run`; see the README's "tmux and Herdr" section and run its smoke test once
+from inside Herdr).
+
+The session-side habit lives in the global CLAUDE.md you merged in step 8
+(silent-stall prevention rules): agents declare deadlines before long waits and
+run final gates in the mother session. The watchdog is the enforcement; the
+rules are the prevention. Details: `onboarding/watchdog/README.md`.
+
 ---
 
 ## B. Laptop (Windows / macOS) — thin client + Obsidian
@@ -211,6 +246,9 @@ headroom perf                              # savings, once traffic has flowed
       healthy*; in a **freshly started** Claude Code session, typing `!env | grep ANTHROPIC_BASE_URL`
       at the Claude Code prompt (the leading `!` tells Claude Code to run the rest as a shell
       command) prints `http://127.0.0.1:8787`.
+- [ ] `systemctl --user status cc-stall-watchdog.timer` shows *active (waiting)*.
+- [ ] In a tmux Claude Code session: `cc-deadman set 1 "smoke test"` → Slack ping within
+      ~6 min (then `cc-deadman clear`).
 
 ---
 

onboarding/watchdog/README.md

@@ -0,0 +1,75 @@
+# cc-watchdog — dead-man's switch for Claude Code sessions
+
+Long autonomous Claude Code sessions can die out silently: the session waits on
+background work (a test suite, a build) whose owner died, and nothing ever wakes
+it — it looks exactly like "done, waiting for you". cc-watchdog catches this.
+
+## How it works
+
+1. Before a session goes quiet while waiting on something, it (or you) declares
+   a deadline: `cc-deadman set 40 "full test suite, ~25 min"` (expected duration
+   + 50% margin).
+2. A systemd user timer runs `cc-stall-watchdog.sh` every 5 minutes:
+   - Session progressed and is idle again → deadline self-clears.
+   - Deadline missed → **Slack ping** (tier 1).
+   - Still nothing an hour later (`GRACE_MIN=60`) → the watchdog **nudges the
+     session itself** (tier 2): Escape if busy, then a recovery prompt asking it
+     to check honest status and continue. Repeat stalls within 24 h back off to
+     notify-only — no injection loops.
+   - Sessions that never declared: if a pane is mid-turn busy for over
+     `BACKSTOP_HOURS=6` h with an equally stale transcript, you get a
+     notify-only ping (max once per 12 h). Legitimate long quiet runs (a 2.75 h
+     suite) can never trip this.
+
+The watchdog is deliberately timer-driven, not Claude-Code-hook-driven: hook
+watchers only wake when a turn ends, which is exactly the blind spot that
+produces multi-hour stalls.
+
+## tmux and Herdr
+
+Sessions under **tmux** and under **Herdr** are both supported. Inside a Herdr
+pane, `cc-deadman` keys the deadline to `$HERDR_PANE_ID` automatically; the
+watchdog then reads Herdr's native `agent_status` (busy = `working`; `blocked`
+counts as NOT idle, so a blocked agent still triggers your tier-1 ping) and
+injects recovery via `herdr pane run`. If `herdr` refuses commands from the
+timer's context, the watchdog degrades gracefully: the pane is assumed alive,
+injection is skipped (logged), and the Slack alert still fires. Run the smoke
+test below once from inside a Herdr pane to confirm external control works on
+your setup. Limitation: the no-declaration backstop scans tmux panes only —
+Herdr sessions are covered by declared deadlines.
+
+## Install
+
+    ./install.sh
+
+Then set your Slack incoming webhook in `~/.cc-watchdog/config`:
+
+    SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
+
+Optional overrides in the same file: `GRACE_MIN`, `BACKSTOP_HOURS`,
+`BACKSTOP_RENOTIFY_HOURS`, `RECOVERY_BACKOFF_HOURS`, `WATCHDOG_DRY=1` (log-only).
+
+## Session-side commands
+
+    cc-deadman set <minutes> "<reason>"   # declare before a long wait
+    cc-deadman extend <minutes>           # work is legitimately taking longer
+    cc-deadman clear                      # first action after resuming
+    cc-deadman list                       # what's armed right now
+
+## Smoke test
+
+1. In a tmux Claude Code session: `cc-deadman set 1 "smoke test"`.
+2. Within ~6 min: Slack ping (tier 1).
+3. To see tier 2 quickly: `echo GRACE_MIN=2 >> ~/.cc-watchdog/config`, wait
+   ~3 more min, watch the recovery prompt land in the pane. Revert the config
+   line and `cc-deadman clear` afterwards.
+
+Recommended first deployment: `WATCHDOG_DRY=1` in the config for 2–3 days;
+watch `~/.cc-watchdog/log` for what it *would* have done, then remove the flag.
+When you remove the flag to arm for real, also clear dry-run artifacts:
+`rm -f ~/.cc-watchdog/state/pane-*.last-recovery` (dry runs stamp recoveries
+too, and a stale stamp would wrongly back off the first real recovery).
+
+## Unit tests
+
+    test/run_tests.sh   # hermetic; fakes tmux/curl/pstree via PATH shims

onboarding/watchdog/cc-deadman

@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+# cc-deadman — declare when this Claude Code session expects to be alive again.
+# Enforced by cc-stall-watchdog.sh (see README.md in this directory).
+set -euo pipefail
+
+WD_HOME="${CC_WATCHDOG_HOME:-$HOME/.cc-watchdog}"
+STATE_DIR="$WD_HOME/state"
+mkdir -p "$STATE_DIR"
+
+usage() {
+  cat >&2 <<'EOF'
+usage: cc-deadman set <minutes> "<reason>"   declare a deadline
+       cc-deadman extend <minutes>           push this session's deadline out
+       cc-deadman clear                      remove this session's deadline
+       cc-deadman list                       show all live deadlines
+EOF
+  exit 2
+}
+
+resolve_pane() {
+  # Herdr injects its pane id into every managed pane; prefer it when present.
+  if [[ "${HERDR_ENV:-}" == 1 && -n "${HERDR_PANE_ID:-}" ]]; then
+    printf 'herdr:%s' "$HERDR_PANE_ID"; return 0
+  fi
+  if [[ -n "${TMUX_PANE:-}" ]]; then printf '%s' "$TMUX_PANE"; return 0; fi
+  # Tool shells often lack $TMUX_PANE: find the pane whose top-level shell
+  # (pane_pid) is an ancestor of this process.
+  local pid=$$ chain=" "
+  while (( pid > 1 )); do
+    chain+="$pid "
+    pid=$(awk '{print $4}' "/proc/$pid/stat" 2>/dev/null) || break
+  done
+  local pane ppid
+  while read -r pane ppid; do
+    [[ "$chain" == *" $ppid "* ]] && { printf '%s' "$pane"; return 0; }
+  done < <(tmux list-panes -a -F '#{pane_id} #{pane_pid}' 2>/dev/null)
+  echo "cc-deadman: cannot resolve tmux pane (not inside tmux?)" >&2
+  return 1
+}
+
+resolve_transcript() {
+  # Claude Code stores transcripts under a cwd-munged directory name.
+  local munged
+  munged=$(pwd | sed 's#[/.]#-#g')
+  # shellcheck disable=SC2012  # ls -t is the point (mtime sort); CC-generated paths contain no newlines
+  ls -t "$HOME/.claude/projects/$munged"/*.jsonl 2>/dev/null | head -1 || true
+}
+
+cmd=${1:-}
+[[ -n "$cmd" ]] || usage
+shift
+case "$cmd" in
+  set)
+    (( $# == 2 )) || usage
+    mins=$1 reason=$2
+    pane=$(resolve_pane)
+    f="$STATE_DIR/$pane.deadline"
+    now=$(date +%s)
+    {
+      echo "pane=$pane"
+      echo "set=$now"
+      echo "deadline=$(( now + mins * 60 ))"
+      echo "reason=$reason"
+      echo "transcript=$(resolve_transcript)"
+    } > "$f"
+    echo "cc-deadman: armed — $mins min for: $reason"
+    ;;
+  extend)
+    (( $# == 1 )) || usage
+    pane=$(resolve_pane)
+    f="$STATE_DIR/$pane.deadline"
+    [[ -f "$f" ]] || { echo "cc-deadman: no deadline set for $pane" >&2; exit 1; }
+    old=$(sed -n 's/^deadline=//p' "$f")
+    grep -Ev '^(deadline|notified|recovered|failed|recovery_kind|baseline)=' "$f" > "$f.tmp" || true
+    echo "deadline=$(( old + $1 * 60 ))" >> "$f.tmp"
+    mv "$f.tmp" "$f"
+    echo "cc-deadman: extended by $1 min"
+    ;;
+  clear)
+    pane=$(resolve_pane)
+    rm -f "$STATE_DIR/$pane.deadline"
+    echo "cc-deadman: cleared"
+    ;;
+  list)
+    shopt -s nullglob
+    now=$(date +%s)
+    for f in "$STATE_DIR"/*.deadline; do
+      d=$(sed -n 's/^deadline=//p' "$f")
+      printf '%s  due in %d min  %s\n' \
+        "$(sed -n 's/^pane=//p' "$f")" "$(( (d - now) / 60 ))" \
+        "$(sed -n 's/^reason=//p' "$f")"
+    done
+    ;;
+  *) usage ;;
+esac

onboarding/watchdog/cc-stall-watchdog.service

@@ -0,0 +1,6 @@
+[Unit]
+Description=Claude Code stall watchdog (dead-man's switch tick)
+
+[Service]
+Type=oneshot
+ExecStart=%h/.local/bin/cc-stall-watchdog.sh