cc-watchdog: dead-man's switch against silent agent-session die-outs by alongd · Pull Request #53 · DanaResearchGroup/DRGScripts

alongd · 2026-07-14T07:40:36Z

Adds onboarding/watchdog/: cc-deadman (session-side deadline declarations, tmux + Herdr pane resolution) + cc-stall-watchdog (systemd user timer, 5 min) with tiered enforcement — baseline-stamped self-clear → Slack notify at deadline miss → pane recovery-injection after a 60-min grace (24 h backoff) — plus a 6 h notify-only backstop for undeclared tmux sessions, hermetic bash unit tests (29, run in CI alongside shellcheck), and onboarding docs (install step 13, smoke checks, silent-stall doctrine in CLAUDE.global.md, MAINTAINING rationale).

Motivation: a 36 h silent die-out — a final test suite owned by a stopped subagent died and nothing ever woke the session. Hook-driven watchers cannot see this stall class (they only wake when a turn ends); an independent timer can. Deadlines declared by the session itself keep false positives near zero (a legitimate 2.75 h quiet suite never trips anything).

Herdr-native: inside a Herdr pane, deadlines key to $HERDR_PANE_ID, busy/idle uses Herdr's agent_status (a blocked agent cannot self-clear — you get pinged), and recovery uses herdr pane run, degrading gracefully to notify-only if external control is unavailable.

Design note: the self-clear condition is baseline-stamped because cc-deadman set runs as a tool call (transcript writes land after the stamp); both naive mtime>set and freshness-window variants were empirically shown to archive every deadline at the first idle tick. See onboarding/MAINTAINING.md and the regression test test_wd_self_clear_baseline_sequence.

Rollout: install via onboarding/watchdog/install.sh, soak 2–3 days with WATCHDOG_DRY=1, then arm (and clear state/pane-*.last-recovery dry-run stamps).

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Member-facing docs for cc-watchdog: ONBOARDING step 13 (install + tmux/Herdr note) with two section-C smoke checks, a MAINTAINING rationale entry, and the silent-stall-prevention doctrine block in CLAUDE.global.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Self-clear must distinguish the declaring turn's own transcript tail writes from later resumed activity: cc-deadman runs as a tool call, so bare mtime>set — and any freshness-vs-now window — archives every declared deadline at the first idle tick, leaving idle-at-prompt stalls unalertable. First idle tick now stamps baseline=<transcript mtime>; self-clear requires strictly newer activity. extend strips the baseline. notify() now escapes backslashes/quotes and flattens control chars so a hostile reason can never cost the tier-1 alert. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Adds a new cc-watchdog onboarding component to detect and remediate “silent stall” Claude Code sessions (dead-man deadlines + periodic watchdog tick), including tmux + Herdr support, hermetic bash unit tests, and onboarding/maintaining documentation plus CI integration.

Changes:

Introduces cc-deadman for session-side deadline declaration/management and cc-stall-watchdog.sh for timer-driven enforcement (Slack notify + optional recovery injection + backstop scanning for undeclared tmux sessions).
Adds a systemd user timer/service and an installer script to deploy symlinks + units under ~/.local/bin and ~/.config/systemd/user.
Adds hermetic bash unit tests (PATH shims for tmux/herdr/curl/etc.), docs updates, and a new CI job running shellcheck + the unit test suite.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
onboarding/watchdog/cc-deadman	Session-side CLI to set/extend/clear/list per-pane deadlines (tmux + Herdr pane keying).
onboarding/watchdog/cc-stall-watchdog.sh	Timer-driven watchdog enforcing deadlines, Slack notify, recovery injection/backoff, and tmux backstop scanning.
onboarding/watchdog/cc-stall-watchdog.service	systemd user oneshot service unit to run the watchdog script.
onboarding/watchdog/cc-stall-watchdog.timer	systemd user timer unit scheduling watchdog execution every 5 minutes.
onboarding/watchdog/install.sh	Idempotent installer wiring scripts into `~/.local/bin` and installing systemd units/config.
onboarding/watchdog/README.md	User-facing documentation: concept, tmux/Herdr behavior, install, smoke test, and unit test instructions.
onboarding/watchdog/test/run_tests.sh	Hermetic unit test runner covering notify, tiering, recovery, backstop, Herdr paths, and install idempotence.
onboarding/watchdog/test/shims/tmux	tmux test shim used by the hermetic unit tests.
onboarding/watchdog/test/shims/pstree	pstree test shim used by the hermetic unit tests.
onboarding/watchdog/test/shims/notify-send	notify-send test shim for non-Slack fallback path testing.
onboarding/watchdog/test/shims/herdr	herdr CLI test shim for Herdr-pane behavior (status/list/run).
onboarding/watchdog/test/shims/curl	curl test shim for Slack webhook posting verification.
onboarding/ONBOARDING.md	Adds a new onboarding step explaining cc-watchdog, install, and verification checks.
onboarding/MAINTAINING.md	Documents cc-watchdog as the repo’s approach to silent-stall detection (timer-driven vs hook-driven).
onboarding/CLAUDE.global.md	Adds a “silent-stall prevention” contract: deadlines before long waits, clear on resume, etc.
.github/workflows/ci.yml	Adds a CI job to shellcheck watchdog scripts and run the hermetic unit tests.
.gitignore	Ignores `docs/superpowers/` working files directory.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=5min


+notify() { # $1 = message — escape for the JSON payload; a dropped alert is
+  # the one unacceptable failure mode, so backslashes/quotes/control chars
+  # must never invalidate the body.


+set_field() {
+  grep -v "^$2=" "$1" > "$1.tmp" || true
+  echo "$2=$3" >> "$1.tmp"
+  mv "$1.tmp" "$1"
+}


+    old=$(sed -n 's/^deadline=//p' "$f")
+    grep -Ev '^(deadline|notified|recovered|failed|recovery_kind|baseline)=' "$f" > "$f.tmp" || true
+    echo "deadline=$(( old + $1 * 60 ))" >> "$f.tmp"
+    mv "$f.tmp" "$f"


Quote herdr status keywords (SC1010), drop the dead setts read left by the baseline-stamp redesign (SC2034), guard test cd calls (SC2164), rework cc-deadman arg parsing away from A&&B||C (SC2015), and add reasoned disables where the flagged code is correct (SC2012 mtime-sort ls, SC2001 per-line sed prefix, SC1091 runtime source). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

alongd and others added 10 commits July 14, 2026 05:31

watchdog: test harness skeleton + local-only superpowers dir

288e6b7

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

watchdog: cc-deadman declaration helper

0a2f92f

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

watchdog: poller scaffolding — config, notify, pane helpers

6d7eb16

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

watchdog: deadline ladder — self-clear, notify, recover, backoff

46edf2c

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

watchdog: multiplexer seam — Herdr backend for the deadline ladder

d7459a6

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

watchdog: notify-only backstop for undeclared stalled sessions

5284438

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

watchdog: systemd units, installer, README

d7ecf5f

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ci: shellcheck + unit tests for cc-watchdog

92e9a13

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

alongd requested a review from Copilot July 14, 2026 08:05

Copilot started reviewing on behalf of alongd July 14, 2026 08:06 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

alongd merged commit cef3b71 into main Jul 14, 2026
3 checks passed

alongd deleted the cc-watchdog branch July 14, 2026 16:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

cc-watchdog: dead-man's switch against silent agent-session die-outs#53

cc-watchdog: dead-man's switch against silent agent-session die-outs#53
alongd merged 11 commits into
mainfrom
cc-watchdog

alongd commented Jul 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

alongd commented Jul 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants