Skip to content

self-update: repo_dir/interpreter contract unguarded — daemon silently ran from a /tmp worktree (08-04 incident) #1000

Description

@olegbrok

What happened (2026-08-04 night)

Two `update_and_restart()` cycles to 26.08.006 appeared to deploy but deps never landed, the second update reported a from-hash (`37bb694`) belonging to a review worktree, and #995's vestigial-venv fallback never logged. Root cause chain:

  1. The launchd plist runs the daemon as `.venv/bin/python -m pinky_daemon` — the project venv is the daemon's interpreter. (`ps` shows the resolved homebrew binary because `.venv/bin/python` is a symlink, which hides this.)
  2. The venv is a uv venv (pip-less by design) — that is the actual root cause of Upgrade claude-agent-sdk 0.1.81 → 0.2.x (breaking, includes CLI 2.1.142 fixes) #508's `No module named pip` failures.
  3. During the daytime fix(admin): fall back from broken venv pip #995 review, a probe ran an editable install into that venv from the review worktree `/private/tmp/pinkybot-508-pip-probe.ANZX36` → `_editable_impl_pinky_ai.pth` repointed at the worktree.
  4. Next restart: the daemon imported worktree code. Since `repo_dir = Path(file)...` (api.py ~11662), the second update git-fetched/checked-out the worktree (hence from-hash 37bb694, worktree HEAD moved to 01e577d, worktree frontend-dist built and served in prod).
  5. fix(admin): fall back from broken venv pip #995's fallback DID fire correctly — but its fallback target is `sys.executable`, which was itself the pip-less venv python → same `No module named pip` error, no vestigial log line (probe skipped: worktree has no `.venv`).

Boot-log tells: `skill_loader: shadowed skill ... /private/tmp/pinkybot-508-pip-probe.ANZX36/...` and `frontend-dist does not exist` on gen-2 boot.

Remediation applied on prod (already done)

  • `ensurepip` into the venv (pip 26.x working), editable install repointed to `/Users/oleg/PinkyBot`, deps verified against pyproject pins, daemon bounced.

Fix items

  1. Observability: `/admin/update` result + log must include `repo_dir` and `sys.executable`. Either alone would have made this a 1-minute diagnosis.
  2. Guard: update should refuse (and owner-alert) when `repo_dir` is not the canonical checkout — e.g. validate against a configured `PINKYBOT_ROOT` or the launchd WorkingDirectory, instead of trusting `file`.
  3. fix(admin): fall back from broken venv pip #995 blind spot: before falling back to `sys.executable`, probe it for pip too; last resort `ensurepip`. A pip-less interpreter can be the running one, not just the venv's.
  4. Process guard: editable installs into the prod venv from a worktree repoint prod imports. Review/probe worktrees must use their own venvs; document in CLAUDE.md / probe tooling.

Cleanup

`/private/tmp/pinkybot-508-pip-probe.ANZX36` still exists (kept for forensics; its frontend-dist served prod until tonight's bounce). Remove after fix items land.

🤖 Opened by Barsik

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions