You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Two `update_and_restart()` cycles to 26.08.006 appeared to deploy but deps never landed, the second update reported a from-hash (`37bb694`) belonging to a review worktree, and #995's vestigial-venv fallback never logged. Root cause chain:
The launchd plist runs the daemon as `.venv/bin/python -m pinky_daemon` — the project venv is the daemon's interpreter. (`ps` shows the resolved homebrew binary because `.venv/bin/python` is a symlink, which hides this.)
During the daytime fix(admin): fall back from broken venv pip #995 review, a probe ran an editable install into that venv from the review worktree `/private/tmp/pinkybot-508-pip-probe.ANZX36` → `_editable_impl_pinky_ai.pth` repointed at the worktree.
Next restart: the daemon imported worktree code. Since `repo_dir = Path(file)...` (api.py ~11662), the second update git-fetched/checked-out the worktree (hence from-hash 37bb694, worktree HEAD moved to 01e577d, worktree frontend-dist built and served in prod).
fix(admin): fall back from broken venv pip #995's fallback DID fire correctly — but its fallback target is `sys.executable`, which was itself the pip-less venv python → same `No module named pip` error, no vestigial log line (probe skipped: worktree has no `.venv`).
Boot-log tells: `skill_loader: shadowed skill ... /private/tmp/pinkybot-508-pip-probe.ANZX36/...` and `frontend-dist does not exist` on gen-2 boot.
Remediation applied on prod (already done)
`ensurepip` into the venv (pip 26.x working), editable install repointed to `/Users/oleg/PinkyBot`, deps verified against pyproject pins, daemon bounced.
Fix items
Observability: `/admin/update` result + log must include `repo_dir` and `sys.executable`. Either alone would have made this a 1-minute diagnosis.
Guard: update should refuse (and owner-alert) when `repo_dir` is not the canonical checkout — e.g. validate against a configured `PINKYBOT_ROOT` or the launchd WorkingDirectory, instead of trusting `file`.
fix(admin): fall back from broken venv pip #995 blind spot: before falling back to `sys.executable`, probe it for pip too; last resort `ensurepip`. A pip-less interpreter can be the running one, not just the venv's.
Process guard: editable installs into the prod venv from a worktree repoint prod imports. Review/probe worktrees must use their own venvs; document in CLAUDE.md / probe tooling.
Cleanup
`/private/tmp/pinkybot-508-pip-probe.ANZX36` still exists (kept for forensics; its frontend-dist served prod until tonight's bounce). Remove after fix items land.
What happened (2026-08-04 night)
Two `update_and_restart()` cycles to 26.08.006 appeared to deploy but deps never landed, the second update reported a from-hash (`37bb694`) belonging to a review worktree, and #995's vestigial-venv fallback never logged. Root cause chain:
Boot-log tells: `skill_loader: shadowed skill ... /private/tmp/pinkybot-508-pip-probe.ANZX36/...` and `frontend-dist does not exist` on gen-2 boot.
Remediation applied on prod (already done)
Fix items
Cleanup
`/private/tmp/pinkybot-508-pip-probe.ANZX36` still exists (kept for forensics; its frontend-dist served prod until tonight's bounce). Remove after fix items land.
🤖 Opened by Barsik