Skip to content

Cookbook task status and crash reports read the tmux pane, not the runner log: finished installs show "running"/"crashed", crash reports capture the user's shell greeting #5695

Description

@catagris

Prerequisites

  • I searched open issues and discussions and did not find an existing report of this bug.
  • This is not a security vulnerability. (Vulnerabilities go to GitHub Security Advisories — see SECURITY.md.)
  • I am running the latest code from the dev branch (the default branch you get on clone, where fixes land first) and the bug still reproduces there. Please git pull the latest dev before filing.

Install Method

Docker (docker compose up)

Operating System

Linux

Steps to Reproduce

  1. Install any pip dependency (e.g. hf_transfer) on a remote server via Cookbook → Dependencies, and leave the task card collapsed (or navigate away from the Running tab). The install finishes in seconds; the runner prints DOWNLOAD_OK and === Process exited with code 0 === to its log, then (by design) drops into an interactive shell so the tmux session stays inspectable.
  2. Watch the task card: it shows "running" indefinitely — completion detection only happens inside _reconnectTask, which returns immediately unless the Running tab is visible AND the card is expanded.
  3. Now kill the tmux server on the target (or let the session die any other way) and reload the page: the session-gone classification reads only the on-screen output (empty after reload), finds no completion markers, and flips the finished task to crashed.
  4. On any task whose session dropped to the interactive shell, use "Copy crash report" / "Copy last 50 lines": the report contains the shell greeting (fastfetch banner — hostname, LAN IP, hardware inventory) instead of the actual output, or claims no log content exists.

Expected Behaviour

Task status reflects what actually happened regardless of whether anyone is watching the card: a finished install flips to done (and a failed one to crashed, with the real error) even when the card is collapsed, the pane was overwritten, or the session is gone. Crash reports and log copies contain the runner's actual output.

Actual Behaviour

Three symptoms, one root cause — everything reads the tmux pane (or the on-screen output element mirroring it) while the ground truth sits in the persisted runner log (/tmp/odysseus-tmux/<sid>.log, written by the runner's tee):

  1. Completed installs show "running" forever. _reconnectTask (which owns the DOWNLOAD_OK → done flip) returns immediately unless the Running tab is visible and the card expanded — completion detection requires an audience. A collapsed pip install that finished in 10 seconds stays "running" for hours (reproduced twice before diagnosing).
  2. Completed installs flip to "crashed" when the session dies. The session-gone path classifies from output.textContent, which is empty after a page reload — so no markers → crashed, even though the log holds the exit-0 sentinel and DOWNLOAD_OK.
  3. Crash reports and "Copy last 50 lines" capture garbage. The runner (by design) execs the user's shell after the process exits so the session stays inspectable — the shell greeting then overwrites the pane. Reports contain a fastfetch banner (hostname, LAN IP, full hardware inventory — a privacy footgun for reports meant to be posted publicly) instead of the error, and the log-copy action claims "No log content available yet" while a 54KB log sits on disk.

Related: #3772 fixed the local flavor of (1)/(2) with output markers, but remote/collapsed/pane-overwritten cases were left; #2709 is the inverse misclassification (false "finished" on connection drop) rooted in the same trust-the-pane design.

Logs / Screenshots

Runner log on the target while the card showed "running" (and later, after a reload, "crashed"):

$ cat /tmp/odysseus-tmux/serve-34df9cb2.log
[odysseus] HF token: NOT SET — ...
Collecting hf_transfer
  Downloading hf_transfer-0.1.9-...whl (3.6 MB)
Installing collected packages: hf_transfer
Successfully installed hf_transfer-0.1.9

DOWNLOAD_OK

=== Process exited with code 0 ===

"Last captured output" in the crash report generated for that same task (abridged):

~ william@Ares
odysseus-venv ❯
(fastfetch banner: CPU/GPU/memory/disk inventory, hostname, local IP)

Model / Backend (if relevant)

No response

Are you willing to submit a fix?

Yes — I can open a PR

Additional Information

Fix ready as a PR, verified end-to-end on a real deployment: a _fetchTaskLogTail() helper fetches the persisted log tail over the existing shell-exec path (remote side wrapped as sh -c for non-POSIX login shells, see #5689), and three consumers fall back to it whenever the on-screen output lacks the === Process exited with code N === sentinel — the session-gone classification in _reconnectTask, "Copy crash report", and "Copy last 50 lines". Plus a 45-second background completion sweep (started idempotently from the Running tab render) that checks running download-type tasks' logs and classifies from the sentinel with no audience required — done flips within a sweep cycle on collapsed cards, mirroring the live path's behaviour (deps refresh + cleanup of the lingering inspection session). Serve tasks are excluded: a live server has no exit sentinel and their semantics belong to the serve poller.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingready for reviewDescription complete — ready for maintainer review

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions