Skip to content

Cookbook remote servers fail with "Permission denied" in Docker under PUID=0 — generated SSH key in /app/.ssh is never offered #5672

Description

@catagris

Prerequisites

  • I searched open issues and discussions and did not find an existing report of this bug.
  • This is not a security vulnerability. (Vulnerabilities go to GitHub Security Advisories — see SECURITY.md.)
  • I am running the latest code from the dev branch (the default branch you get on clone, where fixes land first) and the bug still reproduces there. Please git pull the latest dev before filing.

Install Method

Docker (docker compose up)

Operating System

Linux

Steps to Reproduce

  1. Run the Odysseus Docker image with PUID=0 (e.g. to match root-owned bind mounts on a NAS host) — any deployment where the container runtime user's home directory is not /app reproduces this.
  2. Open Cookbook → Settings → Servers, click "+ Add", enter a reachable Linux server as user@host, and install the Odysseus public key on it using the command from the Key button (key lands correctly in the target's ~/.ssh/authorized_keys).
  3. Click "Check" on the server entry (or try a model download / serve targeting that server).

Expected Behaviour

The Check succeeds using the SSH keypair Cookbook generated (the same one whose public half the Key button installed), and downloads/serves on the remote server launch over key auth.

Actual Behaviour

Check fails with Failed - Permission denied, please try again. even though the public key is correctly installed on the target. Model downloads and serves against the remote host fail the same way. The target server's sshd journal shows the Odysseus container attempting password authentication only — the generated key is never offered (no publickey attempts at all).

Root cause: Cookbook generates and stores its keypair in /app/.ssh (the persisted data/ssh bind mount), but all ssh/scp invocations rely on the client's default $HOME/.ssh lookup — no -i is ever passed (core/platform_compat.py:_ssh_exec_argv and the string-built commands in routes/cookbook_routes.py, routes/cookbook_helpers.py, src/tools/cookbook.py). With the container running as PUID=0, docker/entrypoint.sh resolves the existing root user, so gosu runs the app with HOME=/root — and /root/.ssh is empty. ssh finds no identity, falls back to password auth against a closed stdin, and fails. Any PUID that maps to a pre-existing passwd entry whose home is not /app reproduces this; the bug also makes _repair_cookbook_known_host edit /app/.ssh/known_hosts, a file ssh never reads in that configuration.

Logs / Screenshots

Target server sshd journal during a Cookbook "Check" (odysseus host is 192.168.1.34, target is 192.168.1.161):

Jul 21 19:09:26 Ares sshd-session[345251]: Failed password for william from 192.168.1.34 port 50864 ssh2
Jul 21 19:09:26 Ares sshd-session[345251]: Failed password for william from 192.168.1.34 port 50864 ssh2
Jul 21 19:09:26 Ares sshd-session[345251]: Connection closed by authenticating user william 192.168.1.34 port 50864 [preauth]
Jul 21 19:09:26 Ares sshd[345095]: srclimit_penalise: 192.168.1.34/32: activating ipv4 penalty of 15.841 seconds for penalty: failed authentication

(no publickey attempts from the container at any point)

Inside the container: the key exists at /app/.ssh/id_ed25519, but the process user's HOME is /root and /root/.ssh does not exist.

Model / Backend (if relevant)

No response

Are you willing to submit a fix?

Yes — I can open a PR

Additional Information

Fix is ready in PR #5669: it passes the Cookbook key explicitly (-i <key> -o IdentitiesOnly=yes + -o UserKnownHostsFile when containerized) on every Cookbook ssh/scp invocation, and the entrypoint now symlinks $HOME/.ssh/app/.ssh so frontend-composed ssh commands going through /api/shell/exec are covered as well.

Related quality-of-life fix included there: the manual key-install command shown by the Key button now passes -o PubkeyAuthentication=no, because an ssh-agent holding several unrelated keys exhausts the server's MaxAuthTries before the password prompt ("Too many authentication failures"), which is easy to hit while setting this up.

Update: verified end-to-end — deployed an image built from PR #5669 on the affected PUID=0 setup: Check succeeds, the target's sshd log shows Accepted publickey for the odysseus-cookbook key, and downloads/serves on the remote server work over key auth.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingready for reviewDescription complete — ready for maintainer review

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions