[BUG] Foreman gate pods set XDG_DATA_HOME to a non-writable cache PVC → non-root gates crash

[joryirving]

Bug Description

Foreman verify/gate pods inject XDG_DATA_HOME=/cache/xdg (the gate-cache PVC), but the pods run as the gate image's non-root user with no securityContext.fsGroup. The RWX PVC mounts root-owned, so /cache/xdg is not writable by the gate uid. Any gate tool that writes under XDG_DATA_HOME then fails silently-ish and takes down the whole gate before the real checks run.

Concretely this breaks the Godot (godot-gate) verify used for a Godot project: Godot's user:// resolves under XDG_DATA_HOME, so it cannot create user://logs, which cascades into a hard crash:

ERROR: Could not create directory: user://logs
handle_crash: Program crashed with signal 11
ERROR: FATAL: Index p_index = 1 is out of bounds (size() = 0).
   at: get (core/templates/cowdata.h:158)

Every gate for that repo fails identically, before any test runs — so the real verify result (pass, or a genuine test failure) is completely masked.

Steps to Reproduce

1. A verify gate whose image runs as a non-root user (e.g. USER 65534) and whose tool writes under XDG_DATA_HOME.
2. Foreman creates the gate pod with env XDG_DATA_HOME=/cache/xdg (the gate-cache PVC), securityContext.fsGroup unset, runAsUser unset (so it inherits the image's non-root USER).
3. The tool writes $XDG_DATA_HOME/...; the PVC mount is root:root and not writable by the gate uid → write fails → crash / gate error.

Minimal repro matching production (godot-gate image, USER 65534):

# non-writable XDG (mimics the /cache/xdg PVC) → crash:
export XDG_DATA_HOME=/var/cache/xdg   # root-owned, not writable by uid 65534
godot --headless --path <project> --script res://tests/test_runner.gd
#   -> ERROR: Could not create directory: user://logs
#   -> signal 11 / Index out of bounds size()=0 ; exit 132

# writable XDG → same command runs the tests clean:
export XDG_DATA_HOME=/tmp/.local/share
godot --headless --path <project> --script res://tests/test_runner.gd   # OK

Expected Behavior

Gate pods provide a writable XDG_DATA_HOME, so a non-root gate tool can create its user/data directories (import caches, logs, etc.) and the gate reflects the tool's real result.

Actual Behavior

XDG_DATA_HOME=/cache/xdg points at a root-owned PVC with no fsGroup, so the non-root gate can't write it. The tool crashes on startup and the gate fails before running any check — masking both real passes and real test failures.

Environment

LLMKube Version: 0.9.3

Cluster Type:

• K3s / Talos (bare-metal homelab)

GPU (if applicable):

• Other: AMD Strix Halo gfx1151 iGPU (this is the AMD/foreman fleet), but the bug is GPU-independent — it's a gate-pod filesystem-permissions issue.

Relevant gate-pod facts: securityContext.fsGroup unset, runAsUser unset (inherits image USER 65534); gate-cache PVC (ceph-filesystem, RWX) mounted at /cache; env XDG_DATA_HOME=/cache/xdg.

Logs

Gate step log tail (windowstead / godot-gate):

=== clone ... ===
Cloning into '/work'...
ERROR: Could not create directory: user://logs
handle_crash: Program crashed with signal 11
ERROR: FATAL: Index p_index = 1 is out of bounds (size() = 0).
   at: get (core/templates/cowdata.h:158)
GATE FAIL

Additional Context

Suggested fixes (any one resolves it):

• Set securityContext.fsGroup on gate pods (e.g. matching the gate image's runAsGroup) so the RWX PVC is group-writable — keeps the cross-run cache benefit.
• Or don't force XDG_DATA_HOME onto the shared PVC for gates running as non-root; or ensure the xdg subdir is created world/group-writable.

Workaround in use: re-export XDG_DATA_HOME to a writable /tmp path inside the gate command.

In practice only the Godot gate writes XDG_DATA_HOME today, so it's the visible victim — but this is a latent trap for any non-root gate tool that honors XDG. Happy to send a PR (fsGroup on the gate pod spec seems cleanest).