Display subprocess dies at boot on race with Redis (control:status is None) — DSI shows TTY instead of dashboard

[pythoninthegrass]

Summary

On a fresh boot, control.py's display subprocess reads control:status from Redis before control.py's own main loop has populated it, crashes with TypeError: the JSON object must be str, bytes or bytearray, not NoneType, and is never restarted. The parent control.py stays alive (so supervisor reports RUNNING and does not retry), and on setups driving a DSI panel via pygame/KMSDRM the DSI ends up showing the kernel TTY instead of the dashboard.

A manual sudo supervisorctl restart control after boot (once Redis is populated) always fixes it — confirming the issue is a startup-order race in PiFire itself, not a display/pygame problem.

Environment

• Raspberry Pi 5
• Raspberry Pi OS Lite (Bookworm), Python 3.13.5
• PiFire 1.10.9 (also reproduced on 1.10.1)
• Display: Waveshare 7" DSI LCD 800x480 (dsi_800x480t) via dtoverlay=vc4-kms-dsi-waveshare-800x480
• Redis: redis-server 8.0.2 running on 127.0.0.1:6379, healthy
• Supervisor managing control and webapp as user alex

Reproduction

1. Boot the Pi from power-off with the DSI display module selected in the wizard.
2. Observe dashboard web UI loads fine at http://<pi>/.
3. Observe DSI shows a blinking/login TTY rather than the dashboard.
4. sudo supervisorctl status → control RUNNING (misleading; the inner display subprocess is dead).
5. sudo tail /usr/local/bin/pifire/logs/control.err.log shows the traceback below.
6. sudo supervisorctl restart control → DSI now shows the dashboard; pygame becomes DRM master.

Traceback

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/usr/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/bin/pifire/display/dsi_800x480t.py", line 124, in _display_loop
    self._fetch_data()
  File "/usr/local/bin/pifire/display/base_flex.py", line 598, in _fetch_data
    self.status_data = read_status()
  File "/usr/local/bin/pifire/common/common.py", line 2307, in read_status
    status = json.loads(cmdsts.get('control:status'))
  File "/usr/lib/python3.13/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                    f'not {s.__class__.__name__}')
TypeError: the JSON object must be str, bytes or bytearray, not NoneType

Root cause

common.common.read_status() (non-init path) does:

status = json.loads(cmdsts.get('control:status'))

This assumes control:status always exists in Redis. At boot, the display subprocess (spawned from control.py via multiprocessing.Process) may call this before the main loop has called read_status(init=True) / write_status(...) to populate it. cmdsts.get(...) returns None, json.loads(None) raises TypeError, and the display process exits. The parent process does not notice the child death, so supervisor does not retry.

A similar shape likely exists for other keys read by the display loop in base_flex.py _fetch_data() (e.g. read_current, read_notify_data, etc.) — those currently happen to succeed because they are either exists-guarded or return an empty list from llen-based helpers, but they share the same ordering assumption.

Suggested fix

Make read_status() (and any peer in common.py called from the display loop) tolerant of a not-yet-populated key. Two low-risk options:

1. Treat a missing/None value the same as init=True and return a freshly-constructed default dict (but do not write it back — let the main loop own initialization).
2. Or, in the display subprocess _fetch_data, sleep/retry while required Redis keys are absent before entering the normal loop.

Option 1 is the smaller diff:

def read_status(init=False):
    global cmdsts
    if init or not cmdsts.exists('control:status'):
        # ... build default status dict as today ...
        if init:
            write_status(status)
        return status
    raw = cmdsts.get('control:status')
    if raw is None:
        # Key was deleted between exists() and get(); fall back to defaults.
        return read_status(init=False)  # or build defaults inline
    return json.loads(raw)

Workaround (documented for others hitting this)

Until the ordering is fixed, we drive control from a systemd oneshot that starts it, waits for control:status to appear in Redis, then restarts it so the display subprocess starts fresh with data available. We also unbind the kernel framebuffer console from the VT so the TTY doesn't render on the DSI while pygame is coming up, and we use /dev/dri/by-path/platform-<addr>.dsi-card for SDL_VIDEO_KMSDRM_DEVICE so DRM card renumbering across boots doesn't matter. Full writeup is happy to be shared if useful.

Related observations

• rp1-dsi and vc4-drm DRM card numbers swap between boots on Pi 5; any config that hardcodes /dev/dri/card1 vs card2 will be flaky.
• Updating 1.10.1 → 1.10.9 did not change this behavior.

[nebhead]

Thanks for the detailed sighting. I'll look into it soon. I'm thinking I may not have run into this because my primary Pi is not a Pi5 so it's less tested.

If you have a full write-up available on the display issue that would be great. Getting the DSI display to come up on boot has been an ongoing challenge with regards to 64-bit OSes.

[pythoninthegrass]

This is what claude wrote up for me:

DSI Display Boot-Race Workaround (Pi 5 / Bookworm)

Relates to nebhead/PiFire#242.

Problem

On a Pi 5 running Bookworm with a Waveshare 7" DSI LCD (800x480), a fresh
boot leaves the DSI showing a login TTY (or blinking cursor) instead of the
PiFire dashboard. The web UI works fine and supervisorctl status reports
control RUNNING.

Root cause

Three independent issues combine:

1. Startup race in PiFire. control.py spawns its display subprocess
   via multiprocessing.Process. The subprocess calls read_status() in
   common/common.py, which does json.loads(cmdsts.get('control:status')).
   At boot, the main loop has not yet populated that Redis key, so .get()
   returns None and json.loads(None) raises TypeError. The subprocess
   dies. The parent never notices, so supervisor does not retry.

2. Kernel framebuffer console renders on the DSI. Even with pygame /
   KMSDRM configured, the kernel fbcon is bound to the same VT that the DSI
   is on, so the TTY text renders on top of (or instead of) the pygame
   surface.

3. DRM card numbers swap between boots on Pi 5. rp1-dsi and vc4-drm
   trade /dev/dri/card0 vs card1 nondeterministically. Any config that
   hardcodes a card number is flaky.

A manual sudo supervisorctl restart control after boot (once Redis is
populated) always brings the dashboard up, confirming the race is the primary
cause.

Workaround

The workaround addresses all three issues. It does not patch PiFire source.

1. Autologin on tty1

Prevent agetty from holding a login prompt on the DSI's VT:

sudo mkdir -p /etc/systemd/system/getty@tty1.service.d
sudo tee /etc/systemd/system/getty@tty1.service.d/autologin.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=-/sbin/agetty --autologin <USER> --noclear %I $TERM
EOF
sudo systemctl daemon-reload
sudo systemctl restart getty@tty1.service

Replace <USER> with the user that supervisor runs control.py as (the
user= value in /etc/supervisor/conf.d/control.conf).

2. Use a stable DRM device path

Find your DSI card's by-path symlink:

ls -l /dev/dri/by-path/ | grep dsi

Update SDL_VIDEO_KMSDRM_DEVICE in /etc/supervisor/conf.d/control.conf to
use the by-path link instead of a /dev/dri/cardN path. For example:

environment=SDL_VIDEODRIVER="kmsdrm",SDL_FBDEV="/dev/fb0",SDL_VIDEO_KMSDRM_DEVICE="/dev/dri/by-path/platform-<ADDR>.dsi-card"

3. Disable supervisor autostart for control

sudo sed -i 's/^autostart=true/autostart=false/' /etc/supervisor/conf.d/control.conf
sudo supervisorctl update

4. Install the delayed-start script and systemd unit

The script waits for the DSI DRM device to register, unbinds the fbcon from
the VT, starts control via supervisor, waits for control:status to appear
in Redis, then restarts control so the display subprocess starts fresh with
data available.

sudo tee /usr/local/sbin/pifire-wait-redis.sh >/dev/null <<'SCRIPT'
#!/bin/bash
set -eu

# Wait for DSI DRM device to register (up to 30s)
for i in $(seq 1 30); do
  ls /sys/class/drm/card*-DSI-* >/dev/null 2>&1 && break
  sleep 1
done

# Unbind framebuffer console from VT so TTY text doesn't render on DSI
for vt in /sys/class/vtconsole/vtcon*; do
  if grep -q "frame buffer device" "$vt/name" 2>/dev/null; then
    echo 0 > "$vt/bind" 2>/dev/null || true
  fi
done

# Start control (display subprocess will crash on missing Redis key)
/usr/bin/supervisorctl start control

# Wait for control.py main loop to populate control:status (up to 120s)
for i in $(seq 1 120); do
  if [ "$(redis-cli -h 127.0.0.1 -p 6379 exists control:status 2>/dev/null)" = "1" ]; then
    sleep 3
    # Restart so display subprocess starts with Redis data available
    exec /usr/bin/supervisorctl restart control
  fi
  sleep 1
done

echo "pifire-wait-redis: timed out waiting for control:status in redis" >&2
exit 1
SCRIPT
sudo chmod +x /usr/local/sbin/pifire-wait-redis.sh

sudo tee /etc/systemd/system/pifire-control-delayed.service >/dev/null <<'UNIT'
[Unit]
Description=Start PiFire control after Redis control:status is populated
After=supervisor.service redis-server.service network-online.target
Wants=supervisor.service redis-server.service network-online.target
Requires=supervisor.service redis-server.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pifire-wait-redis.sh
RemainAfterExit=yes
TimeoutStartSec=180

[Install]
WantedBy=multi-user.target
UNIT
sudo systemctl daemon-reload
sudo systemctl enable pifire-control-delayed.service

5. Reboot and verify

sudo reboot

After reboot:

# Oneshot unit ran successfully
systemctl status pifire-control-delayed.service

# Framebuffer console is unbound
for vt in /sys/class/vtconsole/vtcon*; do
  echo "$vt: name=$(cat $vt/name) bind=$(cat $vt/bind)"
done

# pygame holds DRM master on the DSI card
sudo cat /sys/kernel/debug/dri/*/clients

Suggested upstream fix

The lowest-risk fix is in common.common.read_status(): treat a missing or
None value the same as init=True and return a default status dict without
writing it back to Redis (let the main loop own initialization):

def read_status(init=False):
    global cmdsts
    if init or not cmdsts.exists('control:status'):
        # build default status dict ...
        if init:
            write_status(status)
        return status
    raw = cmdsts.get('control:status')
    if raw is None:
        # Key was deleted between exists() and get()
        return read_status(init=False)
    return json.loads(raw)

The same pattern should be applied to any other common.py reader called from
the display loop in base_flex.py _fetch_data() (e.g., read_current,
read_notify_data). Most of those already guard with exists() or
llen()-based helpers, but read_status does not.

An alternative: have the display subprocess retry/sleep while required Redis
keys are absent before entering its normal loop.

Environment tested

• Raspberry Pi 5
• Raspberry Pi OS Lite (Bookworm), Python 3.13.5
• PiFire 1.10.9 (also reproduced on 1.10.1)
• Waveshare 7" DSI LCD 800x480, overlay vc4-kms-dsi-waveshare-800x480
• Redis 8.0.2 on localhost

[nebhead]

I've checked in some changes to the development branch. Please give that a spin and let me know if you run into any issues.