During a Szurubooru conversion, recompute_signatures stopped dead two thirds
of the way through and stayed that way for over ten hours. The process was
alive, the container was healthy, and CPU was zero.
The cause was three ffmpeg child processes deadlocked inside their own
threading. Because nothing in FfmpegSubprocess imposes a timeout, and because
each stuck decode holds a pooled database connection for its whole duration, the
worker pool drained and the entire task stopped.
The files themselves are fine — all three decode cleanly in ~0.01s when run by
hand.
Environment
- Oxibooru server image, bundled
ffmpeg 8.1 (static Alpine build)
- PostgreSQL 18, external to the compose stack
- ~531,000 posts, ~3,000 of them video
- Task:
recompute_signatures, run via ./server --admin
docker top shows the stuck children:
UID PID PPID C STIME TIME CMD
1000 39089 27266 30 00:31 05:12:17 ./server --admin
1000 42914 39089 0 00:34 00:00:04 /opt/app/ffmpeg -loglevel level+info -i /data/posts/.../263436_....webm -vf thumbnail,format=rgb24 -frames:v 1 -f rawvideo - -n
1000 116638 39089 0 02:11 00:00:04 /opt/app/ffmpeg ... 335242_....webm ...
1000 309955 39089 0 08:32 00:00:02 /opt/app/ffmpeg ... 296708_....webm ...
$ for p in 42914 116638 309955; do
printf '%s state=%s wchan=%s\n' "$p" "$(awk '{print $3}' /proc/$p/stat)" "$(cat /proc/$p/wchan)"
done
42914 state=S wchan=futex_do_wait
116638 state=S wchan=futex_do_wait
309955 state=S wchan=futex_do_waitThis one will be I think hard to reproduce, my fix was to kill stuck workers in another tmux session while the signature computation was still in progress:
while sleep 60; do
for p in $(docker top oxibooru-server-1 2>/dev/null | awk '/ffmpeg/{print $2}'); do
et=$(ps -o etimes= -p "$p" 2>/dev/null | tr -d ' ')
if [ -n "$et" ] && [ "$et" -gt 300 ]; then kill -9 "$p"; fi
done
done
During a Szurubooru conversion,
recompute_signaturesstopped dead two thirdsof the way through and stayed that way for over ten hours. The process was
alive, the container was healthy, and CPU was zero.
The cause was three
ffmpegchild processes deadlocked inside their ownthreading. Because nothing in
FfmpegSubprocessimposes a timeout, and becauseeach stuck decode holds a pooled database connection for its whole duration, the
worker pool drained and the entire task stopped.
The files themselves are fine — all three decode cleanly in ~0.01s when run by
hand.
Environment
ffmpeg 8.1(static Alpine build)recompute_signatures, run via./server --admindocker topshows the stuck children:This one will be I think hard to reproduce, my fix was to kill stuck workers in another tmux session while the signature computation was still in progress: