Orchestration deadlocks/parks with a large backlog: workers go idle after 'generating targets', nothing downloads

[bugrax]

Describe the bug

With a large backlog of COMPLETED transfers on put.io (~100+), putioarr stops downloading after a (re)start. The monitor loop queues the transfers (ready for download logged for all of them), a handful of orchestration workers log download started and generating targets, and then nothing — no downloads, no imports, indefinitely. A restart only clears it until the backlog rebuilds.

Key evidence — the process is parked, not busy

While stuck, the container is essentially idle:

• CPU ~0.15%, RSS ~6 MiB, RestartCount=0, not OOM-killed
• only ~7 open file descriptors (fd limit is 1M) → no in-flight HTTP requests / sockets
• download_workers worth of transfers log generating targets and then produce no further log

So the workers are parked on an .await (a lock or channel), not spinning and not resource-exhausted. put.io itself is healthy at the time (a manual files/list returns in ~0.4s), so it is not put.io latency.

generating targets is the first line of get_download_targets; the next thing it does is state.get_download_dir_for_transfer() (a tokio::sync::RwLock read) before any HTTP call — consistent with the tasks parking there (no socket opened, matching the ~7 fds).

Reproduction

Run putioarr against a large Sonarr/Radarr library so ~100+ transfers complete on put.io, then restart. After the resume scan finishes, downloads never start.

Notes

I have separate PRs adding request timeouts and a shared put.io client (#32/#33) and bounding the import watcher (#30/#31); none of them resolve this, which is consistent with it being a lock/channel coordination issue rather than a hung request. Happy to add instrumentation/logging if that would help pin down where the workers park.

[bugrax]

Important context I missed when filing this — it is very likely the real root cause.

The affected put.io account has "clear completed transfers" enabled, so put.io removes the transfer record as soon as a transfer finishes, while leaving the downloaded files in the folder.

putioarr discovers work purely from transfers/list. With clear-completed on, a transfer can be removed from that list before putioarr has pulled it, so:

• putioarr never sees / can never pull those files — they pile up as orphans in the put.io download folder (in my case ~500 GB across 55 files were sitting in the folder, completed but un-pulled, with no corresponding transfer).
• The "parked after generating targets, idle, ~7 fds" symptom in the original report is, I now think, downstream of this: the monitor queues transfers from a transfers/list snapshot, they get cleared underneath it, and processing the now-stale entries leaves the workers stuck.

So the underlying gap is architectural: putioarr is purely transfer-based and has no way to pull a completed file whose transfer record no longer exists. Anyone running put.io with auto-clear (or whose transfers get removed for any reason) loses those downloads.

A fix would be to also scan the configured download folder and pull video files that have no active transfer and have not yet been imported, independently of transfers/list. I am going to prototype that.

[bugrax]

Root cause found and fixed — full write-up.

Root cause: cascading, silent worker deaths

The orchestration and download workers both end their work() loop on the first ? that errors, and the spawned task's Result is discarded (actix_rt::spawn(async move { s.work().await })), so a worker that dies leaves no log at all. The death also cascades:

• Orchestration work() does let targets = t.get_download_targets().await?; and done_rx.recv().await? and self.tx.send(...).await?. Any of these erroring ends the worker.
• Download work() does dtm.tx.send(done_status).await?. If the orchestration worker that owns the matching done_rx is already gone, this send fails and ends the download worker too.

So one failing get_download_targets (e.g. a put.io 4xx/5xx, which is common when "clear completed" removes a transfer mid-flight) kills its orchestration worker → its done channels drop → the download workers feeding it die on send() → orchestration workers waiting on those die on recv() → … until every worker is gone and the process sits idle.

Symptoms this explains

• The process parks completely: ~0.1% CPU, ~6 MiB RSS, ~7 fds, no error in the logs.
• Files are fully downloaded on disk but download done never logs (the download worker finished the file, then died at the send() reporting step).
• A restart "fixes" it (fresh workers) until the next error.
• Larger backlogs hit it sooner (more transfers ⇒ higher chance of a triggering error).

Fix

Make the workers survive per-item errors instead of dying:

• Orchestration: moved the QueuedForDownload body into handle_queued() returning Result, and the worker loop logs its error and continues rather than propagating it.
• Download: dtm.tx.send(done_status) failures are logged and ignored instead of ?-propagated.

After this, with a previously-stalling backlog the workers stay up, download done fires, imports resume, and throughput returns to the NAS write ceiling — verified live.

(Combining this with the request timeouts in #32/#33 is ideal: timeouts turn a hung put.io call into an error, and this change ensures that error no longer takes a worker down.)

PR incoming.