Keep ingestion pipeline alive across transient database failures

[AKercha1]

What happened

On a production deployment the whole ingestion pipeline silently stopped for ~3 months. No data source was synced and newly added ones never ingested, but the API pod stayed Running/Ready the entire time, so nothing restarted it and no alert fired. It was only noticed when a customer reported the chatbot couldn't find documents they had uploaded.

Root cause was a transient database blip — the Azure PostgreSQL server restarting for scheduled maintenance (57P03: the database system is shutting down). That is a normal, expected event, but the way two pieces of code handle it turned it into a permanent outage.

Two defects

1. TaskProcessingHostedService dies permanently on a transient error.
The polling body (GetNew() and the dispatch loop) lived directly inside the while loop, and the only try/catch wrapped the entire loop. So when GetNew() threw during the DB restart, the exception left the loop, "TaskProcessingHostedService crashed." was logged once, and ExecuteAsync returned — the background service was gone for good. The web host kept running, so the pod stayed healthy and Kubernetes never restarted it.

IngestionSchedulerHostedService already does this correctly (its try/catch is inside the loop via RunOnceAsync), which is why the scheduler survived and only the task processor died.

2. Interrupted tasks are never recovered, which deadlocks the scheduler.
When the processor died it left the two tasks it was mid-processing stuck in InProgress. On startup ResetUnfinishedTasks only re-queues InProgress tasks with IsRetriable = true, so these non-retriable ones stayed InProgress forever. The scheduler then permanently skipped them: GetStale().Take(N) always returns the same stalest data sources, sees their orphaned "active" task and continues past them, so it never schedules anything. This means even restarting the pod would not have recovered the fleet — the stuck tasks had to be cleared by hand.

The fix

• TaskProcessingHostedService: move the per-iteration work into RunOnceAsync with its own try/catch. A bad tick is now logged ("...iteration error.") and the loop keeps polling, recovering on the next 5-second tick. Same shape as IngestionSchedulerHostedService.
• TaskProcessor.ResetUnfinishedTasks: reset every InProgress task on startup regardless of IsRetriable. A task still InProgress at startup was interrupted by definition (the worker that owned it is gone) and must be re-queued.

Together: defect 1 means a maintenance restart no longer kills the processor, and defect 2 means anything left mid-flight is picked back up instead of wedging the scheduler.

Verification

Reproduced and confirmed on the affected deployment: a New task had sat unprocessed for 7 days (proving the poll loop was dead, not just idle); restarting the pod revived it instantly, and after clearing the two stuck InProgress tasks the scheduler immediately resumed creating sync tasks. dotnet build -c Release passes (0 errors). The repo has no test project, so no automated test is added.

Suggested follow-ups (not in this PR)

• Tie the liveness probe to a background-worker heartbeat so Kubernetes auto-restarts the pod if a worker loop ever dies again — the strongest safety net, independent of any specific bug.
• Add EnableRetryOnFailure (EF Core connection resiliency) so transient DB errors are retried transparently.
• Alert when any data source's last_sync is older than a threshold.