Summary
Both retention prune CronJobs mount the app's iceberg-data PVC (deploy/k8s/prune-cronjob.yaml:50-62, 99-109), which is ReadWriteOnce (deploy/k8s/pvc.yaml:6) and permanently attached to the running Deployment pod. Neither CronJob has node affinity/topology constraints.
Failure scenario
On any multi-node cluster, a cron pod can be scheduled on a different node than the app and hang in ContainerCreating on volume attach. Compounding it: concurrencyPolicy: Forbid (prune-cronjob.yaml:21,71) plus no activeDeadlineSeconds/backoffLimit means one stuck job suppresses every subsequent scheduled run indefinitely — retention silently stops, which is exactly the failure retention was added to prevent (unbounded audit/PDF growth).
Additionally, iceberg-prune-audit is DB-only under the prod Postgres posture and doesn't need /data at all — the mount (commented "harmless for Postgres") is pure attach-liability. Only prune-renders genuinely needs the PDF volume.
Suggested fix
- Drop the
data mount from prune-audit.
- Give
prune-renders pod affinity to the app: iceberg pod (or document the single-node/RWX constraint as deployment.yaml does).
- Add
activeDeadlineSeconds to both CronJobs so a stuck job can't wedge the schedule.
- Consider
resources requests/limits (a ResourceQuota namespace rejects pods without requests — another silent-stop path).
Found in Fable review of PR #232 → HEAD (flagged independently by two reviewers).
Summary
Both retention prune CronJobs mount the app's
iceberg-dataPVC (deploy/k8s/prune-cronjob.yaml:50-62, 99-109), which isReadWriteOnce(deploy/k8s/pvc.yaml:6) and permanently attached to the running Deployment pod. Neither CronJob has node affinity/topology constraints.Failure scenario
On any multi-node cluster, a cron pod can be scheduled on a different node than the app and hang in
ContainerCreatingon volume attach. Compounding it:concurrencyPolicy: Forbid(prune-cronjob.yaml:21,71) plus noactiveDeadlineSeconds/backoffLimitmeans one stuck job suppresses every subsequent scheduled run indefinitely — retention silently stops, which is exactly the failure retention was added to prevent (unbounded audit/PDF growth).Additionally,
iceberg-prune-auditis DB-only under the prod Postgres posture and doesn't need/dataat all — the mount (commented "harmless for Postgres") is pure attach-liability. Onlyprune-rendersgenuinely needs the PDF volume.Suggested fix
datamount fromprune-audit.prune-renderspod affinity to theapp: icebergpod (or document the single-node/RWX constraint asdeployment.yamldoes).activeDeadlineSecondsto both CronJobs so a stuck job can't wedge the schedule.resourcesrequests/limits (aResourceQuotanamespace rejects pods without requests — another silent-stop path).Found in Fable review of PR #232 → HEAD (flagged independently by two reviewers).