Skip to content

fix(reconciler): exponential crash-loop backoff for whole-pod restarts#63

Merged
dndungu merged 1 commit into
mainfrom
fix/issue-54-crashloop-backoff
Jul 10, 2026
Merged

fix(reconciler): exponential crash-loop backoff for whole-pod restarts#63
dndungu merged 1 commit into
mainfrom
fix/issue-54-crashloop-backoff

Conversation

@dndungu

@dndungu dndungu commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Problem (#54)

Exit-driven restarts had no damping: a pod crashing a few seconds after start was recreated every tick — 128 restarts in ~54 minutes, with each cycle re-running full startup and its external traffic. The existing backoffDelay/retryEligible machinery only gates create failures (podman run errors via StartAttempts); a clean start followed by a crash bypassed it entirely.

Fix

  • nextPodBackoff on the Always / OnFailure-retry transitions: delay doubles 10s → 5m cap (Kubernetes CrashLoopBackOff schedule); reconcilePending holds recreation until the deadline.
  • Reset after a clean run: a pod that ran ≥10 minutes before exiting starts the schedule over (StartedAt-based).
  • The pending event carries the delay (restarting (policy=Always), crash-loop backoff 40s).
  • State is reconciler-memory (same trade-off as the One crash-looping container restarts every container in the pod (k8s restarts only the failed one) #46 per-container backoff), swept on terminal state and pod deletion. Deliberately separate from the persisted create-failure backoff — different reset semantics; both documented in the devlog.

Testing

  • TestCrashLoop_BackoffDelaysRecreate: fake-clock schedule pinning (no recreate at 5s, recreate at 11s, doubled to 20s, event message carries the delay).
  • TestNextPodBackoff_CapsAtMax / _ResetsAfterCleanRun.
  • Existing restart tests updated to advance the clock past the backoff instead of asserting immediate recreate.
  • Full suite -race green, vet + staticcheck clean.

Live acceptance after deploy: a pod with command: ["sh","-c","exit 1"] under Always must show growing gaps between restarts (10s, 20s, 40s...) instead of one per tick.

Closes #54

A pod whose container started fine and then exited nonzero was
recreated at a flat per-tick interval: 128 restarts in ~54 minutes
observed, each re-running full container startup and its external
network calls. The existing backoff machinery only covered create
failures (podman run errors recorded via StartAttempts) — exit-driven
restarts had no damping.

Restart transitions (Always, OnFailure retries) now schedule the
recreate with a CrashLoopBackOff-style delay doubling from 10s to a
5m cap; reconcilePending holds the pod until the deadline. A pod that
ran cleanly for at least 10 minutes before exiting starts the schedule
over. The pending event carries the current delay so operators can see
the loop being damped.

Closes #54
@dndungu dndungu force-pushed the fix/issue-54-crashloop-backoff branch from 90353bf to 43ff8fe Compare July 10, 2026 04:26
@dndungu dndungu merged commit 3798cb8 into main Jul 10, 2026
1 check passed
@dndungu dndungu deleted the fix/issue-54-crashloop-backoff branch July 10, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No backoff on restartPolicy=Always crash loops: 128 flat-interval restarts in ~54 min

1 participant