fix(reconciler): exponential crash-loop backoff for whole-pod restarts#63
Merged
Conversation
A pod whose container started fine and then exited nonzero was recreated at a flat per-tick interval: 128 restarts in ~54 minutes observed, each re-running full container startup and its external network calls. The existing backoff machinery only covered create failures (podman run errors recorded via StartAttempts) — exit-driven restarts had no damping. Restart transitions (Always, OnFailure retries) now schedule the recreate with a CrashLoopBackOff-style delay doubling from 10s to a 5m cap; reconcilePending holds the pod until the deadline. A pod that ran cleanly for at least 10 minutes before exiting starts the schedule over. The pending event carries the current delay so operators can see the loop being damped. Closes #54
90353bf to
43ff8fe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (#54)
Exit-driven restarts had no damping: a pod crashing a few seconds after start was recreated every tick — 128 restarts in ~54 minutes, with each cycle re-running full startup and its external traffic. The existing
backoffDelay/retryEligiblemachinery only gates create failures (podman runerrors viaStartAttempts); a clean start followed by a crash bypassed it entirely.Fix
nextPodBackoffon the Always / OnFailure-retry transitions: delay doubles 10s → 5m cap (Kubernetes CrashLoopBackOff schedule);reconcilePendingholds recreation until the deadline.StartedAt-based).restarting (policy=Always), crash-loop backoff 40s).Testing
TestCrashLoop_BackoffDelaysRecreate: fake-clock schedule pinning (no recreate at 5s, recreate at 11s, doubled to 20s, event message carries the delay).TestNextPodBackoff_CapsAtMax/_ResetsAfterCleanRun.-racegreen, vet + staticcheck clean.Live acceptance after deploy: a pod with
command: ["sh","-c","exit 1"]underAlwaysmust show growing gaps between restarts (10s, 20s, 40s...) instead of one per tick.Closes #54