Fix NPE in FlinkStateSnapshot cleanup causing orphaned CRs and blocked namespace deletion - #27
Merged
Merged
Conversation
Yanisdje
force-pushed
the
yd/state-snapshot-fix
branch
from
March 30, 2026 17:40
1062465 to
21867b8
Compare
| @Override | ||
| public ErrorStatusUpdateControl<FlinkStateSnapshot> updateErrorStatus( | ||
| FlinkStateSnapshot resource, Context<FlinkStateSnapshot> context, Exception e) { | ||
| if (resource.getStatus() == null) { |
There was a problem hiding this comment.
Updating the status before the updateErrorStatus is strange.
Author
There was a problem hiding this comment.
It was added as a guard since updateErrorStatus also calls getFlinkStateSnapshotContext(), which is where the NPE originates.
But the reconcil() method immediately initializes the status as its first action before anything can throw. So status will never be null when updateErrorStatus is called. So it makes sense to remove it. I will only keep the check above.
ryanvanhuuksloot
approved these changes
Mar 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a FlinkStateSnapshot CR is deleted before the FlinkStateSnapshotController has reconciled it, cleanup() throws a NullPointerException on getStatus().toBuilder(). The status is null because the CR was never processed by reconcile() which is the only method that initializes it.
The NPE prevents the finalizer from being removed, causing the CR to be permanently stuck in a terminating state. This blocks namespace deletion and cluster migration. Observed in production with CRs stuck since October 2025.
Root Cause
FlinkStateSnapshot CRs are created without a status (the status subresource is only populated when reconcile() runs). If the CR receives a deletion timestamp before reconcile() runs, JOSDK calls cleanup() directly. reconcile() already has a null status guard:
cleanup() and updateErrorStatus() are missing this guard, causing the NPE.
Fix
Add the same null-status initialization to cleanup() and updateErrorStatus(). A null-status snapshot was never triggered against Flink - no data exists on storage - so cleanup can safely proceed. For checkpoints (the observed case), cleanup immediately returns defaultDelete() via the isCheckpoint() check.
Impact