Skip to content

Fix NPE in FlinkStateSnapshot cleanup causing orphaned CRs and blocked namespace deletion - #27

Merged
Yanisdje merged 4 commits into
mainfrom
yd/state-snapshot-fix
Mar 31, 2026
Merged

Fix NPE in FlinkStateSnapshot cleanup causing orphaned CRs and blocked namespace deletion#27
Yanisdje merged 4 commits into
mainfrom
yd/state-snapshot-fix

Conversation

@Yanisdje

@Yanisdje Yanisdje commented Mar 30, 2026

Copy link
Copy Markdown

Problem

When a FlinkStateSnapshot CR is deleted before the FlinkStateSnapshotController has reconciled it, cleanup() throws a NullPointerException on getStatus().toBuilder(). The status is null because the CR was never processed by reconcile() which is the only method that initializes it.

The NPE prevents the finalizer from being removed, causing the CR to be permanently stuck in a terminating state. This blocks namespace deletion and cluster migration. Observed in production with CRs stuck since October 2025.

Root Cause

FlinkStateSnapshot CRs are created without a status (the status subresource is only populated when reconcile() runs). If the CR receives a deletion timestamp before reconcile() runs, JOSDK calls cleanup() directly. reconcile() already has a null status guard:

// status might be null here
flinkStateSnapshot.setStatus(
        Objects.requireNonNullElseGet(
                flinkStateSnapshot.getStatus(), FlinkStateSnapshotStatus::new));

cleanup() and updateErrorStatus() are missing this guard, causing the NPE.

Fix

Add the same null-status initialization to cleanup() and updateErrorStatus(). A null-status snapshot was never triggered against Flink - no data exists on storage - so cleanup can safely proceed. For checkpoints (the observed case), cleanup immediately returns defaultDelete() via the isCheckpoint() check.

Impact

  • Resolves all currently stuck FlinkStateSnapshot CRs on next operator retry cycle
  • Unblocks terminating namespaces waiting on finalizer removal
  • Prevents future orphaned snapshots from accumulating

@Yanisdje Yanisdje self-assigned this Mar 30, 2026
@Yanisdje
Yanisdje force-pushed the yd/state-snapshot-fix branch from 1062465 to 21867b8 Compare March 30, 2026 17:40
@Override
public ErrorStatusUpdateControl<FlinkStateSnapshot> updateErrorStatus(
FlinkStateSnapshot resource, Context<FlinkStateSnapshot> context, Exception e) {
if (resource.getStatus() == null) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems out of place.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updating the status before the updateErrorStatus is strange.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was added as a guard since updateErrorStatus also calls getFlinkStateSnapshotContext(), which is where the NPE originates.

But the reconcil() method immediately initializes the status as its first action before anything can throw. So status will never be null when updateErrorStatus is called. So it makes sense to remove it. I will only keep the check above.

@Yanisdje
Yanisdje merged commit 850f558 into main Mar 31, 2026
11 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants