FlinkDeployment e2e cleanup can leave stale APIEnablements

[ranxi2001]

Which jobs are flaking:

• CI Workflow / e2e test (v1.34.0)
• Observed in feat: support rotating init-managed certificates #7697 on run https://github.com/karmada-io/karmada/actions/runs/28499042349, job https://github.com/karmada-io/karmada/actions/runs/28499042349/job/84472927003, head SHA 152ab454265ac683f55f04a166e9de9aedaad94c.
• The same PR code path passed after re-triggering CI with an empty commit, so this looks like an e2e timing flake rather than a regression from feat: support rotating init-managed certificates #7697.

Which test(s) are flaking:

The observed failure was:

[EstimatorAssumption] ResourceQuota plugin assumption testing
[It] FlinkDeployment should be unschedulable when assumed workloads exhaust ResourceQuota

The test timed out in WaitResourceBindingFitWith while waiting for the first FlinkDeployment ResourceBinding to become scheduled:

[FAILED] Timed out after 420.000s.
In [It] at: test/e2e/framework/resourcebinding.go:47
test/e2e/suites/base/estimator_test.go:350

This is probably not limited to one It block. Several e2e suites create and delete the same flinkdeployments.flink.apache.org CRD:

• test/e2e/suites/base/estimator_test.go
• test/e2e/suites/base/schedule_multi_template_test.go
• test/e2e/suites/base/federatedresourcequota_test.go

Reason for failure:

The scheduler log from the failed job shows that the ResourceBinding was rejected by the APIEnablement plugin because member1 did not report the FlinkDeployment API:

Cluster(member1) not fit as missing API(flink.apache.org/v1beta1, kind=FlinkDeployment)
ResourceBinding(karmadatest-6cw8j/flinkdeployment-5fc2b-flinkdeployment) scheduled to clusters []
0/3 clusters are available: 1 cluster(s) did not have the API resource, 2 cluster(s) did not match the placement cluster affinity constraint.

The current FlinkDeployment e2e cleanup path only waits for the source CRD to disappear from the Karmada control plane. CRD propagation/removal to member clusters and the Cluster.Status.APIEnablements collection are asynchronous.

I ran a local diagnostic that simulated the old cleanup boundary:

1. Create flinkdeployments.flink.apache.org on the Karmada control plane.
2. Propagate the CRD to two member clusters.
3. Wait until the member CRDs exist and Cluster.Status.APIEnablements reports FlinkDeployment.
4. Delete the ClusterPropagationPolicy.
5. Delete the source CRD from the control plane.
6. Wait only for the control-plane CRD to disappear, which is what the old cleanup waited for.
7. Poll member CRDs and Cluster.Status.APIEnablements.

The diagnostic showed that the old cleanup can return before member-cluster CRD and APIEnablement state has converged:

control-plane CRD disappeared after 1s

after 00s: APIEnablements still reported FlinkDeployment and member CRDs still existed
after 01s: APIEnablements still reported FlinkDeployment; one member CRD was gone
after 02s: member CRDs were gone, but APIEnablements still reported FlinkDeployment
after 03s: member CRDs were gone, but APIEnablements still reported FlinkDeployment
after 04s: APIEnablements also converged

This gives a plausible race:

1. A previous FlinkDeployment e2e deletes the CRD and returns after the control-plane CRD disappears.
2. Cluster.Status.APIEnablements can still report FlinkDeployment from the previous CRD for a short time.
3. The next FlinkDeployment e2e starts and WaitCRDPresentOnClusters() can pass on that stale enabled status instead of a fresh status update for the newly propagated CRD.
4. The scheduler can then observe the member cluster as missing flink.apache.org/v1beta1/FlinkDeployment, causing the ResourceBinding to remain unscheduled until the test times out.

So the cleanup should wait for all state that later tests depend on:

• the source CRD has disappeared from the Karmada control plane;
• the propagated CRD has disappeared from member clusters;
• Cluster.Status.APIEnablements no longer reports FlinkDeployment as enabled.

This follows the same synchronization-barrier idea as #7692, but for the FlinkDeployment CRD/APIEnablements cleanup path.

Anything else we need to know:

I tested a small e2e-only candidate fix that waits for member-cluster CRD disappearance and for Cluster.Status.APIEnablements to stop reporting FlinkDeployment during cleanup.

Validation branch:

• Branch: ranxi2001/karmada@test/estimator-flink-crd-flake
• Commit: f2e7c434bad6d4a970265af79a157afb61e6182e

Local validation:

• git diff --check
• go test ./test/e2e/suites/base -run '^$' -count=0

Fork push CI:

• CI run 28774018375: passed
• Jobs: lint, codegen, compile, unit test, e2e v1.34.0, e2e v1.35.0, and e2e v1.36.1 all passed.

I can send this as a focused e2e flake fix if this issue direction makes sense.

[ranxi2001]

/kind flake

[ranxi2001]

The diagram above shows why this failure is a cleanup race rather than a scheduler regression.

During a normal test run, the FlinkDeployment CRD is created in the Karmada control plane and propagated to the member clusters. The member cluster API information is then collected into Cluster.Status.APIEnablements, which is used by the scheduler's APIEnablement plugin to decide whether a cluster supports flink.apache.org/v1beta1/FlinkDeployment.

The problem is that the current cleanup only waits for the source CRD to disappear from the Karmada control plane.

However, CRD removal is asynchronous:

1. The source CRD disappears from the Karmada control plane.
2. The propagated CRD is removed from each member cluster.
3. Cluster.Status.APIEnablements is updated to remove the FlinkDeployment API.

The cleanup can return after step 1, while steps 2 and 3 are still in progress.

If the next test starts during this window, WaitCRDPresentOnClusters() may see the stale APIEnablement entry left by the previous test and incorrectly assume that the newly created CRD has already been propagated.

The scheduler then checks the actual member cluster API and finds that the FlinkDeployment API is not available yet. As a result, the ResourceBinding remains unscheduled and the test eventually times out.

The proposed fix adds a complete cleanup synchronization barrier. Cleanup should return only after:

• the source CRD is gone from the Karmada control plane;
• the propagated CRD is gone from all member clusters;
• Cluster.Status.APIEnablements no longer reports the FlinkDeployment API.

This ensures that the next test always starts from a clean and fully converged state.