[FLINK-37448] Report actual TaskManager count in status.taskManager.replicas#1146
Open
vsantwana wants to merge 2 commits into
Open
[FLINK-37448] Report actual TaskManager count in status.taskManager.replicas#1146vsantwana wants to merge 2 commits into
vsantwana wants to merge 2 commits into
Conversation
vsantwana
marked this pull request as draft
June 23, 2026 04:48
…eplicas Previously status.taskManager.replicas was computed from the spec at reconcile time (ceil(parallelism / slots)), which is inaccurate for deployments where the TaskManager count fluctuates dynamically (e.g. standalone reactive mode). This now populates the field during observation from the live Flink REST API (actual registered TaskManagers) for both application and session deployments. A new FlinkService#getTaskManagerReplicas method exposes the count, the observer sets the status, and the reconciler only sets the label selector (replicas stay 0 until the cluster is observed). Note: this changes the semantics of status.taskManager.replicas from a spec-derived value to the actual cluster state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vsantwana
marked this pull request as ready for review
June 23, 2026 09:41
Dennis-Mircea
suggested changes
Jul 2, 2026
Dennis-Mircea
left a comment
Contributor
There was a problem hiding this comment.
This PR needs more test coverage as well. The new tests covers only the application observer. The change deliberately covers session clusters too, but there is no session-path test. Consider tightening the test coverage as well.
…TM replicas across reconciles Addresses PR apache#1146 review feedback: - Fold FlinkService#getTaskManagerReplicas into getClusterInfo, which now returns a small ClusterInfo record (info map + observed replica count). Eliminates the duplicate /taskmanagers REST call from the observer path. - ReconciliationUtils#getTaskManagerInfo now preserves the last observed replica count when transitioning through a RUNNING reconcile, so the scale subresource no longer transiently reports 0 after upgrades or spec changes; observation refreshes the count afterwards. - Trim verbose comments in the observer, ReconciliationUtils and the controller test. - Add ReconciliationUtilsTest#taskManagerReplicasPreservedAcrossReconciles.
vsantwana
force-pushed
the
FLINK-37448-tm-replicas
branch
from
July 15, 2026 09:30
9535ecb to
23775ea
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
status.taskManager.replicaspreviously held a value computed from the spec at reconcile time (ceil(parallelism / slots)). This is inaccurate for deployments where the number of TaskManagers fluctuates dynamically rather than being fixed by configuration - most notably standalone clusters running in reactive mode.This change makes the field report the actual number of TaskManagers registered with the running Flink cluster, sourced from the Flink REST API (
/taskmanagers), for both application and session deployments.Brief change log
FlinkService#getTaskManagerReplicas(Configuration), implemented inAbstractFlinkServicevia the existing/taskmanagersREST call;getClusterInfonow routes through it (single source of truth).AbstractFlinkDeploymentObserver#observeClusterInfonow setsstatus.taskManager(label selector + actual replica count) whenever the JobManager is ready, covering both application and session clusters.ReconciliationUtils#getTaskManagerInfono longer derives the count from the spec; at reconcile time it only sets the label selector (replicas stay0until the cluster is observed) and still clears the info when not running.TaskManagerInfo.replicasJavaDoc and the generated CRD eference docs to reflect the new semantics.Note: this changes the semantics of
status.taskManager.replicasfrom a spec-derived value to the observed cluster state.Verifying this change
This change added tests and can be verified as follows:
ApplicationObserverTest#observeReportsActualTaskManagerReplicasverifies thestatus reflects the cluster's actual TaskManager count (independent of the
spec-derived value) and tracks changes across observations.
TestingFlinkServicegained an overridabletaskManagerReplicasstub(defaults to a converged cluster, i.e. the spec-derived count).
Does this pull request potentially affect one of the following parts:
CustomResourceDescriptors: no (the field already exists; only its reported value changes)Documentation