Skip to content

[FLINK-38787][Kubernetes Operator] Introduce Blue Green Deployment Metrics #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
james-kan-shopify wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[FLINK-38787][Kubernetes Operator] Introduce Blue Green Deployment Metrics #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 38 additions & 7 deletions docs/content/docs/operations/metrics-logging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,20 @@ Different operator metrics can be turned on/off individually using the configura
### Flink Resource Metrics
The Operator gathers aggregates metrics about managed resources.

| Scope | Metrics | Description | Type |
|--------------------|-------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
| Scope | Metrics | Description | Type |
|--------------------|-------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
| Namespace | FlinkDeployment/FlinkSessionJob.Count | Number of managed resources per namespace | Gauge |
| Namespace | FlinkDeployment.ResourceUsage.Cpu/Memory | Total resources used per namespace | Gauge |
| Namespace | FlinkDeployment.JmDeploymentStatus.<Status>.Count | Number of managed FlinkDeployment resources per <Status> per namespace. <Status> can take values from: READY, DEPLOYED_NOT_READY, DEPLOYING, MISSING, ERROR | Gauge |
| Namespace | FlinkDeployment.FlinkVersion.<FlinkVersion>.Count | Number of managed FlinkDeployment resources per <FlinkVersion> per namespace. <FlinkVersion> is retrieved via REST API from Flink JM. | Gauge |
| Namespace | FlinkDeployment.ResourceUsage.Cpu/Memory | Total resources used per namespace | Gauge |
| Namespace | FlinkDeployment.JmDeploymentStatus.<Status>.Count | Number of managed FlinkDeployment resources per <Status> per namespace. <Status> can take values from: READY, DEPLOYED_NOT_READY, DEPLOYING, MISSING, ERROR | Gauge |
| Namespace | FlinkDeployment.FlinkVersion.<FlinkVersion>.Count | Number of managed FlinkDeployment resources per <FlinkVersion> per namespace. <FlinkVersion> is retrieved via REST API from Flink JM. | Gauge |
| Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.State.<State>.Count | Number of managed resources currently in state <State> per namespace. <State> can take values from: CREATED, SUSPENDED, UPGRADING, DEPLOYED, STABLE, ROLLING_BACK, ROLLED_BACK, FAILED | Gauge |
| System/Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.State.<State>.TimeSeconds | Time spent in state <State> for a given resource. <State> can take values from: CREATED, SUSPENDED, UPGRADING, DEPLOYED, STABLE, ROLLING_BACK, ROLLED_BACK, FAILED | Histogram |
| System/Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.Transition.<Transition>.TimeSeconds | Time statistics for selected lifecycle state transitions. <Transition> can take values from: Resume, Upgrade, Suspend, Stabilization, Rollback, Submission | Histogram |
| System/Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.State.<State>.TimeSeconds | Time spent in state <State> for a given resource. <State> can take values from: CREATED, SUSPENDED, UPGRADING, DEPLOYED, STABLE, ROLLING_BACK, ROLLED_BACK, FAILED | Histogram |
| System/Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.Transition.<Transition>.TimeSeconds | Time statistics for selected lifecycle state transitions. <Transition> can take values from: Resume, Upgrade, Suspend, Stabilization, Rollback, Submission | Histogram |
| Namespace | FlinkBlueGreenDeployment.BlueGreenState.<State>.Count | Number of managed FlinkBlueGreenDeployment resources currently in state <State> per namespace. <State> can take values from: INITIALIZING_BLUE, ACTIVE_BLUE, SAVEPOINTING_BLUE, TRANSITIONING_TO_GREEN, ACTIVE_GREEN, SAVEPOINTING_GREEN, TRANSITIONING_TO_BLUE | Gauge |
| Namespace | FlinkBlueGreenDeployment.JobStatus.<Status>.Count | Number of managed FlinkBlueGreenDeployment resources currently in JobStatus <Status> per namespace. <Status> can take values from: RUNNING, FAILING, SUSPENDED, FAILED, RECONCILING | Gauge |
| Namespace | FlinkBlueGreenDeployment.Failures | Historical count of failure events (transitions to FAILING state) for all FlinkBlueGreenDeployment resources in the namespace. Counter increments on each transition to FAILING and never decrements. | Counter |
| System/Namespace | FlinkBlueGreenDeployment.Lifecycle.State.<State>.TimeSeconds | Time spent in state <State> for a given FlinkBlueGreenDeployment resource. <State> values same as above. | Histogram |
| System/Namespace | FlinkBlueGreenDeployment.Lifecycle.Transition.<Transition>.TimeSeconds | Time statistics for blue-green lifecycle state transitions. <Transition> can take values from: InitialDeployment, BlueToGreen, GreenToBlue | Histogram |

#### Lifecycle metrics

Expand All @@ -60,6 +65,32 @@ In addition to the simple counts we further track a few selected state transitio
- Rollback : Time from deployed to rolled_back state if the resource was rolled back
- Submission: Flink resource submission time

#### FlinkBlueGreenDeployment Lifecycle metrics

FlinkBlueGreenDeployment resources have their own lifecycle states that track the blue-green deployment process. The operator monitors the following transitions:

- InitialDeployment : Time from leaving INITIALIZING_BLUE to reaching ACTIVE_BLUE (first deployment)
- BlueToGreen : Time from leaving ACTIVE_BLUE to reaching ACTIVE_GREEN (actual transition duration)
- GreenToBlue : Time from leaving ACTIVE_GREEN to reaching ACTIVE_BLUE (actual transition duration)

Transition metrics measure the actual transition time - from when the deployment leaves the source stable state until it reaches the target stable state. This excludes time spent running stably before the transition was initiated.

State time metrics track how long a resource spends in each state (ACTIVE_BLUE, SAVEPOINTING_BLUE, TRANSITIONING_TO_GREEN, etc.), which helps identify bottlenecks in the deployment pipeline.

#### FlinkBlueGreenDeployment JobStatus Tracking

In addition to BlueGreenState tracking, FlinkBlueGreenDeployment resources also expose JobStatus metrics that track the Flink job state:

**JobStatus Gauges**: Current count of deployments per JobStatus (RUNNING, FAILING, SUSPENDED, etc.) - these gauges go up and down as deployments transition between states.

**Failures Counter**: Historical count that increments each time a deployment transitions TO the FAILING state. This counter:
- Never decrements (accumulates total failures since operator start)
- Increments on each new transition to FAILING (even if the same deployment fails multiple times)
- Persists across deployment recoveries (provides historical failure tracking)
- Useful for calculating failure rates and setting up alerts

Example: A deployment goes RUNNING → FAILING → RUNNING → FAILING. The FAILING gauge shows 0 or 1 (current state), while the Failures counter shows 2 (historical events).

### Kubernetes Client Metrics

The Operator gathers various metrics related to Kubernetes API server access.
Expand Down
7 changes: 6 additions & 1 deletion ...kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/FlinkOperator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,12 @@ void registerSnapshotController() {

@VisibleForTesting
void registerBlueGreenController() {
var controller = new FlinkBlueGreenDeploymentController(ctxFactory);
var metricManager =
MetricManager.createFlinkBlueGreenDeploymentMetricManager(baseConfig, metricGroup);
var statusRecorder =
StatusRecorder.createForFlinkBlueGreenDeployment(client, metricManager, listeners);
var controller = new FlinkBlueGreenDeploymentController(ctxFactory, statusRecorder);

registeredControllers.add(operator.register(controller, this::overrideControllerConfigs));
}

Expand Down
22 changes: 17 additions & 5 deletions ...a/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import org.apache.flink.kubernetes.operator.controller.bluegreen.BlueGreenStateHandlerRegistry;
import org.apache.flink.kubernetes.operator.controller.bluegreen.handlers.BlueGreenStateHandler;
import org.apache.flink.kubernetes.operator.service.FlinkResourceContextFactory;
import org.apache.flink.kubernetes.operator.utils.StatusRecorder;

import io.javaoperatorsdk.operator.api.config.informer.InformerEventSourceConfiguration;
import io.javaoperatorsdk.operator.api.reconciler.Context;
Expand Down Expand Up @@ -70,10 +71,16 @@ public class FlinkBlueGreenDeploymentController implements Reconciler<FlinkBlueG

private final FlinkResourceContextFactory ctxFactory;
private final BlueGreenStateHandlerRegistry handlerRegistry;
private final StatusRecorder<FlinkBlueGreenDeployment, FlinkBlueGreenDeploymentStatus>
statusRecorder;

public FlinkBlueGreenDeploymentController(FlinkResourceContextFactory ctxFactory) {
public FlinkBlueGreenDeploymentController(
FlinkResourceContextFactory ctxFactory,
StatusRecorder<FlinkBlueGreenDeployment, FlinkBlueGreenDeploymentStatus>
statusRecorder) {
this.ctxFactory = ctxFactory;
this.handlerRegistry = new BlueGreenStateHandlerRegistry();
this.statusRecorder = statusRecorder;
}

@Override
Expand Down Expand Up @@ -110,9 +117,12 @@ public UpdateControl<FlinkBlueGreenDeployment> reconcile(
josdkContext,
null,
ctxFactory);
return BlueGreenDeploymentService.patchStatusUpdateControl(
context, INITIALIZING_BLUE, null, null)
.rescheduleAfter(0);
UpdateControl<FlinkBlueGreenDeployment> updateControl =
BlueGreenDeploymentService.patchStatusUpdateControl(
context, INITIALIZING_BLUE, null, null)
.rescheduleAfter(0);
statusRecorder.patchAndCacheStatus(bgDeployment, josdkContext.getClient());
return updateControl;
} else {
FlinkBlueGreenDeploymentState currentState = deploymentStatus.getBlueGreenState();
var context =
Expand All @@ -132,7 +142,9 @@ public UpdateControl<FlinkBlueGreenDeployment> reconcile(
context.getDeploymentName());

BlueGreenStateHandler handler = handlerRegistry.getHandler(currentState);
return handler.handle(context);
UpdateControl<FlinkBlueGreenDeployment> updateControl = handler.handle(context);
statusRecorder.patchAndCacheStatus(bgDeployment, josdkContext.getClient());
return updateControl;
}
}

Expand Down
Loading