Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions charts/kubex-automation-engine/BREAKING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Breaking Changes

## 2026-07-20 - GPU enablement defaults and experimental contract

GPU request actions now default to disabled. The following fields changed from `true` to `false`:

- `spec.enablement.gpu.requests.downsize`
- `spec.enablement.gpu.requests.upsize`
- `spec.enablement.gpu.requests.setFromUnspecified`

The GPU/KAI experimental contract changed to `v1alpha1-2026-07`. The previous contract is no longer accepted.

Affected resources:

- `AutomationStrategy`
- `ClusterAutomationStrategy`
- `GpuRebalancingPolicy`
- `ClusterGpuRebalancingPolicy`
- `GpuConsolidationPolicy`

### Migration

After upgrading the CRDs:

1. Replace the existing `spec.experimental.gpuKaiContract` value with `v1alpha1-2026-07` in every affected resource.
2. For `AutomationStrategy` and `ClusterAutomationStrategy`, explicitly set each desired GPU action under `spec.enablement.gpu.requests` to `true`. The GPU policy kinds only require the contract update in step 1.
3. Reapply affected resources.
15 changes: 15 additions & 0 deletions charts/kubex-automation-engine/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,21 @@

All notable changes to the Kubex Automation Engine Helm chart will be documented in this file.

## [1.8.0] - 2026-07-22

### Breaking
- **[GPU enablement defaults and experimental contract](./BREAKING.md#2026-07-20---gpu-enablement-defaults-and-experimental-contract)**: GPU request actions now default to disabled. `spec.enablement.gpu.requests.downsize`, `.upsize`, and `.setFromUnspecified` changed from `true` to `false`. The GPU/KAI experimental contract changed to `v1alpha1-2026-07`; the previous contract is no longer accepted. Affects `AutomationStrategy`, `ClusterAutomationStrategy`, `GpuRebalancingPolicy`, `ClusterGpuRebalancingPolicy`, and `GpuConsolidationPolicy`.
- After upgrading the CRDs, update `spec.experimental.gpuKaiContract` from `v1alpha1-2026-04` to `v1alpha1-2026-07` in every affected resource.
- For `AutomationStrategy`/`ClusterAutomationStrategy`, explicitly set each desired GPU action under `spec.enablement.gpu.requests` to `true` if you rely on GPU request resizing. GPU policy kinds only need the contract update.
- Reapply affected resources after making these changes.

### Added
- Rollback monitoring now reopens automatically when a newer replacement pod appears carrying the same recommendation and matching resources, so a pod recreated by eviction, in-place resize, or a manual restart resumes being tracked instead of starting a brand-new monitoring turn.
- `RollbackPolicy`/`ClusterRollbackPolicy` gain `spec.enableMonitoringReopen` (default `true`) to opt out of this reopen behavior for a given policy.
- Manifest-level resource sizing is now reported to the Kubex backend, giving visibility into a workload's declared (manifest) resources alongside its live/adopted values.

---

## [1.7.0] - 2026-07-14

### Added
Expand Down
2 changes: 1 addition & 1 deletion charts/kubex-automation-engine/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: kubex-automation-engine
description: A Helm chart for deploying kubex-automation-engine for automated workload rightsizing and resource optimization
type: application
version: 1.7.0
version: 1.8.0
icon: https://kubex.ai/wp-content/uploads/kubex-logo-landscape.svg
keywords:
- kubex
Expand Down
11 changes: 9 additions & 2 deletions charts/kubex-automation-engine/docs/Automation-Strategies.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Automation Strategies

> Experimental: GPU/KAI-related fields in this resource are subject to breaking changes. When using them, set `spec.experimental.gpuKaiContract: v1alpha1-2026-04`.
> Experimental: GPU/KAI-related fields in this resource are subject to breaking changes. When using them, set `spec.experimental.gpuKaiContract: v1alpha1-2026-07`.

`AutomationStrategy` defines how resizing is allowed to happen within a namespace.

Expand Down Expand Up @@ -58,6 +58,13 @@ Usage-level `floor` and `ceiling` values apply to all containers by default. Add
| `spec.enablement.memory.limits.ceiling` | none | Maximum memory limit target. |
| `spec.enablement.memory.limits.containers.<name>.floor` | none | Container-specific memory limit minimum that overrides the usage-level floor for that container only. |
| `spec.enablement.memory.limits.containers.<name>.ceiling` | none | Container-specific memory limit maximum that overrides the usage-level ceiling for that container only. |
| `spec.enablement.gpu.requests.downsize` | `false` | EXPERIMENTAL. Allows reducing GPU requests. |
| `spec.enablement.gpu.requests.upsize` | `false` | EXPERIMENTAL. Allows increasing GPU requests. |
| `spec.enablement.gpu.requests.setFromUnspecified` | `false` | EXPERIMENTAL. Allows setting GPU requests when currently unset. |
| `spec.enablement.gpu.requests.floor` | none | EXPERIMENTAL. Minimum GPU request target. |
| `spec.enablement.gpu.requests.ceiling` | none | EXPERIMENTAL. Maximum GPU request target. |
| `spec.enablement.gpu.requests.containers.<name>.floor` | none | EXPERIMENTAL. Container-specific GPU request minimum that overrides the usage-level floor for that container only. |
| `spec.enablement.gpu.requests.containers.<name>.ceiling` | none | EXPERIMENTAL. Container-specific GPU request maximum that overrides the usage-level ceiling for that container only. |
| `spec.kai.queue` | `kubex-unlimited-gpu-queue` | EXPERIMENTAL. Default KAI queue label applied for KAI GPU admission mutation. |
| `spec.kai.setQueueWhenSpecified` | `false` | EXPERIMENTAL. Allows strategy queue value to overwrite an existing `kai.scheduler/queue` label. |
| `spec.kai.vllm` | none | EXPERIMENTAL. Enables admission-time vLLM tuning for KAI GPU-sharing workloads. Requires `spec.experimental.gpuKaiContract`. |
Expand Down Expand Up @@ -180,7 +187,7 @@ Example:
```yaml
spec:
experimental:
gpuKaiContract: v1alpha1-2026-04
gpuKaiContract: v1alpha1-2026-07
kai:
vllm:
gpuMemoryUtilizationBufferPercent: 10
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Cluster Automation Strategies

> Experimental: GPU/KAI-related fields in this resource are subject to breaking changes. When using them, set `spec.experimental.gpuKaiContract: v1alpha1-2026-04`.
> Experimental: GPU/KAI-related fields in this resource are subject to breaking changes. When using them, set `spec.experimental.gpuKaiContract: v1alpha1-2026-07`.

`ClusterAutomationStrategy` defines how resizing is allowed to happen for cluster-scoped policy flows.

Expand Down Expand Up @@ -52,6 +52,13 @@ Usage-level `floor` and `ceiling` values apply to all containers by default. Add
| `spec.enablement.memory.limits.ceiling` | none | Maximum memory limit target. |
| `spec.enablement.memory.limits.containers.<name>.floor` | none | Container-specific memory limit minimum that overrides the usage-level floor for that container only. |
| `spec.enablement.memory.limits.containers.<name>.ceiling` | none | Container-specific memory limit maximum that overrides the usage-level ceiling for that container only. |
| `spec.enablement.gpu.requests.downsize` | `false` | EXPERIMENTAL. Allows reducing GPU requests. |
| `spec.enablement.gpu.requests.upsize` | `false` | EXPERIMENTAL. Allows increasing GPU requests. |
| `spec.enablement.gpu.requests.setFromUnspecified` | `false` | EXPERIMENTAL. Allows setting GPU requests when currently unset. |
| `spec.enablement.gpu.requests.floor` | none | EXPERIMENTAL. Minimum GPU request target. |
| `spec.enablement.gpu.requests.ceiling` | none | EXPERIMENTAL. Maximum GPU request target. |
| `spec.enablement.gpu.requests.containers.<name>.floor` | none | EXPERIMENTAL. Container-specific GPU request minimum that overrides the usage-level floor for that container only. |
| `spec.enablement.gpu.requests.containers.<name>.ceiling` | none | EXPERIMENTAL. Container-specific GPU request maximum that overrides the usage-level ceiling for that container only. |
| `spec.inPlaceResize.enabled` | `true` | Enables the in-place resize execution path. |
| `spec.inPlaceResize.containerRestart` | `false` | Allows in-place resize operations that require container restart. |
| `spec.podEviction.enabled` | `true` | Enables eviction-based fallback. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,7 @@ Use [Global Configuration Reference](./Global-Configuration.md) for the CR field
| `globalConfiguration.webhookProbe.resources` | `{}` | Resource requests and limits for the dry-run webhook probe container |
| `globalConfiguration.webhookProbe.podSecurityContext` | `{}` | Pod security context for the dry-run webhook probe Pod |
| `globalConfiguration.webhookProbe.securityContext` | `{}` | Container security context for the dry-run webhook probe container |
| `experimental.gpuKaiContract` | `v1alpha1-2026-04` | Required acknowledgement token for experimental GPU/KAI CR fields rendered by the chart |
| `experimental.gpuKaiContract` | `v1alpha1-2026-07` | Required acknowledgement token for experimental GPU/KAI CR fields rendered by the chart |

Webhook behavior notes:

Expand Down
10 changes: 5 additions & 5 deletions charts/kubex-automation-engine/docs/GPU-Sharing-with-KAI.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ This guide shows how to configure GPU sharing with KAI and Kubex Automation Engi
Tested with KAI `v0.12.16`.

> [!IMPORTANT]
> GPU/KAI fields and related custom resources are experimental and subject to breaking changes. Set `spec.experimental.gpuKaiContract: v1alpha1-2026-04` on GPU/KAI resources.
> GPU/KAI fields and related custom resources are experimental and subject to breaking changes. Set `spec.experimental.gpuKaiContract: v1alpha1-2026-07` on GPU/KAI resources.

## Prerequisites

Expand Down Expand Up @@ -84,7 +84,7 @@ metadata:
spec:
experimental:
# Required contract version for the current experimental GPU/KAI integration.
gpuKaiContract: v1alpha1-2026-04
gpuKaiContract: v1alpha1-2026-07
enablement:
# Convert matching workloads to the KAI GPU-sharing flow.
overrideScheduler: "kai"
Expand Down Expand Up @@ -137,7 +137,7 @@ metadata:
spec:
experimental:
# Required contract version for the current experimental GPU/KAI integration.
gpuKaiContract: v1alpha1-2026-04
gpuKaiContract: v1alpha1-2026-07
scope:
namespaceSelector:
operator: In
Expand Down Expand Up @@ -229,7 +229,7 @@ Example math:

Behavior:

- requires experimental contract: `spec.experimental.gpuKaiContract: v1alpha1-2026-04`
- requires experimental contract: `spec.experimental.gpuKaiContract: v1alpha1-2026-07`
- only runs for KAI GPU request admission mutation
- only mutates detected vLLM containers
- updates existing `--gpu-memory-utilization=<value>`
Expand Down Expand Up @@ -266,7 +266,7 @@ metadata:
name: gpu-consolidation-pool-a
spec:
experimental:
gpuKaiContract: v1alpha1-2026-04
gpuKaiContract: v1alpha1-2026-07
nodeSelector:
matchLabels:
cloud.google.com/gke-accelerator: nvidia-tesla-t4
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Global Configuration

> Experimental: GPU/KAI-related fields in this resource are subject to breaking changes. When using them, set `spec.experimental.gpuKaiContract: v1alpha1-2026-04`.
> Experimental: GPU/KAI-related fields in this resource are subject to breaking changes. When using them, set `spec.experimental.gpuKaiContract: v1alpha1-2026-07`.

`GlobalConfiguration` defines cluster-wide controller behavior that applies across strategies and policies.

Expand Down
2 changes: 2 additions & 0 deletions charts/kubex-automation-engine/docs/Rollback-Policies.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Use them when you want the controller to monitor workloads after a resize is app
| `spec.backoff.multiplyByTurn` | none (required) | Scales the base duration by the current turn number. Example: with `timePeriod: 1m` and `multiplyByTurn: 2`, turn 1 waits 2m and turn 2 waits 4m. |
| `spec.backoff.maxAttempts` | none (required) | Maximum number of turns before the controller marks the rollback as permanently failed. |
| `spec.weight` | `0` | Higher weight wins when multiple rollback policies match. When weights are equal, older policies win. |
| `spec.enableMonitoringReopen` | `true` | Set to `false` to stop reopening a completed (`monitoringSucceeded`) turn when a newer replacement pod appears carrying the same recommendation and resources. When disabled, only a genuinely new recommendation starts a fresh monitoring turn for an owner that has already succeeded. |

## `ClusterRollbackPolicy.spec`

Expand All @@ -43,6 +44,7 @@ Use them when you want the controller to monitor workloads after a resize is app
| `spec.backoff.multiplyByTurn` | none (required) | Scales the base duration by the current turn number. Example: with `timePeriod: 1m` and `multiplyByTurn: 2`, turn 1 waits 2m and turn 2 waits 4m. |
| `spec.backoff.maxAttempts` | none (required) | Maximum number of turns before the controller marks the rollback as permanently failed. |
| `spec.weight` | `0` | Higher weight wins when multiple rollback policies match. When weights are equal, older policies win. |
| `spec.enableMonitoringReopen` | `true` | Set to `false` to stop reopening a completed (`monitoringSucceeded`) turn when a newer replacement pod appears carrying the same recommendation and resources. When disabled, only a genuinely new recommendation starts a fresh monitoring turn for an owner that has already succeeded. |

## Example: Namespaced Rollback Policy

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# GPU Consolidation Policy

> Experimental: GPU/KAI fields and related custom resources are subject to breaking changes. Set `spec.experimental.gpuKaiContract: v1alpha1-2026-04`.
> Experimental: GPU/KAI fields and related custom resources are subject to breaking changes. Set `spec.experimental.gpuKaiContract: v1alpha1-2026-07`.

`GpuConsolidationPolicy` is a cluster-scoped controller that looks at scheduled pods carrying the `gpu-fraction` annotation and tries to consolidate them off an underutilized node.

Expand Down Expand Up @@ -30,7 +30,7 @@ metadata:
name: gpu-consolidation-pool-a
spec:
experimental:
gpuKaiContract: v1alpha1-2026-04
gpuKaiContract: v1alpha1-2026-07
nodeSelector:
matchLabels:
kubex.ai/gpu-pool: pool-a
Expand All @@ -47,7 +47,7 @@ metadata:
name: gpu-consolidation-l40s
spec:
experimental:
gpuKaiContract: v1alpha1-2026-04
gpuKaiContract: v1alpha1-2026-07
nodeSelector:
matchExpressions:
- key: kubex.ai/gpu-pool
Expand All @@ -67,7 +67,7 @@ metadata:
name: gpu-consolidation-h100
spec:
experimental:
gpuKaiContract: v1alpha1-2026-04
gpuKaiContract: v1alpha1-2026-07
nodeSelector:
matchLabels:
kubex.ai/gpu-pool: training-h100
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# GPU Rebalancing Policies

> Experimental: GPU/KAI fields and related custom resources are subject to breaking changes. Set `spec.experimental.gpuKaiContract: v1alpha1-2026-04`.
> Experimental: GPU/KAI fields and related custom resources are subject to breaking changes. Set `spec.experimental.gpuKaiContract: v1alpha1-2026-07`.

`GpuRebalancingPolicy` and `ClusterGpuRebalancingPolicy` emit GPU rebalancing recommendations (upsize and downsize) from Prometheus utilization.

Expand Down Expand Up @@ -37,7 +37,7 @@ metadata:
namespace: default
spec:
experimental:
gpuKaiContract: v1alpha1-2026-04
gpuKaiContract: v1alpha1-2026-07
scope:
labelSelector:
matchLabels:
Expand Down Expand Up @@ -89,7 +89,7 @@ metadata:
name: gpu-rebalance-cluster
spec:
experimental:
gpuKaiContract: v1alpha1-2026-04
gpuKaiContract: v1alpha1-2026-07
scope:
namespaceSelector:
operator: In
Expand Down
8 changes: 4 additions & 4 deletions charts/kubex-automation-engine/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ crdCheck:
skip: false
# Minimum version of the kubex-crds chart required before this chart can be installed/upgraded.
# Bump this whenever a new CRD field or schema change is required.
minCRDsHelmVersion: "1.7.0"
minCRDsHelmVersion: "1.8.0"
# -- Whether to create the kubex-gateway-config Secret automatically
createSecrets: true

Expand Down Expand Up @@ -54,14 +54,14 @@ image:
# -- Image pull policy
pullPolicy: IfNotPresent
# -- Image tag
tag: "1.7.0"
tag: "1.8.0"

cleanup:
image:
# -- Image repository for the pre-delete cleanup job.
repository: "densify/kubex-automation-cleanup"
# -- Image tag for the pre-delete cleanup job. Defaults to the controller image tag.
tag: "1.7.0"
tag: "1.8.0"
# -- Image pull policy for the pre-delete cleanup job.
pullPolicy: "IfNotPresent"
# -- Optional pod security context for the pre-delete cleanup job
Expand Down Expand Up @@ -323,7 +323,7 @@ crds: {}
# Experimental feature acknowledgements for unstable APIs.
experimental:
# -- Required acknowledgement token for GPU/KAI experimental fields.
gpuKaiContract: "v1alpha1-2026-04"
gpuKaiContract: "v1alpha1-2026-07"

# Global configuration defaults
# These will be used to create the default GlobalConfiguration CR
Expand Down
2 changes: 1 addition & 1 deletion charts/kubex-crds/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: kubex-crds
description: CRDs for Kubex Automation Engine
icon: https://kubex.ai/wp-content/uploads/kubex-logo-landscape.svg
version: 1.7.0
version: 1.8.0
keywords:
- crd
- kubex
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -180,22 +180,22 @@ spec:
Any bound omitted for a specific container falls back to the usage-level floor/ceiling.
type: object
downsize:
default: true
description: downsize controls whether resource downsizing
default: false
description: downsize controls whether GPU request downsizing
actions are permitted.
type: boolean
floor:
description: floor is the minimum allowed resource quantity
for this usage.
type: string
setFromUnspecified:
default: true
default: false
description: setFromUnspecified controls whether an unspecified
resource value may be set.
GPU request may be set.
type: boolean
upsize:
default: true
description: upsize controls whether resource upsizing
default: false
description: upsize controls whether GPU request upsizing
actions are permitted.
type: boolean
type: object
Expand Down Expand Up @@ -330,7 +330,7 @@ spec:
EXPERIMENTAL: this acknowledgement value is required for unstable GPU/KAI behavior.
enum:
- none
- v1alpha1-2026-04
- v1alpha1-2026-07
type: string
required:
- gpuKaiContract
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -180,22 +180,22 @@ spec:
Any bound omitted for a specific container falls back to the usage-level floor/ceiling.
type: object
downsize:
default: true
description: downsize controls whether resource downsizing
default: false
description: downsize controls whether GPU request downsizing
actions are permitted.
type: boolean
floor:
description: floor is the minimum allowed resource quantity
for this usage.
type: string
setFromUnspecified:
default: true
default: false
description: setFromUnspecified controls whether an unspecified
resource value may be set.
GPU request may be set.
type: boolean
upsize:
default: true
description: upsize controls whether resource upsizing
default: false
description: upsize controls whether GPU request upsizing
actions are permitted.
type: boolean
type: object
Expand Down Expand Up @@ -330,7 +330,7 @@ spec:
EXPERIMENTAL: this acknowledgement value is required for unstable GPU/KAI behavior.
enum:
- none
- v1alpha1-2026-04
- v1alpha1-2026-07
type: string
required:
- gpuKaiContract
Expand Down
Loading