From 2d17b73f8fff9a6f740d1143ff653f7b2be97062 Mon Sep 17 00:00:00 2001 From: Renuka Fernando Date: Sat, 13 Jun 2026 12:59:02 +0530 Subject: [PATCH 1/2] feat(helm): make gateway chart pod/service metadata and scheduling configurable Add pod-, service-, and resource-level customization knobs to the gateway-helm-chart and fix two label/annotation rendering bugs. Bugs fixed: - Duplicate label keys when the same key appears in commonLabels and a component's deployment.labels/podLabels (rejected by strict parsers and GitOps tools). Labels are now merged with a clear precedence. - Inconsistent annotation precedence: per-resource annotations now win over commonAnnotations consistently at both resource and pod level. New configuration (controller + gateway-runtime): - topologySpreadConstraints, deployment strategy, terminationGracePeriodSeconds, hostAliases, dnsPolicy/dnsConfig, pod-level automountServiceAccountToken - optional startupProbe (controller template + documented runtime value) - service tunables: clusterIP, externalTrafficPolicy, loadBalancerClass, loadBalancerSourceRanges, ipFamilyPolicy/ipFamilies, static nodePorts (type-gated to capable Service types) - app.kubernetes.io/name added to the standard label set - PVC labels/annotations (e.g. helm.sh/resource-policy: keep) - commonLabels/commonAnnotations now applied to every rendered resource (ServiceAccount, HPAs, PDBs, PVC, Certificate, Issuer, ConfigMaps) Validated with helm lint + helm template (default, values-local, and a conflict-heavy values file): no duplicate keys, correct precedence, selector labels protected from override, type-gated service fields, value coercion. --- kubernetes/helm/gateway-helm-chart/README.md | 7 +- .../gateway-helm-chart/templates/_helpers.tpl | 84 ++++++++++-- .../gateway/controller/certificate.yaml | 4 + .../gateway/controller/deployment.yaml | 46 +++++-- .../templates/gateway/controller/hpa.yaml | 4 + .../templates/gateway/controller/issuer.yaml | 4 + .../templates/gateway/controller/pdb.yaml | 4 + .../templates/gateway/controller/pvc.yaml | 6 +- .../templates/gateway/controller/service.yaml | 44 +++++- .../templates/gateway/gateway-config.yaml | 10 +- .../gateway/gateway-runtime/deployment.yaml | 42 ++++-- .../gateway/gateway-runtime/hpa.yaml | 4 + .../llm-pricing-configmap.yaml | 4 + .../gateway/gateway-runtime/pdb.yaml | 4 + .../gateway/gateway-runtime/service.yaml | 63 +++++++-- .../templates/serviceaccount.yaml | 5 +- .../helm/gateway-helm-chart/values.yaml | 125 ++++++++++++++++++ 17 files changed, 396 insertions(+), 64 deletions(-) diff --git a/kubernetes/helm/gateway-helm-chart/README.md b/kubernetes/helm/gateway-helm-chart/README.md index b6bf8daf1b..1f6485ca91 100644 --- a/kubernetes/helm/gateway-helm-chart/README.md +++ b/kubernetes/helm/gateway-helm-chart/README.md @@ -123,10 +123,11 @@ Each major workload (controller, gateway-runtime) lives in its own nested templa All configurable values are documented in `values.yaml`. Component blocks are fully namespaced so overrides are intuitive: - `gateway.controller.image` / `gateway.gatewayRuntime.image` – container image metadata and pull policies. -- `gateway..deployment.*` – pod-level knobs (replicas, probes, affinities, env overrides, extra volumes) and enable/disable switches. -- `gateway..service.*` – service type/ports plus optional annotations and labels. +- `gateway..deployment.*` – pod-level knobs (replicas, probes incl. optional `startupProbe`, scheduling via `nodeSelector`/`tolerations`/`affinity`/`topologySpreadConstraints`, update `strategy`, `terminationGracePeriodSeconds`, `hostAliases`, `dnsPolicy`/`dnsConfig`, `automountServiceAccountToken`, env overrides, extra volumes) and enable/disable switches. +- `gateway..service.*` – service type/ports plus optional annotations and labels, and network tuning (`clusterIP`, `externalTrafficPolicy`, `loadBalancerClass`, `loadBalancerSourceRanges`, `ipFamilyPolicy`/`ipFamilies`, static `nodePorts.*`). - `gateway..service.expose.*` – per-port toggles for publishing admin/debug ports on the Service. **All default to `false`** so admin surfaces stay pod-internal (reach them with `kubectl port-forward`). Available toggles: `controller.service.expose.admin` (controller admin, 9092), `gatewayRuntime.service.expose.routerAdmin` (Router/Envoy admin, 9901 — includes mutating endpoints, leave off unless trusted), `gatewayRuntime.service.expose.policyEngineAdmin` (policy-engine admin, 9002). Probes are unaffected; they target container ports directly. Note: the controller admin port is no longer exposed by default — set `expose.admin=true` to restore prior behavior. The runtime port key was renamed `envoyAdmin` → `routerAdmin`. -- `gateway.controller.persistence` / `gateway.controller.configMap` – PVC sizing/claims and component configuration payloads. +- `gateway.controller.persistence` / `gateway.controller.configMap` – PVC sizing/claims (plus PVC `labels`/`annotations`, e.g. `helm.sh/resource-policy: keep`) and component configuration payloads. +- `commonLabels` / `commonAnnotations` – applied to every resource the chart renders; per-resource labels/annotations win on key conflicts. - `gateway.controller.controlPlane` and `gateway.controller.logging` – control-plane connectivity plus controller logging level. - `gateway.controller.tls.*` – TLS certificate configuration for HTTPS listener using cert-manager or existing secrets. - `gateway.controller.upstreamCerts.*` – Custom CA certificates for upstream backend TLS verification. diff --git a/kubernetes/helm/gateway-helm-chart/templates/_helpers.tpl b/kubernetes/helm/gateway-helm-chart/templates/_helpers.tpl index 7b3a1fe653..c816738532 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/_helpers.tpl +++ b/kubernetes/helm/gateway-helm-chart/templates/_helpers.tpl @@ -20,14 +20,40 @@ {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" -}} {{- end -}} +{{/* +Render a string-keyed metadata map (labels/annotations) as YAML. Values are +coerced to strings so numbers/bools from values.yaml render quoted (Kubernetes +requires string values). Keys with null values are skipped. +*/}} +{{- define "gateway-operator.renderStringMap" -}} +{{- $out := dict -}} +{{- range $k, $v := . -}} +{{- if not (kindIs "invalid" $v) -}} +{{- $_ := set $out $k (toString $v) -}} +{{- end -}} +{{- end -}} +{{- toYaml $out -}} +{{- end -}} + {{- define "gateway-operator.labels" -}} -helm.sh/chart: {{ include "gateway-operator.chart" . }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -app.kubernetes.io/instance: {{ .Release.Name }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- with .Values.commonLabels }} -{{ toYaml . | indent 0 }} -{{- end }} +{{- $std := dict + "helm.sh/chart" (include "gateway-operator.chart" .) + "app.kubernetes.io/name" (include "gateway-operator.name" .) + "app.kubernetes.io/managed-by" .Release.Service + "app.kubernetes.io/instance" .Release.Name + "app.kubernetes.io/version" .Chart.AppVersion -}} +{{- include "gateway-operator.renderStringMap" (merge (dict) (default (dict) .Values.commonLabels) $std) -}} +{{- end -}} + +{{/* +Standard labels plus extra per-resource labels merged in (extra wins). +Args (list): root, extraLabels (may be nil) +*/}} +{{- define "gateway-operator.resourceLabels" -}} +{{- $root := index . 0 -}} +{{- $extra := default (dict) (index . 1) -}} +{{- $base := fromYaml (include "gateway-operator.labels" $root) -}} +{{- include "gateway-operator.renderStringMap" (merge (dict) $extra $base) -}} {{- end -}} {{- define "gateway-operator.selectorLabels" -}} @@ -35,15 +61,49 @@ app.kubernetes.io/name: {{ include "gateway-operator.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end -}} +{{/* +Standard labels + component label + extra per-resource labels, merged with +precedence: extra > component > commonLabels > standard. +Args (list): root, component, extraLabels (may be nil) +*/}} {{- define "gateway-operator.componentLabels" -}} {{- $root := index . 0 -}} {{- $component := index . 1 -}} {{- $extra := default (dict) (index . 2) -}} -{{ include "gateway-operator.labels" $root }} -app.kubernetes.io/component: {{ $component }} -{{- with $extra }} -{{ toYaml . | indent 0 }} -{{- end }} +{{- $base := fromYaml (include "gateway-operator.labels" $root) -}} +{{- include "gateway-operator.renderStringMap" (merge (dict) $extra (dict "app.kubernetes.io/component" $component) $base) -}} +{{- end -}} + +{{/* +Pod-template labels: selector labels merged over podLabels and commonLabels. +Selector keys always win — the Deployment selector is immutable and pods must +keep matching it regardless of user-supplied labels. +Args (list): root, component, podLabels (may be nil) +*/}} +{{- define "gateway-operator.componentPodLabels" -}} +{{- $root := index . 0 -}} +{{- $component := index . 1 -}} +{{- $podLabels := default (dict) (index . 2) -}} +{{- $selector := fromYaml (include "gateway-operator.componentSelectorLabels" (list $root $component)) -}} +{{- include "gateway-operator.renderStringMap" (merge (dict) $selector $podLabels (default (dict) $root.Values.commonLabels)) -}} +{{- end -}} + +{{/* +Merge commonAnnotations with per-resource annotations (specific wins). Emits +nothing when both are empty so callers can wrap with: + {{- with (include "gateway-operator.annotations" (list . $specific)) }} + annotations: + {{- . | nindent 4 }} + {{- end }} +Args (list): root, specificAnnotations (may be nil) +*/}} +{{- define "gateway-operator.annotations" -}} +{{- $root := index . 0 -}} +{{- $specific := default (dict) (index . 1) -}} +{{- $merged := merge (dict) $specific (default (dict) $root.Values.commonAnnotations) -}} +{{- if $merged -}} +{{- include "gateway-operator.renderStringMap" $merged -}} +{{- end -}} {{- end -}} {{- define "gateway-operator.componentSelectorLabels" -}} diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/certificate.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/certificate.yaml index ea4e2e7acd..e442b337f2 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/certificate.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/certificate.yaml @@ -7,6 +7,10 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-controller-tls labels: {{- include "gateway-operator.componentLabels" (list . "controller" dict) | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . dict)) }} + annotations: + {{- . | nindent 4 }} + {{- end }} spec: secretName: {{ include "gateway-operator.fullname" . }}-controller-tls commonName: {{ $tls.certManager.commonName }} diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/deployment.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/deployment.yaml index 9ae26645d2..bf005decaf 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/deployment.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/deployment.yaml @@ -12,33 +12,30 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-controller labels: {{- include "gateway-operator.componentLabels" (list . "controller" $deployment.labels) | nindent 4 }} - {{- $annotations := merge (deepCopy (default (dict) .Values.commonAnnotations)) (default (dict) $deployment.annotations) }} - {{- if $annotations }} + {{- with (include "gateway-operator.annotations" (list . $deployment.annotations)) }} annotations: - {{- toYaml $annotations | nindent 4 }} + {{- . | nindent 4 }} {{- end }} spec: {{- if not $controller.hpa.enabled }} replicas: {{ $deployment.replicaCount }} {{- end }} + {{- with $deployment.strategy }} + strategy: + {{- toYaml . | nindent 4 }} + {{- end }} selector: matchLabels: {{- include "gateway-operator.componentSelectorLabels" (list . "controller") | nindent 6 }} template: metadata: labels: - {{- with .Values.commonLabels }} - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with $deployment.podLabels }} - {{- toYaml . | nindent 8 }} - {{- end }} - {{- include "gateway-operator.componentSelectorLabels" (list . "controller") | nindent 8 }} - {{- $podAnnotations := merge (deepCopy (default (dict) $deployment.podAnnotations)) (default (dict) .Values.commonAnnotations) }} + {{- include "gateway-operator.componentPodLabels" (list . "controller" $deployment.podLabels) | nindent 8 }} + {{- $podAnnotations := omit (merge (dict) (default (dict) $deployment.podAnnotations) (default (dict) .Values.commonAnnotations)) "checksum/config" }} annotations: checksum/config: {{ include (print $.Template.BasePath "/gateway/gateway-config.yaml") . | sha256sum }} {{- with $podAnnotations }} - {{- toYaml . | nindent 8 }} + {{- include "gateway-operator.renderStringMap" . | nindent 8 }} {{- end }} spec: serviceAccountName: {{ include "gateway-operator.serviceAccountName" . }} @@ -64,6 +61,27 @@ spec: tolerations: {{- toYaml . | nindent 8 }} {{- end }} + {{- with $deployment.topologySpreadConstraints }} + topologySpreadConstraints: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- if hasKey $deployment "terminationGracePeriodSeconds" }} + terminationGracePeriodSeconds: {{ $deployment.terminationGracePeriodSeconds }} + {{- end }} + {{- with $deployment.hostAliases }} + hostAliases: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with $deployment.dnsPolicy }} + dnsPolicy: {{ . }} + {{- end }} + {{- with $deployment.dnsConfig }} + dnsConfig: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- if hasKey $deployment "automountServiceAccountToken" }} + automountServiceAccountToken: {{ $deployment.automountServiceAccountToken }} + {{- end }} containers: - name: gateway-controller image: {{ include "gateway-operator.componentImage" (dict "root" . "repository" $controller.image.repository "defaultRepository" "ghcr.io/wso2/api-platform/gateway-controller" "tag" $controller.image.tag) | quote }} @@ -131,6 +149,10 @@ spec: protocol: TCP livenessProbe: {{- toYaml $deployment.livenessProbe | nindent 12 }} + {{- with $deployment.startupProbe }} + startupProbe: + {{- toYaml . | nindent 12 }} + {{- end }} readinessProbe: {{- toYaml $deployment.readinessProbe | nindent 12 }} resources: diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/hpa.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/hpa.yaml index 4a528f7d8a..7039eb6b97 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/hpa.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/hpa.yaml @@ -10,6 +10,10 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-controller labels: {{- include "gateway-operator.componentLabels" (list . "controller" dict) | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . dict)) }} + annotations: + {{- . | nindent 4 }} + {{- end }} spec: scaleTargetRef: apiVersion: apps/v1 diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/issuer.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/issuer.yaml index 872a6aa476..47a468726b 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/issuer.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/issuer.yaml @@ -11,6 +11,10 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-selfsigned-issuer labels: {{- include "gateway-operator.componentLabels" (list . "controller" dict) | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . dict)) }} + annotations: + {{- . | nindent 4 }} + {{- end }} spec: selfSigned: {} {{- end }} diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/pdb.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/pdb.yaml index 9d25b58fd4..a61afb1d24 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/pdb.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/pdb.yaml @@ -13,6 +13,10 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-controller labels: {{- include "gateway-operator.componentLabels" (list . "controller" dict) | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . dict)) }} + annotations: + {{- . | nindent 4 }} + {{- end }} spec: selector: matchLabels: diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/pvc.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/pvc.yaml index 7818b26389..df279dd225 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/pvc.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/pvc.yaml @@ -6,7 +6,11 @@ kind: PersistentVolumeClaim metadata: name: {{ include "gateway-operator.fullname" . }}-controller-data labels: - {{- include "gateway-operator.componentLabels" (list . "controller" dict) | nindent 4 }} + {{- include "gateway-operator.componentLabels" (list . "controller" $persistence.labels) | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . $persistence.annotations)) }} + annotations: + {{- . | nindent 4 }} + {{- end }} spec: accessModes: {{- range $persistence.accessModes }} diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/service.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/service.yaml index 77322b6ccd..ed714f7ddd 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/service.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/controller/service.yaml @@ -1,5 +1,7 @@ {{- $controller := .Values.gateway.controller -}} {{- $service := $controller.service -}} +{{- $nodePorts := default (dict) $service.nodePorts -}} +{{- $isNodePortCapable := or (eq $service.type "NodePort") (eq $service.type "LoadBalancer") -}} {{- if $controller.deployment.enabled }} apiVersion: v1 kind: Service @@ -7,13 +9,34 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-controller labels: {{- include "gateway-operator.componentLabels" (list . "controller" $service.labels) | nindent 4 }} - {{- $annotations := merge (deepCopy (default (dict) .Values.commonAnnotations)) (default (dict) $service.annotations) }} - {{- if $annotations }} + {{- with (include "gateway-operator.annotations" (list . $service.annotations)) }} annotations: - {{- toYaml $annotations | nindent 4 }} + {{- . | nindent 4 }} {{- end }} spec: type: {{ $service.type }} + {{- with $service.clusterIP }} + clusterIP: {{ . }} + {{- end }} + {{- if and $service.externalTrafficPolicy $isNodePortCapable }} + externalTrafficPolicy: {{ $service.externalTrafficPolicy }} + {{- end }} + {{- if eq $service.type "LoadBalancer" }} + {{- with $service.loadBalancerClass }} + loadBalancerClass: {{ . }} + {{- end }} + {{- with $service.loadBalancerSourceRanges }} + loadBalancerSourceRanges: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- end }} + {{- with $service.ipFamilyPolicy }} + ipFamilyPolicy: {{ . }} + {{- end }} + {{- with $service.ipFamilies }} + ipFamilies: + {{- toYaml . | nindent 4 }} + {{- end }} selector: {{- include "gateway-operator.componentSelectorLabels" (list . "controller") | nindent 4 }} ports: @@ -21,22 +44,37 @@ spec: port: {{ $service.ports.rest }} targetPort: rest protocol: TCP + {{- if and $isNodePortCapable $nodePorts.rest }} + nodePort: {{ $nodePorts.rest }} + {{- end }} - name: xds port: {{ $service.ports.xds }} targetPort: xds protocol: TCP + {{- if and $isNodePortCapable $nodePorts.xds }} + nodePort: {{ $nodePorts.xds }} + {{- end }} - name: policy port: {{ $service.ports.policy }} targetPort: policy protocol: TCP + {{- if and $isNodePortCapable $nodePorts.policy }} + nodePort: {{ $nodePorts.policy }} + {{- end }} {{- if $service.expose.admin }} - name: admin port: {{ $service.ports.admin }} targetPort: admin protocol: TCP + {{- if and $isNodePortCapable $nodePorts.admin }} + nodePort: {{ $nodePorts.admin }} + {{- end }} {{- end }} - name: metrics port: {{ $service.ports.metrics }} targetPort: metrics protocol: TCP + {{- if and $isNodePortCapable $nodePorts.metrics }} + nodePort: {{ $nodePorts.metrics }} + {{- end }} {{- end }} diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-config.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-config.yaml index 5dd0bb1641..c4f2140896 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-config.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-config.yaml @@ -1,4 +1,3 @@ -{{- $annotations := merge (deepCopy (default (dict) .Values.commonAnnotations)) (default (dict) .Values.gateway.configMap.annotations) -}} {{- $gc := .Values.gateway.config.controller -}} {{- $router := .Values.gateway.config.router -}} {{- $pe := .Values.gateway.config.policy_engine -}} @@ -9,13 +8,10 @@ kind: ConfigMap metadata: name: {{ include "gateway-operator.fullname" . }}-config labels: - {{- include "gateway-operator.labels" . | nindent 4 }} - {{- with .Values.gateway.configMap.labels }} - {{- toYaml . | nindent 4 }} - {{- end }} - {{- if $annotations }} + {{- include "gateway-operator.resourceLabels" (list . .Values.gateway.configMap.labels) | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . .Values.gateway.configMap.annotations)) }} annotations: - {{- toYaml $annotations | nindent 4 }} + {{- . | nindent 4 }} {{- end }} data: config.toml: | diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/deployment.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/deployment.yaml index da91f50c94..d111695584 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/deployment.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/deployment.yaml @@ -9,33 +9,30 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-gateway-runtime labels: {{- include "gateway-operator.componentLabels" (list . "gateway-runtime" $deployment.labels) | nindent 4 }} - {{- $annotations := merge (deepCopy (default (dict) .Values.commonAnnotations)) (default (dict) $deployment.annotations) }} - {{- if $annotations }} + {{- with (include "gateway-operator.annotations" (list . $deployment.annotations)) }} annotations: - {{- toYaml $annotations | nindent 4 }} + {{- . | nindent 4 }} {{- end }} spec: {{- if not $unified.hpa.enabled }} replicas: {{ $deployment.replicaCount }} {{- end }} + {{- with $deployment.strategy }} + strategy: + {{- toYaml . | nindent 4 }} + {{- end }} selector: matchLabels: {{- include "gateway-operator.componentSelectorLabels" (list . "gateway-runtime") | nindent 6 }} template: metadata: labels: - {{- with .Values.commonLabels }} - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with $deployment.podLabels }} - {{- toYaml . | nindent 8 }} - {{- end }} - {{- include "gateway-operator.componentSelectorLabels" (list . "gateway-runtime") | nindent 8 }} - {{- $podAnnotations := merge (deepCopy (default (dict) $deployment.podAnnotations)) (default (dict) .Values.commonAnnotations) }} + {{- include "gateway-operator.componentPodLabels" (list . "gateway-runtime" $deployment.podLabels) | nindent 8 }} + {{- $podAnnotations := omit (merge (dict) (default (dict) $deployment.podAnnotations) (default (dict) .Values.commonAnnotations)) "checksum/config" }} annotations: checksum/config: {{ include (print $.Template.BasePath "/gateway/gateway-config.yaml") . | sha256sum }} {{- with $podAnnotations }} - {{- toYaml . | nindent 8 }} + {{- include "gateway-operator.renderStringMap" . | nindent 8 }} {{- end }} spec: serviceAccountName: {{ include "gateway-operator.serviceAccountName" . }} @@ -61,6 +58,27 @@ spec: tolerations: {{- toYaml . | nindent 8 }} {{- end }} + {{- with $deployment.topologySpreadConstraints }} + topologySpreadConstraints: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- if hasKey $deployment "terminationGracePeriodSeconds" }} + terminationGracePeriodSeconds: {{ $deployment.terminationGracePeriodSeconds }} + {{- end }} + {{- with $deployment.hostAliases }} + hostAliases: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with $deployment.dnsPolicy }} + dnsPolicy: {{ . }} + {{- end }} + {{- with $deployment.dnsConfig }} + dnsConfig: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- if hasKey $deployment "automountServiceAccountToken" }} + automountServiceAccountToken: {{ $deployment.automountServiceAccountToken }} + {{- end }} containers: - name: gateway-runtime image: {{ include "gateway-operator.componentImage" (dict "root" . "repository" $unified.image.repository "defaultRepository" "ghcr.io/wso2/api-platform/gateway-runtime" "tag" $unified.image.tag) | quote }} diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/hpa.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/hpa.yaml index c8b84198d7..0e40cbd805 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/hpa.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/hpa.yaml @@ -7,6 +7,10 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-gateway-runtime labels: {{- include "gateway-operator.componentLabels" (list . "gateway-runtime" dict) | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . dict)) }} + annotations: + {{- . | nindent 4 }} + {{- end }} spec: scaleTargetRef: apiVersion: apps/v1 diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/llm-pricing-configmap.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/llm-pricing-configmap.yaml index 4dc9df776c..4cfb281722 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/llm-pricing-configmap.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/llm-pricing-configmap.yaml @@ -6,6 +6,10 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-llm-pricing labels: {{- include "gateway-operator.labels" . | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . dict)) }} + annotations: + {{- . | nindent 4 }} + {{- end }} data: model_prices.json: | {{- .Files.Get "files/llm-pricing/model_prices.json" | nindent 4 }} diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/pdb.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/pdb.yaml index 59531f4b8e..a25b797fa2 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/pdb.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/pdb.yaml @@ -13,6 +13,10 @@ metadata: name: {{ include "gateway-operator.fullname" . }}-gateway-runtime labels: {{- include "gateway-operator.componentLabels" (list . "gateway-runtime" dict) | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . dict)) }} + annotations: + {{- . | nindent 4 }} + {{- end }} spec: selector: matchLabels: diff --git a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/service.yaml b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/service.yaml index 1b9d72ce22..1eb77a2500 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/service.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/service.yaml @@ -1,43 +1,82 @@ {{- $unified := .Values.gateway.gatewayRuntime -}} {{- $deployment := $unified.deployment -}} +{{- $service := $unified.service -}} +{{- $nodePorts := default (dict) $service.nodePorts -}} +{{- $isNodePortCapable := or (eq $service.type "NodePort") (eq $service.type "LoadBalancer") -}} {{- if $deployment.enabled }} apiVersion: v1 kind: Service metadata: name: {{ include "gateway-operator.fullname" . }}-gateway-runtime labels: - {{- include "gateway-operator.componentLabels" (list . "gateway-runtime" $unified.service.labels) | nindent 4 }} - {{- $annotations := merge (deepCopy (default (dict) .Values.commonAnnotations)) (default (dict) $unified.service.annotations) }} - {{- if $annotations }} + {{- include "gateway-operator.componentLabels" (list . "gateway-runtime" $service.labels) | nindent 4 }} + {{- with (include "gateway-operator.annotations" (list . $service.annotations)) }} annotations: - {{- toYaml $annotations | nindent 4 }} + {{- . | nindent 4 }} {{- end }} spec: - type: {{ $unified.service.type }} + type: {{ $service.type }} + {{- with $service.clusterIP }} + clusterIP: {{ . }} + {{- end }} + {{- if and $service.externalTrafficPolicy $isNodePortCapable }} + externalTrafficPolicy: {{ $service.externalTrafficPolicy }} + {{- end }} + {{- if eq $service.type "LoadBalancer" }} + {{- with $service.loadBalancerClass }} + loadBalancerClass: {{ . }} + {{- end }} + {{- with $service.loadBalancerSourceRanges }} + loadBalancerSourceRanges: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- end }} + {{- with $service.ipFamilyPolicy }} + ipFamilyPolicy: {{ . }} + {{- end }} + {{- with $service.ipFamilies }} + ipFamilies: + {{- toYaml . | nindent 4 }} + {{- end }} ports: - - port: {{ $unified.service.ports.http }} + - port: {{ $service.ports.http }} targetPort: http protocol: TCP name: http - - port: {{ $unified.service.ports.https }} + {{- if and $isNodePortCapable $nodePorts.http }} + nodePort: {{ $nodePorts.http }} + {{- end }} + - port: {{ $service.ports.https }} targetPort: https protocol: TCP name: https - - port: {{ $unified.service.ports.policyEngineMetrics }} + {{- if and $isNodePortCapable $nodePorts.https }} + nodePort: {{ $nodePorts.https }} + {{- end }} + - port: {{ $service.ports.policyEngineMetrics }} targetPort: pe-metrics protocol: TCP name: pe-metrics - {{- if $unified.service.expose.routerAdmin }} - - port: {{ $unified.service.ports.routerAdmin }} + {{- if and $isNodePortCapable $nodePorts.policyEngineMetrics }} + nodePort: {{ $nodePorts.policyEngineMetrics }} + {{- end }} + {{- if $service.expose.routerAdmin }} + - port: {{ $service.ports.routerAdmin }} targetPort: router-admin protocol: TCP name: router-admin + {{- if and $isNodePortCapable $nodePorts.routerAdmin }} + nodePort: {{ $nodePorts.routerAdmin }} + {{- end }} {{- end }} - {{- if $unified.service.expose.policyEngineAdmin }} - - port: {{ $unified.service.ports.policyEngineAdmin }} + {{- if $service.expose.policyEngineAdmin }} + - port: {{ $service.ports.policyEngineAdmin }} targetPort: pe-admin protocol: TCP name: pe-admin + {{- if and $isNodePortCapable $nodePorts.policyEngineAdmin }} + nodePort: {{ $nodePorts.policyEngineAdmin }} + {{- end }} {{- end }} selector: {{- include "gateway-operator.componentSelectorLabels" (list . "gateway-runtime") | nindent 4 }} diff --git a/kubernetes/helm/gateway-helm-chart/templates/serviceaccount.yaml b/kubernetes/helm/gateway-helm-chart/templates/serviceaccount.yaml index eb2eb473c8..b62328fb4c 100644 --- a/kubernetes/helm/gateway-helm-chart/templates/serviceaccount.yaml +++ b/kubernetes/helm/gateway-helm-chart/templates/serviceaccount.yaml @@ -5,8 +5,9 @@ metadata: name: {{ include "gateway-operator.serviceAccountName" . }} labels: {{- include "gateway-operator.labels" . | nindent 4 }} - {{- with .Values.serviceAccount.annotations }} + {{- with (include "gateway-operator.annotations" (list . .Values.serviceAccount.annotations)) }} annotations: - {{- toYaml . | nindent 4 }} + {{- . | nindent 4 }} {{- end }} +automountServiceAccountToken: {{ .Values.serviceAccount.automountServiceAccountToken }} {{- end }} diff --git a/kubernetes/helm/gateway-helm-chart/values.yaml b/kubernetes/helm/gateway-helm-chart/values.yaml index c4cf07a2f3..b5e1d758bc 100644 --- a/kubernetes/helm/gateway-helm-chart/values.yaml +++ b/kubernetes/helm/gateway-helm-chart/values.yaml @@ -31,6 +31,9 @@ serviceAccount: create: true annotations: {} name: "" + # Mount the service account token into pods by default (Kubernetes default: true). + # Individual pods can override via gateway..deployment.automountServiceAccountToken. + automountServiceAccountToken: true ## moved to each component's deployment block below @@ -404,6 +407,27 @@ gateway: # readiness probes are unaffected: they hit the container port directly, not the Service. expose: admin: false + # Static cluster IP (e.g. "None" for a headless Service). Empty = auto-assign. + clusterIP: "" + # External traffic policy: "Cluster" or "Local". + # Only rendered when type is LoadBalancer or NodePort. + externalTrafficPolicy: "" + # Load balancer implementation class. Only rendered when type is LoadBalancer. + loadBalancerClass: "" + # CIDR ranges allowed to reach the load balancer. Only rendered when type is LoadBalancer. + loadBalancerSourceRanges: [] + # IP family policy: SingleStack, PreferDualStack or RequireDualStack. Empty = cluster default. + ipFamilyPolicy: "" + # IP families, e.g. ["IPv4"], ["IPv6"] or ["IPv4", "IPv6"]. Empty = cluster default. + ipFamilies: [] + # Static nodePort per named service port. Only rendered when type is NodePort or + # LoadBalancer; empty = auto-allocate. + nodePorts: + rest: "" + xds: "" + policy: "" + admin: "" + metrics: "" controlPlane: # Control plane endpoint. Used directly as the HTTPS/WSS authority, so include # a non-default port here (e.g. "host.docker.internal:8443"); 443 is assumed if omitted. @@ -496,6 +520,13 @@ gateway: - ReadWriteOnce size: 100Mi storageClass: "" + # Extra labels for the PersistentVolumeClaim. + labels: {} + # Extra annotations for the PersistentVolumeClaim (merged with commonAnnotations; + # these win on conflict). Example — keep the claim across helm uninstall: + # annotations: + # helm.sh/resource-policy: keep + annotations: {} deployment: enabled: true replicaCount: 1 @@ -531,6 +562,13 @@ gateway: periodSeconds: 5 timeoutSeconds: 3 failureThreshold: 3 + # Optional startup probe for slow-starting environments. Uncomment to enable. + # startupProbe: + # httpGet: + # path: /api/admin/v0.9/health + # port: admin + # periodSeconds: 5 + # failureThreshold: 30 # Resource limits and requests # Uncomment and adjust based on your workload resources: {} @@ -545,6 +583,36 @@ gateway: nodeSelector: {} tolerations: [] affinity: {} + # Topology spread constraints (list of topologySpreadConstraint objects). + # Example: + # - maxSkew: 1 + # topologyKey: topology.kubernetes.io/zone + # whenUnsatisfiable: ScheduleAnyway + # labelSelector: + # matchLabels: + # app.kubernetes.io/component: controller + topologySpreadConstraints: [] + # Deployment update strategy (full apps/v1 DeploymentStrategy object). + # Empty = Kubernetes default (RollingUpdate, 25% maxSurge / 25% maxUnavailable). + # Example: + # type: RollingUpdate + # rollingUpdate: + # maxSurge: 1 + # maxUnavailable: 0 + strategy: {} + # Seconds the pod is given to terminate gracefully. Uncomment to override the + # Kubernetes default (30). + # terminationGracePeriodSeconds: 30 + # Additional /etc/hosts entries (list of hostAlias objects). + hostAliases: [] + # Pod DNS policy: ClusterFirst, Default, ClusterFirstWithHostNet or None. + # Empty = Kubernetes default. + dnsPolicy: "" + # Pod DNS configuration (nameservers / searches / options); used with dnsPolicy. + dnsConfig: {} + # Override service account token automounting for this pod (takes precedence + # over serviceAccount.automountServiceAccountToken). Uncomment to set. + # automountServiceAccountToken: false # Horizontal Pod Autoscaler for the controller. # Requires gateway.config.controller.storage.type=postgres — SQLite does not support multiple replicas. # When enabled, replicaCount is managed by HPA and the Deployment's replicas field is omitted. @@ -597,6 +665,27 @@ gateway: routerAdmin: false # Policy-engine admin: config_dump + health. policyEngineAdmin: false + # Static cluster IP (e.g. "None" for a headless Service). Empty = auto-assign. + clusterIP: "" + # External traffic policy: "Cluster" or "Local". "Local" preserves client source IPs. + # Only rendered when type is LoadBalancer or NodePort. + externalTrafficPolicy: "" + # Load balancer implementation class. Only rendered when type is LoadBalancer. + loadBalancerClass: "" + # CIDR ranges allowed to reach the load balancer. Only rendered when type is LoadBalancer. + loadBalancerSourceRanges: [] + # IP family policy: SingleStack, PreferDualStack or RequireDualStack. Empty = cluster default. + ipFamilyPolicy: "" + # IP families, e.g. ["IPv4"], ["IPv6"] or ["IPv4", "IPv6"]. Empty = cluster default. + ipFamilies: [] + # Static nodePort per named service port. Only rendered when type is NodePort or + # LoadBalancer; empty = auto-allocate. + nodePorts: + http: "" + https: "" + routerAdmin: "" + policyEngineAdmin: "" + policyEngineMetrics: "" # Policy-related file mounts for the gateway runtime policies: # LLM pricing data for the llm_cost_v1 policy. @@ -642,6 +731,12 @@ gateway: periodSeconds: 5 timeoutSeconds: 3 failureThreshold: 6 + # Optional startup probe for slow-starting environments. Uncomment to enable. + # startupProbe: + # exec: + # command: ["health-check.sh"] + # periodSeconds: 5 + # failureThreshold: 30 # Resource limits and requests # Uncomment and adjust based on your workload resources: {} @@ -656,6 +751,36 @@ gateway: nodeSelector: {} tolerations: [] affinity: {} + # Topology spread constraints (list of topologySpreadConstraint objects). + # Example: + # - maxSkew: 1 + # topologyKey: topology.kubernetes.io/zone + # whenUnsatisfiable: ScheduleAnyway + # labelSelector: + # matchLabels: + # app.kubernetes.io/component: gateway-runtime + topologySpreadConstraints: [] + # Deployment update strategy (full apps/v1 DeploymentStrategy object). + # Empty = Kubernetes default (RollingUpdate, 25% maxSurge / 25% maxUnavailable). + # Example: + # type: RollingUpdate + # rollingUpdate: + # maxSurge: 1 + # maxUnavailable: 0 + strategy: {} + # Seconds the pod is given to terminate gracefully — allow time for Envoy to + # drain in-flight connections. Uncomment to override the Kubernetes default (30). + # terminationGracePeriodSeconds: 30 + # Additional /etc/hosts entries (list of hostAlias objects). + hostAliases: [] + # Pod DNS policy: ClusterFirst, Default, ClusterFirstWithHostNet or None. + # Empty = Kubernetes default. + dnsPolicy: "" + # Pod DNS configuration (nameservers / searches / options); used with dnsPolicy. + dnsConfig: {} + # Override service account token automounting for this pod (takes precedence + # over serviceAccount.automountServiceAccountToken). Uncomment to set. + # automountServiceAccountToken: false # Horizontal Pod Autoscaler for the gateway runtime. # When enabled, replicaCount is managed by HPA and the Deployment's replicas field is omitted. hpa: From a03137dbfd18c5d516ed6ee778460c3f5831b243 Mon Sep 17 00:00:00 2001 From: Renuka Fernando Date: Sat, 13 Jun 2026 13:27:43 +0530 Subject: [PATCH 2/2] docs(helm): fix ConfigMap values path in gateway chart README The Configuration section referenced gateway.controller.configMap, but the values key is gateway.configMap (sibling of gateway.controller). Using the documented path would not apply user overrides. --- kubernetes/helm/gateway-helm-chart/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kubernetes/helm/gateway-helm-chart/README.md b/kubernetes/helm/gateway-helm-chart/README.md index 1f6485ca91..c24039b60b 100644 --- a/kubernetes/helm/gateway-helm-chart/README.md +++ b/kubernetes/helm/gateway-helm-chart/README.md @@ -126,7 +126,7 @@ All configurable values are documented in `values.yaml`. Component blocks are fu - `gateway..deployment.*` – pod-level knobs (replicas, probes incl. optional `startupProbe`, scheduling via `nodeSelector`/`tolerations`/`affinity`/`topologySpreadConstraints`, update `strategy`, `terminationGracePeriodSeconds`, `hostAliases`, `dnsPolicy`/`dnsConfig`, `automountServiceAccountToken`, env overrides, extra volumes) and enable/disable switches. - `gateway..service.*` – service type/ports plus optional annotations and labels, and network tuning (`clusterIP`, `externalTrafficPolicy`, `loadBalancerClass`, `loadBalancerSourceRanges`, `ipFamilyPolicy`/`ipFamilies`, static `nodePorts.*`). - `gateway..service.expose.*` – per-port toggles for publishing admin/debug ports on the Service. **All default to `false`** so admin surfaces stay pod-internal (reach them with `kubectl port-forward`). Available toggles: `controller.service.expose.admin` (controller admin, 9092), `gatewayRuntime.service.expose.routerAdmin` (Router/Envoy admin, 9901 — includes mutating endpoints, leave off unless trusted), `gatewayRuntime.service.expose.policyEngineAdmin` (policy-engine admin, 9002). Probes are unaffected; they target container ports directly. Note: the controller admin port is no longer exposed by default — set `expose.admin=true` to restore prior behavior. The runtime port key was renamed `envoyAdmin` → `routerAdmin`. -- `gateway.controller.persistence` / `gateway.controller.configMap` – PVC sizing/claims (plus PVC `labels`/`annotations`, e.g. `helm.sh/resource-policy: keep`) and component configuration payloads. +- `gateway.controller.persistence` / `gateway.configMap` – PVC sizing/claims (plus PVC `labels`/`annotations`, e.g. `helm.sh/resource-policy: keep`) and component configuration payloads. - `commonLabels` / `commonAnnotations` – applied to every resource the chart renders; per-resource labels/annotations win on key conflicts. - `gateway.controller.controlPlane` and `gateway.controller.logging` – control-plane connectivity plus controller logging level. - `gateway.controller.tls.*` – TLS certificate configuration for HTTPS listener using cert-manager or existing secrets.