fix: glance S3 location churn and undersized memory defaults

docs/reference/ci-cd/ci-workflow.md

@@ -1130,8 +1130,9 @@ OPERATOR=keystone hack/ci-dump-diagnostics.sh   # + operator-specific diagnostic
 ### hack/ci-build-service-image.sh
 
 Builds an OpenStack service container image by resolving upstream source refs, cloning the
-project at the pinned ref, applying constraint overrides, and building the full image chain
-(`python-base` -> `venv-builder` -> service image).
+project at the pinned ref, applying patches from `patches/<service>/<release>/` (the same
+set the Build Images workflow applies), applying constraint overrides, and building the
+full image chain (`python-base` -> `venv-builder` -> service image).
 
 | Environment Variable | Required | Default | Description |
 | --- | --- | --- | --- |

docs/reference/glance/glance-crd.md

@@ -21,7 +21,7 @@ stores are **not** part of this spec — they attach out-of-band through
 | Field | Type | Required | Description |
 | --- | --- | --- | --- |
 | `openStackRelease` | `string` | yes | The OpenStack release the operator deploys and drives; pattern `^\d{4}\.[12]$` (the `YYYY.N` cadence, `N` ∈ {1,2}). Governs the API launch mode (eventlet below `2026.1`, uWSGI from `2026.1`) and install/upgrade schema tracking. Kept separate from the image tag so digest-pinned images still resolve a schema and launch mode |
-| `deployment` | `DeploymentSpec` | no | Shared pod-level knobs: `replicas` (default 3), `resources` (Burstable defaults), `terminationGracePeriodSeconds`, `preStopSleepSeconds`, `strategy`, `topologySpreadConstraints`, `priorityClassName` |
+| `deployment` | `DeploymentSpec` | no | Shared pod-level knobs: `replicas` (default 3), `resources` (defaults: 512Mi request / 1Gi limit memory, 100m/500m CPU), `terminationGracePeriodSeconds`, `preStopSleepSeconds`, `strategy`, `topologySpreadConstraints`, `priorityClassName` |
 | `image` | `ImageSpec` | yes | Container image; exactly one of `tag` or `digest` (shared CEL rule, re-checked by the webhook) |
 | `database` | `DatabaseSpec` | yes | MariaDB connection. Exactly one of `clusterRef` (managed) or `host` (brownfield); `credentialsMode` (`Static` \| `Dynamic`, where `Dynamic` requires `clusterRef`), `secretRef`, and optional `tls`. Mutual-exclusivity and the Dynamic-requires-clusterRef rule are inherited from `commonv1.DatabaseSpec` |
 | `cache` | `CacheSpec` | yes | Memcached backing the Glance image cache. Exactly one of `clusterRef` (managed) or `servers` (brownfield) |
@@ -81,11 +81,16 @@ is never emitted).
 
 ### Defaulting and validation
 
-The mutating webhook applies the shared `DeploymentSpec`/`LoggingSpec` defaults,
-materializes the `PyMemcacheCache` cache backend, fills the `ServiceUserSpec`
-identity defaults (`glance` / `service` / `Default` / `Default`, `secretRef.key`
-→ `password`), and — only when `spec.apiServer.uwsgi` is present — the uWSGI
-sub-field defaults (`processes` 2, `threads` 1, `httpKeepAlive` true).
+The mutating webhook applies the shared `DeploymentSpec`/`LoggingSpec` defaults
+— with one glance-specific deviation: an unset `spec.deployment.resources` is
+filled with 512Mi memory request / 1Gi memory limit (CPU keeps the shared
+100m/500m), because the glance-api container carries the boto3-weighted S3
+store driver and overruns the shared 512Mi baseline under concurrent image
+traffic. It also materializes the `PyMemcacheCache` cache backend, fills the
+`ServiceUserSpec` identity defaults (`glance` / `service` / `Default` /
+`Default`, `secretRef.key` → `password`), and — only when
+`spec.apiServer.uwsgi` is present — the uWSGI sub-field defaults (`processes`
+2, `threads` 1, `httpKeepAlive` true).
 
 The validating webhook accumulates every violation into one admission response,
 reusing the shared validators: replicas floor, image tag/digest XOR, the

docs/reference/horizon/horizon-crd.md

@@ -13,7 +13,7 @@ chart ships a synced copy (`make sync-crds` / `make verify-crd-sync`).
 
 | Field | Type | Required | Description |
 | --- | --- | --- | --- |
-| `deployment` | `DeploymentSpec` | no | Shared pod-level knobs: `replicas` (default 3), `resources` (Burstable defaults), `terminationGracePeriodSeconds`, `preStopSleepSeconds`, `strategy`, `topologySpreadConstraints`, `priorityClassName` |
+| `deployment` | `DeploymentSpec` | no | Shared pod-level knobs: `replicas` (default 3), `resources` (defaults: 256Mi request / 512Mi limit memory, 100m/500m CPU), `terminationGracePeriodSeconds`, `preStopSleepSeconds`, `strategy`, `topologySpreadConstraints`, `priorityClassName` |
 | `image` | `ImageSpec` | yes | Container image; exactly one of `tag` or `digest` (shared CEL rule) |
 | `cache` | `CacheSpec` | yes | Memcached backing the Django cache. Exactly one of `clusterRef` (managed) or `servers` (brownfield); `backend` is a Django cache backend path, defaulted to `django.core.cache.backends.memcached.PyMemcacheCache` |
 | `keystoneEndpoint` | `string` | yes | The Keystone endpoint URL (`OPENSTACK_KEYSTONE_URL`); must match `^https?://` and parse with a host. Consumed server-side by the dashboard pods, so it must be reachable from inside the cluster — for a colocated control plane use the cluster-local Service URL, not an externally routable address |

hack/ci-build-service-image.sh

@@ -5,9 +5,9 @@
 
 # hack/ci-build-service-image.sh — Build an OpenStack service container image.
 #
-# Resolves upstream source refs, clones the project, applies constraint
-# overrides, and builds the full image chain: python-base -> venv-builder ->
-# service image.
+# Resolves upstream source refs, clones the project, applies patches and
+# constraint overrides, and builds the full image chain: python-base ->
+# venv-builder -> service image.
 #
 # Required env vars:
 #   OPERATOR      — OpenStack service name (e.g. keystone)
@@ -63,12 +63,22 @@ git clone --depth 1 --branch "${SERVICE_REF}" \
   "https://github.com/openstack/${OPERATOR}.git" "${SRC_DIR}"
 
 # ---------------------------------------------------------------------------
-# 4. Apply constraint overrides (idempotent, exits 0 if no overrides)
+# 4. Apply patches (mirrors .github/actions/checkout-service-source so the
+#    e2e-built images carry the same source as the published GHCR images)
+# ---------------------------------------------------------------------------
+PATCH_DIR="${REPO_ROOT}/patches/${OPERATOR}/${RELEASE}"
+if [ -d "${PATCH_DIR}" ] && compgen -G "${PATCH_DIR}/*.patch" > /dev/null; then
+  echo "Applying patches from ${PATCH_DIR}"
+  git -C "${SRC_DIR}" apply "${PATCH_DIR}"/*.patch
+fi
+
+# ---------------------------------------------------------------------------
+# 5. Apply constraint overrides (idempotent, exits 0 if no overrides)
 # ---------------------------------------------------------------------------
 "${REPO_ROOT}/scripts/apply-constraint-overrides.sh" "${RELEASE}"
 
 # ---------------------------------------------------------------------------
-# 5. Build image chain: python-base -> venv-builder -> service
+# 6. Build image chain: python-base -> venv-builder -> service
 # ---------------------------------------------------------------------------
 # Reuse existing base images when available (e.g. when a prior invocation in
 # the same job or an artifact load already created them).  Saves ~30s per

internal/common/types/workload.go

@@ -86,9 +86,10 @@ type DeploymentSpec struct {
 	Replicas int32 `json:"replicas,omitempty"`
 
 	// Resources defines the CPU and memory requests and limits for the service API
-	// container. When unset, the defaulting webhook injects sensible defaults
-	// (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and
-	// enable HPA utilization calculations.
+	// container. When unset, the defaulting webhook injects the operator's
+	// documented resource defaults (100m/500m CPU, with memory sized per
+	// service — see the operator's CRD reference) to ensure Burstable QoS class
+	// and enable HPA utilization calculations.
 	// +optional
 	Resources *corev1.ResourceRequirements `json:"resources,omitempty"`
 

operators/glance/api/v1alpha1/glance_webhook.go

@@ -12,7 +12,9 @@ import (
 	"sort"
 
 	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/api/resource"
 	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/apimachinery/pkg/util/validation/field"
 	"k8s.io/utils/ptr"
@@ -53,6 +55,18 @@ const (
 	DefaultEventletWorkers int32 = 2
 )
 
+// Glance-specific container memory defaults, replacing the shared 256Mi/512Mi
+// baseline that keystone and horizon fit in. The glance-api container carries
+// the S3 store driver (boto3/botocore), which raises both the per-process
+// import footprint and the per-request allocation churn: with the bounded two
+// workers the container already idles near 360Mi and, under concurrent image
+// traffic, overruns a 512Mi limit within a minute — an OOM-kill crash loop the
+// gateway surfaces as waves of 503s. CPU keeps the shared defaults.
+var (
+	defaultGlanceMemoryRequest = resource.MustParse("512Mi")
+	defaultGlanceMemoryLimit   = resource.MustParse("1Gi")
+)
+
 // GlanceWebhook implements defaulting and validation webhooks for the Glance
 // CRD. Client is injected at startup for cluster-scoped resource lookups (e.g.
 // PriorityClass validation). Production wiring injects mgr.GetAPIReader() — a
@@ -87,9 +101,27 @@ func (w *GlanceWebhook) SetupWebhookWithManager(mgr ctrl.Manager) error {
 // filled when explicitly present, except spec.logging which is materialized so
 // downstream reconciler code never sees a nil pointer.
 func (w *GlanceWebhook) Default(_ context.Context, obj *Glance) error {
-	// Shared-type defaults (replicas, container resources) are applied by the
-	// commonv1.DeploymentSpec Default method so they cannot drift across
-	// operators.
+	// Fill spec.deployment.resources with the glance-specific memory defaults
+	// before the shared DeploymentSpec defaults run — Deployment.Default()
+	// would otherwise inject the shared 256Mi/512Mi baseline, which the
+	// S3-backed glance-api overruns under load. Same nil-or-empty condition as
+	// the shared method so an explicit user value is never clobbered.
+	if obj.Spec.Deployment.Resources == nil ||
+		(len(obj.Spec.Deployment.Resources.Requests) == 0 && len(obj.Spec.Deployment.Resources.Limits) == 0) {
+		obj.Spec.Deployment.Resources = &corev1.ResourceRequirements{
+			Requests: corev1.ResourceList{
+				corev1.ResourceMemory: defaultGlanceMemoryRequest.DeepCopy(),
+				corev1.ResourceCPU:    commonv1.DefaultCPURequest(),
+			},
+			Limits: corev1.ResourceList{
+				corev1.ResourceMemory: defaultGlanceMemoryLimit.DeepCopy(),
+				corev1.ResourceCPU:    commonv1.DefaultCPULimit(),
+			},
+		}
+	}
+	// Shared-type defaults (replicas, remaining container resources) are
+	// applied by the commonv1.DeploymentSpec Default method so they cannot
+	// drift across operators.
 	obj.Spec.Deployment.Default()
 	if obj.Spec.Cache.Backend == "" {
 		obj.Spec.Cache.Backend = commonv1.DefaultCacheBackend

operators/glance/api/v1alpha1/glance_webhook_test.go

@@ -10,6 +10,7 @@ import (
 
 	"github.com/onsi/gomega"
 	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/utils/ptr"
 
@@ -60,14 +61,40 @@ func TestGlanceDefault_MaterializesServiceUserAndLoggingDefaults(t *testing.T) {
 	g.Expect(obj.Spec.ServiceUser.ProjectDomainName).To(gomega.Equal("Default"))
 	g.Expect(obj.Spec.ServiceUser.SecretRef.Key).To(gomega.Equal("password"))
 
-	// Shared-block defaults come along too.
+	// Shared-block defaults come along too — with the glance-specific memory
+	// values (512Mi request / 1Gi limit for the boto3-weighted S3 path)
+	// replacing the shared 256Mi/512Mi baseline, while CPU keeps the shared
+	// defaults.
 	g.Expect(obj.Spec.Deployment.Resources).NotTo(gomega.BeNil())
+	g.Expect(obj.Spec.Deployment.Resources.Requests.Memory().String()).To(gomega.Equal("512Mi"))
+	g.Expect(obj.Spec.Deployment.Resources.Limits.Memory().String()).To(gomega.Equal("1Gi"))
+	g.Expect(obj.Spec.Deployment.Resources.Requests.Cpu().String()).To(gomega.Equal("100m"))
+	g.Expect(obj.Spec.Deployment.Resources.Limits.Cpu().String()).To(gomega.Equal("500m"))
 	g.Expect(obj.Spec.Cache.Backend).To(gomega.Equal(commonv1.DefaultCacheBackend))
 	g.Expect(obj.Spec.Logging).NotTo(gomega.BeNil())
 	g.Expect(obj.Spec.Logging.Format).To(gomega.Equal("text"))
 	g.Expect(obj.Spec.Logging.Level).To(gomega.Equal("INFO"))
 }
 
+func TestGlanceDefault_PreservesExplicitResources(t *testing.T) {
+	g := gomega.NewWithT(t)
+	w := &GlanceWebhook{}
+
+	obj := validGlance()
+	obj.Spec.Deployment.Resources = &corev1.ResourceRequirements{
+		Limits: corev1.ResourceList{
+			corev1.ResourceMemory: resource.MustParse("256Mi"),
+		},
+	}
+
+	g.Expect(w.Default(context.Background(), obj)).To(gomega.Succeed())
+
+	// A non-empty resources block is left verbatim: neither the glance memory
+	// defaults nor the shared baseline overwrite an explicit value.
+	g.Expect(obj.Spec.Deployment.Resources.Limits.Memory().String()).To(gomega.Equal("256Mi"))
+	g.Expect(obj.Spec.Deployment.Resources.Requests).To(gomega.BeEmpty())
+}
+
 func TestGlanceDefault_PreservesExplicitServiceUserValues(t *testing.T) {
 	g := gomega.NewWithT(t)
 	w := &GlanceWebhook{}

operators/glance/config/crd/bases/glance.openstack.c5c3.io_glances.yaml

@@ -452,9 +452,10 @@ spec:
                   resources:
                     description: |-
                       Resources defines the CPU and memory requests and limits for the service API
-                      container. When unset, the defaulting webhook injects sensible defaults
-                      (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and
-                      enable HPA utilization calculations.
+                      container. When unset, the defaulting webhook injects the operator's
+                      documented resource defaults (100m/500m CPU, with memory sized per
+                      service — see the operator's CRD reference) to ensure Burstable QoS class
+                      and enable HPA utilization calculations.
                     properties:
                       claims:
                         description: |-

operators/glance/helm/glance-operator/crds/glance.openstack.c5c3.io_glances.yaml

@@ -455,9 +455,10 @@ spec:
                   resources:
                     description: |-
                       Resources defines the CPU and memory requests and limits for the service API
-                      container. When unset, the defaulting webhook injects sensible defaults
-                      (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and
-                      enable HPA utilization calculations.
+                      container. When unset, the defaulting webhook injects the operator's
+                      documented resource defaults (100m/500m CPU, with memory sized per
+                      service — see the operator's CRD reference) to ensure Burstable QoS class
+                      and enable HPA utilization calculations.
                     properties:
                       claims:
                         description: |-

operators/horizon/config/crd/bases/horizon.openstack.c5c3.io_horizons.yaml

@@ -181,9 +181,10 @@ spec:
                   resources:
                     description: |-
                       Resources defines the CPU and memory requests and limits for the service API
-                      container. When unset, the defaulting webhook injects sensible defaults
-                      (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and
-                      enable HPA utilization calculations.
+                      container. When unset, the defaulting webhook injects the operator's
+                      documented resource defaults (100m/500m CPU, with memory sized per
+                      service — see the operator's CRD reference) to ensure Burstable QoS class
+                      and enable HPA utilization calculations.
                     properties:
                       claims:
                         description: |-

operators/horizon/helm/horizon-operator/crds/horizon.openstack.c5c3.io_horizons.yaml

@@ -184,9 +184,10 @@ spec:
                   resources:
                     description: |-
                       Resources defines the CPU and memory requests and limits for the service API
-                      container. When unset, the defaulting webhook injects sensible defaults
-                      (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and
-                      enable HPA utilization calculations.
+                      container. When unset, the defaulting webhook injects the operator's
+                      documented resource defaults (100m/500m CPU, with memory sized per
+                      service — see the operator's CRD reference) to ensure Burstable QoS class
+                      and enable HPA utilization calculations.
                     properties:
                       claims:
                         description: |-

operators/keystone/config/crd/bases/keystone.openstack.c5c3.io_keystones.yaml

@@ -478,9 +478,10 @@ spec:
                   resources:
                     description: |-
                       Resources defines the CPU and memory requests and limits for the service API
-                      container. When unset, the defaulting webhook injects sensible defaults
-                      (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and
-                      enable HPA utilization calculations.
+                      container. When unset, the defaulting webhook injects the operator's
+                      documented resource defaults (100m/500m CPU, with memory sized per
+                      service — see the operator's CRD reference) to ensure Burstable QoS class
+                      and enable HPA utilization calculations.
                     properties:
                       claims:
                         description: |-

operators/keystone/helm/keystone-operator/crds/keystone.openstack.c5c3.io_keystones.yaml

@@ -481,9 +481,10 @@ spec:
                   resources:
                     description: |-
                       Resources defines the CPU and memory requests and limits for the service API
-                      container. When unset, the defaulting webhook injects sensible defaults
-                      (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and
-                      enable HPA utilization calculations.
+                      container. When unset, the defaulting webhook injects the operator's
+                      documented resource defaults (100m/500m CPU, with memory sized per
+                      service — see the operator's CRD reference) to ensure Burstable QoS class
+                      and enable HPA utilization calculations.
                     properties:
                       claims:
                         description: |-