diff --git a/docs/reference/ci-cd/ci-workflow.md b/docs/reference/ci-cd/ci-workflow.md index 1553abcef..b4c3c18b3 100644 --- a/docs/reference/ci-cd/ci-workflow.md +++ b/docs/reference/ci-cd/ci-workflow.md @@ -1130,8 +1130,9 @@ OPERATOR=keystone hack/ci-dump-diagnostics.sh # + operator-specific diagnostic ### hack/ci-build-service-image.sh Builds an OpenStack service container image by resolving upstream source refs, cloning the -project at the pinned ref, applying constraint overrides, and building the full image chain -(`python-base` -> `venv-builder` -> service image). +project at the pinned ref, applying patches from `patches///` (the same +set the Build Images workflow applies), applying constraint overrides, and building the +full image chain (`python-base` -> `venv-builder` -> service image). | Environment Variable | Required | Default | Description | | --- | --- | --- | --- | diff --git a/docs/reference/glance/glance-crd.md b/docs/reference/glance/glance-crd.md index fcde5340f..cafd5984e 100644 --- a/docs/reference/glance/glance-crd.md +++ b/docs/reference/glance/glance-crd.md @@ -21,7 +21,7 @@ stores are **not** part of this spec — they attach out-of-band through | Field | Type | Required | Description | | --- | --- | --- | --- | | `openStackRelease` | `string` | yes | The OpenStack release the operator deploys and drives; pattern `^\d{4}\.[12]$` (the `YYYY.N` cadence, `N` ∈ {1,2}). Governs the API launch mode (eventlet below `2026.1`, uWSGI from `2026.1`) and install/upgrade schema tracking. Kept separate from the image tag so digest-pinned images still resolve a schema and launch mode | -| `deployment` | `DeploymentSpec` | no | Shared pod-level knobs: `replicas` (default 3), `resources` (Burstable defaults), `terminationGracePeriodSeconds`, `preStopSleepSeconds`, `strategy`, `topologySpreadConstraints`, `priorityClassName` | +| `deployment` | `DeploymentSpec` | no | Shared pod-level knobs: `replicas` (default 3), `resources` (defaults: 512Mi request / 1Gi limit memory, 100m/500m CPU), `terminationGracePeriodSeconds`, `preStopSleepSeconds`, `strategy`, `topologySpreadConstraints`, `priorityClassName` | | `image` | `ImageSpec` | yes | Container image; exactly one of `tag` or `digest` (shared CEL rule, re-checked by the webhook) | | `database` | `DatabaseSpec` | yes | MariaDB connection. Exactly one of `clusterRef` (managed) or `host` (brownfield); `credentialsMode` (`Static` \| `Dynamic`, where `Dynamic` requires `clusterRef`), `secretRef`, and optional `tls`. Mutual-exclusivity and the Dynamic-requires-clusterRef rule are inherited from `commonv1.DatabaseSpec` | | `cache` | `CacheSpec` | yes | Memcached backing the Glance image cache. Exactly one of `clusterRef` (managed) or `servers` (brownfield) | @@ -81,11 +81,16 @@ is never emitted). ### Defaulting and validation -The mutating webhook applies the shared `DeploymentSpec`/`LoggingSpec` defaults, -materializes the `PyMemcacheCache` cache backend, fills the `ServiceUserSpec` -identity defaults (`glance` / `service` / `Default` / `Default`, `secretRef.key` -→ `password`), and — only when `spec.apiServer.uwsgi` is present — the uWSGI -sub-field defaults (`processes` 2, `threads` 1, `httpKeepAlive` true). +The mutating webhook applies the shared `DeploymentSpec`/`LoggingSpec` defaults +— with one glance-specific deviation: an unset `spec.deployment.resources` is +filled with 512Mi memory request / 1Gi memory limit (CPU keeps the shared +100m/500m), because the glance-api container carries the boto3-weighted S3 +store driver and overruns the shared 512Mi baseline under concurrent image +traffic. It also materializes the `PyMemcacheCache` cache backend, fills the +`ServiceUserSpec` identity defaults (`glance` / `service` / `Default` / +`Default`, `secretRef.key` → `password`), and — only when +`spec.apiServer.uwsgi` is present — the uWSGI sub-field defaults (`processes` +2, `threads` 1, `httpKeepAlive` true). The validating webhook accumulates every violation into one admission response, reusing the shared validators: replicas floor, image tag/digest XOR, the diff --git a/docs/reference/horizon/horizon-crd.md b/docs/reference/horizon/horizon-crd.md index ac6e86afe..f71a619da 100644 --- a/docs/reference/horizon/horizon-crd.md +++ b/docs/reference/horizon/horizon-crd.md @@ -13,7 +13,7 @@ chart ships a synced copy (`make sync-crds` / `make verify-crd-sync`). | Field | Type | Required | Description | | --- | --- | --- | --- | -| `deployment` | `DeploymentSpec` | no | Shared pod-level knobs: `replicas` (default 3), `resources` (Burstable defaults), `terminationGracePeriodSeconds`, `preStopSleepSeconds`, `strategy`, `topologySpreadConstraints`, `priorityClassName` | +| `deployment` | `DeploymentSpec` | no | Shared pod-level knobs: `replicas` (default 3), `resources` (defaults: 256Mi request / 512Mi limit memory, 100m/500m CPU), `terminationGracePeriodSeconds`, `preStopSleepSeconds`, `strategy`, `topologySpreadConstraints`, `priorityClassName` | | `image` | `ImageSpec` | yes | Container image; exactly one of `tag` or `digest` (shared CEL rule) | | `cache` | `CacheSpec` | yes | Memcached backing the Django cache. Exactly one of `clusterRef` (managed) or `servers` (brownfield); `backend` is a Django cache backend path, defaulted to `django.core.cache.backends.memcached.PyMemcacheCache` | | `keystoneEndpoint` | `string` | yes | The Keystone endpoint URL (`OPENSTACK_KEYSTONE_URL`); must match `^https?://` and parse with a host. Consumed server-side by the dashboard pods, so it must be reachable from inside the cluster — for a colocated control plane use the cluster-local Service URL, not an externally routable address | diff --git a/hack/ci-build-service-image.sh b/hack/ci-build-service-image.sh index 3cda72ec2..51c594c4d 100755 --- a/hack/ci-build-service-image.sh +++ b/hack/ci-build-service-image.sh @@ -5,9 +5,9 @@ # hack/ci-build-service-image.sh — Build an OpenStack service container image. # -# Resolves upstream source refs, clones the project, applies constraint -# overrides, and builds the full image chain: python-base -> venv-builder -> -# service image. +# Resolves upstream source refs, clones the project, applies patches and +# constraint overrides, and builds the full image chain: python-base -> +# venv-builder -> service image. # # Required env vars: # OPERATOR — OpenStack service name (e.g. keystone) @@ -63,12 +63,22 @@ git clone --depth 1 --branch "${SERVICE_REF}" \ "https://github.com/openstack/${OPERATOR}.git" "${SRC_DIR}" # --------------------------------------------------------------------------- -# 4. Apply constraint overrides (idempotent, exits 0 if no overrides) +# 4. Apply patches (mirrors .github/actions/checkout-service-source so the +# e2e-built images carry the same source as the published GHCR images) +# --------------------------------------------------------------------------- +PATCH_DIR="${REPO_ROOT}/patches/${OPERATOR}/${RELEASE}" +if [ -d "${PATCH_DIR}" ] && compgen -G "${PATCH_DIR}/*.patch" > /dev/null; then + echo "Applying patches from ${PATCH_DIR}" + git -C "${SRC_DIR}" apply "${PATCH_DIR}"/*.patch +fi + +# --------------------------------------------------------------------------- +# 5. Apply constraint overrides (idempotent, exits 0 if no overrides) # --------------------------------------------------------------------------- "${REPO_ROOT}/scripts/apply-constraint-overrides.sh" "${RELEASE}" # --------------------------------------------------------------------------- -# 5. Build image chain: python-base -> venv-builder -> service +# 6. Build image chain: python-base -> venv-builder -> service # --------------------------------------------------------------------------- # Reuse existing base images when available (e.g. when a prior invocation in # the same job or an artifact load already created them). Saves ~30s per diff --git a/internal/common/types/workload.go b/internal/common/types/workload.go index aced5eec5..a32b2509e 100644 --- a/internal/common/types/workload.go +++ b/internal/common/types/workload.go @@ -86,9 +86,10 @@ type DeploymentSpec struct { Replicas int32 `json:"replicas,omitempty"` // Resources defines the CPU and memory requests and limits for the service API - // container. When unset, the defaulting webhook injects sensible defaults - // (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and - // enable HPA utilization calculations. + // container. When unset, the defaulting webhook injects the operator's + // documented resource defaults (100m/500m CPU, with memory sized per + // service — see the operator's CRD reference) to ensure Burstable QoS class + // and enable HPA utilization calculations. // +optional Resources *corev1.ResourceRequirements `json:"resources,omitempty"` diff --git a/operators/glance/api/v1alpha1/glance_webhook.go b/operators/glance/api/v1alpha1/glance_webhook.go index 0bff6abcf..c1f1b5e4e 100644 --- a/operators/glance/api/v1alpha1/glance_webhook.go +++ b/operators/glance/api/v1alpha1/glance_webhook.go @@ -12,7 +12,9 @@ import ( "sort" appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/utils/ptr" @@ -53,6 +55,18 @@ const ( DefaultEventletWorkers int32 = 2 ) +// Glance-specific container memory defaults, replacing the shared 256Mi/512Mi +// baseline that keystone and horizon fit in. The glance-api container carries +// the S3 store driver (boto3/botocore), which raises both the per-process +// import footprint and the per-request allocation churn: with the bounded two +// workers the container already idles near 360Mi and, under concurrent image +// traffic, overruns a 512Mi limit within a minute — an OOM-kill crash loop the +// gateway surfaces as waves of 503s. CPU keeps the shared defaults. +var ( + defaultGlanceMemoryRequest = resource.MustParse("512Mi") + defaultGlanceMemoryLimit = resource.MustParse("1Gi") +) + // GlanceWebhook implements defaulting and validation webhooks for the Glance // CRD. Client is injected at startup for cluster-scoped resource lookups (e.g. // PriorityClass validation). Production wiring injects mgr.GetAPIReader() — a @@ -87,9 +101,27 @@ func (w *GlanceWebhook) SetupWebhookWithManager(mgr ctrl.Manager) error { // filled when explicitly present, except spec.logging which is materialized so // downstream reconciler code never sees a nil pointer. func (w *GlanceWebhook) Default(_ context.Context, obj *Glance) error { - // Shared-type defaults (replicas, container resources) are applied by the - // commonv1.DeploymentSpec Default method so they cannot drift across - // operators. + // Fill spec.deployment.resources with the glance-specific memory defaults + // before the shared DeploymentSpec defaults run — Deployment.Default() + // would otherwise inject the shared 256Mi/512Mi baseline, which the + // S3-backed glance-api overruns under load. Same nil-or-empty condition as + // the shared method so an explicit user value is never clobbered. + if obj.Spec.Deployment.Resources == nil || + (len(obj.Spec.Deployment.Resources.Requests) == 0 && len(obj.Spec.Deployment.Resources.Limits) == 0) { + obj.Spec.Deployment.Resources = &corev1.ResourceRequirements{ + Requests: corev1.ResourceList{ + corev1.ResourceMemory: defaultGlanceMemoryRequest.DeepCopy(), + corev1.ResourceCPU: commonv1.DefaultCPURequest(), + }, + Limits: corev1.ResourceList{ + corev1.ResourceMemory: defaultGlanceMemoryLimit.DeepCopy(), + corev1.ResourceCPU: commonv1.DefaultCPULimit(), + }, + } + } + // Shared-type defaults (replicas, remaining container resources) are + // applied by the commonv1.DeploymentSpec Default method so they cannot + // drift across operators. obj.Spec.Deployment.Default() if obj.Spec.Cache.Backend == "" { obj.Spec.Cache.Backend = commonv1.DefaultCacheBackend diff --git a/operators/glance/api/v1alpha1/glance_webhook_test.go b/operators/glance/api/v1alpha1/glance_webhook_test.go index b39be211b..4fbb7fcfc 100644 --- a/operators/glance/api/v1alpha1/glance_webhook_test.go +++ b/operators/glance/api/v1alpha1/glance_webhook_test.go @@ -10,6 +10,7 @@ import ( "github.com/onsi/gomega" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/utils/ptr" @@ -60,14 +61,40 @@ func TestGlanceDefault_MaterializesServiceUserAndLoggingDefaults(t *testing.T) { g.Expect(obj.Spec.ServiceUser.ProjectDomainName).To(gomega.Equal("Default")) g.Expect(obj.Spec.ServiceUser.SecretRef.Key).To(gomega.Equal("password")) - // Shared-block defaults come along too. + // Shared-block defaults come along too — with the glance-specific memory + // values (512Mi request / 1Gi limit for the boto3-weighted S3 path) + // replacing the shared 256Mi/512Mi baseline, while CPU keeps the shared + // defaults. g.Expect(obj.Spec.Deployment.Resources).NotTo(gomega.BeNil()) + g.Expect(obj.Spec.Deployment.Resources.Requests.Memory().String()).To(gomega.Equal("512Mi")) + g.Expect(obj.Spec.Deployment.Resources.Limits.Memory().String()).To(gomega.Equal("1Gi")) + g.Expect(obj.Spec.Deployment.Resources.Requests.Cpu().String()).To(gomega.Equal("100m")) + g.Expect(obj.Spec.Deployment.Resources.Limits.Cpu().String()).To(gomega.Equal("500m")) g.Expect(obj.Spec.Cache.Backend).To(gomega.Equal(commonv1.DefaultCacheBackend)) g.Expect(obj.Spec.Logging).NotTo(gomega.BeNil()) g.Expect(obj.Spec.Logging.Format).To(gomega.Equal("text")) g.Expect(obj.Spec.Logging.Level).To(gomega.Equal("INFO")) } +func TestGlanceDefault_PreservesExplicitResources(t *testing.T) { + g := gomega.NewWithT(t) + w := &GlanceWebhook{} + + obj := validGlance() + obj.Spec.Deployment.Resources = &corev1.ResourceRequirements{ + Limits: corev1.ResourceList{ + corev1.ResourceMemory: resource.MustParse("256Mi"), + }, + } + + g.Expect(w.Default(context.Background(), obj)).To(gomega.Succeed()) + + // A non-empty resources block is left verbatim: neither the glance memory + // defaults nor the shared baseline overwrite an explicit value. + g.Expect(obj.Spec.Deployment.Resources.Limits.Memory().String()).To(gomega.Equal("256Mi")) + g.Expect(obj.Spec.Deployment.Resources.Requests).To(gomega.BeEmpty()) +} + func TestGlanceDefault_PreservesExplicitServiceUserValues(t *testing.T) { g := gomega.NewWithT(t) w := &GlanceWebhook{} diff --git a/operators/glance/config/crd/bases/glance.openstack.c5c3.io_glances.yaml b/operators/glance/config/crd/bases/glance.openstack.c5c3.io_glances.yaml index 309c089fe..207fd65c3 100644 --- a/operators/glance/config/crd/bases/glance.openstack.c5c3.io_glances.yaml +++ b/operators/glance/config/crd/bases/glance.openstack.c5c3.io_glances.yaml @@ -452,9 +452,10 @@ spec: resources: description: |- Resources defines the CPU and memory requests and limits for the service API - container. When unset, the defaulting webhook injects sensible defaults - (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and - enable HPA utilization calculations. + container. When unset, the defaulting webhook injects the operator's + documented resource defaults (100m/500m CPU, with memory sized per + service — see the operator's CRD reference) to ensure Burstable QoS class + and enable HPA utilization calculations. properties: claims: description: |- diff --git a/operators/glance/helm/glance-operator/crds/glance.openstack.c5c3.io_glances.yaml b/operators/glance/helm/glance-operator/crds/glance.openstack.c5c3.io_glances.yaml index 1ba2b823f..c9e7ae5b5 100644 --- a/operators/glance/helm/glance-operator/crds/glance.openstack.c5c3.io_glances.yaml +++ b/operators/glance/helm/glance-operator/crds/glance.openstack.c5c3.io_glances.yaml @@ -455,9 +455,10 @@ spec: resources: description: |- Resources defines the CPU and memory requests and limits for the service API - container. When unset, the defaulting webhook injects sensible defaults - (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and - enable HPA utilization calculations. + container. When unset, the defaulting webhook injects the operator's + documented resource defaults (100m/500m CPU, with memory sized per + service — see the operator's CRD reference) to ensure Burstable QoS class + and enable HPA utilization calculations. properties: claims: description: |- diff --git a/operators/horizon/config/crd/bases/horizon.openstack.c5c3.io_horizons.yaml b/operators/horizon/config/crd/bases/horizon.openstack.c5c3.io_horizons.yaml index c0d49c0e2..6c8d0724e 100644 --- a/operators/horizon/config/crd/bases/horizon.openstack.c5c3.io_horizons.yaml +++ b/operators/horizon/config/crd/bases/horizon.openstack.c5c3.io_horizons.yaml @@ -181,9 +181,10 @@ spec: resources: description: |- Resources defines the CPU and memory requests and limits for the service API - container. When unset, the defaulting webhook injects sensible defaults - (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and - enable HPA utilization calculations. + container. When unset, the defaulting webhook injects the operator's + documented resource defaults (100m/500m CPU, with memory sized per + service — see the operator's CRD reference) to ensure Burstable QoS class + and enable HPA utilization calculations. properties: claims: description: |- diff --git a/operators/horizon/helm/horizon-operator/crds/horizon.openstack.c5c3.io_horizons.yaml b/operators/horizon/helm/horizon-operator/crds/horizon.openstack.c5c3.io_horizons.yaml index d4ca40a1c..89842fe8c 100644 --- a/operators/horizon/helm/horizon-operator/crds/horizon.openstack.c5c3.io_horizons.yaml +++ b/operators/horizon/helm/horizon-operator/crds/horizon.openstack.c5c3.io_horizons.yaml @@ -184,9 +184,10 @@ spec: resources: description: |- Resources defines the CPU and memory requests and limits for the service API - container. When unset, the defaulting webhook injects sensible defaults - (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and - enable HPA utilization calculations. + container. When unset, the defaulting webhook injects the operator's + documented resource defaults (100m/500m CPU, with memory sized per + service — see the operator's CRD reference) to ensure Burstable QoS class + and enable HPA utilization calculations. properties: claims: description: |- diff --git a/operators/keystone/config/crd/bases/keystone.openstack.c5c3.io_keystones.yaml b/operators/keystone/config/crd/bases/keystone.openstack.c5c3.io_keystones.yaml index 0ce329cfb..7b6ebd982 100644 --- a/operators/keystone/config/crd/bases/keystone.openstack.c5c3.io_keystones.yaml +++ b/operators/keystone/config/crd/bases/keystone.openstack.c5c3.io_keystones.yaml @@ -478,9 +478,10 @@ spec: resources: description: |- Resources defines the CPU and memory requests and limits for the service API - container. When unset, the defaulting webhook injects sensible defaults - (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and - enable HPA utilization calculations. + container. When unset, the defaulting webhook injects the operator's + documented resource defaults (100m/500m CPU, with memory sized per + service — see the operator's CRD reference) to ensure Burstable QoS class + and enable HPA utilization calculations. properties: claims: description: |- diff --git a/operators/keystone/helm/keystone-operator/crds/keystone.openstack.c5c3.io_keystones.yaml b/operators/keystone/helm/keystone-operator/crds/keystone.openstack.c5c3.io_keystones.yaml index e808f82d6..229b63eeb 100644 --- a/operators/keystone/helm/keystone-operator/crds/keystone.openstack.c5c3.io_keystones.yaml +++ b/operators/keystone/helm/keystone-operator/crds/keystone.openstack.c5c3.io_keystones.yaml @@ -481,9 +481,10 @@ spec: resources: description: |- Resources defines the CPU and memory requests and limits for the service API - container. When unset, the defaulting webhook injects sensible defaults - (256Mi/512Mi memory, 100m/500m CPU) to ensure Burstable QoS class and - enable HPA utilization calculations. + container. When unset, the defaulting webhook injects the operator's + documented resource defaults (100m/500m CPU, with memory sized per + service — see the operator's CRD reference) to ensure Burstable QoS class + and enable HPA utilization calculations. properties: claims: description: |- diff --git a/patches/glance/2025.2/0001-normalize-scheme-prefixed-s3-host-in-location-repair.patch b/patches/glance/2025.2/0001-normalize-scheme-prefixed-s3-host-in-location-repair.patch new file mode 100644 index 000000000..3fce38195 --- /dev/null +++ b/patches/glance/2025.2/0001-normalize-scheme-prefixed-s3-host-in-location-repair.patch @@ -0,0 +1,46 @@ +SPDX-FileCopyrightText: Copyright 2026 SAP SE or an SAP affiliate company +SPDX-License-Identifier: Apache-2.0 + +Normalize the scheme-prefixed s3_store_host in _construct_s3_url + +s3_store_host must carry its http:// or https:// prefix for non-AWS +endpoints (boto3 requires a scheme in endpoint_url), but the S3 location +URLs stored per image embed only the bare authority — the store strips +the prefix when it builds Store._url_prefix. _construct_s3_url +concatenates the raw option value instead, so the expected URL never +equals any stored location URL: every API request that touches image +locations logs "S3 URL mismatch for image ..., updating URL" and +rewrites the location row, and a genuine credential rotation is +indistinguishable from the permanent false positive. + +Strip the scheme prefix exactly the way Store._url_prefix does before +building the comparison URL. + +Applies to glance 31.1.0 (2025.2) and 32.0.0 (2026.1); the function is +identical in both. + +Upstream status: not yet proposed. + +diff --git a/glance/common/store_utils.py b/glance/common/store_utils.py +index 57f417d..036d705 100644 +--- a/glance/common/store_utils.py ++++ b/glance/common/store_utils.py +@@ -283,6 +283,18 @@ def _construct_s3_url(store_instance, scheme, path): + s3_host = getattr(store_instance, 's3_host') + bucket = getattr(store_instance, 'bucket') + ++ # The store keeps s3_store_host verbatim, and the option may carry an ++ # http:// or https:// prefix (boto3 requires a scheme in endpoint_url for ++ # non-AWS endpoints). Location URLs embed only the bare authority (see ++ # Store._url_prefix), so strip the prefix here as well — otherwise the ++ # comparison in _update_s3_location_and_store_id never converges and ++ # every API request logs a spurious "S3 URL mismatch" and rewrites the ++ # image location. ++ if s3_host.startswith('https://'): ++ s3_host = s3_host[len('https://'):] ++ elif s3_host.startswith('http://'): ++ s3_host = s3_host[len('http://'):] ++ + # Construct the full URL with the object path + return "%s://%s:%s@%s/%s%s" % ( + scheme, access_key, secret_key, s3_host, bucket, path) diff --git a/patches/glance/2026.1/0001-normalize-scheme-prefixed-s3-host-in-location-repair.patch b/patches/glance/2026.1/0001-normalize-scheme-prefixed-s3-host-in-location-repair.patch new file mode 100644 index 000000000..3fce38195 --- /dev/null +++ b/patches/glance/2026.1/0001-normalize-scheme-prefixed-s3-host-in-location-repair.patch @@ -0,0 +1,46 @@ +SPDX-FileCopyrightText: Copyright 2026 SAP SE or an SAP affiliate company +SPDX-License-Identifier: Apache-2.0 + +Normalize the scheme-prefixed s3_store_host in _construct_s3_url + +s3_store_host must carry its http:// or https:// prefix for non-AWS +endpoints (boto3 requires a scheme in endpoint_url), but the S3 location +URLs stored per image embed only the bare authority — the store strips +the prefix when it builds Store._url_prefix. _construct_s3_url +concatenates the raw option value instead, so the expected URL never +equals any stored location URL: every API request that touches image +locations logs "S3 URL mismatch for image ..., updating URL" and +rewrites the location row, and a genuine credential rotation is +indistinguishable from the permanent false positive. + +Strip the scheme prefix exactly the way Store._url_prefix does before +building the comparison URL. + +Applies to glance 31.1.0 (2025.2) and 32.0.0 (2026.1); the function is +identical in both. + +Upstream status: not yet proposed. + +diff --git a/glance/common/store_utils.py b/glance/common/store_utils.py +index 57f417d..036d705 100644 +--- a/glance/common/store_utils.py ++++ b/glance/common/store_utils.py +@@ -283,6 +283,18 @@ def _construct_s3_url(store_instance, scheme, path): + s3_host = getattr(store_instance, 's3_host') + bucket = getattr(store_instance, 'bucket') + ++ # The store keeps s3_store_host verbatim, and the option may carry an ++ # http:// or https:// prefix (boto3 requires a scheme in endpoint_url for ++ # non-AWS endpoints). Location URLs embed only the bare authority (see ++ # Store._url_prefix), so strip the prefix here as well — otherwise the ++ # comparison in _update_s3_location_and_store_id never converges and ++ # every API request logs a spurious "S3 URL mismatch" and rewrites the ++ # image location. ++ if s3_host.startswith('https://'): ++ s3_host = s3_host[len('https://'):] ++ elif s3_host.startswith('http://'): ++ s3_host = s3_host[len('http://'):] ++ + # Construct the full URL with the object path + return "%s://%s:%s@%s/%s%s" % ( + scheme, access_key, secret_key, s3_host, bucket, path)