Skip to content

fix: glance S3 location churn and undersized memory defaults#726

Merged
berendt merged 2 commits into
mainfrom
fix/glance-s3-host-and-memory-defaults
Jul 23, 2026
Merged

fix: glance S3 location churn and undersized memory defaults#726
berendt merged 2 commits into
mainfrom
fix/glance-s3-host-and-memory-defaults

Conversation

@berendt

@berendt berendt commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Problem

make dizzy-glance against the quick-start ControlPlane kept failing even after #725 bounded the eventlet worker count: 416 of 929 image operations returned HTTP 503, in waves where whole 30-second buckets failed completely. Live diagnosis showed two independent defects:

  1. The glance-api container is OOM-killed under load. The bounded two workers are three Python processes carrying boto3/botocore; the container idles near 360Mi of the shared 512Mi limit and overruns it within a minute of concurrent image traffic. Each OOM kill takes the single replica down, and the gateway answers 503 until the crash-loop backoff lets it return.
  2. Every image request logs S3 URL mismatch for image ..., updating URL and rewrites the location row. Glance's S3 credential-rotation repair (_construct_s3_url) concatenates the raw s3_store_host — which must keep its http:// prefix for boto3's endpoint_url on non-AWS endpoints like Garage — while the stored location URLs embed only the bare authority (Store._url_prefix strips the prefix). The comparison never converges, so the repair fires on every request, forever.

Fix

Commit 1 introduces the repo's first source patches under patches/glance/{2025.2,2026.1}/: strip the scheme prefix in _construct_s3_url exactly the way Store._url_prefix does. The function is identical in glance 31.1.0 and 32.0.0; both patch files were validated with git apply --check against both upstream tags. It also teaches hack/ci-build-service-image.sh to apply patches after cloning, mirroring .github/actions/checkout-service-source — without this, e2e- and locally-built images would silently diverge from the published GHCR images the moment a patch exists.

Commit 2 gives glance service-specific memory defaults: the defaulting webhook fills an unset spec.deployment.resources with a 512Mi request / 1Gi limit (CPU keeps the shared 100m/500m) before the shared DeploymentSpec defaults run, with the same nil-or-empty guard so explicit values are never clobbered. The shared Resources godoc no longer hardcodes 256Mi/512Mi for every operator; the keystone/horizon/glance CRDs and their Helm chart copies are regenerated, and the concrete values are documented in the per-operator CRD references. Pre-existing CRs keep their materialized resources — the webhook only fills empty blocks.

Verification

Reproduced and verified live on the quick-start stack (kind, Garage S3 backend, dizzy small chaos scenario, 5-minute soak):

512Mi limit (before) 1Gi limit (after)
Image operations 513/929 OK, 416 × http_503 1034/1034 OK
glance-api restarts during soak 4 (OOMKilled, exit 137) 0
Container memory peak OOM at 512Mi 576Mi
Throughput 1.7 ops/s 3.4 ops/s

The 576Mi peak sits above the old limit and comfortably below the new one, confirming the old default could not survive the soak regardless of worker count.

Test plan

  • go test ./operators/glance/... ./internal/common/... (new webhook unit tests cover the glance-specific defaulting and the explicit-value passthrough)
  • make verify-crd-sync, gofumpt -l, shellcheck hack/ci-build-service-image.sh all clean
  • Both patch files validated with git apply --check against pristine glance 31.1.0 and 32.0.0 trees (including the SPDX preamble)

🤖 Generated with Claude Code

berendt added 2 commits July 23, 2026 19:31
Glance's S3 credential-rotation repair (_update_s3_location_and_store_id
in glance/common/store_utils.py) rebuilds the expected location URL by
concatenating the raw s3_store_host option. That option must carry its
http:// or https:// prefix for non-AWS endpoints — boto3 requires a
scheme in endpoint_url — but the location URLs stored per image embed
only the bare authority, because the store strips the prefix when it
builds Store._url_prefix. The two URLs therefore never converge: every
API request that touches image locations logs "S3 URL mismatch for
image ..., updating URL" and rewrites the location row in the database.
Against the operator-rendered Garage backend that is one spurious
UPDATE per GET/PATCH, and a genuine credential rotation is
indistinguishable from the permanent false positive.

Introduce the repo's first source patches (the patches/ mechanism has
existed in the Build Images workflow since the start but was unused):
strip the scheme prefix in _construct_s3_url exactly the way
Store._url_prefix does. The function is identical in glance 31.1.0
(2025.2) and 32.0.0 (2026.1), so both release directories carry the
same patch, validated with git apply against both upstream tags.

Also teach hack/ci-build-service-image.sh to apply patches after
cloning, mirroring .github/actions/checkout-service-source — without
this the e2e- and locally-built service images would silently diverge
from the published GHCR images the moment a patch exists. Sync the
script's step numbering and the CI reference page.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
The shared DeploymentSpec baseline (256Mi request / 512Mi limit) is
calibrated for keystone and horizon. The glance-api container carries
the S3 store driver on top: boto3/botocore raise the per-process import
footprint (three Python processes with the bounded two workers) and add
per-request client-construction churn. Measured on the quick-start
stack, the container idles near 360Mi of the 512Mi limit and, under
dizzy's concurrent image traffic, is OOM-killed within a minute — a
crash loop the gateway surfaces as waves of 503s (416 of 929 image
operations failed in a five-minute soak). With a 1Gi limit the same
soak passes 1034/1034 with a 576Mi peak.

Fill spec.deployment.resources in the glance defaulting webhook with a
512Mi request / 1Gi limit (CPU keeps the shared 100m/500m) before the
shared DeploymentSpec defaults run, using the same nil-or-empty guard
so an explicit user value is never clobbered. The request moves up with
the limit because the shared 256Mi request sits below glance's idle
footprint. Pre-existing CRs keep their materialized 512Mi limit — the
webhook only fills empty resource blocks.

The shared Resources godoc hardcoded the 256Mi/512Mi literals for every
operator; reword it to point at the per-operator CRD reference and
regenerate the keystone/horizon/glance CRDs plus their Helm chart
copies. Document the concrete values in the glance and horizon CRD
references (keystone already lists its own) and cover the new
defaulting and the explicit-value passthrough with webhook unit tests.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
@berendt
berendt merged commit 4f8c227 into main Jul 23, 2026
89 checks passed
@berendt
berendt deleted the fix/glance-s3-host-and-memory-defaults branch July 23, 2026 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant