feat: adopt the shared expand-migrate-contract upgrade flow for glance

.claude/skills/prepare-new-service/SKILL.md

@@ -48,7 +48,14 @@ applies and which decisions need a Phase-0 spike:
 
 - **Database?** Which migration tool (alembic `db_sync` vs Django
   `migrate` vs none)? No DB ⇒ drop the database/db-sync/upgrade
-  sub-reconcilers entirely. A DB service also inherits the **dynamic
+  sub-reconcilers entirely. Which zero-downtime upgrade mechanism does
+  the manage tool support (expand-migrate-contract, single-pass
+  `db sync`, online data migrations), and which phases must straddle the
+  workload rollout? A database-backed service adopts the shared
+  `database.ReconcileUpgrade` flow (`internal/common/database/upgrade.go`),
+  keying it off the image tag (keystone) or `spec.openStackRelease`
+  (glance), or documents in its reference set why single-pass is
+  acceptable. A DB service also inherits the **dynamic
   credential chain**: the shared `credentialsMode` on `DatabaseSpec`, a
   per-service `databaseCredentialsMode` override on its c5c3 service spec,
   a `provision_service_tenant` leg in
@@ -151,8 +158,8 @@ a second (or third) time?** Classify keystone internals into:
 1. thin wrappers over `internal/common` — copy as pattern, fine;
 2. generic logic living in keystone (pipeline/status machinery, workload
    builders, watch mappers, webhook validators) — **extraction candidates**;
-3. genuinely keystone-specific (fernet, bootstrap, trust-flush,
-   expand-migrate-contract) — leave alone, rule of three.
+3. genuinely keystone-specific (fernet, bootstrap, trust-flush) — leave
+   alone, rule of three.
 
 If category 2 is non-empty, file (or update) a **separate refactor issue**
 listing the candidates with file:line references, S/M/L effort, and a

docs/.vitepress/config.ts

@@ -114,6 +114,7 @@ export default defineConfig({
               { text: 'Backend CRD', link: '/reference/glance/glance-backend-crd' },
               { text: 'Controller Events', link: '/reference/glance/glance-events' },
               { text: 'Reconciler Architecture', link: '/reference/glance/glance-reconciler' },
+              { text: 'Upgrade Flow', link: '/reference/glance/glance-upgrade-flow' },
             ],
           },
           {

docs/reference/glance/glance-crd.md

@@ -115,16 +115,16 @@ owns via `spec.serviceUser.secretRef`.
 | `conditions` | List-map keyed by `type`; see the [reconciler reference](./glance-reconciler.md#conditions) for the vocabulary |
 | `observedGeneration` | The `.metadata.generation` last reconciled |
 | `endpoint` | The Glance API URL: `https://{gateway.hostname}/` when a gateway is set, otherwise the cluster-local Service URL |
-| `installedRelease` | The OpenStack release whose schema is currently installed, promoted to `spec.openStackRelease` after a successful `db sync` |
-| `targetRelease` | The `spec.openStackRelease` being converged to during an active release transition |
+| `installedRelease` | The OpenStack release whose schema is currently installed, promoted to `spec.openStackRelease` after the upgrade completes (or after the first `db sync` on a fresh install) |
+| `targetRelease` | The `spec.openStackRelease` being upgraded to during an active release transition. Set when the upgrade initiates, cleared on completion or abort; empty in steady state |
+| `upgradePhase` | The current expand-migrate-contract phase during an active release upgrade (`Expanding`, `Migrating`, `RollingUpdate`, `Contracting`); empty when no upgrade is in flight |
 
 ::: info
-Unlike Keystone, Glance carries **no `upgradePhase` field**: its migrations run
-in a single `glance-manage db sync`, so a release transition is tracked only
-through `installedRelease`/`targetRelease`, with no expand-migrate-contract
-phase state. That is a trade-off, not only simpler status: without the phase
-split, the outgoing release's pods serve against the already-migrated schema for
-the duration of the roll — see [Database](./glance-reconciler.md#database).
+A release transition (a `spec.openStackRelease` bump with the image in lockstep)
+walks the shared expand-migrate-contract phase machine, tracked through
+`upgradePhase`/`targetRelease`. See the
+[Glance Upgrade Flow](./glance-upgrade-flow.md) for the phases, condition
+reasons, events, and abort semantics.
 :::
 
 ## Sub-Resource Naming Convention

docs/reference/glance/glance-events.md

@@ -84,13 +84,31 @@ All events follow these conventions:
 > schema-check Job, so — unlike Keystone — **`SchemaDriftDetected` never fires
 > for Glance**.
 
-### Release Validation
+### Upgrade
+
+A release transition (a `spec.openStackRelease` bump with the image in lockstep)
+walks the shared expand-migrate-contract flow, which emits these events on the
+Glance CR. For the phase machine see the
+[Glance Upgrade Flow](./glance-upgrade-flow.md).
 
 | Reason | Type | Trigger Condition | Example Message |
 | --- | --- | --- | --- |
-| `InvalidReleaseTransition` | Warning | The requested `spec.openStackRelease` is a downgrade, or a jump that is neither patch-only nor a single sequential step, relative to `status.installedRelease` | `downgrade from 2026.1 to 2025.2 is not supported` |
-
-**Source:** `rejectReleaseTransition` in `reconcile_database.go`
+| `UpgradeInitiated` | Normal | An accepted release bump starts the upgrade | `Upgrade initiated: 2025.2 → 2026.1` |
+| `ExpandComplete` | Normal | The expand phase Job succeeded | `Expand phase complete: 2025.2 → 2026.1` |
+| `MigrateComplete` | Normal | The migrate phase Job succeeded | `Migrate phase complete: 2025.2 → 2026.1` |
+| `DeploymentRolloutComplete` | Normal | The Deployment rolled out; the phase flips to Contracting | `Deployment rollout complete during upgrade 2025.2 → 2026.1` |
+| `UpgradeComplete` | Normal | The contract phase Job succeeded; the upgrade finished | `Upgrade complete: 2025.2 → 2026.1` |
+| `UpgradeAborted` | Normal | `spec.openStackRelease` reverted to the installed release, cancelling the upgrade | `Upgrade 2025.2 → 2026.1 aborted: spec release reverted to installed release 2025.2` |
+| `VersionParseError` | Warning | The installed or target release is not a valid `YYYY.N` string | `Failed to parse target release "latest": ...` |
+| `DowngradeNotSupported` | Warning | The target release is older than the installed release | `Downgrade from 2026.1 to 2025.2 is not supported` |
+| `UpgradePathInvalid` | Warning | The requested jump is not a single sequential step | `Upgrade from 2024.2 to 2026.1 is not sequential` |
+| `UpgradeTargetChanged` | Warning | `spec.openStackRelease` changed to a third value during an active upgrade | `Spec release changed to 2026.2 during active upgrade 2025.2 → 2026.1` |
+| `ExpandFailed` | Warning | The expand phase Job failed permanently | `Expand job glance-db-expand failed: ...` |
+| `MigrateFailed` | Warning | The migrate phase Job failed permanently | `Migrate job glance-db-migrate failed: ...` |
+| `ContractFailed` | Warning | The contract phase Job failed permanently | `Contract job glance-db-contract failed: ...` |
+
+**Source:** the shared expand-migrate-contract flow in
+`internal/common/database/upgrade.go`
 
 ### Finalization
 
@@ -116,8 +134,8 @@ Use `kubectl get events --field-selector` to filter by reason:
 # Watch for db-sync failures
 kubectl get events --field-selector reason=DBSyncFailed -w
 
-# Watch for a rejected release transition
-kubectl get events --field-selector reason=InvalidReleaseTransition -w
+# Watch for a rejected release upgrade path
+kubectl get events --field-selector reason=UpgradePathInvalid -w
 
 # Watch for a skipped image-store backend
 kubectl get events --field-selector reason=GlanceBackendSkipped -w
@@ -148,18 +166,18 @@ groups:
           summary: "Glance db-sync failed"
           description: "The Glance db-sync Job has failed. Check the Job logs for details."
 
-      - alert: GlanceInvalidReleaseTransition
+      - alert: GlanceUpgradePhaseFailed
         expr: |
           increase(kube_event_count{
-            reason="InvalidReleaseTransition",
+            reason=~"ExpandFailed|MigrateFailed|ContractFailed",
             involved_object_kind="Glance"
           }[5m]) > 0
         for: 0m
         labels:
-          severity: warning
+          severity: critical
         annotations:
-          summary: "Glance release transition rejected"
-          description: "spec.openStackRelease requests an unsupported transition from the installed release."
+          summary: "Glance database upgrade phase failed"
+          description: "An expand, migrate, or contract Job failed during a Glance release upgrade. Check the phase Job logs and consider aborting the upgrade."
 ```
 
 ---
@@ -180,9 +198,16 @@ GlanceReconciler.Reconcile()
   │     └─ spec.extraConfig overrides operator-owned keys → Warning ExtraConfigOwnedKeyOverride
   │       (gated on transition into ExtraConfigHealthy=False, Reason=OwnedKeysOverridden)
   │
-  └── reconcileDatabase()
-        ├─ Invalid release transition → Warning InvalidReleaseTransition
-        ├─ db_sync fails              → Warning DBSyncFailed
-        ├─ db_sync succeeds           → Normal  DatabaseSynced
-        └─ Job-UID patch fails        → Warning DBSyncMetricEmissionDeferred
+  ├── reconcileDatabase()
+  │     ├─ db_sync fails              → Warning DBSyncFailed
+  │     ├─ db_sync succeeds           → Normal  DatabaseSynced
+  │     ├─ Job-UID patch fails        → Warning DBSyncMetricEmissionDeferred
+  │     └─ release upgrade            → Normal  UpgradeInitiated / ExpandComplete /
+  │                                             MigrateComplete / UpgradeComplete / UpgradeAborted
+  │                                      Warning VersionParseError / DowngradeNotSupported /
+  │                                             UpgradePathInvalid / UpgradeTargetChanged /
+  │                                             ExpandFailed / MigrateFailed / ContractFailed
+  │
+  └── reconcileDeployment()
+        └─ rollout ready mid-upgrade  → Normal  DeploymentRolloutComplete
 ```

docs/reference/glance/glance-reconciler.md

@@ -33,8 +33,8 @@ Secrets ──► DBConnectionSecret ──► Backends ──► Config ──
 | DBConnectionSecret | Materializes the pymysql DSN into the derived `{name}-db-connection` Secret and digests it; **reports through `SecretsReady`** | `SecretsReady` |
 | Backends | Aggregates the attached, credential-ready `GlanceBackend`s into the content-hashed backends Secret. Waiting states never short-circuit the pipeline (first-install proceeds; a backend status flip re-enqueues via the watch) | `BackendsReady` |
 | Config | Renders `glance-api.conf` / `glance-api-paste.ini` (plus `policy.yaml` / `logging.conf` when applicable) into an immutable content-addressed ConfigMap. An invalid projection keeps the live Deployment's last-good names instead of re-rendering. **Reports through `SecretsReady`** (failure reason `ConfigError`) | `SecretsReady` |
-| Database | Provisions/migrates the schema (MariaDB gate, `Database`/`User`/`Grant`, one `glance-manage db sync` Job); promotes `installedRelease` | `DatabaseReady` |
-| Deployment | Ensures the API Deployment (both launch modes), Service (port 9292), and PDB; stamps `status.endpoint` | `DeploymentReady` |
+| Database | Provisions/migrates the schema (MariaDB gate, `Database`/`User`/`Grant`, one `glance-manage db sync` Job); a release bump instead runs the shared expand-migrate-contract flow; promotes `installedRelease` | `DatabaseReady` |
+| Deployment | Ensures the API Deployment (both launch modes), Service (port 9292), and PDB; stamps `status.endpoint`. Mid-upgrade it flips the `RollingUpdate` phase to `Contracting` once the rolled-out Deployment reports ready | `DeploymentReady` |
 | HTTPRoute | Full `spec.gateway` lifecycle; reflects the Gateway's Accepted condition | `HTTPRouteReady` |
 | HealthCheck | HTTP GET of the cluster-local `/healthcheck` through the shared TTL probe cache | `GlanceAPIReady` |
 | HPA | Creates/deletes the HorizontalPodAutoscaler | `HPAReady` |
@@ -55,7 +55,7 @@ eight sub-conditions are `True`; otherwise `False` (`NotAllReady`).
 | --- | --- | --- |
 | `SecretsReady` | `SecretsAvailable` | `SecretStoreNotReady`, `WaitingForDBCredentials`, `WaitingForServiceUserCredentials`, `ConfigError` |
 | `BackendsReady` | `AllBackendsProjected` | `WaitingForBackends`, `NoDefaultBackend` |
-| `DatabaseReady` | `DatabaseSynced` | `ClusterNotReady`, `WaitingForDatabase`, `DBSyncFailed`, `DBSyncInProgress`, `InvalidReleaseTransition`, `WaitingForBackends` |
+| `DatabaseReady` | `DatabaseSynced` | `ClusterNotReady`, `WaitingForDatabase`, `DBSyncFailed`, `DBSyncInProgress`, `WaitingForBackends`, `ImageReleaseMismatch`, `VersionParseError`, `DowngradeNotSupported`, `UpgradePathInvalid`, `UpgradeTargetChanged`, `ExpandInProgress`, `MigrateInProgress`, `UpgradeRollingUpdate`, `ContractInProgress`, `ExpandFailed`, `MigrateFailed`, `ContractFailed` |
 | `DeploymentReady` | `DeploymentReady` | `WaitingForDeployment`, `WaitingForBackends` |
 | `GlanceAPIReady` | `APIHealthy` | `APIUnhealthy`, `EndpointNotReady`, `HealthCheckTimeout`, `ConnectionFailed`, `HealthCheckFailed` |
 | `HPAReady` | `HPAReady`, `HPANotRequired` | — (errors propagate) |
@@ -116,27 +116,24 @@ The Database step runs the shared provisioning + sync flow: a MariaDB cluster
 gate and `Database`/`User`/`Grant` in managed mode (a no-op in brownfield),
 then a single `{name}-db-sync` Job running `glance-manage db sync` followed by
 `db load_metadefs`. Because Glance's migrations apply in one idempotent pass,
-there is **no schema-check Job** — so, unlike Keystone, `SchemaDriftDetected`
-never fires for Glance. On Job success `installedRelease` is promoted to
-`spec.openStackRelease`.
-
-A release transition is validated against `installedRelease`: a downgrade, or a
-jump that is neither patch-only nor a single sequential step, is rejected with
-`DatabaseReady=False / InvalidReleaseTransition` and an `InvalidReleaseTransition`
-Warning event. The rejection short-circuits the pipeline (non-zero requeue, no
-error) so the Deployment is never rolled to new code against an un-migrated
-schema; it stays rejected until the spec changes.
-
-**The reverse window is not closed.** `db sync` is a single pass with no
-expand/migrate/contract split, and the pipeline runs `Database ──► Deployment`,
-so on an accepted release bump the schema advances to the new release while the
-outgoing release's pods keep serving — for the whole rolling update, and
-`deployment.replicas` defaults to 3. Surviving pods therefore serve *old code
-against the migrated schema*. A release transition is consequently **not a
-zero-downtime operation**: drain or scale down before bumping
-`spec.openStackRelease` if request failures during the roll are unacceptable.
-There is no downgrade path to fall back to — `InvalidReleaseTransition` rejects
-it.
+there is **no schema-check Job**, so, unlike Keystone, `SchemaDriftDetected`
+never fires for Glance. On fresh installs and patch bumps `installedRelease` is
+promoted to `spec.openStackRelease` on Job success.
+
+A release transition (a `spec.openStackRelease` bump with the image in lockstep)
+instead dispatches to the shared expand-migrate-contract flow
+(`internal/common/database`), the same phase machine Keystone runs. The step
+validates the path against `installedRelease`, rejecting a downgrade or a
+non-sequential jump with `VersionParseError`, `DowngradeNotSupported`, or
+`UpgradePathInvalid`, then walks `Expanding → Migrating → RollingUpdate →
+Contracting` with `glance-manage db expand|migrate|contract` phase Jobs on the
+new image. The outgoing pods keep serving the expanded schema across the
+rollout, so the API stays available, and `installedRelease` is promoted only
+after contract completes. The Deployment step owns the `RollingUpdate →
+Contracting` flip: once `reconcileDeployment` sees the rolled-out Deployment
+ready, it advances the phase to `Contracting` and requeues so the contract Job
+runs. See the [Glance Upgrade Flow](./glance-upgrade-flow.md) for the phase
+table, condition reasons, events, and abort semantics.
 
 ## Requeue semantics