Manually test every guide against its declared devstack

[berendt]

Category: feature | Scope: Large

Description

This issue tracks a one-time manual verification of every published guide.
The Acceptance Criteria below are the tracking checklist: one checkbox per
guide, grouped by the devstack the guide declares. A checkbox is ticked when
the guide has been walked by hand once, end to end — ticked means tested,
not passed (per the author). A failed walk is recorded at the entry and
filed as its own defect issue; the fix lands outside this issue.

Concrete boundaries:

1. Already exists: 23 guides under docs/guides/ — 6 top-level, 11 under
   keystone/, 2 under horizon/, 4 under glance/ — all reachable from the
   docs nav (docs/.vitepress/config.ts). Each guide declares exactly one
   devstack in a ::: info Devstack block in its Prerequisites section and
   ends with a ## Tested by section naming the chainsaw suites that mirror
   it (for example docs/guides/day-2-operations.md:300).
   This issue adds: a recorded manual walk of each guide, with a result
   note per checklist entry. No repository file changes; the deliverable is
   the completed checklist on this issue, not a pull request.
2. Already exists: CI coverage that mirrors each guide — the Tested-by
   suites apply equivalent fixture manifests and assert the same end state.
   They never execute the guide's literal command lines, its host-side steps
   (browser logins, curl against nip.io hostnames, openstack CLI calls),
   or its prose claims about expected output.
   This issue adds: the literal-execution check — every command run
   verbatim, every claimed output compared against what a reader actually
   sees.

Test protocol

Each checklist entry below is walked under this protocol:

1. Fresh devstack per guide. Run make teardown-infra (deletes the kind
   cluster, hack/teardown-infra.sh:70), then the exact bring-up line from
   the guide's ::: info Devstack block, then follow the quick-start that
   block references through its final Verify step. Never reuse a cluster
   across entries: guides mutate their devstack (key and password rotations,
   credential migrations, second ControlPlanes), and a ControlPlane
   re-deployed onto a used cluster can wedge on leftover PVC and OpenBao
   state. The registry pull-through cache (WITH_REGISTRY_CACHE opt-in in
   hack/deploy-infra.sh) persists only image blobs and may stay enabled
   across walks to speed up repeated bring-ups.
2. Verbatim execution. Run every command exactly as printed. The only
   permitted substitutions are the ones the guide itself instructs (for
   example swapping fernet for credential in
   docs/guides/keystone/keystone-key-rotation.md).
3. Output comparison. At every verify step, compare the observed output
   against what the guide claims. A mismatch is a failure even when the
   overall procedure ends in a working system.
4. Result note. Append to the checklist entry: test date, the main
   commit SHA tested, and pass or fail.
5. On failure, classify the defect and file one issue per distinct cause,
   referencing this issue, then tick the box:
   • guide defect — the text is wrong (stale command, renamed field, wrong
     claimed output);
   • product defect — the guide is right and the operator misbehaves;
   • environment defect — host-specific interference unrelated to both
     (for example container-runtime ulimits on a local kind node).
     Link the filed issue at the entry.
6. Bring-up failure. A failure before the guide's first own step counts
   against the referenced quick-start, not the guide: file it as a defect
   issue, leave the entry unticked, and retry the walk once a devstack comes
   up.

Motivation

The guides are the operator-facing surface of this project: they are the
first thing a platform operator executes against a real cluster. Every one of
them carries a Tested-by section, so it is tempting to read them as verified —
but the suites mirror the flow with fixture manifests; nobody and nothing runs
the printed words. A renamed flag, a changed status output, or a stale
hostname ships silently, and the first person to find it is a reader stranded
mid-procedure with a half-mutated cluster.

The gap is a single recorded fact per guide: has a human walked this page, as
written, against the devstack it names, and did it behave as claimed? Today
that fact exists for no guide.

This issue closes the gap with one campaign: 23 manual walks, one fresh
devstack each, each result recorded at its checklist entry and each failure
turned into a filed, linked defect issue. When the last box is ticked, every
published guide has a known, dated test status.

Affected Areas

• docs/guides/ (all 23 pages — the subject under test; read-only, no edits
  land from this issue)
• this issue's body (checklist entries accrue result notes and defect-issue
  links as walks complete)
• hack/deploy-infra.sh, hack/teardown-infra.sh (the bring-up and teardown
  tooling every walk uses; unchanged)

Acceptance Criteria

Campaign-level:

• Tear down and re-create the devstack before every walk (make teardown-infra, then the guide's declared bring-up line) — no cluster is
  reused across entries.
• Record at every ticked entry a result note with test date, main
  commit SHA, and pass or fail.
• File one defect issue per distinct failure cause, classified as guide,
  product, or environment defect, referencing this issue and linked at the
  affected entry.
• File a bring-up failure that occurs before a guide's first own step as
  a defect against the referenced quick-start and leave the affected entry
  unticked until a devstack comes up.

Per guide — Quick Start devstack
(KIND_HOST_PORT=8443 make deploy-infra):

• Walk docs/guides/keystone/keystone-admin-password-scheduled-rotation.md
  (Schedule Keystone Admin Password Rotation) per the test protocol; append
  the result note here.
• Walk docs/guides/keystone/ldap-domain-backend.md (Attach an LDAP
  Domain Backend) per the test protocol; append the result note here.
• Walk docs/guides/keystone/saml-federation.md (Attach a SAML
  Federation Backend) per the test protocol; append the result note here.
• Walk docs/guides/keystone/enable-keystone-operator-networkpolicy.md
  (Enable Keystone Operator NetworkPolicy) per the test protocol; append the
  result note here.

Per guide — Quick Start + Prometheus devstack
(KIND_HOST_PORT=8443 WITH_PROMETHEUS=true make deploy-infra):

• Walk docs/guides/keystone/enable-keystone-operator-metrics.md
  (Enable Keystone Operator Metrics) per the test protocol; append the
  result note here.

Per guide — Quick Start (Extended) devstack
(kind create cluster --name forge --config hack/kind-config.yaml, then
make deploy-infra):

• Walk docs/guides/observability.md (Observability & Diagnostics) per
  the test protocol; append the result note here.

Per guide — Quick Start (ControlPlane) devstack
(KIND_HOST_PORT=8443 WITH_CONTROLPLANE=true make deploy-infra):

• Walk docs/guides/advanced-configuration.md (Advanced Configuration)
  per the test protocol; append the result note here.
• Walk docs/guides/day-2-operations.md (Day 2 Operations) per the test
  protocol; append the result note here.
• Walk docs/guides/dedicated-service-namespaces.md (Deploy Services
  into Dedicated Namespaces) per the test protocol; append the result note
  here.
• Walk docs/guides/end-to-end-sso.md (End-to-End SSO) per the test
  protocol; append the result note here.
• Walk docs/guides/multi-tenant-deployment.md (Multi-Tenant
  Deployment) per the test protocol; append the result note here.
• Walk docs/guides/keystone/keystone-key-rotation.md (Rotate Keystone
  Keys) per the test protocol; append the result note here.
• Walk docs/guides/keystone/keystone-admin-password-rotation.md
  (Rotate Keystone Admin Password) per the test protocol; append the result
  note here.
• Walk docs/guides/keystone/oidc-federation.md (Attach an OIDC
  Federation Backend) per the test protocol; append the result note here.
• Walk docs/guides/keystone/enable-keystone-database-tls.md (Enable
  Keystone Database TLS) per the test protocol; append the result note here.
• Walk docs/guides/keystone/migrate-keystone-db-to-dynamic-credentials.md
  (Migrate Keystone DB to Dynamic Credentials) per the test protocol; append
  the result note here.
• Walk docs/guides/keystone/adopt-external-keystone.md (Adopt an
  External Keystone) per the test protocol; append the result note here.
• Walk docs/guides/horizon/enable-horizon-operator-networkpolicy.md
  (Enable Horizon Operator NetworkPolicy) per the test protocol; append the
  result note here.
• Walk docs/guides/glance/glance-s3-multistore.md (Attach S3
  Multi-Store Backends to Glance) per the test protocol; append the result
  note here.
• Walk docs/guides/glance/enable-glance-operator-networkpolicy.md
  (Enable Glance Operator NetworkPolicy) per the test protocol; append the
  result note here.
• Walk docs/guides/glance/migrate-glance-db-to-dynamic-credentials.md
  (Migrate Glance DB to Dynamic Credentials) per the test protocol; append
  the result note here.

Per guide — Quick Start (ControlPlane) + Prometheus devstack
(KIND_HOST_PORT=8443 WITH_CONTROLPLANE=true WITH_PROMETHEUS=true make deploy-infra):

• Walk docs/guides/horizon/enable-horizon-operator-metrics.md (Enable
  Horizon Operator Metrics) per the test protocol; append the result note
  here.
• Walk docs/guides/glance/enable-glance-operator-metrics.md (Enable
  Glance Operator Metrics) per the test protocol; append the result note
  here.

Non-Goals

• Fixing the defects a walk finds. Each failure lands in its own filed
  issue; this issue records discovery, not repair. Ticked entries may point
  at open defect issues.
• Checklist entries for the three quick-starts (docs/quick-start.md,
  docs/quick-start-extended.md, docs/quick-start-controlplane.md) — per
  the author. Every walk executes its guide's quick-start through the final
  Verify step as a prerequisite, so they are exercised implicitly; bring-up
  failures are still filed against them (protocol step 6).
• Sub-issues per guide — per the author, the tracking lives in this flat
  checklist.
• New automation. No new chainsaw suites or CI jobs; the Tested-by suites
  already carry the automated half, and the point of this campaign is the
  manual half.
• Pages outside docs/guides/ (docs/reference/, docs/contributing/,
  docs/future/, docs/index.md) — they are not step-by-step procedures.

References

• docs/.vitepress/config.ts — the nav inventory the 23-guide list is drawn
  from
• docs/quick-start.md, docs/quick-start-extended.md,
  docs/quick-start-controlplane.md — the devstacks the guides build on
• hack/deploy-infra.sh, hack/teardown-infra.sh — bring-up and teardown
  used by every walk

---

Elaborated by planwerk-agent with Claude:claude-fable-5