feat(aso-overlay): accept AEM CS preview-tier -prev service suffix#2832
Conversation
AEM Cloud Service injects `-prev` into the `AEM_SERVICE` env var on preview
publish pods (e.g. `cm-p154709-e1629980-prev`) for its own internal tier
routing. Skyline dispatcher builds the overlay URL from that env var
verbatim, so preview traffic hits us with `/config/cm-pN-eN-prev/redirects.txt`.
Current state on the aso-overlay Splunk board (2026-07-16): ~273/hr 401s
for `-prev` URLs — the Fastly VCL regex `[^/]+` accepts the suffix, but our
`AsoOverlayKeyHandler.OVERLAY_ROUTE` regex and `RedirectsController.SERVICE_RE`
both reject anything not exactly `cm-pN-eN`, so:
1. auth handler doesn't match the path → falls through → 401 at authWrapper
2. even with correct edge auth, controller `SERVICE_RE` would 400
Mystique writes exactly one overlay per (program, environment) — publish and
preview tiers share it. So we accept the suffix at both the auth + controller
boundaries and strip it before all downstream lookups (Site lookup, S3 key,
Surrogate-Key). Origin-side stripping avoids requiring a Fastly VCL change
(dispatcher-team territory) and lands the fix on our side alone.
Fastly cache footprint doubles per tenant (preview URL and publish URL become
separate cache entries under raw-URL keying). Acceptable because:
1. Both share `Surrogate-Key: aso-overlay-<canonical>`, so mystique's
purge-on-Deploy invalidates both in a single call.
2. Both fetch the same S3 object (canonical), so upstream is not doubled
beyond initial per-URL fills.
3. At 10k tenants × ~20% preview coverage the extra entries are cheap.
Changes:
- `SERVICE_RE`: accept optional trailing `-prev`, keep case-sensitive
- `RedirectsController.getRedirects`: derive `canonicalService` from
captured program/environment ids; use it for S3 key + Surrogate-Key on
both 200 and 304 branches
- `AsoOverlayKeyHandler.OVERLAY_ROUTE`: accept the `-prev` suffix so the
auth handler doesn't 401 preview traffic before it reaches the controller
- Unit tests: -prev happy path, 304 on -prev, malformed variants still 400,
auth handler accepts -prev
- IT test: -prev end-to-end against the seeded canonical overlay
- OpenAPI: pattern + description updated
SITES-48140
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Regression fix from post-review audit on this PR: **Bug:** On the 200 path, `emitS3Duration(SUCCESS)` fires right after `s3Client.send()` resolves. If the subsequent `response.Body.transformToString()` throws (SDK stream error, aborted download, decode failure), the outer catch falls into the UNEXPECTED branch and calls `emitS3Duration(UNEXPECTED)` for the same request — producing two `AsoOverlayS3ReadDurationMs` samples with contradictory `S3Result` dimensions. On a rollout where a fraction of requests fail mid-body, the S3-tier histogram would over-count and the SUCCESS bucket would carry noise from what were actually failures. **Fix:** Wrap `emitS3Duration` in an idempotency flag — first call wins. The caller-observable outcome is still correctly recorded by `emitFinal` on the final status, so the request-total chart is unaffected. The change is scoped: one flag, one guard, no path re-shuffling. Also update OpenAPI to document the `Surrogate-Key` response header on both 200 and 304, since Mystique's purge-on-Deploy contract depends on it and it was previously undocumented for consumers reading the spec. Regression test asserts exactly one `AsoOverlayS3ReadDurationMs` envelope is emitted when the body stream throws post-send, and that the SUCCESS result wins (first-writer semantics). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
This PR will trigger a patch release when merged. |
|
Converting to Draft based on Cornel's Slack feedback earlier today (quoted at the top of the PR body): preview pods aren't supposed to be polling Self-review of this PR agreed with Cornel's read and recommended closing rather than merging. The concern is two-fold:
Leaving the PR as Draft (not closed) so it can be re-marked ready in the future if:
Both audit-follow-up fixes (I1 metric double-emit + I4 OpenAPI Next step: I'll ping @boston and @Slavic per Cornel's suggestion to align on direction. |
|
Reframed and un-drafted. See the top of the PR body for the defense-in-depth positioning. Short version: this PR does not weaken auth. The concern raised in the earlier review — "future auth-chain regression could auto-allow keyless overlay traffic" — applies equally to the canonical Fastly edge remains the primary diagnostic surface for "preview pods are misconfigured": the 401 rate by URL there is unchanged by this PR (we only change origin behavior). Skyline can still see the signal. Bundled audit follow-ups still land here regardless:
|
…-overlay-preview-tier # Conflicts: # docs/index.html
# Conflicts: # docs/index.html
# [1.660.0](v1.659.1...v1.660.0) (2026-07-16) ### Features * **aso-overlay:** accept AEM CS preview-tier -prev service suffix ([#2832](#2832)) ([4890495](4890495)) * **referral-traffic:** plumb categoryName through controller | LLMO-6026 ([#2790](#2790)) ([a77092f](a77092f)), closes [Hi#level](https://github.com/Hi/issues/level)
|
🎉 This PR is included in version 1.660.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
AEM Cloud Service preview publish pods send
AEM_SERVICE=cm-pN-eN-prev. Skyline dispatcher builds the overlay URL from that env var verbatim, so preview traffic hits us with/config/cm-pN-eN-prev/redirects.txt. Two api-service regexes reject the shape today:AsoOverlayKeyHandler.OVERLAY_ROUTEdoes not match-prev, so the request falls through the auth chain and 401s atauthWrapperbefore the controller runs.RedirectsController.SERVICE_REwould 400 the shape.Mystique writes exactly one overlay per (program, environment) — publish and preview tiers share it. This PR accepts the suffix at both boundaries and strips it before all downstream lookups (Site lookup, S3 key, Surrogate-Key). Origin-side stripping avoids requiring a Fastly VCL change and lands the fix on our side alone.
Auth model is preserved end-to-end.
AsoOverlayKeyHandlerstill requiresX-ASO-API-Keyregardless of URL shape — accepting-prevat the route regex does not weaken any authentication guarantee. A future auth-chain regression that accidentally auto-allowed keyless overlay-route requests would be the actual security problem, and that risk exists forcm-pN-eNas much as forcm-pN-eN-prev.Fastly cache footprint
Fastly caches on raw URL, so preview and publish create two separate cache entries per tenant. Trade-off is acceptable because:
Surrogate-Key: aso-overlay-<canonical>, so mystique's purge-on-Deploy invalidates both in a single call — no divergence risk.A future normalization at Fastly VCL (
regsubinvcl_recvbefore cache-key derivation) would collapse to one entry per tenant, but that requires Fastly-admin territory. Origin-side is the pragmatic bridge.Changes
SERVICE_RE(src/controllers/redirects.js:43): accept optional trailing-prev, case-sensitiveRedirectsController.getRedirects: derivecanonicalServicefrom captured program/environment ids; use it for S3 key + Surrogate-Key on both 200 and 304 branchesAsoOverlayKeyHandler.OVERLAY_ROUTE(src/support/aso-overlay-key-handler.js:33): accept-prevso the auth handler doesn't 401 preview traffic before it reaches the controllers3DurationEmittedidempotency guard (audit follow-up): fix double-emission ofAsoOverlayS3ReadDurationMswhen the body stream throws post-send(). Test inredirects.test.jsasserts exactly one emission with SUCCESS result (first-writer-wins).Surrogate-Keydocumentation (audit follow-up): document the header on both 200 and 304 response schemas so the mystique purge-on-Deploy contract is discoverable-prevhappy path, 304 on-prev, malformed variants still 400 (-preview,-PREV,-prev-2, trailing hyphen), auth handler accepts-prev-prevend-to-end against the seeded canonical overlay-prevJira: SITES-48140
Test plan
redirects.test.jspass, all auth handler tests green-prevURL returns the same canonical body +surrogate-key: aso-overlay-cm-pN-eNheader via curl on dev-prevURLs is unchanged (the edge signal for upstream misconfig is preserved)Post-review audit findings
Self-review by staff-engineer-level reviewer flagged:
AsoOverlayS3ReadDurationMs— fixed in this PR vias3DurationEmittedguardSurrogate-Keydoc — fixed in this PRresolveEnvironmentsilent fallback — split out to feat(metrics-emf): warn once when AWS_ENV/ENV missing #2835 (independent concern)🤖 Generated with Claude Code