Sunset un-paginated GET /sites: migrate remaining legacy-shape consumers

[rainer-friederich]

Background

GET /sites recently gained optional pagination (limit/cursor) via PR #2497 (commit ac0dab47, SITES-41367). When called without limit/cursor, the endpoint still returns the legacy flat array (full fetchAllPages scan) for backwards compatibility, and emits a [sites][legacy-shape] log line recording the caller's clientId.

We want to sunset the un-paginated path. This issue tracks the remaining consumers that still call GET /sites without pagination params so we can migrate them before removing the legacy shape.

Method

Coralogix, prod (spacecat-services-prod, account aws-640168421876, function /spacecat-services/api-service/latest), aggregating the [sites][legacy-shape] marker by clientId.

• Window: 2026-06-08 → 2026-06-11 (last 3 days)
• The legacy-shape log line carries only clientId — no User-Agent, headers, or source IP (all null).
• S2S clientId values resolved to consumer names via the prod consumers Postgres table (idx_consumers_client_id).

Findings — consumers of the un-paginated GET /sites (last 3 days)

Named S2S consumers (migration targets)

Calls	Consumer name	clientId	Notes
169	s2s-llmo-data-retrieval-service	6a342b9f35a84f4ab60e4f7f82dfeda0	DRS — BP monitor, site-name maps, schedulers
71	AMS Sites Optimizer ESE Validator Script and Bot	a3c925e9aaeb43e7b96c9f7b5c1d7f3c	Registered 2026-06-03
34	S2S LLMO Usage Report Generator	500bb9199427463f9c7f6dd022d2248b	LLMO usage/MAU reporting
8	S2S AEM-Trustcenter	2f61fa914f324902b9b5bc0c96bfcb4f	AEM Trustcenter integration
1	AEM Engineering Operations DaaS ETL	1980219afe204b13b84116f118bcd33f	Registered 2026-06-10 — already on legacy shape

All five are ACTIVE, under IMS org 908936ED5D35CC220A495CD4@AdobeOrg, and hold site:readAll.

Admin path — admin-bypass (4,539 calls)

The vast majority of un-paginated calls came through the shared admin credential (admin x-api-key / admin IMS JWT). These all log clientId=admin-bypass, and because the legacy-shape log line carries no user-agent/source, they cannot be attributed to a specific consumer from these logs alone. This bucket is internal tooling / UI / scripts / monitors sharing the admin key.

Open items

• Attribute the admin-bypass bucket. It is ~95% of legacy-shape volume and is currently opaque. Break it down via API Gateway access logs (they carry User-Agent + source IP keyed by the same requestId). Consider also enriching the [sites][legacy-shape] log line with User-Agent so future attribution is possible directly.
• Migrate the 5 named S2S consumers to pass ?limit=&cursor= (paginated envelope):
  • s2s-llmo-data-retrieval-service (DRS)
  • AMS Sites Optimizer ESE Validator Script and Bot
  • S2S LLMO Usage Report Generator
  • S2S AEM-Trustcenter
  • AEM Engineering Operations DaaS ETL
• Flag to the DaaS ETL owners that their newly registered consumer (2026-06-10) is already integrating against the legacy shape — fix before sunset.
• Run the same breakdown for stage/dev before removal.
• Define the deprecation/sunset timeline and remove the legacy flat-array path once consumers are migrated.

Notes

• This analysis is prod only.
• The legacy path performs a full fetchAllPages scan (filtering internal orgs to stay under the 6MB Lambda limit), which is the cost motivating the sunset.

[martinst06]

admin-bypass bucket attribution

The admin-bypass calls trace to a single frontend, adobe/project-elmo-ui (the LLMO Dashboard), not multiple separate callers. Evidence from Fastly CDN access logs in Coralogix (window 2026-06-04 to 06-11):

• Host: the requests hit llmo.experiencecloud.live, not spacecat.experiencecloud.live.
• User-agent: browser UAs across many Chrome, Safari, and Firefox versions, desktop and mobile.
• Referer: llmo.now (2,014), localhost:3000 (3,148), pr-NN.d2ikwb7s634epv.amplifyapp.com (~430 combined), llmo-dashboard.adobecqms.net, llmo.adobe.com.
• Repo mapping: Amplify app d2ikwb7s634epv and llmo.experiencecloud.live/api/v1 map to adobe/project-elmo-ui.
• Code path: project-elmo-ui src/contexts/SiteContext.tsx:455 calls spacecatApi.adminsGetAllSites().then(allSites => allSites.filter(isLlmoEnabled)). It fetches the unpaginated GET /api/v1/sites and filters client-side. Auth is the user's admin or read-only-admin IMS token (SpaceCatAuthProvider.tsx:279, jwtUtils.ts:104), which the API logs as admin-bypass.

The remaining <2% is ad-hoc tooling: Postman, Insomnia, curl, Python scripts, chatgpt-user, and Slack link unfurling.

Migration list: project-elmo-ui, llmo-data-retrieval-service, and the four external consumers (AMS Sites Optimizer ESE Validator, S2S LLMO Usage Report Generator, S2S AEM-Trustcenter, AEM Engineering Operations DaaS ETL).

[martinst06]

DRS (s2s-llmo-data-retrieval-service) — resolved by adobe-rnd/llmo-data-retrieval-service#2142

DRS's legacy-shape traffic came from the content scraper's site-id resolution, which called the bare GET /sites?baseURL= and filtered client-side (the baseURL filter returns all accessible sites, so it pulled the full list). PR 2142 replaces that with a direct GET /sites/by-base-url/{b64} lookup, dropping the unpaginated list call entirely — cleaner than paginating.

DRS's other GET /sites callers already pass ?limit=&cursor= (the shared SpaceCat client list methods and the SLA picker site_directory), so the content scraper was the only un-migrated path.

One caveat to verify post-deploy: CDN logs show a bp data quality monitor - spacecat/1.0 user-agent also hitting the bare /api/v1/sites (~16 hits over 7 days), which 2142 doesn't touch and may authenticate as admin rather than the DRS consumer (so it could fall in the admin-bypass bucket). Once 2142 ships I'll confirm clientId 6a342b9f… drops to zero on [sites][legacy-shape].

https://github.com/adobe-rnd/llmo-data-retrieval-service/pull/2142
https://github.com/adobe-rnd/llmo-data-retrieval-service/pull/2033
https://github.com/adobe-rnd/llmo-data-retrieval-service/pull/2421

[martinst06]

S2S LLMO Usage Report Generator — migrated

S2S LLMO Usage Report Generator (500bb9199427463f9c7f6dd022d2248b) is the llmo-usage-report-node CLI (x-client-type: llmo-usage-report-cli).

As of OneAdobe/llmo-usage-report-node#23 (merged) it now calls GET /sites?limit=500&cursor=… and walks the full paginated envelope instead of the legacy flat array (500 = server MAX_LIMIT). This was its only GET /sites call site. Since it's an operator-run CLI, legacy-shape hits will taper off as people pick up the new version rather than dropping at a single deploy — safe to tick once prod logs show zero hits.

[martinst06]

AEM Engineering Operations DaaS ETL and AMS Sites Optimizer ESE Validator Script and Bot have been notified, will mark as done when merged

[martinst06]

S2S AEM-Trustcenter migrated 2026-06-10 - @bisswanger

[martinst06]

AEM Engineering Operations DaaS ETL migrated 2026-06-17 - @dgranber

[martinst06]

AEM Engineering Operations DaaS ETL and AMS Sites Optimizer ESE Validator Script and Bot have been notified, will mark as done when merged

AMS Sites Optimizer ESE Validator Script and Bot migrated 2026-06-18 / 2026-06-19 - ams-sites-optimizer/pull/186 and 2026-06-16 - ams-sites-optimizer/pull/195

[rainer-friederich]

Log re-check (2026-06-29) — legacy-shape traffic is not yet zero, plus one untracked consumer

I re-ran the [sites][legacy-shape] breakdown against current logs. Findings below, including the observability caveats that made this harder than the original Coralogix pass.

Observability caveats (important for whoever owns the sunset)

• Coralogix is gone — the original data source no longer exists.
• api-service does not ship to Splunk. The prod backend sourcetype (dx_aem_sites_spacecat_backend_prod) only carries workers (content-scraper, audit-worker, import-worker, auth-service); there is no api-service service field. A Splunk query for the marker returns a misleading zero.
• The only working source is CloudWatch (/aws/lambda/spacecat-services--api-service, prod acct 640168421876), and its retention is only 7 days. So the window below is 2026-06-22 → 06-29 and no longer-term trend is recoverable. This matters because the code's own exit criterion (sites.js, the legacy branch: "remove once Coralogix shows zero hits for 30 consecutive days") is currently unmeasurable — Coralogix is gone, Splunk doesn't see api-service, and CloudWatch only holds 7 days.

Legacy-shape callers, last 7 days (2026-06-22 → 06-29)

clientId	Consumer	Calls	Daily (06-22 → 06-29)	Read
admin-bypass	shared admin key (mostly project-elmo-ui + ad-hoc tooling)	580	93, 135, 112, 78, 108, 21, 20, 13	Dropped ~80 percent on 06-27 — consistent with the project-elmo-ui paginated migration deploying. Residual is the ad-hoc/admin-key tail. The drop is sharper than that weekend's overall traffic dip (~43M to ~23M lines/day), so a real deploy, not just lower volume, drove it down.
324cf5195c7d4bc6ae2720bd521706a9	LLMO Activation and Usage Dashboard	40	0, 3, 10, 3, 17, 2, 2, 3	Not tracked in this issue. Consumer registered 2026-06-16 (after the original analysis), ACTIVE, IMS org 908936ED5D35CC220A495CD4@AdobeOrg, holds site:readAll. A brand-new integration built directly onto the legacy shape — same pattern as the DaaS ETL already flagged here.
6a342b9f35a84f4ab60e4f7f82dfeda0	s2s-llmo-data-retrieval-service (DRS)	48	0, 10, 9, 0, 8, 11, 10, 0	Steady ~10/day, not zero. Matches the BP data-quality-monitor (bp data quality monitor - spacecat/1.0) residual that the DRS comment above flagged as untouched by PR 2142. The DRS box is ticked but this path still emits.
500bb9199427463f9c7f6dd022d2248b	S2S LLMO Usage Report Generator	46	0, 28, 0, 2, 16, 0, 0, 0	Sporadic — operator-run CLI tapering as people pick up the new version, as predicted.

What I think still needs doing

• Add a new migration target: "LLMO Activation and Usage Dashboard" (324cf5195c7d4bc6ae2720bd521706a9) — registered after this issue, integrating against the legacy shape, ~40 calls in 7 days and steady.
• DRS is not actually at zero — the BP monitor path is still ~10/day. Worth confirming whether that path will be migrated or is acceptable until sunset.
• The two unticked open items (stage/dev breakdown, sunset timeline) are still genuinely open.
• The suggested log enrichment was not implemented — the legacy line still logs only requestId + clientId, no User-Agent, so admin-bypass stays unattributable from these logs.

@martinst06 — could you take this further when you have a cycle? Specifically:

1. Identify and reach out to the owners of LLMO Activation and Usage Dashboard (324cf519…) and add it to the migration checklist.
2. Decide whether the DRS BP-monitor residual (~10/day) gets migrated or is accepted until removal.
3. Before we can trust a "zero hits" signal again, we need an observability home for this marker now that Coralogix is gone and api-service isn't in Splunk — either get api-service Lambda logs shipping to Splunk, or bump the CloudWatch retention on /aws/lambda/spacecat-services--api-service beyond 7 days, otherwise the 30-consecutive-days exit gate can't be evaluated.
4. Run the stage/dev breakdown and propose the sunset timeline.

(Numbers from CloudWatch Logs Insights over the api-service prod log group; consumer names resolved via the prod consumers table.)