fix: allow S2S consumers with site:readAll to submit authenticated scrape jobs

[aenascut]

Problem

POST /tools/scrape/jobs with a siteId (required when options.enableAuthentication: true) applies a per-site org-membership check (hasAccess(site)) that always fails for cross-tenant S2S principals. A service token carries the principal's own IMS org in its tenants list, not the customer's org — so hasOrganization(customerImsOrgId) is unconditionally false regardless of what capabilities the consumer holds.

Every other cross-tenant S2S route in this service (sites.js, organizations.js, configuration.js) already has the hasS2SCapability(CAP_SITE_READ_ALL) fallback. The scrape job controller was missed.

Evidence

Traced via Splunk (index=dx_aem_engineering sourcetype=dx_aem_sites_mystique_backend_prod) and CloudWatch (SpaceCat prod account 640168421876) during incident SITES-47964 (twilio.com, site 275d1b4e, org 138eaa2a).

Scan c474b9f1 — authenticated path, DRS job dde32831

Mystique submits batch at 10:23:41. Six seconds later the poller logs:

DRS job dde32831-a4aa-4f3a-a672-b8ad28eb8404 failed:
Exception: All 10 SpaceCat sub-job submission(s) failed —
first error: SpaceCat auth error (HTTP 403) during submit —
check S2S credentials: {"message":"User does not have access to this site"}

DRS's S2S principal passed L1 (scrapeJob:write capability present, consumer ACTIVE) but hit the org-membership wall at L2 (hasAccess(site)). All 10 sub-job POSTs returned HTTP 403. Fast-fail at 13.9s.

Scan 8339a60b — unauthenticated path (no siteId), DRS job 870b4218

No siteId → hasAccess not reached → job accepted (HTTP 200), 10 job IDs 019f5b0a-* created, 100 URL messages queued on spacecat-scrape-content-processing-jobs.fifo. The content-scraper Lambda logged zero events for all 10 job IDs across the 4-hour window. DRS backstop tripped at 14400s:

DRS job 870b4218-66a7-4705-b2e7-9d0c0840e718 failed:
SpaceCat jobs exceeded max age of 14400s still non-terminal:
['019f5b0a-98ff-7035-8ef4-bc2876cdcbb3', '019f5b0a-9911-7bc4-a463-2b06a0455641',
 '019f5b0a-98f2-7a20-bdff-8a9c8b3ac256', '019f5b0a-991c-7d51-bd1c-bbd23ece1091',
 '019f5b0a-9947-72c2-b117-f3e41097d823', '019f5b0a-9a01-7b89-96f5-51cc4ed29d3f',
 '019f5b0a-9a37-7b50-b79f-9ae55c8ea5d1', '019f5b0a-9a75-77e7-9291-21086110933d',
 '019f5b0a-9a42-7662-b817-497048dbc3cb', '019f5b0a-9a53-7981-bc0a-89948bb64687']

Root cause for this path: spacecat-scrape-queue-1.fifo had ~175k messages backlogged at the time. The 100 twilio messages were enqueued but expired (4-day SQS retention) before the content-scraper reached them. This is a separate scaling issue tracked in SITES-47964.

What changed

createScrapeJob in scrapeJob.js now mirrors the access-control pattern used by every other cross-tenant S2S route:

// Before — org-membership check only, always fails for S2S cross-tenant principals
const accessControlUtil = AccessControlUtil.fromContext(requestContext);
if (!await accessControlUtil.hasAccess(site)) {
  return forbidden('User does not have access to this site');
}

// After — S2S capability fallback added
const accessControlUtil = AccessControlUtil.fromContext(requestContext);
const isAdmin = accessControlUtil.hasAdminReadAccess();
const s2sResult = isAdmin
  ? { allowed: false, reason: 'admin-bypass' }
  : await accessControlUtil.hasS2SCapability(CAP_SITE_READ_ALL);
if (!isAdmin && !s2sResult.allowed && !await accessControlUtil.hasAccess(site)) {
  return forbidden('User does not have access to this site');
}

hasS2SCapability re-fetches the Consumer record from the DB on every call (stale token cannot grant access). An S2S consumer without site:readAll still falls through to hasAccess(site) — existing behaviour for all other consumers is unchanged.

Auth flow after this fix

DRS token → POST /tools/scrape/jobs (with siteId + enableAuthentication: true)

L1 — s2sAuthWrapper
  Consumer found by (clientId, imsOrgId), status ACTIVE
  capabilities includes 'scrapeJob:write' ✅

L2 — createScrapeJob (this PR)
  isAdmin = false
  hasS2SCapability('site:readAll')
    re-fetches Consumer from DB
    capabilities includes 'site:readAll' ✅  ← requires follow-up data grant below
    → { allowed: true, reason: 'granted' }
  s2sResult.allowed = true → skips forbidden() ✅

Follow-up required — grant DRS the site:readAll capability

This code change is necessary but not sufficient. The DRS Consumer record must be updated to include site:readAll before the fix takes effect. Without the data grant, hasS2SCapability returns { allowed: false, reason: 'missing-capability' } and the 403 persists.

DRS is already registered as an S2S consumer with scrapeJob:write (that is how it passes L1). The update only adds the cross-tenant read capability.

Steps (requires is_s2s_admin token):

1. Look up the DRS consumer by its clientId:

GET /api/v1/consumers/by-client-id/{drs_client_id}

2. Note the returned consumerId. Then patch capabilities:

PATCH /api/v1/consumers/{consumerId}
Content-Type: application/json

{
  "capabilities": ["scrapeJob:write", "scrapeJob:read", "site:readAll"]
}

Valid capabilities follow entity:operation format where operation ∈ { read, write, delete, readAll, create }. site:readAll is the existing capability used by all other cross-tenant S2S routes (e.g. GET /sites, GET /sites/:siteId/identity) — no new capability string is introduced.

The PATCH endpoint is admin-gated (hasS2SAdminAccess() required). It re-validates capabilities on write and logs the change. The update takes effect immediately on the next DRS request — no deploy needed after the data grant.

Suggested verification after deploy + data grant:
Re-run a DRS authenticated bundle scan against twilio prod site 275d1b4e. Expect HTTP 200 on POST /tools/scrape/jobs and SpaceCat job IDs transitioning to COMPLETED within the normal scrape window.

Secondary issue (not fixed here)

spacecat-scrape-queue-1.fifo had ~175k messages backlogged at investigation time (2026-07-20). spacecat-scrape-queue-2.fifo exists but has no event source mapping to content-scraper. Adding one would double consumer throughput with no infra changes. The DLQ (spacecat-scraper-dlq.fifo) held 277 messages from failed scrapes across multiple customers. Both items tracked in SITES-47964.

Related

• SITES-47964 — twilio.com prod DRS failure (incident this was traced from)
• SITES-46195 — twilio prod onboarding
• SITES-39521 — Mysticat epic