feat: Semrush Elements prompts (count) endpoint for URL Inspector

[rainer-friederich]

Summary

Add a new Semrush Elements API wrapper endpoint that returns the prompts (and their count) for the URL Inspector / Brand Presence dashboards, filtered by topic tag, AI model, and one or more Semrush projects.

This is the next endpoint in the Semrush Elements wrapper family introduced by:

• feat: introduce semrush elements apis wrapper for llmo ui #2666 — feat: introduce semrush elements apis wrapper for llmo ui (the src/support/elements/* layer + ElementsController)
• feat: Semrush Elements weeks endpoint for URL Inspector | LLMO-6011 #2755 — feat: Semrush Elements weeks endpoint for URL Inspector | LLMO-6011 (the closest reference implementation — copy its shape)

It wraps a new Semrush element that is not yet registered in src/support/elements/element-ids.js:

POST .../products/ai/elements/406ba6e0-0de2-475e-80d9-42fab8616032

Observed upstream contract

The UI/POC calls the Semrush element directly today. This is the request/response we must reproduce through the SpaceCat wrapper (SpaceCat mints the IMS token and injects the workspace).

Request (raw Semrush v4-raw/external-api, filters block only):

{
  "render_data": {
    "filters": {
      "simple": {},
      "advanced": {
        "op": "and",
        "filters": [
          { "col": "tags", "op": "contains", "val": "topic:AI" },
          { "op": "or", "filters": [ { "op": "eq", "val": "search-gpt", "col": "CBF_model" } ] },
          { "op": "or", "filters": [
            { "op": "eq", "val": "b558a5e8-d9cb-4ace-907d-825eb4f9c0db", "col": "CBF_project" },
            { "op": "eq", "val": "5f0e8c91-dbd5-4b96-91e4-803fc920a589", "col": "CBF_project" }
          ] }
        ]
      }
    }
  }
}

Response (type: "table"):

{
  "type": "table",
  "blocks": {
    "data": [
      { "primary_intent": "informational", "prompt": "can i make ai influencer for free", "prompt_topic": "AI Instagram Influencers", "volume": 2119 },
      { "primary_intent": "informational", "prompt": "What is the best AI free image generator?", "prompt_topic": "AI Image Generators", "volume": 997 }
    ]
  },
  "config": { "data": null }
}

Gateway note: the curl above hits the public apis/v4-raw/external-api/v1/... path and wraps the payload in render_data. The SpaceCat transport (src/support/elements/elements-transport.js#fetchElement) posts to the internal v3 gateway /enterprise/pages/api/v3/workspaces/{workspaceId}/products/ai/elements/{elementId}/data and sends the inner object directly (no render_data wrapper) — exactly like the existing buildMarketsPayload / buildTopicsPayload / buildWeeksPayload builders. The new payload builder must return the un-wrapped { comparison_data_formatting, filters: { advanced: {...} } } shape, not the render_data-wrapped shape.

Proposed endpoint

GET /v2/orgs/:spaceCatId/serenity/all/brand-presence/prompts

Sits alongside the existing org-scoped elements routes in src/routes/index.js (lines 241–242):

GET /v2/orgs/:spaceCatId/serenity/all/brand-presence/url-inspector/filter-dimensions  → listUrlInspectorFilterDimensions
GET /v2/orgs/:spaceCatId/serenity/all/brand-presence/weeks                            → listWeeks

Query parameters

Name	Required	Maps to	Notes
spaceCatId (path)	✅	workspace resolution	Resolved to semrush_workspace_id via authorizeOrg (already implemented)
model / platform	❌	CBF_model (OR group)	Run through resolveElementModel() (UI platform code → Semrush model, default search-gpt). model wins if both sent — mirror buildWeeksPayload
topic	❌	tags contains topic:<value>	The topic: prefix is added server-side (the filter-dimensions topics array returns bare labels). Support one or more (repeated param or CSV) → multiple contains filters
projectId	❌	CBF_project (OR group)	Semrush project UUID(s). Support one or more (repeated param or CSV). The UI already has these as semrush_project_id from the filter-dimensions regions array

Open question (resolve with product / UI before build):

• Should the endpoint also accept a siteId and reverse-map it to the site's brand's Semrush projects (CBF_project), the way listWeeks reverse-maps siteId → brand → CBF_ws_brand? The observed payload passes explicit CBF_project UUIDs, so explicit projectId is the primary contract; siteId resolution would be an additive convenience. Recommend shipping explicit projectId first.
• Response shape / naming. Branch is feat/prompt_count_endpoint. Confirm whether consumers want the count only, the prompt list, or both. Recommendation below returns both so one endpoint serves both needs.

Recommended response

{
  "count": 6,
  "prompts": [
    { "prompt": "can i make ai influencer for free", "topic": "AI Instagram Influencers", "intent": "informational", "volume": 2119 },
    { "prompt": "What is the best AI free image generator?", "topic": "AI Image Generators", "intent": "informational", "volume": 997 }
  ]
}

count = prompts.length. Field rename prompt_topic → topic, primary_intent → intent keeps the wrapper contract UI-facing rather than leaking raw Semrush column names (consistent with how the other definitions reshape their raw rows). Confirm final field names with the UI.

Implementation checklist

Mirror the #2755 (weeks) diff — it touched the exact same set of files.

1. src/support/elements/element-ids.js — register the new element:

   // Topic Prompts (URL Inspector prompt list / count)
   PROMPTS_TABLE: '406ba6e0-0de2-475e-80d9-42fab8616032', // confirm canonical constant name

2. src/support/elements/definitions/prompts.js (new) — buildPromptsPayload({ model, platform, topics, projectIds }) returning the un-wrapped payload (advanced and group: tags contains topic:* filters + CBF_model OR group via resolveElementModel + CBF_project OR group), and transformPromptsResponse(raw) mapping raw.blocks.data[] → { prompt, topic, intent, volume } and computing count. Guard against a missing/empty blocks.data (return { count: 0, prompts: [] }).
3. src/support/elements/definitions/index.js — re-export the two new functions.
4. src/support/elements/elements-service.js — add getPrompts(workspaceId, params) that calls transport.fetchElement(workspaceId, ELEMENT_IDS.PROMPTS_TABLE, buildPromptsPayload(params)) and returns transformPromptsResponse(raw).
5. src/controllers/elements.js — add a listPrompts(ctx) handler modeled on listWeeks: authorizeOrg → extractQuery → parse topic/projectId (repeated + CSV) → buildService(ctx).getPrompts(...) → ok(result) → mapError. Export it from the controller factory.
6. src/routes/index.js — register GET /v2/orgs/:spaceCatId/serenity/all/brand-presence/prompts → elementsController.listPrompts.
7. src/routes/facs-capabilities.js — add 'GET .../brand-presence/prompts': 'llmo/can_view'.
8. src/routes/required-capabilities.js — add 'GET .../brand-presence/prompts': 'organization:read'.
9. test/routes/index.test.js — add the new route to the expected-routes list (the route-map test fails the build otherwise).
10. Docs — extend docs/llmo-semrush-apis/filter-dimensions-apis.md (add a "List Prompts" section) and add the new element row to docs/elements/semrush-elements-api-reference.md.

FACS param classification

No new dynamic :param is introduced (only the existing :spaceCatId, already classified), so the routeFacsCapabilities disjointness test needs no new alias — but the route must still be added to both capability maps in steps 7–8 (the FACS capability test enumerates all routes).

Auth model — REQUIRED (per platform guidance)

Any Semrush-wrapping API must use x-promise-token as the request header for the upstream Semrush call, and authenticate to SpaceCat itself with the spacecat JWT session token on Authorization. This is live in prod now.

Concretely, for this endpoint:

• AuthN to SpaceCat: caller sends a spacecat JWT session token on Authorization: Bearer <jwt> (a non-IMS credential — authInfo.getType() !== 'ims').
• AuthN to Semrush: caller sends x-promise-token (minted by POST /auth/v2/promise). The controller exchanges it for an IMS token and forwards that upstream.

The shared elements layer already implements exactly this and needs no new work — just make sure the new listPrompts handler goes through it (as listWeeks does):

• resolveElementsImsToken (src/controllers/elements.js) checks x-promise-token first via the shared resolveSemrushImsToken helper; when present, the IMS-type gate in requireImsBearer is never hit, so a JWT-authenticated caller passes. When the header is absent it falls back to requiring IMS-type auth and returns the PROMISE_TOKEN_REQUIRED_ERROR_CODE 401 pointing the caller at the promise-token flow. Introduced in feat: add support for promise token for semrush api authentication #2753.
• buildService(ctx) → resolveElementsImsToken(ctx) → createElementsTransport({ env, imsToken }). Do not re-read Authorization directly in the handler.

Implication for the checklist: step 5's listPrompts must call buildService(ctx) (not build a transport itself), so the promise-token path is inherited. Add a controller test asserting the promise-token path reaches getPrompts and the no-token/non-IMS path returns the PROMISE_TOKEN_REQUIRED_ERROR_CODE 401.

Other infra (already handled by the shared layer — no new work)

• Access control: authorizeOrg (org lookup + AccessControlUtil.hasAccess + workspace resolution) — reuse as-is.
• Error mapping: reuse mapError (upstream 401/403 → passthrough, other upstream → 502, ElementsTransportError/ErrorWithStatusCode handled).

Testing

• Unit (test/controllers/elements.js, test/support/elements/...): payload-builder shape (topic prefixing, OR groups for model + multiple projects, model resolution/fallback), response transform (field rename, count, empty-blocks.data → { count: 0, prompts: [] }), and controller auth/error branches. Note feat: Semrush Elements weeks endpoint for URL Inspector | LLMO-6011 #2755 marked its handler c8 ignore as a POC; prefer real unit tests here unless this is explicitly a throwaway POC.
• Integration (test/it/): if the serenity Semrush mock supports this element, add a wiring test; otherwise note the mock gap.
• Run npm run docs:lint / npm run docs:build after editing OpenAPI/docs, and npm test.

Open questions (block or clarify before merge)

1. Canonical element-ids.js constant name for 406ba6e0-... (is this Semrush's "Topic Prompts" element? cross-check the Serenity wiki element table).
2. Final response field names + whether count alone, prompts alone, or both.
3. Whether siteId → brand projects resolution is in scope for v1 (recommend: no — explicit projectId).
4. Are topic and projectId single-value or multi-value in the UI's actual call? The observed payload shows a single topic and two projects (OR). Confirm cardinality so the query-param parsing matches.
5. Jira key — link this to the LLMO epic (the weeks endpoint was LLMO-6011); confirm the tracking issue for prompt count.

---

Context gathered from the current main (spacecat-api-service @ 2ee0efb9) elements wrapper layer and PRs #2666 / #2755.

[rainer-friederich]

Resolved decisions + implementation

Implemented on branch feat/prompt_count_endpoint. The four open questions were resolved as:

1. Response shape → both { count, prompts } (count = prompts.length). The count is the headline output; the endpoint is deliberately generic so it can back the prompt healthcheck metrics — intent % (group returned rows on primary_intent) and branded % (topic-filtered count ÷ unfiltered count).
2. Filter contract → explicit Semrush project id(s), not brand/site resolution. Rationale: the elements layer already surfaces semrush_project_id to the UI via the URL Inspector filter-dimensions regions (in fact the sample payload's 5f0e8c91-… is one of those), so passing project ids straight back keeps the endpoint symmetric with the rest of the elements wrapper and avoids a BrandSemrushProject fan-out. Params: model/platform, topic (CSV → tags contains topic:<name>), projectId/project_id (CSV → CBF_project OR-group; omitted → all projects in the workspace).
3. Field names → Semrush passthrough (prompt, prompt_topic, primary_intent, volume).
4. Testing → real unit tests (no c8 ignore), covering the payload builder, response transform (incl. empty/missing blocks.data), CSV parsing, and the promise-token vs missing-token auth branches.

Naming

Element constant PROMPTS and endpoint .../brand-presence/prompts kept generic (not intent/healthcheck-specific) precisely so the same element serves both the intent-% and branded-% calculations. Purpose is documented in code + docs.

Element ID

Registered 406ba6e0-0de2-475e-80d9-42fab8616032 as PROMPTS in element-ids.js.

Incidental fix

While mirroring listWeeks I found it had a latent bug: await buildService(ctx).getWeeks(...) reads .getWeeks off the Promise returned by the async buildService, throwing a TypeError → 500 on every call. It was never caught because listWeeks is c8 ignore'd with no tests. Fixed both listWeeks and listPrompts to const service = await buildService(ctx) first. (Weeks remains c8 ignore'd POC — worth productionizing/testing separately.)

Status

All unit tests green (3216 passing), lint clean, FACS + required-capabilities route tests pass. Diff ~268 LOC. No OpenAPI/test/it/ added — consistent with the merged filter-dimensions (#2666) and weeks (#2755) elements-family precedent.

Known caveat

The topic filter uses tags contains topic:<name> (faithful to the observed upstream payload). If one topic label is a prefix of another (e.g. AI vs AI Generators), contains could over-match — verify against real topic data if it surfaces.

[rainer-friederich]

Prod validation → brand-scoped redesign

Ran the actual new code (buildPromptsPayload → transport POST → transformPromptsResponse) against prod Semrush for the Lovesac org (THE LOVESAC COMPANY).

Scenario	Workspace	Result
Both projects, search-gpt	brand sub-workspace 3cbb3c36…	✅ count 1246
Both projects, search-gpt	org parent ad76b37d…	⚠️ count 0
EN project only	sub-workspace	✅ 1246 → 1197
No project filter (workspace-wide)	sub-workspace	✅ 1246

The mechanics are proven end-to-end (payload shape, CBF_project filtering, transform, and the intent distribution the healthcheck needs: { informational: 1061, commercial: 71, navigational: 105, task: 9 }).

Key finding: Semrush projects/prompts live only in a brand's sub-workspace, never in the org's parent workspace — same payload returns 1246 on the sub-workspace and 0 on the parent. My original org-scoped route resolved the parent workspace → would return count: 0 for every real (sub-workspace) customer.

Change: org-scoped → brand-scoped + sub-workspace validation

• Route: GET /v2/orgs/:spaceCatId/brands/:brandId/serenity/brand-presence/prompts (was /serenity/all/…).
• Resolves the brand's sub-workspace via the existing resolveBrandWorkspace (org access check → resolveBrandUuid → sub-workspace), mirroring the serenity controller.
• Validation guard (the ask): refuses to run against an org workspace — mode !== 'subworkspace' → 404 subWorkspaceRequired; sub-workspace pointer == org parent → 409 workspaceMisconfigured; non-UUID brandId → 400; brand not in org → 404.
• FACS: :brandId already classified as LLMO.brand (no alias change). Capabilities: llmo/can_view + organization:read (matches the other brand-scoped /serenity/* routes).

For Lovesac this now resolves sub-workspace 3cbb3c36… and returns the 1246 prompts; a flat-mode brand is correctly rejected.

Note for the team: the sibling weeks / url-inspector/filter-dimensions endpoints (#2755, #2666) are still org-scoped against the parent workspace and share this same gap — they'll return empty for sub-workspace customers. Worth the same brand-scoped treatment.

Status: 3222 unit tests green (incl. the new validation branches), lint clean, diff ~439 LOC.

[rainer-friederich]

Filter fix so the endpoint actually powers the two healthcheck metrics

Probing the raw element + tags in prod (Lovesac sub-workspace) settled what the endpoint needs for Branded/Unbranded and Intent coverage (both per project):

PROMPTS element columns are only primary_intent, prompt, prompt_topic, volume — no branded column. The branded signal is a tag: type:branded / type:non-branded. The TOPICS element for this brand is populated with category:… (nested via __), intent:…, source:…, and type:… — but no topic: tags at all.

Two problems found

1. Branded% was impossible — my payload builder could only emit tags contains topic:<name>, so it couldn't ask for type:branded.
2. The topic filter was broken for real brands — it hardcoded the topic: prefix, but this brand's tags are category:/type:/intent:/source:. (That was the "empty topics" I misread earlier.)

Fix

Replaced the rigid topic param with a generic tag filter that takes the full prefixed value (tags contains <value>, AND-ed). No prefix assumed — the taxonomy varies by brand.

Result — both metrics, per project, through the real service method (prod)

Project	total	branded (tag=type:branded)	branded%	intent coverage (group by primary_intent)
US-en	1197	510	43%	informational 1020, commercial 66, navigational 102, task 9
US-es	49	17	35%	informational 41, commercial 5, navigational 3

US-en 43% matches the product UI's "43% Branded" exactly. type:branded + type:non-branded partition the total (510 + 687 = 1197).

• Intent coverage needs no filter — primary_intent is on every row; group per projectId (4 intents present; the "of 6" is the canonical framework the consumer applies).
• Branded% = ?projectId=<id>&tag=type:branded ÷ ?projectId=<id>.

Docs updated with a "Computing the healthcheck metrics" section. 3227 tests green, lint clean.