scripts(drift-check): add v1 static detector for provider guides#608
Open
caiopizzol wants to merge 8 commits into
Open
scripts(drift-check): add v1 static detector for provider guides#608caiopizzol wants to merge 8 commits into
caiopizzol wants to merge 8 commits into
Conversation
This was referenced May 18, 2026
caiopizzol
added a commit
that referenced
this pull request
May 26, 2026
The sample at github.com/amp-labs/samples lives at /zendeskSupport/ (camelCase), not /zendesk-support/ (hyphenated). Both links in the guide were 404'ing. Surfaced by the drift-check detector (#608) and triaged in #609 as the lone deterministic docs fix among the broken-sample-link findings. Verified: https://github.com/amp-labs/samples/tree/main/zendeskSupport returns 200.
Static three-source diff between provider guides and the connectors catalog. v1 scope (static only, no runtime, no provider public-doc scraping). Module-aware: a guide claim is valid if either the provider-level or any module-level support flag matches. Files: - scripts/drift-check/index.ts CLI entry; modes full / per-pr / provider - scripts/drift-check/catalog.ts fetch connectors catalog from main; module-aware capability helpers - scripts/drift-check/docs.ts scan src/provider-guides/*.mdx, extract frontmatter and the four canonical action-link patterns; case- insensitive resolution of file slug to catalog key - scripts/drift-check/samples.ts HEAD-check each sample manifest link - scripts/drift-check/recipes.ts sparse slugOverrides for hyphen-vs- camelCase and family aliases; undocumentedAllowed list with reasons and expirations - scripts/drift-check/report.ts Finding type, JSON canonical output, Markdown derived First sweep on main surfaces 42 findings: 32 undocumented providers, 7 broken sample links, 2 write overclaims, 1 proxy overclaim. All appear legitimate; no false positives observed. Adds pnpm run drift-check. Default mode is full; runs report-only (does not fail CI). PR 3 will add connector-source inspection (connectors.ts) and CI wiring.
- recipes.ts: fix payPalSandBox case (catalog uses payPalSandBox, not
paypalSandbox); the typo caused a false-positive undocumented-provider
- catalog.ts: add 'search' to Capability with object-shape handling
(search support is {operators: {equals: bool}}, not a flat boolean)
- docs.ts: extract all distinct sample links per guide, not just the
first. Accept both amp.yaml and amp.yml extensions. Surfaces google,
netsuite-m2m, and salesforce multi-sample cases
- docs.ts: add Search Actions to the action-link pattern set
- index.ts: enforce module-aware schema by injecting module: 'all'
when the provider has modules and the finding does not specify one
- index.ts: accept both 'drift-check --out X' and the
'pnpm run drift-check -- --out X' forms by stripping leading '--'
Findings: 42 -> 41 (false positive removed). aws and seismic findings
now carry module: 'all'. CLI forms both work; --fail-on-error exits
nonzero when errors are present.
The detector compares docs against the connector catalog. It does not validate the deployed API against its declared shape, and it does not prove that read/write/proxy/subscribe calls succeed at runtime. List the specific drift surfaces left out of v1 so users do not overstate what the green report means. Verified today: connector catalog.json (227 providers) and live /v1/providers (227 providers) are in lockstep across keys, authType, baseURL, displayName, defaultModule, and top-level support flags. So catalog-vs-live drift is currently theoretical; it can become a separate cheap check in a follow-up PR.
…EADME samples.ts: HEAD against GitHub raw is occasionally flaky (transient 5xx or network blips). Trust a 404 immediately, but retry once on non-404 errors with a short backoff to avoid false-positive broken-sample-link findings in CI. README: add an explicit one-line summary at the top of the 'does not prove' section: this check uses the generated catalog from main, not the live /v1/providers API or runtime behavior.
- README trimmed from 118 to 66 lines: cut file inventory and repeated CI behavior; compressed the out-of-scope table; kept the slugOverrides example and the does-not-prove disclaimer. - index.ts: drop the file header (README covers it); rename cp -> provider for readability; drop the unused _level parameter and rename the helper to providerLevelSupport; drop the unused GuideDoc import. - catalog.ts: shorten supports() comment to one line; rename hasAnySearchOperator -> supportsSearch. - docs.ts: shorten the resolveProviderKey comment; rename SAMPLE_LINK_RE_G -> SAMPLE_LINK_RE; drop trailing field comments. - samples.ts: compress the blob-to-raw and retry comments. - recipes.ts: drop interface/inline commentary on obvious entries; keep the comments that explain non-obvious mappings (instantly, jira, highlevel). - report.ts: unchanged. Total LOC 744 -> 572 (-23%). Detector still produces the same 41 findings against main.
Full-content audit of the 31 undocumented findings turned up one real alias miss: the docusign.mdx guide explicitly walks through creating a DocuSign Developer Account, so the docusignDeveloper catalog entry is covered there. Same pattern as instantly/instantlyAI. The other audit candidates were false positives from shared docs domains: - adobe -> marketo.mdx (Adobe owns Marketo's docs URLs) - greenhouse -> greenhouseJobBoard.mdx (Harvest API vs Job Board API, different connectors) - payPal -> braintree.mdx (PayPal owns Braintree's docs URLs) microsoftClientCredentials surfaced in microsoft.mdx but only as a generic mention; the existing guide documents the Authorization Code flow against Microsoft Graph, not Client Credentials. Treat as genuinely missing (precedent: salesforce-jwt.mdx exists separately from salesforce.mdx).
caiopizzol
force-pushed
the
caiopizzol/drift-check-v1
branch
from
d283181 to
fc9ada9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a static drift-check script for provider guides. Compares provider-guide claims against the generated
amp-labs/connectorscatalog and validates linked sample manifests. Writes both a machine-readable JSON report and a Markdown rollup.What it checks
amp.yaml/amp.ymlsample linksThe check is module-aware: a capability claim is valid if either the provider-level flag is true or any module-level flag is true. Catches cases like Google's Gmail submodule.
What this does not prove
Static validation only. Does not compare against the deployed
/v1/providersAPI, inspect connector Go source, or run provider runtime calls. The full out-of-scope list is inscripts/drift-check/README.md.Current output
A full run on current
mainreports 37 findings:(Initial run when the PR was opened reported 40; subsequent merges of #610 and #612, plus new provider guides for accuLynx and procore on
main, account for the difference.)The mismatches are intentionally left as report output rather than auto-fixed because the correct fix may live in docs, catalog metadata, connector source, or product release state. A separate triage issue will categorize the findings for follow-up.
Verification
pnpm run drift-check # 37 findings (37 error, 0 warning, 0 info)Screenshot
Not applicable. This PR adds a CLI script under
scripts/and does not modify any documentation content.mint devrenders the docs site identically tomain.