Authenticated External Link Resolution (linkAuth) — Design
- Date: 2026-06-01
- Status: Design — ready to file as a VAT issue for implementation
- Surface:
vibe-agent-toolkit.config.yaml (resources.linkAuth), @vibe-agent-toolkit/utils, @vibe-agent-toolkit/resources
- Audience: contributors implementing the feature. Public/open-source. No adopter-specific names —
github.com and *.sharepoint.com are used purely as representative enterprise test points.
1. Summary
Add a config-declared engine that can rewrite and authenticate external URLs so VAT can (a) health-check private links (alive / dead / indeterminate) during vat resources validate, and (b) fetch private content for future consumers. The capability is driven entirely by a VAT project's own vibe-agent-toolkit.config.yaml — VAT never stores secrets; config only declares where to read a token (an env var or a command to run).
The core is a generic vocabulary (match → rewrite → auth → token → check). Built-in support for specific hosts (GitHub, SharePoint) ships as macros — named bundles of that same generic config, with no privileged code path. This guarantees the generic vocabulary is sufficient for real-world hosts and makes "support another host" a config-only pull request.
2. Motivation
Agent skills, CLAUDE.md files, and knowledge resources routinely link to private resources — internal GitHub repos, SharePoint/OneDrive documents. Today VAT's external-link check can only HEAD/GET anonymously, so every private link reports as dead (404), making external-link validation useless for enterprise projects.
The naive fix — "attach an Authorization header" — does not work, and this was established empirically:
| Target |
Valid repo-scoped token in Authorization header |
github.com/<o>/<r>/blob/<ref>/<path> (web HTML) |
404 ❌ — web pages authenticate via session cookies, not bearer tokens |
github.com/<o>/<r>/tree/..., repo root (web HTML) |
404 ❌ |
raw.githubusercontent.com/<o>/<r>/<ref>/<path> |
200 ✅ (files only) |
api.github.com/repos/<o>/<r>/contents/<path>?ref=<ref> |
200 ✅ (files and directories) |
The real fix is two moves, not one: rewrite the web URL to an API/raw host, then attach the token. The GitHub Contents API is the most general endpoint (200 for both files and /tree/ directories). SharePoint requires a different rewrite (sharing URL → Microsoft Graph /shares/{id}/driveItem) whose share id is a base64url encoding of the original URL — so the two hosts together exercise both plain substitution and a transform, forcing the vocabulary to be genuinely general.
3. Goals / Non-goals
Goals
- A generic, config-expressible rewrite + auth + token-source vocabulary.
- GitHub and SharePoint support shipped as macros over that vocabulary, not as bespoke code.
- Health-check consumer wired into the existing
checkUrlLinks / externalUrls validation path.
- A content-fetch primitive (returns bytes + metadata) for future consumers, with a short-TTL content cache and a force-refresh bypass.
- Distinguished outcomes and configurable severity so opt-in checking is useful without breaking builds for contributors who lack credentials or access.
- Tokens never logged or persisted.
Non-goals (v1)
- VAT performing OAuth flows itself (device-code, refresh-token dances). Delegated to ambient tools (
gh, az).
- Token caching/refresh across runs — fresh resolution per run.
- Wiring the content-fetch primitive into asset-reference resolution / private-resource bundling — the primitive ships; those consumers are a follow-up.
- Rewriting links in output/distribution artifacts — unrelated to the existing link-template feature.
- Any GUI or secret-vault integration.
This is not vat audit <git-url>. That command audits arbitrary repos and is deliberately credential-free (auth = git's job). linkAuth is a property of a configured VAT project, not an ad-hoc auditor; there is no standalone "authenticate this random URL" command.
4. The generic vocabulary (the engine)
resources.linkAuth.providers[]. Each provider is pure config with exactly five verbs:
resources:
linkAuth:
cache:
ttlMinutes: 30 # content/metadata cache TTL (see §6)
providers:
- match: { host: "github.com" } # 1. MATCH — glob on URL host
rewrite: # 2. REWRITE — ordered; first matching `when` wins
- when: "^https://github\\.com/(?<owner>[^/]+)/(?<repo>[^/]+)/(?:blob|tree)/(?<ref>[^/]+)/(?<path>.+)$"
to: "https://api.github.com/repos/${owner}/${repo}/contents/${path}?ref=${ref}"
auth: # 3. AUTH — header templates over ${token} + captures
headers:
Authorization: "Bearer ${token}"
Accept: "application/vnd.github+json"
token: # 4. TOKEN — ordered sources, first non-empty wins
- { command: "gh auth token" }
- { env: "GITHUB_TOKEN" }
check: # 5. CHECK — classifier ONLY: status → outcome → code
method: GET
aliveStatus: [200]
notFoundMeaning: ambiguous # GitHub masks no-access as 404 → 404 emits
# LINK_AUTH_DEAD_OR_UNAUTHORIZED, not LINK_AUTH_DEAD
# No severity knobs here: `check` only decides WHICH code an outcome becomes (host HTTP semantics).
# code → severity is the framework's job — each LINK_AUTH_* code's CODE_REGISTRY default, tunable
# per project via validation.severity.LINK_AUTH_* / validation.allow (see §7).
Verbs:
match.host — glob matched against the URL's host, with optional match.excludeHost globs. Selects which provider claims a URL. Example: claim *.sharepoint.com team sites but not *-my.sharepoint.com personal OneDrive.
rewrite[] — ordered list; each entry has a when regex (named capture groups), an optional vars map (computed via transforms over captures), and a to URL template. The first entry whose when matches is applied. If none match, the provider does not claim the URL for rewriting → outcome unsupported.
auth.headers — a map of header name → template string. Templates interpolate ${token} and any named captures. Produces the headers attached to the fetch.
token[] — ordered token sources. Each is { env: "NAME" } or { command: argv } (argv array preferred; convenience string form is whitespace-tokenized, never shelled — see §6.1). Resolved in order; the first that yields a non-empty value wins. If none yield a value → outcome unverified.
check — a pure classifier (status → outcome → code): method (GET/HEAD), aliveStatus (status codes meaning "alive"), and notFoundMeaning (ambiguous | dead — how to read a 404 on this host; see §7). check carries no severity knobs; its only job is to decide which LINK_AUTH_* code an outcome becomes. Severity (code → severity) is owned entirely by the framework — each code's CODE_REGISTRY default, tunable via validation.severity.LINK_AUTH_* and validation.allow, exactly like every other VAT code. One severity surface, not two.
4.1 Transforms (closed allowlist)
Templates may call a fixed, whitelisted set of transform functions. v1 set:
| Transform |
Purpose |
base64url(x) |
URL-safe base64 without padding — required for SharePoint share ids |
urlencode(x) |
percent-encode a capture for safe path/query insertion |
lower(x) |
lowercase (host normalization) |
An unknown transform name in config is a hard config error, never arbitrary evaluation. The allowlist grows only by VAT PR — this is the one place new host needs can require engine code, and SharePoint is the v1 driver for base64url.
5. Macros — built-ins as config
Built-in host support is not a privileged code path. It is a named bundle of the §4 vocabulary, shipped in a single file (packages/.../link-auth-macros.yaml) that maps a short name → the exact generic config it expands to. Adopter config references a macro and overrides keys as needed:
resources:
linkAuth:
providers:
- use: github # expands to the full GitHub provider block from the macro file
- use: sharepoint # expands to the SharePoint provider block
token: # override just the token source for this environment
- { command: "az account get-access-token --resource https://graph.microsoft.com --query accessToken -o tsv" }
Expansion rule: use: <name> loads the macro's provider config, then deep-merges any sibling keys from the adopter entry on top (adopter wins). A provider entry uses either use: (+ optional overrides) or a full inline provider — both produce the same internal shape.
Design invariants this enforces:
- A built-in has zero capability an adopter's inline config lacks. If a macro needs something the vocabulary can't express, that is a signal to extend the vocabulary (e.g. a new transform), not to special-case the built-in.
- Supporting a new host (GitLab, Bitbucket, a corporate Artifactory, etc.) is a PR that appends one entry to the macro file — config, not engine code. This is intended to make contributions trivial.
5.1 Shipped macros (v1)
github — the block in §4 (plain substitution, Contents API rewrite, gh/GITHUB_TOKEN token sources).
sharepoint — the forcing function for transforms:
- match: { host: "*.sharepoint.com", excludeHost: ["*-my.sharepoint.com"] } # team sites, not personal OneDrive
rewrite:
- when: "^(?<u>https://.+)$"
vars: { shareId: "u!${base64url(u)}" }
to: "https://graph.microsoft.com/v1.0/shares/${shareId}/driveItem"
auth:
headers: { Authorization: "Bearer ${token}" }
token: [] # NO default — SharePoint has no zero-config token source (see note). Adopter supplies one.
check: { method: GET, aliveStatus: [200], notFoundMeaning: dead } # classifier only; severity is framework-owned (§7)
GitHub needs only ${capture} substitution; SharePoint needs base64url(). Designing both concurrently is what proves the vocabulary sufficient.
SharePoint token source — empirically established constraint. Reading SharePoint via Graph requires a token carrying Sites.Read.All / Files.Read.All (or Sites.Selected). The stock Azure CLI token does NOT work and cannot be made to work: az account get-access-token --resource https://graph.microsoft.com yields a token with directory/user scopes but no Sites.*, and requesting --scope .../Sites.Read.All fails with AADSTS65002 (the Azure CLI first-party app is not pre-authorized for Graph Sites scopes). So — unlike GitHub, where gh auth token is a genuine zero-config source — SharePoint requires an Entra app registration with the Sites scope and a token minted from that app (client-credentials, or a delegated flow via a tool whose app carries the scope, e.g. the CLI for Microsoft 365: m365 util accesstoken get --resource https://graph.microsoft.com). The sharepoint macro therefore ships with an empty token and a doc pointer; the adopter supplies the command for their app. This asymmetry is a useful artifact in itself: the vocabulary cleanly separates "rewrite/auth mechanics" (which the macro provides) from "where a scoped token comes from" (environment-specific, not defaultable).
5.2 Rewrite & fetch hardening
- Strip fragment and query before matching. A web URL may carry
#L10-L20 line ranges or ? query; the rewrite matches against the bare path and the to template re-adds only what it needs (e.g. ?ref=). A greedy (?<path>.+) must not swallow #…/?… into the API path.
- Document the multi-segment-ref limitation. GitHub
blob/<ref>/<path> is genuinely ambiguous when a branch name contains / (e.g. release/1.0); the github macro targets single-segment refs and documents this. Resolving arbitrary refs via the API is deferred.
- Macros are host-scoped, not host-family-scoped.
github.com and *.sharepoint.com are explicit. Enterprise/self-hosted variants (github.example.com → /api/v3, GitLab, Bitbucket) are separate macro entries, never assumed by the shipped ones.
- Honor
429 / Retry-After in the authed fetch branch. The anonymous markdown-link-check path retried on 429 for free; the new direct-fetch branch must reimplement it, since Graph and the GitHub API both throttle under batch checking.
6. Resolver pipeline, module placement & caching
The pipeline is a thin I/O shell around a pure core, so the "is the config sufficient?" proof is unit-testable without a network:
| Step |
Function |
Pure? |
Home |
| 1. Select provider |
selectProvider(url, providers) — host-glob match |
✅ |
utils |
| 2. Expand macros |
expandMacro(use, overrides) — deep-merge |
✅ |
utils |
| 3. Rewrite |
first matching when → compute vars (transforms) → render to |
✅ |
utils |
| 4. Build headers |
render auth.headers over ${token} + captures |
✅ |
utils |
| 5. Resolve token |
ordered env / command sources, first non-empty |
❌ I/O |
utils |
| 6. Fetch + classify |
request rewritten URL w/ headers → outcome (+ content) |
❌ I/O |
resources |
The pure engine (steps 1–4 + macro expansion + transforms) is a self-contained link-auth module in @vibe-agent-toolkit/utils — no internal deps, the natural home for a primitive that both link-checking and future content-fetch consume. Single entry point:
resolveAuthenticatedUrl(url, linkAuthConfig)
→ { fetchUrl: string, headers: Record<string,string> } // ready to fetch
| { outcome: 'unsupported' } // no provider claims this host
| { outcome: 'unverified', reason: string } // claimed, but no token resolved
6.1 Token resolution — command execution
token.command resolves a token by running a command. Use VAT's own command-execution utility — safeExecSync (packages/utils/src/safe-exec.ts), enforced by the no-child-process-execSync ESLint rule. Its security model is shell: false: the binary is resolved to an absolute path (via which) and arguments are passed as an argv array, so there is no shell to inject into. This is deliberately the opposite of a shell-string approach.
Because there is no shell, a token.command is expressed as argv, not a shell line:
token:
- { command: ["gh", "auth", "token"] }
- { command: ["az", "account", "get-access-token", "--resource",
"https://graph.microsoft.com", "--query", "accessToken", "-o", "tsv"] }
A convenience string form (command: "gh auth token") is accepted and whitespace-tokenized into argv — but it is not passed to a shell, so shell operators (|, &&, $(...)) do not work. A token source that genuinely needs a shell pipeline is a sharp edge: document it and require an explicit opt-in rather than silently enabling shell: true. Trim trailing whitespace/newlines from command output before use.
vibe-validate's spawnCommand (packages/core/src/process-utils.ts) is reference prior art, not a dependency — VAT and vv are cousins, and VAT carries its own utils. Two differences from vv's model are intentional and worth calling out:
- vv uses
shell: true (it needs shell operators in user-configured validation steps); linkAuth token sources do not, so VAT's safer shell: false argv model is preferred.
- vv scrubs
GIT_* from the child env because it runs commands from inside git hooks, where GIT_DIR / GIT_WORK_TREE / etc. are set and pollute nested git calls (especially in worktrees). Whether that applies here must be evaluated, not assumed: vat resources validate can run inside a pre-commit hook, and gh / git credential helpers read git config — so if a token command shells out to git, GIT_* pollution could surface. Treat GIT_* scrubbing as a considered option for git-backed token commands, not a default copied from vv.
Required system test (cross-platform dispatch). The token dispatcher must invoke real binaries that are .cmd shims on Windows (gh, az, git), where safeExecSync switches to shell: true. Ship a system test that runs a harmless command — gh --help and git --help — through the dispatcher on every CI platform. This mirrors vibe-validate's proven model (its config commands run cross-platform with no per-OS variants) and ensures the Windows matrix catches any .cmd-resolution regression before release. The security claim ("no shell to inject into") is precise on POSIX; on Windows it reduces to argv + windowsShellQuote, with config as the trust boundary.
6.2 Two consumers of the one primitive
Health-check — ExternalLinkValidator (packages/resources/src/external-link-validator.ts) gains an authenticated branch. If the engine claims the host, it issues a direct fetch() of fetchUrl with headers and classifies the status (bypassing markdown-link-check, which can't express the rewrite or distinguish 401/403/404 the way we need). URLs no provider claims keep today's anonymous markdown-link-check path untouched.
Content fetch — a sibling fetchAuthenticated(url, config) → { bytes, metadata } for future consumers. Ships as the primitive; wiring it into asset-reference / bundling is out of scope for v1.
Health-check and content-fetch may need different headers (the >1 MiB lesson). For GitHub, the health-check uses Accept: application/vnd.github+json — which returns 200 for files and directories regardless of size (a >1 MiB file is still alive). But that same response omits the bytes above 1 MiB (encoding:"none", empty content), so content-fetch must instead send Accept: application/vnd.github.raw (which streams the bytes inline at 200, on api.github.com, even for multi-MiB files) or follow download_url. So a provider declares a fetch-mode header override distinct from its check-mode headers; the macro's auth.headers covers health-check, and an optional fetch.headers covers content retrieval. (GitHub raw stays same-host, so it poses no cross-host token-leak; the only GitHub cross-host path is download_url, which carries its own short-lived token — see §8.)
6.3 Caching
Two caches, both keyed by the rewritten URL (the original blob/ URL 404s — caching it would poison results):
- Status cache — reuses
ExternalLinkCache. Stores outcome + status code. Default TTL from the existing external-URL cache config.
- Content cache — stores fetched bytes + metadata (status, content-type, ETag/Last-Modified if present,
fetchedAt, rewritten URL). Default TTL 30 minutes (resources.linkAuth.cache.ttlMinutes), giving cheap cache hits within a working session. A force-refresh bypass (--refresh / --no-cache on the relevant CLI command, and an option on fetchAuthenticated) skips the cache and performs a fresh fetch.
Never cache unverified / token-absent outcomes — they flip the moment a token appears. Never persist tokens in either cache.
Cache location & identity scoping. Both caches live under os.tmpdir() (resolved via the safe path primitive), scoped by the OS user running the checks — the cache key incorporates the OS user so two users on a shared machine or CI host never read each other's authenticated results. The cache is deliberately not keyed on the web identity or token (no secrets at rest). If a user authenticates as the wrong web identity — common where one person holds several logins — the remedy is the force-refresh bypass, not a finer key. Identity is the OS user; a different answer means busting the cache.
7. Outcomes, codes, severity & reporting
Post-#114 note (2026-06-02). The unified validation framework now lives in
@vibe-agent-toolkit/agent-schema (CODE_REGISTRY in validation-codes.ts), and PR #114 already migrated
the external-URL codes into it (see the Framework integration note below). The five LINK_AUTH_* codes
below are new CODE_REGISTRY entries added there — there is no separate codes file, and external-URL
migration is no longer this feature's job.
Seven outcomes, surfaced as codes in the unified validation framework (codes, not schema):
| Outcome |
Trigger |
Default severity |
Code |
alive |
status in aliveStatus |
— |
— |
dead |
404 / 410 with valid auth, on a provider where notFoundMeaning: dead |
error |
LINK_AUTH_DEAD |
dead_or_unauthorized |
404 with valid auth, on a provider where notFoundMeaning: ambiguous |
warn |
LINK_AUTH_DEAD_OR_UNAUTHORIZED |
forbidden |
403 (authed, no access) |
warn, downgradable to ignore |
LINK_AUTH_FORBIDDEN |
unauthorized |
401 (token missing/bad) |
warn (error on strict lanes) |
LINK_AUTH_UNAUTHORIZED |
unverified |
no token source resolved |
warn |
LINK_AUTH_UNVERIFIED |
unsupported |
no provider claims host |
falls through to existing anonymous check |
EXTERNAL_URL_DEAD (existing registry code) |
Status semantics are per-provider — this is empirically forced, not a preference. Probing confirmed that GitHub masks access-denied as 404: an existing-but-inaccessible file and a genuinely-missing path both return 404, and GitHub never returns 403 for private-repo content (it avoids leaking existence). Microsoft Graph is the opposite — it cleanly separates 401 (bad/expired token), 403 (accessDenied — authenticated but unauthorized), 404 (missing), and 200. A single shared outcome→code mapping would therefore be wrong — the per-host difference lives in classification, not severity. The notFoundMeaning knob captures it: hosts that give an honest 404 use dead (404 → dead → error); hosts that mask access use ambiguous (404 → dead_or_unauthorized → warn, because the checker cannot prove the link is rotted vs merely inaccessible). The shipped macros set github → ambiguous, sharepoint → dead.
Why warn-by-default: external-URL checking is opt-in (the project asked VAT to check these), so silently skipping is wrong. But the failure reason matters: a 403 means the requester is authenticated but lacks access to that resource — not link rot — so it is downgradable to ignore; a 401 usually means a configuration/token problem; a confident 404 (honest-404 host) is genuine rot and a hard error; an ambiguous 404 (masking host) can only be a warning. A pleasant consequence: on GitHub, a contributor lacking repo access gets a warn, not a red build — GitHub emits no 403, and its 404 routes (via notFoundMeaning: ambiguous) to LINK_AUTH_DEAD_OR_UNAUTHORIZED (warn), so no downgrade knob is needed. Severity is set in exactly one place — the framework. Each LINK_AUTH_* code carries a CODE_REGISTRY default, and a project tunes it with the ordinary validation.severity.LINK_AUTH_* override (e.g. promote LINK_AUTH_UNAUTHORIZED to error on a strict CI lane) or validation.allow (downgrade LINK_AUTH_FORBIDDEN to ignore with a reason). The provider's check block never sets severity — it only routes an outcome to a code, notFoundMeaning being the one host-semantics knob, and it picks the code, not the severity. Per-host differences are thus expressed as different codes, never as per-provider severity on the same code.
Framework integration — one severity system, not a parallel path. These outcomes do not introduce a separate reporting lane. VAT's pre-existing external-URL issues were once a distinct lowercase issue-type that was filtered out and never failed validation — but PR #114 (validation-code consolidation) already migrated them into the unified framework as EXTERNAL_URL_DEAD / EXTERNAL_URL_TIMEOUT / EXTERNAL_URL_ERROR registry codes (default severity warning) and removed the "informational only, never fails" filter; the validate exit code is now purely severity-based (hasErrors). So that migration is no longer part of linkAuth's scope — this work only adds the five LINK_AUTH_* codes to the same CODE_REGISTRY in @vibe-agent-toolkit/agent-schema. The result is unchanged: external-link codes do not fire when link checking isn't requested, and when it is requested every code — the already-migrated external-URL codes and the new LINK_AUTH_* codes — is configurable exactly like any other VAT code (downgrade a noisy one, promote a critical one), rather than being unconditionally suppressed or unconditionally fatal.
Reporting for each issue: original URL, redacted rewritten URL, provider name, HTTP status, outcome, and a remediation hint (unauthorized → "run gh auth login" / "az login"; unverified → "no token source resolved — configure token or log in").
Required docs anchors. Each new LINK_AUTH_* registry code needs a matching heading section in docs/validation-codes.md (#link_auth_dead, #link_auth_dead_or_unauthorized, #link_auth_forbidden, #link_auth_unauthorized, #link_auth_unverified). A test (packages/agent-schema/test/docs/validation-codes.test.ts) iterates CODE_REGISTRY and fails CI for any code lacking its doc anchor — so the doc sections ship in the same PR as the registry entries. (The external-URL codes were already documented by PR #114.)
8. Security
token.command runs a command from config. Trust model: vibe-agent-toolkit.config.yaml is author-controlled — the same trust level as package.json scripts or the repo's build — not attacker input. Execution goes through safeExecSync with shell: false + argv (§6.1), so the command-injection surface is closed at the spawn layer rather than relying on the trust model alone. An opt-out env (VAT_LINKAUTH_ALLOW_COMMAND=0) still lets a context force env-only token resolution.
- Tokens never leak across hosts. On any redirect to a different origin, the
Authorization header is stripped — a token minted for one host is never replayed to a redirect target. Headers are only ever sent to the matched provider host.
- No secrets at rest. Neither cache stores token values.
base64url(url) embeds a path, not a secret — safe to persist.
- Transforms are a closed allowlist. Unknown transform → hard config error, never arbitrary evaluation.
- Redaction is structural. The
Authorization header value is redacted in all log/report/serialized output and never enters a cache entry. (This generalizes the prototype's discipline of only ever printing status codes.)
9. Testing
- Unit (the bulk): macro expansion (
use: github → expected provider config; override deep-merge); every GitHub URL form (blob / tree / raw) → Contents API; SharePoint sharing URL → Graph URL with correct base64url; header templating; token-source ordering (env vs command, first non-empty); transform allowlist rejection; status → outcome → severity mapping; redaction assertion (Authorization never appears in any serialized issue or cache entry); cross-origin redirect strips auth header.
- Integration:
ExternalLinkValidator authenticated branch against a mocked fetch returning 200/401/403/404 → correct outcomes + severities; status & content caches keyed by rewritten URL; content-cache TTL + force-refresh bypass; no-token → unverified.
- System / e2e: gated behind
NET_AVAILABLE=1 (mirrors the git-url feature) against a real public GitHub file via the Contents API — proves the rewrite + fetch path with no secret involved. Authenticated real-host runs are validated separately, out of this repo.
- Fixtures: synthetic only —
acme/widgets, contoso.sharepoint.com. No real org/site names in committed artifacts.
10. Contribution model
Adding a host is intended to be a low-friction PR:
- Append a provider entry to the macro file mapping a short name → generic config.
- Add unit tests for its rewrite(s) and header template.
- If — and only if — the host needs a transform the allowlist lacks, add the transform (with tests) and document why.
If a host cannot be expressed without engine changes beyond a transform, that is itself a finding: the vocabulary is missing a verb, and the gap should be designed into the vocabulary rather than worked around.
Appendix A — Worked examples
GitHub (plain substitution):
in: https://github.com/acme/widgets/blob/main/docs/api.md
when: ^https://github\.com/(?<owner>[^/]+)/(?<repo>[^/]+)/(?:blob|tree)/(?<ref>[^/]+)/(?<path>.+)$
out: https://api.github.com/repos/acme/widgets/contents/docs/api.md?ref=main
hdr: Authorization: Bearer <token> Accept: application/vnd.github+json
200 → alive | 404 → dead | 403 → forbidden | 401 → unauthorized
SharePoint (transform):
in: https://contoso.sharepoint.com/sites/Eng/Shared%20Documents/spec.docx
vars: shareId = "u!" + base64url(in)
out: https://graph.microsoft.com/v1.0/shares/u!<base64url>/driveItem
hdr: Authorization: Bearer <graph-token>
200 → alive | 404 → dead | 403 → forbidden | 401 → unauthorized
Appendix B — Empirical basis
The §2 status table was produced by live curl probes (status codes only, token never printed) against a real private repository, confirming that github.com web URLs ignore bearer tokens (404) while api.github.com/.../contents and raw.githubusercontent.com accept them (200), and that the Contents API uniquely handles both files and directories. This is the empirical foundation for the "rewrite, then authenticate" two-move requirement.
A second probe established the per-provider status semantics behind §7. On the GitHub Contents API, an existing-but-inaccessible file (no/invalid token) and a genuinely-missing path both returned 404, and no 403 was ever observed for private-repo content — so a 404 there is dead_or_unauthorized, not provably dead. Microsoft Graph, by contrast, returned a clean 401 for a garbage token, a true 403 accessDenied for an authenticated request to a resource the identity could not access, and 404 only for genuinely-missing items — so Graph 404 is honestly dead. This asymmetry is the empirical basis for the notFoundMeaning knob (github → ambiguous, sharepoint → dead).
A third probe validated the SharePoint macro end-to-end. The Graph /shares/u!{base64url(url)}/driveItem rewrite resolved every readable canonical document URL to 200 with a round-trip id match (the resolved driveItem.id equalled the item's own id) once the caller presented a token with a SharePoint read scope (Files.Read.All/Sites.Read.All) — confirming the base64url rewrite is correct and that a 403 is purely an authorization/scope signal, not an encoding error. The same probe confirmed R7: GET …/driveItem/content returned 302 to a different storage host than graph.microsoft.com, empirically demonstrating the cross-origin redirect that an auto-following client would follow while replaying Authorization — which is why §6.2/§8 require redirect: 'manual' with same-origin-only re-attachment. (As established above, no stock cloud-CLI token carries the SharePoint scope; the scoped token must come from a purpose-registered app — see §5.1.)
A fourth probe confirmed the GitHub >1 MiB Contents-API cliff against a 2.3 MiB file: the +json Contents response returned 200 with encoding:"none" and empty content (so health-check stays 200/alive at any size, but byte-fetch via .content fails above 1 MiB), while Accept: application/vnd.github.raw returned the bytes inline at 200 on api.github.com with no cross-host redirect. This grounds §6.2's split between check-mode (+json) and fetch-mode (raw) headers, and confirms GitHub's only cross-host content path is the self-tokenized download_url — leaving the SharePoint /content 302 as the redirect case the §8 mitigation guards.
Authenticated External Link Resolution (
linkAuth) — Designvibe-agent-toolkit.config.yaml(resources.linkAuth),@vibe-agent-toolkit/utils,@vibe-agent-toolkit/resourcesgithub.comand*.sharepoint.comare used purely as representative enterprise test points.1. Summary
Add a config-declared engine that can rewrite and authenticate external URLs so VAT can (a) health-check private links (alive / dead / indeterminate) during
vat resources validate, and (b) fetch private content for future consumers. The capability is driven entirely by a VAT project's ownvibe-agent-toolkit.config.yaml— VAT never stores secrets; config only declares where to read a token (an env var or a command to run).The core is a generic vocabulary (match → rewrite → auth → token → check). Built-in support for specific hosts (GitHub, SharePoint) ships as macros — named bundles of that same generic config, with no privileged code path. This guarantees the generic vocabulary is sufficient for real-world hosts and makes "support another host" a config-only pull request.
2. Motivation
Agent skills, CLAUDE.md files, and knowledge resources routinely link to private resources — internal GitHub repos, SharePoint/OneDrive documents. Today VAT's external-link check can only HEAD/GET anonymously, so every private link reports as dead (404), making external-link validation useless for enterprise projects.
The naive fix — "attach an
Authorizationheader" — does not work, and this was established empirically:repo-scoped token inAuthorizationheadergithub.com/<o>/<r>/blob/<ref>/<path>(web HTML)github.com/<o>/<r>/tree/..., repo root (web HTML)raw.githubusercontent.com/<o>/<r>/<ref>/<path>api.github.com/repos/<o>/<r>/contents/<path>?ref=<ref>The real fix is two moves, not one: rewrite the web URL to an API/raw host, then attach the token. The GitHub Contents API is the most general endpoint (200 for both files and
/tree/directories). SharePoint requires a different rewrite (sharing URL → Microsoft Graph/shares/{id}/driveItem) whose share id is abase64urlencoding of the original URL — so the two hosts together exercise both plain substitution and a transform, forcing the vocabulary to be genuinely general.3. Goals / Non-goals
Goals
checkUrlLinks/externalUrlsvalidation path.Non-goals (v1)
gh,az).This is not
vat audit <git-url>. That command audits arbitrary repos and is deliberately credential-free (auth = git's job).linkAuthis a property of a configured VAT project, not an ad-hoc auditor; there is no standalone "authenticate this random URL" command.4. The generic vocabulary (the engine)
resources.linkAuth.providers[]. Each provider is pure config with exactly five verbs:Verbs:
match.host— glob matched against the URL's host, with optionalmatch.excludeHostglobs. Selects which provider claims a URL. Example: claim*.sharepoint.comteam sites but not*-my.sharepoint.compersonal OneDrive.rewrite[]— ordered list; each entry has awhenregex (named capture groups), an optionalvarsmap (computed via transforms over captures), and atoURL template. The first entry whosewhenmatches is applied. If none match, the provider does not claim the URL for rewriting → outcomeunsupported.auth.headers— a map of header name → template string. Templates interpolate${token}and any named captures. Produces the headers attached to the fetch.token[]— ordered token sources. Each is{ env: "NAME" }or{ command: argv }(argv array preferred; convenience string form is whitespace-tokenized, never shelled — see §6.1). Resolved in order; the first that yields a non-empty value wins. If none yield a value → outcomeunverified.check— a pure classifier (status → outcome → code):method(GET/HEAD),aliveStatus(status codes meaning "alive"), andnotFoundMeaning(ambiguous | dead— how to read a 404 on this host; see §7).checkcarries no severity knobs; its only job is to decide whichLINK_AUTH_*code an outcome becomes. Severity (code → severity) is owned entirely by the framework — each code'sCODE_REGISTRYdefault, tunable viavalidation.severity.LINK_AUTH_*andvalidation.allow, exactly like every other VAT code. One severity surface, not two.4.1 Transforms (closed allowlist)
Templates may call a fixed, whitelisted set of transform functions. v1 set:
base64url(x)urlencode(x)lower(x)An unknown transform name in config is a hard config error, never arbitrary evaluation. The allowlist grows only by VAT PR — this is the one place new host needs can require engine code, and SharePoint is the v1 driver for
base64url.5. Macros — built-ins as config
Built-in host support is not a privileged code path. It is a named bundle of the §4 vocabulary, shipped in a single file (
packages/.../link-auth-macros.yaml) that maps a short name → the exact generic config it expands to. Adopter config references a macro and overrides keys as needed:Expansion rule:
use: <name>loads the macro's provider config, then deep-merges any sibling keys from the adopter entry on top (adopter wins). A provider entry uses eitheruse:(+ optional overrides) or a full inline provider — both produce the same internal shape.Design invariants this enforces:
5.1 Shipped macros (v1)
github— the block in §4 (plain substitution, Contents API rewrite,gh/GITHUB_TOKENtoken sources).sharepoint— the forcing function for transforms:GitHub needs only
${capture}substitution; SharePoint needsbase64url(). Designing both concurrently is what proves the vocabulary sufficient.5.2 Rewrite & fetch hardening
#L10-L20line ranges or?query; the rewrite matches against the bare path and thetotemplate re-adds only what it needs (e.g.?ref=). A greedy(?<path>.+)must not swallow#…/?…into the API path.blob/<ref>/<path>is genuinely ambiguous when a branch name contains/(e.g.release/1.0); thegithubmacro targets single-segment refs and documents this. Resolving arbitrary refs via the API is deferred.github.comand*.sharepoint.comare explicit. Enterprise/self-hosted variants (github.example.com→/api/v3, GitLab, Bitbucket) are separate macro entries, never assumed by the shipped ones.429/Retry-Afterin the authed fetch branch. The anonymousmarkdown-link-checkpath retried on 429 for free; the new direct-fetchbranch must reimplement it, since Graph and the GitHub API both throttle under batch checking.6. Resolver pipeline, module placement & caching
The pipeline is a thin I/O shell around a pure core, so the "is the config sufficient?" proof is unit-testable without a network:
selectProvider(url, providers)— host-glob matchutilsexpandMacro(use, overrides)— deep-mergeutilswhen→ computevars(transforms) → rendertoutilsauth.headersover${token}+ capturesutilsenv/commandsources, first non-emptyutilsresourcesThe pure engine (steps 1–4 + macro expansion + transforms) is a self-contained
link-authmodule in@vibe-agent-toolkit/utils— no internal deps, the natural home for a primitive that both link-checking and future content-fetch consume. Single entry point:6.1 Token resolution — command execution
token.commandresolves a token by running a command. Use VAT's own command-execution utility —safeExecSync(packages/utils/src/safe-exec.ts), enforced by theno-child-process-execSyncESLint rule. Its security model isshell: false: the binary is resolved to an absolute path (viawhich) and arguments are passed as an argv array, so there is no shell to inject into. This is deliberately the opposite of a shell-string approach.Because there is no shell, a
token.commandis expressed as argv, not a shell line:A convenience string form (
command: "gh auth token") is accepted and whitespace-tokenized into argv — but it is not passed to a shell, so shell operators (|,&&,$(...)) do not work. A token source that genuinely needs a shell pipeline is a sharp edge: document it and require an explicit opt-in rather than silently enablingshell: true. Trim trailing whitespace/newlines from command output before use.vibe-validate's
spawnCommand(packages/core/src/process-utils.ts) is reference prior art, not a dependency — VAT and vv are cousins, and VAT carries its own utils. Two differences from vv's model are intentional and worth calling out:shell: true(it needs shell operators in user-configured validation steps); linkAuth token sources do not, so VAT's safershell: falseargv model is preferred.GIT_*from the child env because it runs commands from inside git hooks, whereGIT_DIR/GIT_WORK_TREE/ etc. are set and pollute nested git calls (especially in worktrees). Whether that applies here must be evaluated, not assumed:vat resources validatecan run inside a pre-commit hook, andgh/ git credential helpers read git config — so if a token command shells out to git,GIT_*pollution could surface. TreatGIT_*scrubbing as a considered option for git-backed token commands, not a default copied from vv.Required system test (cross-platform dispatch). The token dispatcher must invoke real binaries that are
.cmdshims on Windows (gh,az,git), wheresafeExecSyncswitches toshell: true. Ship a system test that runs a harmless command —gh --helpandgit --help— through the dispatcher on every CI platform. This mirrors vibe-validate's proven model (its config commands run cross-platform with no per-OS variants) and ensures the Windows matrix catches any.cmd-resolution regression before release. The security claim ("no shell to inject into") is precise on POSIX; on Windows it reduces to argv +windowsShellQuote, with config as the trust boundary.6.2 Two consumers of the one primitive
Health-check —
ExternalLinkValidator(packages/resources/src/external-link-validator.ts) gains an authenticated branch. If the engine claims the host, it issues a directfetch()offetchUrlwithheadersand classifies the status (bypassingmarkdown-link-check, which can't express the rewrite or distinguish 401/403/404 the way we need). URLs no provider claims keep today's anonymousmarkdown-link-checkpath untouched.Content fetch — a sibling
fetchAuthenticated(url, config) → { bytes, metadata }for future consumers. Ships as the primitive; wiring it into asset-reference / bundling is out of scope for v1.Health-check and content-fetch may need different headers (the >1 MiB lesson). For GitHub, the health-check uses
Accept: application/vnd.github+json— which returns200for files and directories regardless of size (a >1 MiB file is stillalive). But that same response omits the bytes above 1 MiB (encoding:"none", emptycontent), so content-fetch must instead sendAccept: application/vnd.github.raw(which streams the bytes inline at200, onapi.github.com, even for multi-MiB files) or followdownload_url. So a provider declares a fetch-mode header override distinct from its check-mode headers; the macro'sauth.headerscovers health-check, and an optionalfetch.headerscovers content retrieval. (GitHubrawstays same-host, so it poses no cross-host token-leak; the only GitHub cross-host path isdownload_url, which carries its own short-lived token — see §8.)6.3 Caching
Two caches, both keyed by the rewritten URL (the original
blob/URL 404s — caching it would poison results):ExternalLinkCache. Stores outcome + status code. Default TTL from the existing external-URL cache config.fetchedAt, rewritten URL). Default TTL 30 minutes (resources.linkAuth.cache.ttlMinutes), giving cheap cache hits within a working session. A force-refresh bypass (--refresh/--no-cacheon the relevant CLI command, and an option onfetchAuthenticated) skips the cache and performs a fresh fetch.Never cache
unverified/ token-absent outcomes — they flip the moment a token appears. Never persist tokens in either cache.Cache location & identity scoping. Both caches live under
os.tmpdir()(resolved via the safe path primitive), scoped by the OS user running the checks — the cache key incorporates the OS user so two users on a shared machine or CI host never read each other's authenticated results. The cache is deliberately not keyed on the web identity or token (no secrets at rest). If a user authenticates as the wrong web identity — common where one person holds several logins — the remedy is the force-refresh bypass, not a finer key. Identity is the OS user; a different answer means busting the cache.7. Outcomes, codes, severity & reporting
Seven outcomes, surfaced as codes in the unified validation framework (codes, not schema):
alivealiveStatusdeadnotFoundMeaning: deadLINK_AUTH_DEADdead_or_unauthorizednotFoundMeaning: ambiguousLINK_AUTH_DEAD_OR_UNAUTHORIZEDforbiddenignoreLINK_AUTH_FORBIDDENunauthorizederroron strict lanes)LINK_AUTH_UNAUTHORIZEDunverifiedLINK_AUTH_UNVERIFIEDunsupportedEXTERNAL_URL_DEAD(existing registry code)Status semantics are per-provider — this is empirically forced, not a preference. Probing confirmed that GitHub masks access-denied as 404: an existing-but-inaccessible file and a genuinely-missing path both return 404, and GitHub never returns 403 for private-repo content (it avoids leaking existence). Microsoft Graph is the opposite — it cleanly separates 401 (bad/expired token), 403 (
accessDenied— authenticated but unauthorized), 404 (missing), and 200. A single shared outcome→code mapping would therefore be wrong — the per-host difference lives in classification, not severity. ThenotFoundMeaningknob captures it: hosts that give an honest 404 usedead(404 →dead→ error); hosts that mask access useambiguous(404 →dead_or_unauthorized→ warn, because the checker cannot prove the link is rotted vs merely inaccessible). The shipped macros setgithub → ambiguous,sharepoint → dead.Why warn-by-default: external-URL checking is opt-in (the project asked VAT to check these), so silently skipping is wrong. But the failure reason matters: a 403 means the requester is authenticated but lacks access to that resource — not link rot — so it is downgradable to
ignore; a 401 usually means a configuration/token problem; a confident 404 (honest-404 host) is genuine rot and a hard error; an ambiguous 404 (masking host) can only be a warning. A pleasant consequence: on GitHub, a contributor lacking repo access gets a warn, not a red build — GitHub emits no 403, and its 404 routes (vianotFoundMeaning: ambiguous) toLINK_AUTH_DEAD_OR_UNAUTHORIZED(warn), so no downgrade knob is needed. Severity is set in exactly one place — the framework. EachLINK_AUTH_*code carries aCODE_REGISTRYdefault, and a project tunes it with the ordinaryvalidation.severity.LINK_AUTH_*override (e.g. promoteLINK_AUTH_UNAUTHORIZEDtoerroron a strict CI lane) orvalidation.allow(downgradeLINK_AUTH_FORBIDDENto ignore with a reason). The provider'scheckblock never sets severity — it only routes an outcome to a code,notFoundMeaningbeing the one host-semantics knob, and it picks the code, not the severity. Per-host differences are thus expressed as different codes, never as per-provider severity on the same code.Framework integration — one severity system, not a parallel path. These outcomes do not introduce a separate reporting lane. VAT's pre-existing external-URL issues were once a distinct lowercase issue-type that was filtered out and never failed validation — but PR #114 (validation-code consolidation) already migrated them into the unified framework as
EXTERNAL_URL_DEAD/EXTERNAL_URL_TIMEOUT/EXTERNAL_URL_ERRORregistry codes (default severitywarning) and removed the "informational only, never fails" filter; the validate exit code is now purely severity-based (hasErrors). So that migration is no longer part of linkAuth's scope — this work only adds the fiveLINK_AUTH_*codes to the sameCODE_REGISTRYin@vibe-agent-toolkit/agent-schema. The result is unchanged: external-link codes do not fire when link checking isn't requested, and when it is requested every code — the already-migrated external-URL codes and the newLINK_AUTH_*codes — is configurable exactly like any other VAT code (downgrade a noisy one, promote a critical one), rather than being unconditionally suppressed or unconditionally fatal.Reporting for each issue: original URL, redacted rewritten URL, provider name, HTTP status, outcome, and a remediation hint (
unauthorized→ "rungh auth login" / "az login";unverified→ "no token source resolved — configuretokenor log in").Required docs anchors. Each new
LINK_AUTH_*registry code needs a matching heading section indocs/validation-codes.md(#link_auth_dead,#link_auth_dead_or_unauthorized,#link_auth_forbidden,#link_auth_unauthorized,#link_auth_unverified). A test (packages/agent-schema/test/docs/validation-codes.test.ts) iteratesCODE_REGISTRYand fails CI for any code lacking its doc anchor — so the doc sections ship in the same PR as the registry entries. (The external-URL codes were already documented by PR #114.)8. Security
token.commandruns a command from config. Trust model:vibe-agent-toolkit.config.yamlis author-controlled — the same trust level aspackage.jsonscripts or the repo's build — not attacker input. Execution goes throughsafeExecSyncwithshell: false+ argv (§6.1), so the command-injection surface is closed at the spawn layer rather than relying on the trust model alone. An opt-out env (VAT_LINKAUTH_ALLOW_COMMAND=0) still lets a context force env-only token resolution.Authorizationheader is stripped — a token minted for one host is never replayed to a redirect target. Headers are only ever sent to the matched provider host.base64url(url)embeds a path, not a secret — safe to persist.Authorizationheader value is redacted in all log/report/serialized output and never enters a cache entry. (This generalizes the prototype's discipline of only ever printing status codes.)9. Testing
use: github→ expected provider config; override deep-merge); every GitHub URL form (blob / tree / raw) → Contents API; SharePoint sharing URL → Graph URL with correctbase64url; header templating; token-source ordering (env vs command, first non-empty); transform allowlist rejection; status → outcome → severity mapping; redaction assertion (Authorization never appears in any serialized issue or cache entry); cross-origin redirect strips auth header.ExternalLinkValidatorauthenticated branch against a mocked fetch returning 200/401/403/404 → correct outcomes + severities; status & content caches keyed by rewritten URL; content-cache TTL + force-refresh bypass; no-token →unverified.NET_AVAILABLE=1(mirrors the git-url feature) against a real public GitHub file via the Contents API — proves the rewrite + fetch path with no secret involved. Authenticated real-host runs are validated separately, out of this repo.acme/widgets,contoso.sharepoint.com. No real org/site names in committed artifacts.10. Contribution model
Adding a host is intended to be a low-friction PR:
If a host cannot be expressed without engine changes beyond a transform, that is itself a finding: the vocabulary is missing a verb, and the gap should be designed into the vocabulary rather than worked around.
Appendix A — Worked examples
GitHub (plain substitution):
SharePoint (transform):
Appendix B — Empirical basis
The §2 status table was produced by live
curlprobes (status codes only, token never printed) against a real private repository, confirming thatgithub.comweb URLs ignore bearer tokens (404) whileapi.github.com/.../contentsandraw.githubusercontent.comaccept them (200), and that the Contents API uniquely handles both files and directories. This is the empirical foundation for the "rewrite, then authenticate" two-move requirement.A second probe established the per-provider status semantics behind §7. On the GitHub Contents API, an existing-but-inaccessible file (no/invalid token) and a genuinely-missing path both returned 404, and no 403 was ever observed for private-repo content — so a 404 there is
dead_or_unauthorized, not provably dead. Microsoft Graph, by contrast, returned a clean 401 for a garbage token, a true 403accessDeniedfor an authenticated request to a resource the identity could not access, and 404 only for genuinely-missing items — so Graph 404 is honestlydead. This asymmetry is the empirical basis for thenotFoundMeaningknob (github → ambiguous,sharepoint → dead).A third probe validated the SharePoint macro end-to-end. The Graph
/shares/u!{base64url(url)}/driveItemrewrite resolved every readable canonical document URL to 200 with a round-trip id match (the resolveddriveItem.idequalled the item's own id) once the caller presented a token with a SharePoint read scope (Files.Read.All/Sites.Read.All) — confirming thebase64urlrewrite is correct and that a 403 is purely an authorization/scope signal, not an encoding error. The same probe confirmed R7:GET …/driveItem/contentreturned 302 to a different storage host thangraph.microsoft.com, empirically demonstrating the cross-origin redirect that an auto-following client would follow while replayingAuthorization— which is why §6.2/§8 requireredirect: 'manual'with same-origin-only re-attachment. (As established above, no stock cloud-CLI token carries the SharePoint scope; the scoped token must come from a purpose-registered app — see §5.1.)A fourth probe confirmed the GitHub >1 MiB Contents-API cliff against a 2.3 MiB file: the
+jsonContents response returned200withencoding:"none"and emptycontent(so health-check stays200/alive at any size, but byte-fetch via.contentfails above 1 MiB), whileAccept: application/vnd.github.rawreturned the bytes inline at200onapi.github.comwith no cross-host redirect. This grounds §6.2's split between check-mode (+json) and fetch-mode (raw) headers, and confirms GitHub's only cross-host content path is the self-tokenizeddownload_url— leaving the SharePoint/content302 as the redirect case the §8 mitigation guards.