feat: executable conformance — black-box harness + CI validation#16
Merged
Conversation
Close the gap between the turnkey crypto path and the wire/transport
layer, where most "get it right first try" friction lives for both
humans and LLMs implementing AVP.
- openapi.yaml: OpenAPI 3.1 description of the HTTP/JSON profile — every
path, method, status code, and error response — referencing the JSON
Schema shapes so the route surface is machine-readable and
stub-generatable. The gRPC profile had proto/avp.proto; HTTP had only a
prose route table.
- schema + SPEC §6: define the error body { error, code?, detail? } and a
status→code table, and state that every non-2xx is terminal while a
stale-version push is a 200 with conflict:true. The five reference
servers already emit { error, detail? }; `code` is additive, so they
stay conformant.
- SPEC §11: reword conformance to map each requirement to the vector that
actually exists (Ed25519 anchors the challenge signature; the
payload-AEAD epoch-tamper case anchors rotation correctness) instead of
implying standalone challenge/token and rotation vector files that do
not exist. Dedicated vectors for those paths are noted as welcome
additions.
- vectors/index.json: machine-readable index of the vectors (file, kind,
spec section, RFC anchors) so a harness can enumerate them without
hardcoding filenames.
- Taskfile.yml: `task test` runs the conformance suite and every
language's example tests in one command, mirroring the CI jobs.
Cross-linked from README, IMPLEMENTING.md, llms.txt, examples/README.md,
and vectors/README.md; recorded under CHANGELOG [Unreleased].
Make "conformant" executable rather than self-attested, and stop the machine-readable artifacts from silently drifting apart. harness/ — a black-box conformance suite. Point it at any running AVP server (`bun harness/conformance.ts --server URL`) and it drives the full HTTP/JSON wire contract, asserting the normative MUSTs instead of just printing a transcript: - §3 auth: nonce shape, valid-token, single-use-nonce rejection, the classic "signed the base64 string not the raw nonce" rejection, 401 without a token. - §6 vault: createRepo sole-member/duplicate rules, pull unchanged vs envelope, push accept + version bump, a stale push as a 200 conflict (never a 4xx), non-member 403, addMember/fetchMemberKey, removed-member 403. - §4/§10 crypto + invariants: envelope round-trips to our plaintext; a second member unwraps and decrypts; zero-knowledge (the plaintext never appears in the server's JSON or stored ciphertext); removeMember bumps the epoch and a departed member's old key cannot decrypt the rotated envelope; a remaining member can. The only AVP-specific code is the client crypto, reused from the vector-tested examples/typescript/client, so the harness stays a thin wire driver. Verified locally: 24/24 against the TypeScript reference server. .github/workflows/conformance.yml — runs the vector runner (which had no CI until now), lints openapi.yaml with Spectral, checks schema/example/ index/openapi consistency, and runs the harness against the Go and TypeScript reference servers (SHA-pinned actions, the interop workflow's server-start/poll pattern). examples/conformance — a new test asserts every examples/*.json body validates against its schema $def, that vectors/index.json lists exactly the vector files present, and that every openapi.yaml $ref resolves to a schema $def (adds ajv + yaml dev-deps). Cross-linked from README, IMPLEMENTING.md (§6 checklist), SPEC.md (§11), llms.txt, and Taskfile.yml (`task harness`, `task openapi-lint`); recorded under CHANGELOG [Unreleased].
CLAUDE.md rule: no em dashes. Replace the em dashes introduced with the openapi/error/index/harness work (and one pre-existing in SPEC §6) with commas, colons, semicolons, or parentheses. No semantic change.
Add openapi.yaml, the HTTP error body ($defs/Error), vectors/index.json, the harness/ black-box suite, the Taskfile runner, and the conformance CI workflow to the AI-agent mental model; list Go among the examples; extend the "one protocol, keep in sync" note to cover openapi.yaml.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft for review. Stacked on #15 (base =
docs/impl-ergonomics) — merge #15 first, then this retargets tomainautomatically. Diff shown here is only the conformance work.Turns "conformant" from self-attested into executable, and wires the machine-readable artifacts so they can't silently drift.
What's in it
harness/.github/workflows/conformance.ymlopenapi.yaml, checks schema/example/index/openapi consistency, and runs the harness against the Go + TypeScript reference servers.examples/conformance/test/schema.test.ts(+ ajv, yaml)examples/*.jsonvalidates against its schema$def;vectors/index.jsonlists exactly the files present; everyopenapi.yaml$refresolves..spectral.yamlHarness coverage (24 checks)
The only AVP-specific code is the client crypto, reused from the vector-tested
examples/typescript/client— the harness stays a thin wire driver.Verified locally
bun harness/conformance.ts).examples/conformancebun run test: 24 tests pass (vectors + new schema/example/index/openapi checks);typecheckclean.harnesstsc --noEmit: clean (cross-dir import resolves).Go-server CI path couldn't be run locally (no Go toolchain here) — it reuses the proven server-start/poll pattern from
examples-interop.ymlwith SHA-pinned actions; CI is the check.Still open (from the menu, not in this PR)
Reference IdP + persistent server; federation vectors (invite/locator base64url,
avp://URI);THREATMODEL.md; vector generator script; negative/"MUST reject" vector bank; gRPC example; browser/WASM demo;llms-full.txt; spec quardrails (size limits, idempotency,Retry-After).