P0: Schema `required` must match the reference serializer's omit-when-empty fields — audit every wire type, and validate serialized reference values

[macanderson]

What surfaced this

#44 (ADR 0006) was the first code in the repo to serialize a frame from the reference contextgraph-types type and validate that JSON against schema/contextgraph-envelope.schema.json. It immediately failed: the schema listed provenance and relations as globally required on a ContextFrame, but the reference serializer omits both when empty (#[serde(skip_serializing_if = "Vec::is_empty")]), and no frame-validity check (SPEC §6, F1–F5) requires either. A frame with no graph edges — every ingest frame — was therefore un-representable in conformant JSON.

Fixed in #44 for the frame (required is now exactly what the serializer always emits: id, kind, title, score, token_cost). But the class of bug is unaddressed: the schema's required lists were never checked against the reference serializer's actual output, because schema validation only ran over hand-authored example transcripts (schema/validate-examples.py over examples/*), never over Rust-serialized values.

The systemic fix

• Audit every $defs.*.required in the schema against its Rust type. Any field that is skip_serializing_if = "Vec::is_empty" / Option::is_none and is required by no conformance check must not be globally required. Candidates to check: ContextQueryResult, Provenance, ContentRef, Capabilities, DataFlow, VerifyResponse, the envelope variants.
• Add a serialize-and-validate conformance test that constructs a reference value of each wire envelope/type — including the minimal form (empty vecs, None options) — serializes it, and validates against the schema. This catches the whole class going forward, not just the frame. The lightweight required-key guard added in contextgraph-conformance/tests/ingest_conformance.rs is the seed; generalize it (ideally with real JSON-Schema validation, mirroring validate-examples.py, so type/pattern/allOf are covered too, not just required-keys).

This is squarely "conformance that actually catches things" (#33): the schema, the examples, and the reference types describe one wire — a test should prove they agree on serialized reference output, not only on curated fixtures. Related: #48 (extensibility/unknown-field rules), #2 (CI schema validation).

[macanderson]

Re-verified 2026-07-26 against origin/main @ 1ee9b1c.

⚠️ Live defect: the bug class this issue predicts is already firing, in ContextQuery

ContextQuery's schema required lists kinds and anchors, but both fields carry skip_serializing_if = "Vec::is_empty" on the reference serializer.

So an ordinary unfiltered, unanchored query — the single most common query anyone will ever send — serializes without those keys and is INVALID against the repo's own schema. Adding explicit empty arrays makes it validate. This is not a hypothetical audit finding; it is the default path.

The remaining 13 candidate wire types audit clean, so ContextQuery is the only remaining offender. That turns the first checkbox from an open-ended sweep into a small, bounded fix with a known answer.

Recommended scope: keep, and add the found instance to the body so the first checkbox has a concrete failing case to fix rather than an audit to perform. The "validate serialized reference values" checkbox is the regression guard that stops it recurring — and it overlaps #55's golden NDJSON.