Proposal: a §9 conformance corpus (valid + invalid example bundles + a runner)

[wey-gu]

okf/tests/test_document.py unit-tests OKFDocument on inline strings, which is
great for the parser. What the repo does not yet have is a corpus of whole example
bundles that pins down the bundle-level conformance rules in SPEC §9. I would like to
contribute one, if it is welcome.

What it is. A small tests/conformance/ corpus plus a parametrized
test_conformance.py that validates each bundle with the existing OKFDocument (no
new dependencies):

• valid/ bundles that MUST be conformant: a spec-minimal one (only type), a full
  one (recommended fields, a file-relative cross-link, # Citations, log.md,
  okf_version on the root index), a unicode/CJK one, a broken-link one (§5.3:
  consumers MUST tolerate broken links), and an extra-keys one (§4.1).
• invalid/ bundles that MUST be flagged, each isolating one violation: missing
  type, empty type, no frontmatter, unterminated frontmatter, non-mapping
  frontmatter, and a README.md left frontmatter-less (it is not a reserved name per
  §3.1, so it reads as a malformed concept — a gotcha real producers hit).

It tests the spec's §9 bar (parseable frontmatter mapping + non-empty type),
which is deliberately looser than OKFDocument.validate()'s producer bar (that also
requires title/description/timestamp), so spec-minimal producers are not failed.

Why it helps. A conformance corpus is how a young spec pins down what it actually
means: it documents §9 by example, catches parser regressions, and gives anyone
writing a producer a concrete target to test against. It slots into the existing
okf/tests/ pytest setup and adds no dependencies.

Two questions it surfaces, that I would align the corpus to your answers on:

1. Does a file with no frontmatter at all violate §9.1 (no block) or just §9.2
   (no type)? OKFDocument.parse treats a no-frontmatter file as an empty mapping
   rather than raising, so today the corpus catches that case via the missing-type
   check. If §9.1 is meant to require the presence of a block, the check (or
   parse) may want to distinguish "no block" from "empty block."
2. Reserved-file shape (§6 index.md, §7 log.md) is not machine-checked yet. Worth
   a follow-up, or out of scope for §9 conformance?

I have the corpus and runner drafted and passing against the current OKFDocument
(5 valid bundles conformant, 6 invalid flagged). Happy to open the PR and sign the
CLA if the direction is agreeable. Context: I work on Nowledge Mem, which exports OKF
bundles, so this came out of validating real output.

Thanks for publishing OKF.

Wey Gu

[tcconnally]

Thanks for this proposal — this is a great idea and the corpus you describe would catch exactly the kind of spec-compliance drift we just found and fixed.

Re: the validate() discrepancy you noticed — PR #64 (opened today) fixes this. REQUIRED_FRONTMATTER_KEYS was incorrectly set to ("type", "title", "description", "timestamp") when the spec only requires type (OKF §4.1). After the fix, validate() only checks for type, matching the §9 bar. Your spec-minimal bundle (type-only) will now pass validation cleanly.

On your two questions:

1. No-frontmatter files: I agree §9.1 should require the presence of a frontmatter block. parse() returning an empty mapping for no-frontmatter files conflates "no block" with "empty block." These are different cases — a file with ---\n---\n (empty but present) vs raw markdown with no --- delimiters. Your corpus distinguishing these would be valuable. I'd suggest OKFDocument.parse() raise a distinct error for missing frontmatter, or return a sentinel, rather than silently treating the whole file as body.

2. Reserved-file shape: My instinct is that §9 conformance should be machine-checked — index.md format (no frontmatter except okf_version) and log.md format (ISO 8601 date headings) are structural rules, not prose conventions. But "don't let perfect be the enemy of started" — a corpus without reserved-file checks is still a huge improvement over the current inline-string-only test setup. Ship v1, add reserved-file shape in a follow-up.

One suggestion: add a conformance/valid/spec-minimal/ bundle with a single concept doc that has type as its only frontmatter key. This is the spec's absolute minimum bar and the most common failure mode for programmatic producers that don't have human-readable descriptions. If a producer emits {"type": "API Endpoint"} with no other fields, that SHOULD pass §9. Your proposal already includes this ("a spec-minimal one"), just highlighting that it's the single most important test case.

[Harsh-Gharewal]

Great!!

[distorx]

+1, and a data point that the §9 bar you're pinning is the right one.

We run that exact check (parseable frontmatter mapping + non-empty type) over a ~7,500-concept bundle: it currently sits at 72.6% conformant, and the entire non-conforming tail is freeform / auto-generated notes that simply lack a type. That's precisely why the §9 bar has to be looser than a producer's bar — your spec-minimal valid/ case (only type) captures the distinction exactly, and it's the thing that keeps a conformance runner from failing legitimately-minimal producers.

Your invalid set matches the real producer bugs — the present-but-empty type (distinct from missing) one especially bit us, as did the frontmatter-less README.md reading as a malformed concept. Happy to contribute valid/invalid samples from our corpus if it helps round out tests/conformance/.