Conformance baselines as a public contract: document format + aggregation semantics for the website conformance page

[nickna]

Context

sharpts-www (the marketing/playground site) is adding a /conformance page: a collapsible per-category/per-method explorer with pass-percentage bars, generated from this repo's committed conformance baselines. The site consumes SharpTS as a pinned submodule, so the baselines are already version-matched to exactly the build the playground runs — no CI artifact or pipeline needed. The website parses these files at build time:

• SharpTS.Test262/baselines/interpreted.txt
• SharpTS.Test262/baselines/compiled.txt
• SharpTS.TypeScriptConformance/baselines/interpreted.txt

That makes the baseline files an externally consumed contract, not just internal harness state. This issue covers the (small) SharpTS-side work to make that consumption safe and the numbers well-defined.

Work

• Document the baseline file format as externally consumed. Add a note to SharpTS.Test262/README.md and SharpTS.TypeScriptConformance/README.md: line format is <test-path> <Bucket[:reason]> with a single # header line; the bucket vocabulary is closed (Test262: Pass, Fail, RuntimeError, ParseError, Timeout, Skipped[:reason]; TS conformance: Pass, Fail, ParseError, TypeCheckError, Skipped:<reason>, HarnessError). Format/vocabulary changes should bump a version marker in the header comment so the website parser can fail loudly instead of silently miscounting.
• Define the official aggregation semantics (one place, e.g. a short section in each README):
  • Denominator: all tests excluding Skipped:* (skips are policy, not capability — counting them either way fudges the number).
  • Numerator: Pass only. Fail, RuntimeError, ParseError, Timeout, TypeCheckError, HarnessError all count as not-passing.
  • Skipped counts are reported separately/footnoted, never folded into the percentage.
    Any consumer (the website, badges, future dashboards) must compute identical percentages from the same files.
• (Optional) baseline lint in CI: a trivial check that every non-comment line parses as <path> <known-bucket> — protects the contract against hand-edit typos and harness drift.

Non-goals

• No JSON/artifact generation in SharpTS CI. The raw committed baselines are the interface; aggregation, tree-building, and presentation live entirely in the website build.
• No new test infrastructure. Subset growth is already tracked elsewhere and feeds the page automatically:
  • Test262: broader coverage rollout (grow the committed subset) #69 (Test262 rollout) — each folder added appears on the site with zero extra plumbing
  • Epic: Add TypeScript conformance test runner (microsoft/TypeScript corpus) #80 (TS conformance epic) — currently 79 tests / 2 folders; as this grows, the site gains a "Type system" top-level section
  • TS Conformance Phase 1.5 (deferred): load tsc lib.d.ts files into type checker #99 (lib.d.ts loading) — shrinks Skipped:lib-drift noise, tightening the published numbers

Current numbers (for reference, from committed baselines)

Suite	Mode	Pass	Total (excl. skipped)
Test262	compiled	7,849	~10,505 (~75%)
Test262	interpreted	3,898	~10,505 (~37%)
TS conformance	interpreted	32	78

(Website issue link to follow.)

[nickna]

Website-side counterpart (aggregator + /conformance page): nickna/sharpts-www#6