Add listing conformance harness (scripts/conformance)#634
Open
tintinthong wants to merge 3 commits into
Open
Conversation
Standalone Node 20 stdlib harness with 14 static gates (structure, JSON:API shapes, realm-bricking relationship classes, date-shape contract, formats, CQ fitted, thunk links, theme tokens, icon CDN, reuse inventory, AI-slop signals) and 5 dynamic gate wrappers (boxel parse/lint, module-load probe, typed-search counts, prerender smoke). Waiver file support, machine-readable report (report-schema.json), broken-listing fixture + selftest. Full-catalog static sweep: 26 listings in ~1s, 0 harness errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new scripts/conformance/ Node 20 (stdlib-only) harness for validating catalog listing packages, producing human-readable output plus a JSON report suitable for CI gating and automated fixing/waiving workflows.
Changes:
- Introduces
run.mjs(harness CLI) andselftest.mjs(fixture-based harness health check). - Implements static gates (S01–S14) and dynamic gates (D01–D05) with a report schema, waiver support, and reuse/slop rule tables.
- Adds fixtures plus repo ignore rules for generated reports/history.
Reviewed changes
Copilot reviewed 35 out of 36 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/conformance/selftest.mjs | Self-test runner that validates harness behavior against broken + known-good fixtures. |
| scripts/conformance/run.mjs | Main CLI entrypoint: loads listing dir, runs gates, applies waivers, emits report + exit codes. |
| scripts/conformance/rules/reuse-rules.mjs | Heuristic detectors for “hand-rolled UI” reuse warnings (S13). |
| scripts/conformance/rules/boxel-ui-manifest.json | Manifest of Boxel UI components used for reuse suggestions/inventory. |
| scripts/conformance/rules/base-field-manifest.json | Manifest of base fields to discourage duplicate local FieldDefs. |
| scripts/conformance/report-schema.json | JSON schema for machine-readable conformance reports. |
| scripts/conformance/README.md | Documentation for running, waiving, and extending the harness. |
| scripts/conformance/lib/report.mjs | Waiver parsing/application plus report aggregation and human output formatting. |
| scripts/conformance/lib/json-walk.mjs | Listing dir walker + JSON/GTS source loader and JSON helpers. |
| scripts/conformance/lib/gts-scan.mjs | Regex/brace-based scanners for classes, formats, imports, and field declarations. |
| scripts/conformance/lib/exec.mjs | npx boxel … wrapper used by dynamic gates with timeouts. |
| scripts/conformance/gates/s14-slop-signals.mjs | Static warnings for deterministic “AI slop” signals in templates/data. |
| scripts/conformance/gates/s13-reuse-inventory.mjs | Static reuse inventory + duplication heuristics; attaches inventory to report. |
| scripts/conformance/gates/s12-icon-cdn.mjs | Static gate that HEAD-checks imported icons exist on the CDN (skippable via --offline). |
| scripts/conformance/gates/s11-cardinfo.mjs | Static warning when instances omit attributes.cardInfo. |
| scripts/conformance/gates/s10-theme-vars.mjs | Static warning for color literals in scoped CSS (encourages theme tokens). |
| scripts/conformance/gates/s09-thunk-links.mjs | Static error when kit-internal linksTo/linksToMany aren’t thunk-wrapped. |
| scripts/conformance/gates/s08-fitted-cq.mjs | Static warnings for fitted templates lacking container queries or using fixed px dimensions. |
| scripts/conformance/gates/s07-formats-present.mjs | Static errors/warnings for missing required formats on exported defs. |
| scripts/conformance/gates/s06-date-shape.mjs | Static errors for DateField vs DateTimeField schema/value mismatches. |
| scripts/conformance/gates/s05-relative-refs.mjs | Static errors/warnings for dev-realm leftovers, localhost, and external module URLs. |
| scripts/conformance/gates/s04-no-inline-media.mjs | Static errors for inline media/base64 blobs in JSON attributes. |
| scripts/conformance/gates/s03-instance-shape.mjs | Static errors for realm-bricking relationship shapes/URLs. |
| scripts/conformance/gates/s02-json-validity.mjs | Static JSON parse + JSON:API card document shape validation. |
| scripts/conformance/gates/s01-structure.mjs | Static listing anatomy/spec/example/imagery checks. |
| scripts/conformance/gates/d05-prerender.mjs | Dynamic prerender smoke checks (requires --realm). |
| scripts/conformance/gates/d04-typed-search.mjs | Dynamic typed-search vs on-disk instance count consistency checks (requires --realm). |
| scripts/conformance/gates/d03-module-load.mjs | Dynamic module-load probe to catch runtime-evaluation failures (requires --realm). |
| scripts/conformance/gates/d02-boxel-lint.mjs | Dynamic realm lint via npx boxel lint (requires --realm). |
| scripts/conformance/gates/d01-boxel-parse.mjs | Dynamic parse/typecheck via npx boxel parse. |
| scripts/conformance/fixtures/broken-listing/venue.gts | Fixture def used to seed a known failure class. |
| scripts/conformance/fixtures/broken-listing/Event/gala.json | Fixture instance intentionally violating seeded rules. |
| scripts/conformance/fixtures/broken-listing/event.gts | Fixture card def intentionally violating seeded rules. |
| scripts/conformance/fixtures/broken-listing/CardListing/broken.json | Fixture listing intentionally missing required structure relationships. |
| .gitignore | Ignores generated .conformance-report.json and .boxel-history/. |
| .boxelignore | Excludes conformance artifacts and .boxel-history from realm sync. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+19
to
+23
| for (const raw of text.split('\n')) { | ||
| const line = raw.replace(/#.*$/, '').trimEnd(); | ||
| if (!line.trim() || /^waivers\s*:\s*$/.test(line.trim())) continue; | ||
| const item = line.match(/^\s*-\s+(\w+)\s*:\s*(.+)$/); | ||
| const cont = line.match(/^\s+(\w+)\s*:\s*(.+)$/); |
Comment on lines
+57
to
+61
| const rootDir = resolve(args.positional[0]); | ||
| if (!existsSync(rootDir)) { | ||
| console.error(`No such directory: ${rootDir}`); | ||
| process.exit(2); | ||
| } |
| } | ||
| } | ||
|
|
||
| // ---- inventory block (attached to the gate result by run.mjs via findings meta) |
The monorepo Lint job (ci-lint.yaml) runs eslint+prettier over the whole checkout including conformance fixtures; the fixture's seeded bugs are conformance-level, not formatting-level, so formatting can comply. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A standalone, stdlib-only Node 20 conformance harness for catalog listing packages — the keystone of the card-factory pipeline (15 new high-quality listings incoming as follow-up PRs).
Exit 0 = pass (warnings allowed) · 1 = errors · 2 = harness fault.
Gates
Static (S01–S14, <1s per listing, no npm install): listing anatomy + Spec coverage · JSON:API shape · realm-bricking relationship classes (linksToMany array shape, external URLs in
links.self) · inline base64/media · reference hygiene (dev-realm leftovers) · DateField/DateTimeField schema↔value contract · required formats · CQ fitted discipline · thunk-wrapped kit-internal links · theme-token usage · icon CDN resolution · component/field reuse inventory · AI-slop signals.Dynamic (D01–D05, local pre-PR): thin wrappers over
npx boxel parse/lint, module-load probe (Gate A), typed-search counts (Gate B), prerender smoke.Warnings never fail a run, but each carries a suggestion + fixRef; the factory workflow requires every warning to be fixed or waived with a reason in
<dir>/.conformance-waivers.yaml(realm-sync-excluded, travels with the PR so reviewers see justifications).Verification
node scripts/conformance/selftest.mjs— broken fixture fails with all 12 seeded finding classes;d29736-tier-listpasses with zero errors.Next PRs
.github/workflows/conformance.yaml— PR check on changed listing dirs + nightly whole-catalog sweep.🤖 Generated with Claude Code