Fix markdown serializer boundary-whitespace defects on the…#287
Merged
Conversation
* docs(open-knowledge): add serialize-fidelity substrate spec
Approved /spec planning record (SPEC.md + evidence + meta) for the serialize-output fidelity test substrate and 7-rung fidelity ladder: non-normalizing byte comparator, maximalist reachable enumeration, serialize-output catalog and ratchet, INV-7 registry flip, bug-3 detection tiers, and the minimize-the-ratchet execution model (decisions D1 through D11).
* [US-001] add non-normalizing serialize-fidelity byte comparator
New packages/core/src/markdown/serialize-fidelity.test-helper.ts exports
assertFragmentSerializesTo, assertNoBoundaryLeak, and assertMarkSurvives: raw
byte assertions over a ProseMirror fragment serialized through the production
MarkdownManager, with zero normalization, so a leaked boundary  /	
can never be masked by a shared trimEnd before comparison. The subject is
always a fragment (the mint), never serialize(parse(x)).
The co-located test exercises all three exports with green cases, non-vacuity
bites, and a self-proving anti-masking guard: a byte-exact pin whose throw
holds only while the comparator stays byte-exact, so a future trimEnd in the
comparison path fails loud.
Regenerates ng-anchors-catalog.json: the new .test-helper.ts is a file in the
NG-anchors production-source scan glob, so its productionFileCount advances
from 82 to 83.
* [US-002] add maximalist boundary-fidelity reachability enumeration + classifier
New packages/md-conformance/src/boundary-fidelity/reachability.ts enumerates
the full six-axis cross-product of the live-bridge boundary-whitespace / escape
fidelity class (ws-type x position x construct x write-entry x Observer-A
direction x significance) and classifies every cell residual / guard /
theoretical, so the class is a discovery engine rather than a hand-picked list
that re-introduces the blind-one-path gap.
The classifier is pure (no engine) and cites each verdict, mirroring the
FORMAT-DOF and NG taxonomy split. The RES discipline: R-DIR collapses every
door but the WYSIWYG fragment serialize mint; the over-fidelity pole is scoped
to the verified leaking constructs (plain-text, list-item, blockquote, found by
running the real serializer); heading (ATX strips trailing whitespace) and
table-cell (padding absorbs it) are cited theoretical exclusions, not leaks;
the under-fidelity pole is the exposed mark set (strike, highlight) on a
whitespace boundary; emphasis / strong and the leading tab are must-preserve
guards. The discovered siblings (the TAB leak, the backslash over-escape, the
strike / highlight silent loss) are first-class residuals.
The co-located test pins determinism (stable id-sorted product, unique ids),
totality (every cell classified with a non-empty reason), the known residual
register, the guards, and non-vacuous cited exclusions. Empirical grounding of
each residual / guard lands downstream (the serialize-output catalog's expected
bytes and the ladder rungs), as the substrate design intends.
* [US-003] add serialize-output boundary-fidelity catalog with live-binding ratchet
New serialize-OUTPUT catalog for the boundary-whitespace/escape fidelity class. Runs the real MarkdownManager over every reachable cell from the reachability enumeration and records the emitted bytes, at the serializer's produced-output altitude rather than the construct-presence CONSTRUCT_PROBES altitude.
Live-catalog binding (P-A-706 analog): the committed JSON deep-equals a fresh build, and diffCoverage flags any reachable cell missing a covering entry, with planted-negative (dropped entry) and planted-positive (total absence) controls proving the predicate has teeth. 3-tier S/G/M provenance; the seven hand-found register cells carry Tier-M citations. Guard grounding asserts re-parse properties rather than leak presence, staying decoupled from the witness-manifest ratchet that owns the known-failing semantics.
Adds the enumerate:boundary-fidelity-catalog materializer script.
* [US-004] add boundary-fidelity ladder + 3-tier witness manifest
Five floor rungs (byte-exact, meaning-tolerant, no-escape-leak,
no-mark-loss, fixed-point) over the boundary catalog corpus, each
asserting through the US-001 comparator with a NUL non-vacuity control
and a substrate-identity guard. A separate boundary-fidelity witness
manifest owns the ratchet because the strict byte oracle is blind to the
PM-fragment serialize reds: the known leaks and mark drops are recorded
as producer-fixable-residual tier-2 membership (10 guards, 17 reds, 0
accepted-divergent) with a freshness gate. The gate stays green with the
bugs present and reds on a new unrecorded leak or an un-removed green flip.
* [US-005] flip INV-7 from gap to covered-at-boundary in invariant-oracle registry
INV-7 (no-visible-escape-leak) now has a live oracle: the boundary-fidelity
ladder no-escape-leak rung detects over-fidelity escapes minted on the WYSIWYG
fragment-to-serialize path. Point its registry resolution at the editor-app-e2e
boundary and regenerate the pinned catalog. Counts move covered-at-boundary 3 to 4
and gap 2 to 1; engine invariants stay untouched per the non-engine layer gate.
* fix(open-knowledge): escape literal NUL in boundary-fidelity ladder non-vacuity test
The U+0000 non-vacuity control embedded a literal NUL byte in startDoc, making the source file binary to git (broke diffs and review). Use the \u0000 escape: identical runtime string, proper UTF-8 source.
* chore(open-knowledge): post-review cleanup for boundary-fidelity substrate
Strip the US-001 process marker from the ladder test header comment (comment-discipline keeps story IDs out of source). Clarify SPEC IS-4: the 5 floor rungs land in this story; the 2 bug-3 family rungs ship with IS-6. Add the post-implementation review record.
* fix(open-knowledge): add deterministic freshness gates for boundary-fidelity catalog and witness manifest
The serialize-fidelity substrate committed two JSON artifacts,
boundary-fidelity-catalog.json and boundary-fidelity-witness-manifest.json,
but unlike every other committed JSON in the md-conformance package neither
had a standalone deterministic freshness-check in the bun run check chain.
Their freshness was enforced only at the test tier, which turbo caches, so a
stale commit could pass the gate on a warm cache.
This adds two check scripts that re-derive each artifact in-memory via its
production builder and deep-compare against the committed JSON, excluding the
non-deterministic generatedAt timestamp, mirroring the established
check-catalog-fresh and check-witness-manifest-fresh precedent. They are
wired through a new check:boundary-fidelity-fresh script into both the check
and check:full:parallel chains, immediately after check:conformance-gates, so
drift fails fast and deterministically at the standalone check tier.
* docs(open-knowledge): add substrate-patterns evidence brief
The code-verified machinery brief that grounded the substrate implementation: the rung template, 3-tier witness manifest, invariant-oracle registry, corpus/P-A-706 binding, mae.ts contract, the live-bridge harness, and the CI-script wiring.
* fix(open-knowledge): drop insignificant boundary whitespace + encode strike/highlight marks in serializer (BF-1, BF-3)
The WYSIWYG-fragment to markdown serializer had two boundary-fidelity defects on
the Observer A mint path:
BF-1 (over-fidelity): an insignificant boundary space or tab at a phrasing to
block edge serialized to a visible numeric char-ref (a literal   / 	)
because mdast-util-to-markdown's state.unsafe encodes it to survive re-parse.
That whitespace re-parses away at a block edge, so the encode is over-broad.
safeText now drops the two phrasing boundary-space rules and the TRAILING
phrasing tab rule, emitting a literal. The LEADING tab rule stays in force: a
line-initial tab is the indented-code-block trigger, so its char-ref must
survive. The load-bearing inside-attention space is preserved for free by
encodeAttentionBoundaries, which takes over once safeText stops masking it.
BF-3 (under-fidelity): strike (~~) and highlight (==) emitted their inner value
raw, so a boundary space broke GFM / highlight delimiter-run flanking and the
mark dropped silently on re-parse. encodeAttentionBoundaries, the same inside
whitespace encode emphasis/strong use, now wraps the strike and highlight inner
value too, so a boundary space is char-ref-encoded and the mark survives. The
encode handles a missing neighbor (a direct handler invocation with an empty
info) as the document edge.
serialize-fidelity.test.ts (the unit test of the byte comparators) updates its
anti-masking pin and non-vacuity bites to the significant inside-emphasis mint,
the inline-whitespace char-ref that legitimately remains after BF-1.
* test(open-knowledge): flip boundary-fidelity substrate from RED-recorder to GREEN-guard (BF-1, BF-3 landed; BF-2 escalated)
The serializer fixes flip the recorded reds, so the derived catalog, witness
manifest, and ladder are regenerated and re-pointed.
reachability.ts: the BF-1 cells (insignificant trailing space/tab on
plain-text/list-item/blockquote) and the BF-3 cells (strike/highlight
significant boundary space/tab) reclassify from residual to guard — the green
the fix now protects. The attention-boundary encode set widens to all four
attention marks; the over-fidelity classifier returns a guard for the now-clean
trailing whitespace. The producer-fixable-residual tier drops from 17 to 3.
The 3 remaining residuals are the BF-2 interior backslash-punct over-escape,
ESCALATED: the `!`-before-`[` escape disambiguates a shortcut reference image
the moment a matching `[x]:` definition exists anywhere in the document (bare
`![x]` + `[x]: /u` re-parses as an imageReference; the escaped form as plain
text plus a linkReference). The per-text-node serializer has no whole-document
reference-definition context to gate on, so narrowing the escape correctly needs
a design decision on the escape contract. It stays a tracked residual; the tier
never relaxes to accepted-divergent (which ships empty).
witness-manifest.ts adds the escape-leak-guard role and re-points guardsFix for
the strike/highlight guards to BF-3. The ladder rungs that asserted the reds via
toThrow() now assert the fixed cells are clean (rung 3 no-escape-leak, rung 4
no-mark-loss), keep the BF-2 over-escape red, and keep every NUL non-vacuity
control biting. Rung 1 (strict byte-exact) covers the char-ref-bearing guards;
the BF-1 literal-whitespace guards are byte-unstable by design and ride rungs 2
and 5. The classifier and manifest unit tests flip to the post-fix partition.
* chore(open-knowledge): changeset for boundary-whitespace serializer fidelity fix
* chore(open-knowledge): drop unused ladder builder + collapse witness-manifest JSON arrays
Biome lint fixes for the boundary-fidelity substrate flip: the rung-4
non-vacuity probe no longer constructs a strike fragment, so the local strike
builder is unused; the regenerated witness manifest's single-element closingGaps
arrays collapse to biome's canonical one-line form.
* test(open-knowledge): bug-3 source-mode write-back guards (re-indent closed by #1991)
Live Observer-A integration guards proving bug 3's re-indent facet is closed on the current base: the faithful <Steps> is a serialize fixed-point, an isolated source keystroke stays byte-verbatim (V1), and a concurrent WYSIWYG fragment commit (non-vacuous: it fires Observer A) does not re-indent the <Steps> in Y.Text. Reads raw Y.Text bytes (every shared comparator trimEnds). Guards a regression that re-opens the write-back; the cursor-jump facet is downstream of the same write-back (Y.Text-delta-driven), so guarding the bytes guards the caret.
* fix(open-knowledge): keep leading boundary-space char-ref in serializer
safeText dropped both phrasing-space unsafe rules, so a paragraph whose text
starts with four or more spaces serialized to a literal indented run that
CommonMark re-parses as an indented code block, a serialize-to-reparse
block-type change that corrupted docs on close-and-reopen. Narrow the space
clause to the trailing edge only (matching the sibling tab clause) so the
leading-space rule stays in force and the first space mints a   char-ref
that holds the re-parse to a paragraph. Trailing-space and strike/highlight
behavior are unchanged.
Guard it in the boundary-fidelity substrate: classify leading-space-on-block-
text as a must-preserve guard (mirroring the leading-tab guard), add the catalog
and witness cells, add a ladder rung asserting the minted char-ref keeps the
re-parse a paragraph, and regenerate the catalog and witness manifest. Run
check:boundary-fidelity-fresh in CI and add src JSON to the turbo test inputs so
a stale committed catalog cannot be served from cache.
Also tidy exhaustiveness with assertNever, share the catalog output-path getter,
correct two stale serializer comments, and align the md-conformance scripts with
sibling conventions.
* chore(open-knowledge): refresh ng-anchors catalog test-file count
The committed md-audit ng-anchors catalog recorded 178 test files while a fresh
build counts 179; a test file was added earlier without regenerating the
catalog. The deep-equal freshness assertion runs only in the turbo-cached test
tier, so the drift was served green from cache and surfaced only on a cold
build. Regenerate so the freshness test passes. Pre-existing and unrelated to
the serializer fix.
* chore(open-knowledge): strip stray NUL byte from two spec evidence docs
* fix(open-knowledge): drop mirror-excluded md-conformance ref from check scripts
main #2069 mirror-excludes packages/md-conformance. The boundary-fidelity-fresh freshness gate still runs in CI via public-open-knowledge-validation.yml; removing the package.json check/check:full:parallel refs restores the manifest full-value replacement match so the public clone package.json stays valid.
* test(open-knowledge): pin Steps serialization to a literal, not a round-trip oracle
main added a biome no-roundtrip-identity-oracle rule forbidding serialize(parse(x)).toBe(x) in public tests. Pin the exact expected <Steps> serialization as a literal public-contract assertion; the byte-identity oracle stays in the private fidelity suite.
* test(core): add BF-3 boundary-space cases; clarify leading-space guard comments
Close out two non-blocking review suggestions on the serialize-fidelity fix.
- serialize-fidelity.test.ts: the under-fidelity oracle cases used clean
content (strike('word')), vacuous for BF-3 since they stay green without
encodeAttentionBoundaries. Add strike('word ')/highlight('word ')
boundary-space cases that exercise the trailing-space mark-survival path.
- to-markdown-handlers.ts: define the mdast-util-to-markdown after/before
field convention up front so the (after [\r\n]) parentheticals read
correctly, and drop process vocabulary from the leading-tab guard comment.
---------
GitOrigin-RevId: c74100b910580131aedfe6dbe5281331914e7073
![inkeep-internal-ci[bot]](https://avatars.githubusercontent.com/in/2806322?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated approval from agents-private public-mirror-sync (run: https://github.com/inkeep/agents-private/actions/runs/28203247584). Source of truth is the monorepo; direct edits on inkeep/open-knowledge are overwritten on next sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix markdown serializer boundary-whitespace defects on the WYSIWYG-to-source path. An insignificant trailing space or tab at a block edge now serializes to a literal character instead of a visible
 /	character reference, so source mode no longer shows the escape where a space was typed (a leading tab, or a leading run of four or more spaces, still encodes, since it would otherwise trigger an indented code block on re-parse and turn the paragraph into a code block). Strikethrough (~~) and highlight (==) marks authored with boundary whitespace now char-ref-encode that whitespace so the mark survives re-parse instead of silently dropping. Emphasis and strong already behaved correctly and are unchanged.