v4 Roadmap: Tabular Generalization

[johannschopplich]

This was generated by AI during triage.

v4 Roadmap: Tabular Generalization

Three open RFCs hit adjacent variants of the same gap. TOON's tabular collapse in §9 only handles uniform-primitive arrays, and several contributors have prototyped extensions targeting the more common mixed or nested shapes.

This issue is for coordination. Proposals stay on their own threads; this thread is for sequencing, not voting.

The three RFCs

• [RFC]: Nested tabular headers for uniform nested objects #46 (Nested tabular headers) – orders[2]{id,customer{name,country},total}:. Targets uniform nested objects. Single ABNF change to §6 fields-seg, depth-first leaf walk, byte-identical output where inapplicable, fail-closed on old decoders. Reference impl in feat: add nested table syntax for uniform nested objects toon#296. Smallest blast radius. Open versioning question: the RFC currently targets v3.1 (MINOR), but per VERSIONING.md, pre-v3 decoders hard-fail on the new syntax (the RFC body admits "Decoders built for v3 will hard-fail"), which is arguably an incompatible behavior change and therefore MAJOR (v4). Worth settling before the implementation PR lands.

• [RFC] v4.0: Mixed columnar arrays, objectArrayLayout, ignoreNullOrEmpty, excludeEmptyArrays #48 (Mixed columnar + options) – columnar layout with spill lines for arrays where most fields are primitive but some are complex. Targets mixed-shape arrays. Bundled with three other features (objectArrayLayout option, ignoreNullOrEmpty lossy-by-default, excludeEmptyArrays lossy-by-default) which should be unbundled. Backed by production .NET deployment data.

• RFC: Object Schema Headers (Nest-Collapse) for Keyed Object Collections #45 (Object schema headers / Nest-Collapse) – key{f1,f2}*: for keyed object collections (Record<string, Object>). Targets keyed dictionaries. Most novel semantics: introduces a * sigil and a ~ absent-sentinel. Working prototype, 21–26% savings measured.

Proposed sequencing

1. [RFC]: Nested tabular headers for uniform nested objects #46 first. Smallest grammar change, reference impl exists, fail-closed semantics keep risk low. Resolve the open questions (depth cap, empty-group handling, versioning classification).
2. Reassess [RFC] v4.0: Mixed columnar arrays, objectArrayLayout, ignoreNullOrEmpty, excludeEmptyArrays #48's columnar. Once nested headers are in, check whether the mixed-columnar case is still distinct or becomes a degenerate form. Unbundle the lossy options regardless – those want separate MINOR PRs with false defaults.
3. RFC: Object Schema Headers (Nest-Collapse) for Keyed Object Collections #45 last. The keyed-object form is the most novel and the ~ sentinel needs more design work. May be subsumed by a path-expansion convention rather than a new sigil.

Out of scope here

• [RFC]: Add \uXXXX unicode escape sequences for control characters in strings #39 (\uXXXX escapes) ships independently as v3.1, orthogonal to the tabular discussion.
• Empty arrays (eg matches[0]:) are often misinterpreted by LLMs #19 (empty-array LLM confusion) ships as a v3.x clarification.

How to engage

Keep design discussion on the individual RFC threads. Use this issue for cross-cutting concerns, sequencing pushback, or unification proposals that span multiple RFCs.

Thanks to @Turtle-dev3, @vhafdal, and @metafishTV. Three independent contributors hitting the same gap from different angles is the strongest signal there is that v4 needs to address tabular generalization.