Skip to content

feat: residual-size splitting with flattened placement#23

Merged
NickMoignard merged 1 commit into
masterfrom
feat/residual-size-flatten-split
Jul 13, 2026
Merged

feat: residual-size splitting with flattened placement#23
NickMoignard merged 1 commit into
masterfrom
feat/residual-size-flatten-split

Conversation

@NickMoignard

Copy link
Copy Markdown
Owner

Summary

Reworks split's file-extraction engine from a single top-down pass into two passes so it stops emitting one-line !include wrapper files.

Previously a node's size was measured before its children were extracted, and the walk only descended into nodes it had already decided to extract. To reach a dataset's deep custom SQL, the quicksight preset force-extracted the whole PhysicalTableMap chain, so every intermediate level became a one-line !include wrapper — ~40% of the output on real QuickSight datasets.

What changed

  • Phase A (decide) judges each node on its residual size — what it holds once its own extracted descendants collapse to one-line !include placeholders — so a passthrough wrapper stays inline while its large child is lifted out.
  • Phase B (place) writes files and rewrites includes, flattening extracted descendants up to the nearest ancestor that owns a file (dataset/customsql.yml, not dataset/physicaltablemap/<uuid>/customsql.yml).
  • Rules.LeadsBeyond separates reach from forcing a file: the walk can descend past --max-depth toward a deep rule target without forcing a file at each level. ForceExtract keeps meaning "always a file", so the dashboard sheet/category layout is unchanged.
  • Trimmed the quicksight dataset rules to DataSet, CustomSql, and the SqlQuery sidecar.
  • join is untouched.

Results

On the target datasets: .yml files 55 → 34, all 21 one-line stubs gone, sidecars and byte-for-byte round-trip intact.

Docs

New ADR-0003 documents the residual-size + flatten design; README and the config guide updated.

🤖 Generated with Claude Code

Split's file decision was top-down and coupled to recursion: a node's
size was measured before its children were extracted, and the walk only
descended into nodes it had already decided to extract. To reach a
dataset's deep custom SQL the quicksight preset force-extracted the whole
PhysicalTableMap chain, so every intermediate level became a one-line
!include wrapper file — ~40% of the output on real QuickSight datasets.

Rework the engine into two passes. Phase A (decide) judges each node on
its residual size — what it holds once its own extracted descendants
collapse to one-line !include placeholders — so a passthrough wrapper
stays inline while its large child is lifted out. Phase B (place) writes
files and rewrites includes, flattening extracted descendants up to the
nearest ancestor that owns a file (dataset/customsql.yml, not
dataset/physicaltablemap/<uuid>/customsql.yml).

Separate reach from forcing a file: add Rules.LeadsBeyond so the walk can
descend past --max-depth toward a deep rule target without forcing a file
at each level, and trim the quicksight dataset rules to DataSet, CustomSql,
and the SqlQuery sidecar. ForceExtract keeps meaning "always a file", so
the dashboard sheet/category layout is unchanged. join is untouched.

On the target datasets: .yml files 55->34, all 21 one-line stubs gone,
sidecars and byte-for-byte round-trip intact.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@NickMoignard NickMoignard merged commit 313264a into master Jul 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant