feat: SQL sidecar (.sql) extraction with local fidelity guard (#16)#20
Merged
Conversation
Add a Sidecar rule action that lifts a scalar's text into a standalone non-YAML file (e.g. a dataset's CustomSql.SqlQuery into a .sql file), replacing it with an !include. Extraction is guarded so it never risks the round-trip invariant: before writing a sidecar, canReconstruct builds the exact scalar join will rebuild (yamltree.RawScalar) and only commits when it re-renders identically to the original; otherwise the scalar stays inline. SQL with embedded CRLF re-quotes deterministically and passes; an int-looking scalar, say, would not and is kept. - yamltree: SaveRaw (verbatim write) + RawScalar (the shared reconstruction node), used by both the guard and join so the guard predicts join exactly. - join: a non-.yml/.yaml include resolves to its raw bytes as a string scalar. - quicksight preset: force the DataSet physical-table chain open so the walk reaches the SQL, then a Sidecar rule (.sql) on CustomSql.SqlQuery. - cli: --no-sidecar drops sidecar rules for a structural-only split; verify now exercises the real quicksight preset (rules + sidecars), so it gates sidecar fidelity, not just the bare engine. Sidecar applies to scalars, handled before the isComplex guard that governs file-per-subtree extraction. Closes #16. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds “sidecar” extraction for scalar values (notably QuickSight custom SQL) so large/escaped strings can be lifted into readable non-YAML files (e.g., .sql) while preserving the join(split(x)) ≡ x fidelity guarantee via a per-scalar reconstruction guard. It also updates join to resolve non-.yml/.yaml includes as raw string scalars and makes verify exercise the real QuickSight preset (including rules/sidecars).
Changes:
- Introduces Sidecar rules (preset-level) and split-side scalar extraction guarded by a render-equality reconstruction check.
- Extends join to treat non-YAML
!includetargets as raw bytes spliced back as a string scalar. - Adds CLI support for
--no-sidecarand expands tests (unit + txtar script) around SQL sidecars and guard behavior.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/yamltree/yamltree.go | Adds raw sidecar writer (SaveRaw) and shared reconstruction node helper (RawScalar). |
| internal/split/split.go | Adds preset-derived options, Sidecar extraction, and the per-scalar fidelity guard. |
| internal/split/split_test.go | Adds tests covering SQL sidecar extraction and guard fallback (e.g., int-like scalars). |
| internal/preset/rule.go | Extends rule model with Sidecar action and Ext field. |
| internal/preset/rule_test.go | Asserts the QuickSight preset includes the .sql Sidecar rule for CustomSql.SqlQuery. |
| internal/preset/preset.go | Updates QuickSight preset to force-open dataset paths and apply .sql Sidecar on SqlQuery. |
| internal/join/join.go | Resolves non-YAML includes by reading raw bytes and constructing a string scalar. |
| internal/join/join_test.go | Adds a test verifying raw include reconstruction preserves CR/LF bytes. |
| internal/cli/testdata/script/preset-quicksight-dataset.txtar | End-to-end script test for SQL sidecars and --no-sidecar. |
| internal/cli/commands.go | Adds --no-sidecar, factors OptionsFromPreset, and makes verify use QuickSight preset rules/sidecars. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…riants Address review of the sidecar slice: - Document the two yaml.v3 properties the fidelity guard's soundness rests on (content-only style selection, and disabled line-wrapping), with an explicit warning not to add SetWidth to yamltree.Save without re-deriving the argument. - Add a SaveRaw byte-verbatim test (empty, trailing newline, CRLF, leading space, no trailing newline) — the guard's exactness depends on this. - Add a test that a Sidecar rule with no Ext defaults to .txt. The guard was independently verified sound against yaml.v3 v3.0.1 (no false positive commits a lossy sidecar). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lot) A Sidecar rule whose Ext is .yml/.yaml would be written raw but re-parsed as YAML on join (join picks raw-vs-YAML by extension), breaking the round-trip while the canReconstruct guard reasoned about a raw read. Reject that misconfiguration in extractSidecar, and make split.IsRawInclude the single source of truth for the raw-vs-YAML extension decision so split and join cannot disagree (join now calls it instead of a private copy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Third slice of preset-based semantic explosion (#16). Lifts a QuickSight dataset's custom SQL out of an unreadable
\r\n-escaped YAML scalar into a readable.sqlfile — without ever risking thejoin(split(x)) ≡ xinvariant.How
internal/preset/rule.go): lifts a scalar's text into a standalone non-YAML file with the rule'sExt, replacing it with an!include. Handled before theisComplexguard (which governs file-per-subtree extraction).canReconstruct,internal/split/split.go): before committing a sidecar, it builds the exact nodejoinwill rebuild (yamltree.RawScalar) and only extracts when it re-renders identically to the original scalar; otherwise the scalar stays inline. Local, per-scalar, no global fallback — the invariant is never weakened. (Reasoning: same value + tag ⇒ yaml.v3 picks style from content alone, so a standalone match holds in any parent context; SQL with CRLF re-quotes deterministically and passes.)yamltree:SaveRaw(verbatim write) +RawScalar(the one shared reconstruction node used by both the guard and join, so the guard predicts join exactly).join: a non-.yml/.yamlinclude resolves to its raw bytes as a string scalar (reusing the existing traversal/cycle guards).DataSet.PhysicalTableMap.*.CustomSqlchain open so the walk reaches the SQL, then aSidecarrule (.sql) onSqlQuery.--no-sidecardrops sidecar rules (structural-only split);verifynow runs the real quicksight preset (rules + sidecars), so it gates sidecar fidelity, not just the bare engine.Result (real DataSet fixture)
Verification
go build ./...,go test ./...,golangci-lint runall green.!includewiring + round-trip on a synthetic dataset, join raw-include reconstruction (CRLF preserved), preset sidecar-rule assertion, andpreset-quicksight-dataset.txtarend-to-end (incl.--no-sidecar).verifyok under the now rule-aware verify.Notes
.sqlfiles are written byte-exact (no added trailing newline) so the guard's reconstruction matches; this is what keeps the round-trip exact.verifynow always uses the quicksight preset (it has no--presetflag); harmless on other shapes since the rules are root-anchored.Closes #16.
🤖 Generated with Claude Code