perf: enable parquet lazy vectors - #787
Open
lizhen-0710 wants to merge 2 commits into
Open
Conversation
Enable Parquet readers to produce top-level LazyVectors for projected columns so table scan can defer payload decode/materialization and aggregation can consume lazy columns through ValueHook. Set the Parquet root reader as top-level and carry the upstream lazy-reader correctness fixes needed for safe lazy loading: page-level hook row bias, offset-aware iota rows, string decoder value counts, and decoded string dictionary hook values. Keep scan behavior unchanged for filter columns: pushed-down filter inputs remain eager while projected payload columns can stay lazy and remaining-filter outputs can preserve wrapped lazy vectors. Add focused coverage for reader lazy output, complex projected columns, stale lazy lifecycle failures, small-page lazy loads, string dictionary ValueHook row/value alignment, DWRF string dictionary hooks, scan filter/lazy boundaries, remaining-filter wrappers, aggregation pushdown metrics, and iota offset fallback. Verified with targeted Parquet reader/table-scan tests, DWRF string dictionary hook test, RawVector iota test, staged clang-tidy, and Parquet E2E filter/value-hook coverage across direct and dictionary encodings.
lizhen-0710
force-pushed
the
feat-parquet-lazy-vector-oss
branch
2 times, most recently
from
July 28, 2026 05:11
551f1ed to
5a78e3a
Compare
Add the follow-up correctness fixes needed after enabling Parquet lazy vectors. Vector lifecycle and wrapper handling: - clear stale nulls when reusing LazyVector loaders - make the lazy-wrapper consistency flag atomic and release it from constant/dictionary wrappers - unwrap lazy vectors in flattenVector and load wrapper-over-lazy vectors before compare() Execution and scan materialization: - load reused lazy inputs in FilterProject and remaining-filter scan paths before they can be partially loaded - keep HashProbe filter inputs temporary so lazy wrappers do not outlive filter evaluation - load lazy input before OrderBy sorting and stop lazy hook pushdown across pre-grouped aggregation split remainders - avoid double-counting lazy load time in noMoreInput timing Parquet and connector boundaries: - keep DCMap and Variant parquet readers eager where children are immediately combined - materialize Paimon split rows before merge-engine copy/resize/aggregate paths - unwrap top-level lazy vectors at public Arrow export entry points Tests: - cover lazy wrapper reuse, compare, flatten, FilterProject reuse, OrderBy, aggregation hook, Arrow export, and parquet scan/reader paths - update common query assertions to materialize lazy output consistently Co-authored-by: TRAE CLI <noreply@bytedance.com>
lizhen-0710
force-pushed
the
feat-parquet-lazy-vector-oss
branch
from
July 28, 2026 06:20
5a78e3a to
36c841a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Enable Parquet readers to produce top-level LazyVectors for projected columns so table scan can defer payload decode/materialization and aggregation can consume lazy columns through ValueHook.
Type of Change
Description
Set the Parquet root reader as top-level and carry the upstream lazy-reader correctness fixes needed for safe lazy loading: page-level hook row bias, offset-aware iota rows, string decoder value counts, and decoded string dictionary hook values. Keep scan behavior unchanged for filter columns: pushed-down filter inputs remain eager while projected payload columns can stay lazy and remaining-filter outputs can preserve wrapped lazy vectors.
Add focused coverage for reader lazy output, complex projected columns, stale lazy lifecycle failures, small-page lazy loads, string dictionary ValueHook row/value alignment, DWRF string dictionary hooks, scan filter/lazy boundaries, remaining-filter wrappers, aggregation pushdown metrics, and iota offset fallback.
Verified with targeted Parquet reader/table-scan tests, DWRF string dictionary hook test, RawVector iota test, staged clang-tidy, and Parquet E2E filter/value-hook coverage across direct and dictionary encodings.
Performance Impact
No Impact: This change does not affect the critical path (e.g., build system, doc, error handling).
Positive Impact: I have run benchmarks.
Click to view Benchmark Results
Negative Impact: Explained below (e.g., trade-off for correctness).
Release Note
Please describe the changes in this PR
Release Note:
Checklist (For Author)
Breaking Changes
No
Yes (Description: ...)
Click to view Breaking Changes