Wide-schema parquet: O(1) parquet_column + cacheable arrow-reader-metadata primitives - #9882
Open
adriangb wants to merge 4 commits into
Open
Wide-schema parquet: O(1) parquet_column + cacheable arrow-reader-metadata primitives#9882adriangb wants to merge 4 commits into
adriangb wants to merge 4 commits into
Conversation
This was referenced May 2, 2026
adriangb
force-pushed
the
adrian/wide-schema-perf
branch
2 times, most recently
from
June 8, 2026 18:44
55d6f50 to
97e0b4b
Compare
`parquet_column` previously did an O(N) linear scan over all parquet leaf columns to find the leaf belonging to a given root field. For wide schemas (hundreds/thousands of columns) this is called once per column, making per-file statistics setup O(N^2). Precompute a `root_to_first_leaf` map on `SchemaDescriptor` at construction so `parquet_column` is O(1). Expose it via `SchemaDescriptor::root_first_leaf_index`. Update the `memory_size` tests to account for the extra cache field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adriangb
force-pushed
the
adrian/wide-schema-perf
branch
from
June 8, 2026 18:56
97e0b4b to
0173350
Compare
adriangb
changed the title
adriangb
marked this pull request as ready for review
June 8, 2026 18:57
Add building blocks so callers (e.g. a metadata-caching reader) can construct an `ArrowReaderMetadata` from precomputed parts instead of re-walking every parquet leaf on each file open: - `parquet_to_arrow_schema_and_field_levels`: produce the arrow `Schema` and `FieldLevels` in a single schema walk. - `ArrowReaderMetadata::from_field_levels`: build an `ArrowReaderMetadata` from already-computed `(Schema, FieldLevels)`. - Public accessors on `ArrowReaderOptions` (`virtual_columns`, `supplied_schema`, `skip_arrow_metadata`) so a cache-aware reader can decide whether its cached arrow view applies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a low-overhead `StatisticsConverter` constructor that takes an already-resolved `(arrow field, parquet leaf index)` pair, skipping the name-based `parquet_column` lookup that `try_new` performs. Lets callers that have precomputed the logical->leaf mapping build converters in O(1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a `get_arrow_reader_metadata` method to `AsyncFileReader` whose default implementation delegates to `ArrowReaderMetadata::try_new`, and route `load_async` through it. Cache-aware readers can override it to return a precomputed/cached `ArrowReaderMetadata` (built via `from_field_levels`) instead of re-walking the parquet schema per open. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adriangb
force-pushed
the
adrian/wide-schema-perf
branch
from
June 8, 2026 19:50
0173350 to
41d7b4a
Compare
Contributor
|
are we still keen on this PR? i'm not as familiar with the context but can help review if needed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of the wide-schema parquet read performance work tracked in
#9722 and apache/datafusion#21968.
Rationale for this change
Reading parquet files with very wide schemas (hundreds/thousands of
columns) pays per-file CPU costs that scale with the schema width even
when a query touches only a handful of columns. Two of those costs live
in
parquet:parquet_columndoes an O(N) linear scan over parquet leaves; calledonce per column during per-file statistics setup it becomes O(N^2).
Schema+ dremel field levels (ArrowReaderMetadata::try_new), withno way for a cache-aware reader to reuse a precomputed arrow view.
This PR adds the
parquet-side primitives to fix both. The DataFusionside that consumes them lives in apache/datafusion#21987.
What changes are included in this PR?
Stacked into reviewable commits:
parquet_columnO(1) — precompute aroot_to_first_leafmap onSchemaDescriptor(exposed viaroot_first_leaf_index).parquet_to_arrow_schema_and_field_levels(one schema walk →(Schema, FieldLevels)),ArrowReaderMetadata::from_field_levels,and public
ArrowReaderOptionsaccessors.StatisticsConverter::from_arrow_field— low-overhead constructortaking a resolved
(field, leaf index).AsyncFileReader::get_arrow_reader_metadata— new method(default delegates to
try_new;load_asyncroutes through it) so acache-aware reader can short-circuit the per-leaf walk.
Are these changes tested?
Yes — existing parquet tests pass (including the updated
memory_sizetests for the new
SchemaDescriptorcache field). Each commit buildsgreen on its own.
Are there any user-facing changes?
New public API only (the items above); no breaking changes.