Skip to content

Prune unread nested parquet leaves when a projected column is cast to a narrower type (nested schema pruning) #23398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
adriangb wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Prune unread nested parquet leaves when a projected column is cast to a narrower type (nested schema pruning) #23398

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c153c6a
bench: parquet scan with a table schema narrower than a nested column
adriangb Jul 8, 2026
5e737f2
refactor: extract parquet projection read plan into its own module
adriangb Jul 8, 2026
71f5cd2
feat: prune unread nested parquet leaves when a projected column is c…
adriangb Jul 8, 2026
5e55b4f
bench: add pruning-disabled variants to parquet_nested_schema_pruning
adriangb Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions datafusion/common/src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1181,6 +1181,14 @@ config_namespace! {
/// (reading) Use any available bloom filters when reading parquet files
pub bloom_filter_on_read: bool, default = true

/// (reading) If true, when a projected nested column (struct, or
/// struct nested in lists) is read through a cast to a narrower
/// nested type — as happens when the table schema declares fewer
/// struct fields than the parquet file contains — the reader only
/// fetches and decodes the leaf columns the cast retains, instead of
/// the entire column
pub nested_projection_pruning: bool, default = true

@mbutrovich mbutrovich Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flag is responsible for a large share of the PR's surface: the six proto-* files, parquet_writer.rs, file_formats.rs, information_schema.slt, configs.md, and the threading through source -> morselizer -> PreparedParquetOpen -> opener -> DecoderProjection. It's also what triggers the cargo-semver-checks major-version failure CI already reported (new pub field on the externally-constructible ParquetOptions).

The PR description itself notes there's "no scenario where a correctly-working clip should be disabled" and that the flag could be deprecated after a release or two. For a change that's semantically invisible (IO reduction only, worst case = today's behavior), a permanent serialized knob seems disproportionate.

Options, roughly in order of preference:

  1. Drop the flag entirely. The total-fallback design already makes this safe.
  2. If a bisection escape hatch is genuinely wanted, make it a non-serialized/session-only setting so it stays off the proto wire format and out of the semver surface.

Happy to be wrong if there's a concrete reason it must be plan-serialized, but as-is it roughly doubles the file count and forces a major-version bump for an optimization that's invisible to results.


/// (reading) The maximum predicate cache size, in bytes. When
/// `pushdown_filters` is enabled, sets the maximum memory used to cache
/// the results of predicate evaluation between filter evaluation and
Expand Down
4 changes: 4 additions & 0 deletions datafusion/common/src/file_options/parquet_writer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,7 @@ impl ParquetOptions {
maximum_parallel_row_group_writers: _,
maximum_buffered_record_batches_per_stream: _,
bloom_filter_on_read: _, // reads not used for writer props
nested_projection_pruning: _, // reads not used for writer props
schema_force_view_types: _,
binary_as_string: _, // not used for writer props
coerce_int96: _, // not used for writer props
Expand Down Expand Up @@ -500,6 +501,7 @@ mod tests {
maximum_buffered_record_batches_per_stream: defaults
.maximum_buffered_record_batches_per_stream,
bloom_filter_on_read: defaults.bloom_filter_on_read,
nested_projection_pruning: defaults.nested_projection_pruning,
schema_force_view_types: defaults.schema_force_view_types,
binary_as_string: defaults.binary_as_string,
skip_arrow_metadata: defaults.skip_arrow_metadata,
Expand Down Expand Up @@ -620,6 +622,8 @@ mod tests {
maximum_buffered_record_batches_per_stream: global_options_defaults
.maximum_buffered_record_batches_per_stream,
bloom_filter_on_read: global_options_defaults.bloom_filter_on_read,
nested_projection_pruning: global_options_defaults
.nested_projection_pruning,
max_predicate_cache_size: global_options_defaults
.max_predicate_cache_size,
schema_force_view_types: global_options_defaults.schema_force_view_types,
Expand Down
5 changes: 5 additions & 0 deletions datafusion/core/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,11 @@ harness = false
name = "parquet_struct_query"
required-features = ["parquet"]

[[bench]]
harness = false
name = "parquet_nested_schema_pruning"
required-features = ["parquet"]

[[bench]]
harness = false
name = "parquet_struct_projection"
Expand Down
Loading
Loading