Extract RowFilterGenerator + push-decoder module from opener.rs - #6
Closed
adriangb wants to merge 3 commits into
Closed
Conversation
This decouples the helper from opener.rs internals: it now takes its inputs directly (predicate, schema, metadata, reorder flag, metrics) rather than a `&PreparedParquetOpen`, so it lives alongside the underlying `build_row_filter` it wraps. opener.rs constructs it from the prepared state. No behavior change.
Move the configuration struct and its builder construction out of opener.rs into a new `push_decoder` module. The helper becomes a `build(prepared_access_plan, metadata)` method on the config, which reads more naturally at the call site. No behavior change.
Co-locate the per-file stream driver with the builder configuration it consumes. Add `into_stream` so the opener doesn't need to name the unfold/fuse type; it just hands off the state and gets back a `BoxStream`. After this commit, push_decoder.rs owns the full push-decoder lifecycle (builder setup + stream driving), and build_stream in opener.rs reads as orchestration: prepare access plans, build decoders, hand off to the stream, optionally wrap in EarlyStoppingStream. No behavior change.
|
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on top of your apache#22191. Three small commits that move the pieces that don't depend on opener-internal types out of
opener.rs, so the file gets closer to pure state-machine orchestration.Summary
RowFilterGeneratortorow_filter.rs. Decouples it fromPreparedParquetOpen— it now takes its inputs directly (predicate, schema, metadata, reorder flag, metrics), so it lives next to thebuild_row_filterit wraps. (I considered a literalimpl From<&PreparedParquetOpen>per Adrian's suggestion, but the constructor also needsfile_metadatawhich is a sibling ofpreparedrather than a field on it, so a genericnew(...)was the cleanest decoupling. Happy to fold into aFrom<(&PreparedParquetOpen, &ParquetMetaData)>tuple impl if you'd prefer.)DecoderBuilderConfiginto a newpush_decoder.rsmodule. Same struct + fields; the freebuild_decoder_builderbecomes a.build(prepared_access_plan, metadata)method on the config, which reads more naturally at the call site.PushDecoderStreamStateintopush_decoder.rs. Co-locates the per-file stream driver with the builder configuration it consumes. Addsinto_stream(self) -> BoxStream<...>so the opener doesn't need to name the unfold/fuse type — it just hands off the state and gets back a boxed stream. Drops the now-unusedparquet::DecodeResult,ParquetPushDecoder,Projector,Gauge,DataFusionError, etc. imports fromopener.rs.After this,
push_decoder.rsowns the full push-decoder lifecycle (builder setup + stream driving), andbuild_streaminopener.rsreads as orchestration: prepare access plans, build decoders, hand off to the stream, optionally wrap inEarlyStoppingStream.The commits are stacked one-move-per-commit so each is reviewable independently.
No behavior change
Every move is pure code motion; no semantics change. The tests called out in apache#22191's body still pass:
cargo fmt --all --check— cleancargo clippy -p datafusion-datasource-parquet --all-targets --all-features -- -D warnings— cleancargo test -p datafusion-datasource-parquet --lib fully_matched— 5/5 passcargo test -p datafusion-datasource-parquet --lib test_page_pruning_predicate_respects_enable_page_index— 1/1 pass(Full
--librun has 16 unrelated failures from a missingparquet-testingsubmodule fixture in my worktree — same failures on the base commit.)Are there any user-facing changes?
No. All moved items are
pub(crate). The publicPagePruningAccessPlanFilter::prune_plan_with_page_indexAPI restored in apache#22191 is untouched.