Optimize Parquet lazy repdef window decode - #784
Conversation
3e62034 to
7389117
Compare
| void PageReader::decodeRepDefs(int32_t numTopLevelRows) { | ||
| if (definitionLevels_.empty() && maxDefine_ > 0) { | ||
| preloadRepDefs(); | ||
| const auto startNs = getCurrentTimeNano(); |
There was a problem hiding this comment.
This changes the core rep/def preload/decode path, but I don't see any tests added in this PR. Can we add coverage for window mode vs the existing path on nested repeated/optional data? The cases I would expect at minimum are: window mode disabled, window mode enabled across multiple pages, and skip/seek after only part of the window has been decoded.
| return numRepDefsInPage; | ||
| } | ||
|
|
||
| auto bitReader = ::arrow::bit_util::BitReader( |
There was a problem hiding this comment.
This looks like a second implementation of Parquet def-level RLE decoding. Can we reuse the existing level decoder path, or extract a shared helper, instead of hand-parsing the RLE runs here?
This is a correctness-sensitive format detail and easy to drift from the normal path.
| } | ||
|
|
||
| void PageReader::compactConsumedLeafNulls() { | ||
| constexpr int32_t WordBits = 64; |
There was a problem hiding this comment.
This duplicates the leaf-null compaction logic, including the glibc/FSRM memcpy workaround from the existing path. Can we extract that existing block into a shared helper and call it from both
paths?
| void PageReader::preloadRepDefs() { | ||
| const auto startNs = getCurrentTimeNano(); | ||
| hasChunkRepDefs_ = true; | ||
| windowRepDefMode_ = false; |
There was a problem hiding this comment.
These window-mode fields are tightly coupled. Can we group them into a small private state struct, maybe with a reset() method? That would make the invariants easier to follow.
|
The optimization direction looks reasonable, but I think this PR needs more work before it is easy to review and maintain. A few concerns:
Overall, I think the PR should aim for: enough reuse with the existing rep/def decode machinery, tests for the new behavior, and a microbenchmark showing the performance win. |
Summary
This PR optimizes Parquet repetition/definition level decoding by adding a lazy windowed decode path.
The change is split into three logical commits:
Validation
Validated locally with:
Both default mode and window preload mode passed.