Skip to content

fix(parquet): dense-fetch fallback when a projected column lacks page locations - #87

Open
fsdvh wants to merge 1 commit into
brent/arrow-57from
fix/sparse-fetch-missing-page-locations
Open

fix(parquet): dense-fetch fallback when a projected column lacks page locations#87
fsdvh wants to merge 1 commit into
brent/arrow-57from
fix/sparse-fetch-missing-page-locations

Conversation

@fsdvh

@fsdvh fsdvh commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Problem

The async row-selection reader defaults to the Mask selection strategy. Whenever an offset index is present, it fetches only the data pages a RowSelection touches — a sparse column chunk (InMemoryRowGroup::fetch_ranges).

That sparse path was taken for every projected column based purely on the offset-index array being present, without checking that each column actually carries page locations.

A column whose OffsetIndexMetaData has empty page_locations — e.g. metadata synthesized from a manifest that lacks a page index for that column — then produces no scan ranges via scan_ranges. The column is fetched empty and the subsequent read fails with:

Invalid offset in sparse column chunk data: <n>, no matching page found.
If you are using a `SelectionStrategyPolicy::Mask`, ensure that the OffsetIndex
is provided when creating the InMemoryRowGroup.

The existing should_force_selectors safety net does not cover this case — selection_skips_any_page returns false for a column with empty page locations. And forcing Selectors would not help anyway, because the sparse fetch in fetch_ranges is chosen independently of the resolved selection strategy (get_offset_index in data.rs derives it straight from the metadata).

Fix

Guard the sparse path in fetch_ranges so it is only taken when every projected, not-yet-fetched column has non-empty page locations. Otherwise fall back to fetching whole column chunks (dense), which is always correct. A column that lacks page locations but is not projected does not disable sparse fetches for the remaining columns.

This restores the invariant that a sparse (Mask-strategy) fetch relies on: if the offset index drives the fetch, every fetched column has real page locations.

Tests

Adds unit tests on fetch_ranges:

  • sparse fetch when every projected column has page locations,
  • dense fallback when a projected column lacks page locations (verifies it reads full column byte ranges),
  • sparse fetch preserved when the column lacking page locations is not projected.

The dense-fallback test fails without the guard (the sparse path is taken and the read would later fail with "no matching page found").

Context

Surfaced downstream in Coralogix's DataPrime query engine, which synthesizes ParquetMetaData from its own manifest format and can leave a column's offset index empty when the manifest carries no page index for it. Under arrow v57's Mask default, any row-selection scan (e.g. full-text index selections) projecting such a column failed.

…page locations

The async row-selection reader defaults to the `Mask` selection strategy,
which whenever an offset index is present fetches only the data pages a
`RowSelection` touches (a "sparse" column chunk). `InMemoryRowGroup::fetch_ranges`
took that sparse path for every projected column purely based on the offset
index array being present, without checking that each column actually carries
page locations.

A column whose `OffsetIndexMetaData` has empty `page_locations` (for example
metadata synthesized from a manifest that lacks a page index for that column)
then produces no scan ranges: the column is fetched empty and the subsequent
read fails with `Invalid offset in sparse column chunk data: <n>, no matching
page found`. The existing `should_force_selectors` safety net does not cover
this case (it returns false for empty page locations), and forcing `Selectors`
would not help since the sparse fetch is chosen independently of the strategy.

Guard the sparse path so it is only taken when every projected, not-yet-fetched
column has non-empty page locations; otherwise fall back to fetching whole
column chunks (dense), which is always correct. Columns lacking page locations
that are not projected do not disable sparse fetches for the rest.

Adds unit tests covering the sparse path, the dense fallback, and the
unprojected-column case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant