Skip to content

fix(table): guard DV filtering against stale row counts - #1624

Open
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:fix/table-dv-stale-row-count
Open

fix(table): guard DV filtering against stale row counts#1624
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:fix/table-dv-stale-row-count

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Prevent deletion-vector filtering from panicking when a data file emits more rows than its manifest record_count.

Problem

The contiguous fast path builds a keep mask sized from DataFile.Count() and slices it using the rows emitted by each Arrow batch. If the manifest count is stale or otherwise smaller than the physical file, the batch end falls outside the mask and array.NewSlice panics.

The row-group-pruned path already handles this mismatch by preserving positions beyond the mask. The contiguous path should behave the same way.

Fix

Keep the existing zero-copy mask slice while a batch is within the declared row count. If a batch crosses that boundary, build a bounded mask for that batch and keep rows whose positions are beyond the declared count. Once a batch is entirely beyond the mask, pass it through without allocating an all-true filter. In-range deletion-vector positions are still applied normally.

Tests

Added coverage across three batches: one within the manifest count, one crossing it, and one entirely beyond it. The test verifies in-range deletes while preserving rows beyond the declared count.

go test ./table -count=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant