Skip to content

fix(deletes): Implement filtering position delete files by file path - #2936

Open
brgr-s wants to merge 2 commits into
apache:mainfrom
brgr-s:brgr-s/pos-delete-by-file-path
Open

fix(deletes): Implement filtering position delete files by file path#2936
brgr-s wants to merge 2 commits into
apache:mainfrom
brgr-s:brgr-s/pos-delete-by-file-path

Conversation

@brgr-s

@brgr-s brgr-s commented Jul 31, 2026

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

The PR introduces (uncomments) pos_deletes_by_path in PopulatedDeleteFileIndex and populates the HashMap. To determine if a position delete file belongs in the map, it uses a helper position_delete_target that investigates

  1. is referenced_data_file set? if yes, we know the delete file applies to that path
  2. is lower == upper for the reserved field RESERVED_FIELD_ID_DELETE_FILE_PATH? If yes, we know that all entries point to the same path

As a small ergonomic imrpovement it changes eq_deletes_by_partition and pod_deletes_by_partition to take a (i32, Struct) as key, where the i32 is the partition_spec_id of the delete file.

get_deletes_for_data_file can then use pos_deletes_by_path to lookup delete files that match that specific data_file.

PR also adds a debug trace that could be usefull to determine if a read on a table is slow or OOMs (see linked issue) due to a large number of created delete file references.

Are these changes tested?

I have added unit tests.

The change was also tested locally in conjuncture with PR #2620. The task was compacting a merge-on-read table with a huge number of delete files. The test scenario can be reproduced without PR #2620, see Issue #2935.

AI Disclosure

I used AI to investigate how Spark handles delete files, to review my changes and add more tests. I also used AI to write parts of the Issue and double-check the math.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scan planning matches every position delete to every data file in the partition, making planning memory O(data files × delete files)

1 participant