Align interleaved source locators with Parquet FILE

[VibhuJawa]

Summary

Align interleaved source locators with the Parquet FILE logical-type shape, simplify materialization, and improve actual Parquet reader/writer throughput.

Previous field	New location
source_ref.path	source_ref.uri
source_ref.byte_offset	source_ref.offset
source_ref.byte_size	source_ref.size
source_ref.member	adjacent source_member column
source_ref.frame_index	adjacent source_frame_index column

source_ref has all six optional FILE children from the specification: uri, offset, size, content_type, checksum, and inline.

What changed

• WebDataset emits a typed FILE-compatible struct, preserving canonical Arrow child order.
• Existing JSON-string locators are migrated once at the shared schema-alignment boundary.
• Legacy Nemotron {page, bbox} provenance migrates to typed adjacent page_number and source_bbox columns instead of being discarded.
• uri + size now correctly resolves [0, size) when offset is absent.
• Whole-object and ranged references share one deduplicated fs.cat_ranges() path, with one call per filesystem backend.
• Parquet output is opened through the configured fsspec filesystem, preserving remote protocols and storage_options.
• Arrow inputs bypass redundant Arrow → pandas → Arrow conversion when materialization/error filtering is unnecessary.
• Dictionary encoding defaults to low-cardinality interleaved fields; callers can still override use_dictionary.

Archive-member and TIFF-frame metadata remain adjacent because FILE is a closed group. source_member is only needed for tar extraction when a usable FILE byte range is unavailable; ranged and whole-object FILE reads do not depend on it.

Compatibility and scope

The change keeps binary_content unchanged. The inline FILE child is present in the canonical schema, but moving existing payload storage or materialization to inline is outside this PR.

PyArrow 19 can write the exact physical group but cannot emit the new LogicalType.FILE footer annotation. The FILE specification merged on 2026-07-26, and the cited parquet-java and arrow-rs implementations are still open/unreleased. This PR therefore avoids a custom Thrift footer rewriter and can adopt native Arrow support when it lands.

• Parquet FILE specification
• parquet-java proof of concept
• arrow-rs proof of concept
• cuDF FILE tracking
• cuDF binary/inline prerequisite

Performance validation

Compared latest upstream main (5732455) with this branch on the same CPU host. These are real InterleavedParquetWriterStage and InterleavedParquetReaderStage runs over 200,000 typed Arrow rows; values are medians of seven warm runs.

Metric	Upstream main	This PR	Change
Parquet write throughput	4.78M rows/s	6.24M rows/s	+30.7%
Parquet read throughput	3.43M rows/s	4.03M rows/s	+17.6%
Parquet file size	3,128,791 bytes	1,592,172 bytes	−49.1%

Remote HTTP materialization used a pinned, range-capable 466 KB object and verified every returned payload:

• 32 rows / 8 unique byte ranges: combined alternating-run median was 0.109 s here versus 0.115 s upstream (network-sensitive, no regression).
• 32 duplicate whole-object references: combined median was 0.174 s here versus 0.273 s upstream; individual runs were network-noisy, but the unified path showed no consistent slowdown.

Validation

• Ruff passes across nemo_curator and interleaved tests.
• 209 top-level interleaved CPU tests pass, covering locator and PDF-provenance migration, FILE semantics, size-only ranges, materialization, fsspec output, Parquet/WebDataset/Lance readers and writers, and round trips.
• Latest fork main, upstream main, and the PR base all resolve to 5732455; the branch is based directly on current main.

Diff

• 18 files changed
• 358 additions / 362 deletions
• net reduction of 4 lines

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

The PR aligns interleaved source locators with the Parquet FILE physical shape while retaining compatibility with existing serialized locators.

• Migrates legacy JSON source_ref values at the shared schema-alignment boundary.
• Moves archive-member, TIFF-frame, PDF-page, and bounding-box metadata into adjacent columns.
• Consolidates whole-object and range materialization through deduplicated filesystem range reads.
• Updates WebDataset and Parquet I/O to use the typed locator representation and Arrow-native write path.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported legacy-alignment, locator-caller, and PDF-provenance issues are addressed at the current head.

Important Files Changed

Filename	Overview
nemo_curator/stages/interleaved/utils/schema.py	Migrates legacy locator JSON into the typed FILE-compatible struct and preserves adjacent source and PDF provenance fields.
nemo_curator/tasks/interleaved.py	Defines the canonical FILE-compatible Arrow struct and updates locator construction and expansion APIs.
nemo_curator/stages/interleaved/utils/materialization.py	Uses typed locator fields to deduplicate whole-object and ranged reads per filesystem backend.
nemo_curator/stages/interleaved/pdf/nemotron_parse/utils.py	Preserves PDF page and bounding-box provenance in dedicated passthrough columns.
nemo_curator/stages/interleaved/io/writers/tabular.py	Writes Arrow tables through the configured filesystem with tuned Parquet dictionary defaults.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Legacy JSON source_ref] --> B[align_interleaved_table]
  B --> C[FILE-compatible source_ref struct]
  B --> D[source_member / source_frame_index]
  B --> E[page_number / source_bbox]
  C --> F[Materialization]
  D --> F
  C --> G[Parquet or Lance output]
  D --> G
  E --> G

Loading

_{Reviews (7): Last reviewed commit: "fix(interleaved): preserve PDF provenanc..." | Re-trigger Greptile}

[VibhuJawa]

/okay to test 7a5a8e9