Add GPU Lance IO with bulk sparse reads

[VibhuJawa]

Summary

• Adds experimental cuDF C++ and Python Lance v2.2 read/write APIs.
• Keeps the data path on the GPU: KvikIO device reads, nvCOMP ZSTD batches, GPU sparse copies, and direct cuDF column output.
• Supports non-null fixed-width columns plus canonical FullZip large_binary image data represented as LIST<UINT8>.
• Adds read_lance_bulk for one sparse request spanning many Lance files while preserving source order and requested row order.
• Adds a lookup-oriented image writer profile using FullZip with compression=NONE, avoiding redundant compression around JPEG/PNG payloads.

Bulk Image Path

The multi-file reader now:

• parses file metadata concurrently across active sources
• fetches FullZip repetition indexes concurrently across touched pages
• buckets requested rows by source and page
• sorts and coalesces ZSTD payload ranges by physical file offset
• physically orders uncompressed requests within each file
• uses one contiguous GPU staging allocation instead of one allocation per range
• uses one stream fence for the entire bulk request and submits each file's ranges as a batch
• batches all selected ZSTD image values into nvCOMP
• reads uncompressed image payloads directly into their final cuDF child buffer, with no staging or scatter copy
• defaults the sparse coalescing gap to the measured 64 KiB

The API reports requested output bytes, compressed bytes touched, decompressed bytes, physical bytes read, ranges before/after coalescing, touched pages, files, and columns.

MINT1T Image Benchmark

Input: 32 canonical MINT1T Lance files, 227,344 rows, 28.8 GB on disk. Each case performs random sparse lookups of the actual image bytes. Results are median-of-three warm-process timings.

Files	Rows/file	Python Lance	cuDF bulk C++	Speedup
16	32	170.4 ms	21.8 ms	7.83x
16	64	284.7 ms	34.4 ms	8.29x
16	128	730.2 ms	60.9 ms	11.98x
32	32	210.6 ms	33.6 ms	6.27x
32	64	390.6 ms	51.4 ms	7.60x
32	128	1063.7 ms	94.5 ms	11.25x

The lookup-oriented files written by this PR are faster again in absolute time because they omit the redundant outer ZSTD layer:

Files	Rows/file	Canonical ZSTD	Lookup profile
16	32	21.8 ms	11.8 ms
16	64	34.4 ms	21.3 ms
16	128	60.9 ms	38.7 ms
32	32	33.6 ms	20.0 ms
32	64	51.4 ms	36.0 ms
32	128	94.5 ms	68.1 ms

Current Scope

• Lance file version 2.2.
• Fixed-width MiniBlock columns support NONE and ZSTD.
• Image columns use canonical FullZip large_binary; reading supports NONE and ZSTD, while writing currently requires NONE.
• Bulk image projection currently supports one image column and cannot mix image and fixed-width columns in the same call.
• Null masks and other nested/string types are not yet supported.

Validation

• Rebuilt the full cudf target successfully after the retained scheduler change.
• Compiled the focused LANCE_TEST source, including a two-source uncompressed image regression test with reordered and empty values.
• Passed exact-byte C++ image writer/readback smoke tests.
• Confirmed cuDF-written files load as non-null large_binary in Python Lance and match canonical image bytes exactly.
• Passed canonical ZSTD and uncompressed bulk image read smoke tests.
• Passed git diff --check.

The complete LANCE_TEST executable was not linked in this environment because enabling the test suite invalidated and triggered a near-full libcudf rebuild; the focused test object compiled successfully.

Global Lookup Order

The bulk C++ API also accepts row_locations(std::vector<lance_row_location>), a flat list of (source_index, row_index) positions in the required caller order. It buckets these internally for sparse multi-file scheduling, then writes each image directly to its global output row. This removes the caller-side read-in-source-order plus GPU gather/reorder step used by a URL left join.

On the same 32-file MINT1T image input, direct global-order output took 22.4–94.2 ms for 512–4,096 images. The equivalent source-order read followed by cudf::gather took 27.5–158.7 ms: direct placement is 1.23x–1.68x faster for this portion of the pipeline. The largest case was 94.2 ms versus 158.7 ms (1.68x).

The image writer profile is conventional Lance FullZip large_binary with outer compression=NONE; it keeps already-compressed JPEG/PNG image bytes as direct-copy values. It does not change the image bytes, add an index, or define a second file format.

Python API And Environment

The public cuDF API now exposes the native global-order reader as cudf.read_lance_bulk(paths, row_locations=[(source_index, row_index), ...], columns=["image"]). It requires exactly one of rows_per_source or row_locations; the latter is the direct handoff from a URL join.

The binding declares and invokes the C++ lance_row_location builder API without a Python data-path shim. It also returns list child metadata for LIST<UINT8> image output, and to_lance now admits homogeneous non-null list<uint8> image columns. The benchmark environment has been installed with the local libcudf headers/library and rebuilt pylibcudf Lance module.

Validation: full python/cudf/cudf/tests/input_output/test_lance.py passes (20 tests); the focused C++ Lance test object compiles; real MINT1T two-file image order smoke passes.