[FEA] Support the Parquet FILE logical type for external/blob payloads

[VibhuJawa]

DRAFT for upstream rapidsai/cudf. Roadmap/design ask — nothing to reproduce, because no Python-side implementation exists yet (verified: neither pyarrow 23.0.1 nor 25.0.0 has a FILE logical type). Filed to find out whether this is on the roadmap before we invest further in a bespoke locator format. Before filing: swap the VibhuJawa/cudf#9 cross-reference for its upstream number once that issue is filed, and delete this banner.

Is your feature request related to a problem? Please describe.

apache/parquet-format#585 added a FILE logical type to the Parquet spec — merged 2026-07-26 as 19: FileType FILE in parquet.thrift, with the field semantics in LogicalTypes.md.

FILE annotates a group representing a reference to a range of bytes, which may be stored inline in the value, elsewhere within the same Parquet file, or in an external file. Every field is optional, in both the schema and the data; a group need only define the fields it uses:

field	type	meaning
uri	STRING	RFC 3986 URI-reference. If not set, the value is a self-reference to the current Parquet file.
offset	INT64	Start of the byte range. Required for a self-reference.
size	INT64	Byte length. Required whenever offset is set.
content_type	STRING	RFC 2046 media type; defaults to application/octet-stream.
checksum	STRING	<algorithm>:<digest>; recognized algorithms are ETAG, MD5, CRC32, CRC32C, SHA-256.
inline	BYTE_ARRAY	The bytes themselves, stored in the value. When set, any locator fields are provenance only.

Additional engine- or table-format-specific metadata must be stored adjacent to the group, not inside it.

It is the standards-track way to keep large unstructured payloads — images, audio, video, PDFs — addressable from a tabular column without inlining them. cuDF has no support for it and, as far as I can find, no tracking issue.

Ecosystem status. The spec is merged; the two PoC implementations referenced by the PR — apache/parquet-java#3608 and apache/arrow-rs#10109 — are both still open. pyarrow has nothing (checked 23.0.1 and 25.0.0), so there is currently no Python reference data to test against. cuDF is not behind here; there is a window to be early.

Why this matters for GPU data pipelines

Multimodal curation is the workload. A canonical image table looks like:

url          string
image        <payload>          <- ~140 KB median
mime_type    string
width,height int32
sha256       string
embedding    list<float32>[1024]

Today we express the payload reference by hand: a JSON locator string per row carrying {path, member, byte_offset, byte_size}, parsed on the host, with byte-range gathers issued through fsspec. That works, but it is bespoke (no other engine can read our table without our parser), unvalidated (a stale path surfaces as a runtime error, not a null), and opaque to cuDF — payload resolution happens entirely off-GPU. FILE is that locator, standardized and typed.

Note that FILE also touches the binary gap: the inline field is a BYTE_ARRAY, and cuDF has no binary column type at all (#9 — Arrow binary/large_binary unsupported, Parquet binary read as string). So inline FILE values need that work too, while the locator-only variants do not — which makes the reference-based path the cheaper starting point.

Describe the solution you'd like

Staged, in value order:

1. Read/round-trip fidelity. Read a FILE column as a struct column and write it back without loss. Even with no payload resolution, this lets cuDF sit in a pipeline where another stage resolves payloads — and avoids the silent type-rewriting we already hit with binary columns.
2. Payload resolution. An opt-in read mode that follows uri + offset + size and materializes bytes, batched and coalesced rather than one request per row. Self-references (payload inside the same Parquet file) are the cheapest case and a sensible first target.
3. Integration with hybrid scan / sparse-page I/O. Resolving a FILE column for a filtered subset of rows is precisely the sparse-gather pattern Enable page-level payload I/O in hybrid scan rapidsai/cudf#23362 and Add sparse page I/O to hybrid scan rapidsai/cudf#23375 are building: plan byte ranges, coalesce them, issue them concurrently. FILE gives that machinery a typed, spec-blessed column to operate on.
4. Writer support, including checksum computation.

Describe alternatives you've considered

• Inline large_binary — the obvious approach, but read amplification is severe when you only want metadata, and compaction rewrites terabytes of payload. Also blocked in cuDF today by [FEA] No binary column type: Arrow binary/large_binary unsupported, Parquet binary silently becomes string #9.
• A plain string path column — what we do now. No validation, no engine interop, and the reader cannot plan I/O.
• Lance blob encoding — works, and we use it. Our own measurements, offered as design input rather than as a claim about cuDF: blob encoding beats inline binary on local NVMe (~4× on read_blobs, 16.4K vs 4.0K img/s) but loses over S3 (~500 img/s vs ~2,466 img/s for plain binary + row-id), because per-blob fetch RTT eats the API-level saving. The lesson we would carry into any cuDF FILE implementation: the win is in request coalescing and planning, not in the encoding itself. An implementation that issues one GET per row will be slower than inlining, regardless of how good the type is.

Additional context

Priority is below sparse-page I/O (rapidsai#23362 / rapidsai#23375) — that work unblocks us now; this is the schema story for the next 6–12 months. They converge at stage 3 above.

Related logical-type work for precedent: rapidsai#22312 (ingest of Parquet VARIANT columns), rapidsai#23251 (JSON reader output → VARIANT), rapidsai#23182 (VARIANT logical-type inspection). Related from us: #9 (no binary type).

[VibhuJawa]

Filed upstream as rapidsai#23494