[FEA] No binary column type: Arrow binary/large_binary unsupported, Parquet binary silently becomes string

[VibhuJawa]

DRAFT for upstream rapidsai/cudf. Verified on cuDF 25.10.00 (pyarrow 25.0.0) and 26.08.00a990 nightly (pyarrow 23.0.1), H100 80GB. Before filing: swap the VibhuJawa/cudf#6 cross-reference for its upstream number once that issue is filed, and delete this banner.

Is your feature request related to a problem? Please describe.

cuDF has no binary column type. Arrow binary / large_binary are neither representable as a cuDF dtype nor convertible from Arrow, and Parquet binary columns are read as strings.

Binary payload columns are now standard in multimodal / AI training datasets. Our interleaved schema is:

pa.field("binary_content", pa.large_binary(), nullable=True)   # encoded image bytes, ~140 KB/row
pa.field("text_content",   pa.string(),       nullable=True)
pa.field("embedding",      pa.list_(pa.float32()))

The payload column has no faithful representation in cuDF.

What already works, stated up front: reading Parquet binary through cuDF is byte-accurate. Verified by SHA-256 over random bytes, embedded NULs, and empty values; data survives a GPU gather, .str.byte_count() returns correct lengths, and a cuDF write-back preserves bytes exactly — on both versions, for both binary and large_binary. This is not a data-loss bug. It is a type-system and interop gap.

Verified behaviour

Probe output, cuDF 26.08.00a990:

--- is there a binary dtype in the cudf namespace? ---
  cudf.*binary*/byte*  : NONE

--- does cudf accept a binary dtype string? ---
  dtype='binary'       -> TypeError: data type 'binary' not understood
  dtype='large_binary' -> TypeError: data type 'large_binary' not understood
  dtype='bytes'        -> TypeError: Unsupported type |S0
  dtype='|S10'         -> TypeError: Unsupported type |S10

--- constructing a Series from python bytes ---
  cudf.Series([b"ab", b"cd"])
  -> ValueError: CUDF failure at: cpp/src/interop/arrow_utilities.cpp:61:
     Unsupported type_id conversion to cudf

--- reader options exposing a binary mode ---
  cudf.read_parquet kwargs                  : no binary-related kwarg
  plc.io.parquet.ParquetReaderOptions(Builder): no binary-related option

--- what does cudf call a parquet binary column? ---
  parquet says : large_binary
  cudf says    : str   (arrow: large_string)

Parquet binary and large_binary behave identically. What the column is called on the way out differs by version, but neither is binary:

	cuDF dtype after read	Arrow type after read	Parquet type after cuDF write-back
25.10.00	object	string	string
26.08.00a990	str	large_string	string

So there are four distinct gaps:

1. No dtype. Nothing in the cuDF namespace represents binary; no dtype string is accepted.
2. No Arrow interop. cudf.DataFrame.from_arrow / cudf.Series on binary or large_binary raises Unsupported type_id conversion to cudf (cpp/src/interop/arrow_utilities.cpp:72 in 25.10, :61 in 26.08). The Parquet path is the only way in.
3. Type identity is silently lost. large_binary in → string out. A pipeline that reads and rewrites a multimodal table silently rewrites its schema, so downstream consumers see string where the producer wrote large_binary. Round-tripping is not schema-preserving.
4. libcudf's binary mode is not reachable from Python. libcudf has reader_column_schema::set_convert_binary_to_strings(false) (cpp/include/cudf/io/types.hpp, default true), which returns binary Parquet columns as list<uint8>. Nothing equivalent is exposed on cudf.read_parquet or on ParquetReaderOptions / ParquetReaderOptionsBuilder in either version — there is no reader_column_schema in pylibcudf.io at all. Separately, a cuDF list<uint8> column round-trips Parquet as list<element: uint8>, i.e. a LIST rather than binary, so it is not a usable substitute either.

Steps/code to reproduce

import cudf, pyarrow as pa, pyarrow.parquet as pq, tempfile, os

# 1. no dtype
for spec in ["binary", "large_binary", "bytes", "|S10"]:
    try:
        cudf.Series([b"ab"], dtype=spec)
    except Exception as e:
        print(spec, "->", type(e).__name__, e)

# 2. no arrow interop
try:
    cudf.Series([b"ab", b"cd"])
except Exception as e:
    print("from bytes ->", type(e).__name__, e)

# 3. type identity lost through parquet (but bytes are fine)
payload = [os.urandom(4096), b"\xff\xd8\x00\xff", b""]
with tempfile.TemporaryDirectory() as d:
    src, dst = os.path.join(d, "s.parquet"), os.path.join(d, "d.parquet")
    pq.write_table(pa.table({"x": pa.array(payload, type=pa.large_binary())}), src)

    df = cudf.read_parquet(src)
    print("cudf dtype:", df.dtypes.iloc[0])                       # str (26.08) / object (25.10)
    arr = df["x"].to_arrow()
    print("arrow type:", arr.type)                                # large_string (26.08) / string (25.10)
    got = arr.cast(pa.large_binary()).to_pylist()
    print("bytes intact:", got == payload)                        # True

    df.to_parquet(dst, index=False)
    print("in :", pq.read_schema(src).field("x").type)            # large_binary
    print("out:", pq.read_schema(dst).field("x").type)            # string  <-- identity lost

Describe the solution you'd like

In rough priority order:

1. Preserve binary type identity through a Parquet round-trip. Even without a new dtype, a column read from a binary / large_binary Parquet field should be written back as binary rather than becoming string. Smallest fix, largest correctness payoff.
2. Arrow interop for binary / large_binary in from_arrow / to_arrow. This is what blocks the zero-copy handoff between cuDF and the Arrow-based parts of our pipeline.
3. A first-class binary dtype, so payload columns are not conflated with text. Practical consequences of the conflation today: .to_pylist() raises UnicodeDecodeError on non-UTF-8 payloads (callers must know to .cast(pa.binary()) first), and every string kernel is nominally applicable to data that is not text.

Exposing the existing set_convert_binary_to_strings knob through pylibcudf would be a cheaper partial step than (3), though list<uint8> does not round-trip Parquet as binary, so on its own it does not fix (1).

Describe alternatives you've considered

• Treat it as a string column — what we do today. Works for byte fidelity, but loses the type on write and makes the API misleading.
• list<uint8> — faithful in memory, but Parquet round-trips it as a LIST, not binary, so it does not interoperate with non-cuDF readers.
• Keeping payloads out of cuDF entirely — also what we do today for the multimodal path. Means cuDF cannot participate in payload-bearing tables at all.

Additional context

• [FEA] Support >2^31 elements in list column children (large-offset list columns) #6 — the list child-element cap. Same multimodal schema, separate root cause; tracked separately.
• ParquetDatasetWriter max_file_size overestimates list columns and creates too many files rapidsai/cudf#23378 — ParquetDatasetWriter max_file_size overestimates list columns (open); relevant to the same multimodal write path.
• The Parquet FILE logical type (merged into the spec 2026-07-26) defines an inline field of Parquet type BYTE_ARRAY, so binary support is a prerequisite for reading that variant of FILE columns.

[VibhuJawa]

Filed upstream as rapidsai#23492