[FEA] Preserve Polars Array dtype in cudf-polars pass-through plans

[0guban0v]

Is your feature request related to a problem? Please describe.

cudf_polars rejects pl.Array during schema conversion. This causes whole query to fall back to CPU even when no Array operation is requested and column only passes through otherwise supported GPU operations. This prevents GPU filtering or projection of scalar metadata while retaining fixed-size embedding columns.

Describe the solution you'd like

Support lossless pass-through for one-dimensional, all-valid pl.Array columns with fixed-width scalar dtypes already supported by libcudf.

Initial scope:

• in-memory DataFrame scan
• projection
• filtering on another scalar column
• slice
• exact values, row order, dtype, and width
• CPU fallback for unsupported Array cases

No Array arithmetic or reduction is needed.

Before implementing this, I'd like guidance from cudf_polars maintainers on representation boundary:
Should first implementation adapt Arrow fixed-size-list to regular libcudf LIST at cudf_polars ingress, retain original pl.Array shape in existing DataType, and reconstruct fixed-size-list output? Or should fixed-size-list support first be added to libcudf Arrow interoperability?

Describe alternatives you've considered

Users can manually convert Array to List before GPU execution and cast result back afterward. This loses fixed-width schema contract during execution and requires application-level conversion.

Current CPU fallback preserves correctness but prevents otherwise supported scalar filtering and projection from using GPU.

Adding native fixed-size-list support to libcudf Arrow interoperability would provide broader support, but may be larger than pass-through-only cudf_polars adapter.

Additional context

import polars as pl
from polars.testing import assert_frame_equal

df = pl.DataFrame(
    {
        "keep": [True, False, True],
        "embedding": pl.Series(
            [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]],
            dtype=pl.Array(pl.Float32, 2),
        ),
    }
)

query = df.lazy().filter("keep").select("embedding")

expected = query.collect()
result = query.collect(
    engine=pl.GPUEngine(
        executor="in-memory",
        raise_on_fail=True,
    )
)

assert_frame_equal(result, expected)
assert result.schema == expected.schema

Nullable or nested arrays, Array expressions such as arr.dot, file scans, and distributed execution are separate follow-up work.

[wence-]

I think the most appealing option would be to support fixed width lists in libcudf natively. However, one would need to balance that against the potential for blowing up compile/binary sizes for, potentially, not a large performance gain.

So I think a reasonable first step would be to convert at the boundaries.

I think the arrow interop would be a good place, since that's reasonably self-contained. To do Arrow Array -> libcudf List we don't need extra metadata. But to go in the other direction we do. That might necessitate an API change (or extension).

Question: what is the set of operations in Polars that are supported on pl.Array but not pl.List?

[0guban0v]

Thanks, converting at the boundaries sounds like reasonable first step.

On current Polars main, Array and List have almost the same computational namespace. The only Array method without corresponding List method by name is arr.to_list(), which is just conversion itself.

pola-rs/polars#28504 proposes arr.dot; if merged, that would be Array-specific operation.

For this request, important difference is type contract rather than available operations. Array(Float32, 768) records and guarantees embedding dimension in the schema, while List(Float32) does not. Permanently returning it as List loses that contract and requires users to cast it back manually before using Array-specific consumers.

For initial pass-through scope, representing physical GPU column as libcudf LIST seems sufficient. cudf_polars would retain original Polars inner dtype and shape as metadata, then reconstruct fixed-size-list/pl.Array representation at the output boundary. Array expressions would remain unsupported and fall back to CPU.

I understand that libcudf-to-Arrow direction needs additional metadata or small API extension to reconstruct the fixed-size-list schema. If this scope sounds appropriate, I can investigate that boundary further.

Today:
Array present → entire plan falls back to CPU

After this feature request:
GPU filtering/projection → Array preserved → CPU arr.dot

Possible future:
GPU filtering/projection → GPU arr.dot

Array pass-through is prerequisite for clean GPU embedding pipeline, but it is not GPU acceleration for arr.dot itself. Hopefully, it makes sense.