Skip to content

map Iceberg Binary to Arrow Binary instead of LargeBinary #2698

Description

@parthchandra

Follow up from #2668

Summary

iceberg-rust currently maps PrimitiveType::Binary to DataType::LargeBinary in the Arrow type conversion (type_to_arrow_type). This is inconsistent with:

  1. Java's implementationArrowSchemaUtil.java#L134 maps Iceberg Binary to Arrow Binary.
  2. parquet-rs behavior — parquet-rs reads binary columns from Parquet files as DataType::Binary, not LargeBinary.

This mismatch causes unnecessary casts when reading Binary columns and creates inconsistency with the Java ecosystem.

Scope

  • Change PrimitiveType::Binary mapping from DataType::LargeBinary to DataType::Binary in crates/iceberg/src/arrow/schema.rs
  • Update create_primitive_array_repeated in crates/iceberg/src/arrow/value.rs — the existing DataType::Binary arm already handles this, but the LargeBinary arm added in feat: add support for _partition metadata column #2668 can be removed or kept as a fallback
  • Audit all places that construct or expect LargeBinary arrays for Iceberg Binary columns
  • Update tests that assert LargeBinary output

Context

Raised in review of #2668 by @advancedxy. The LargeBinary arm in create_primitive_array_repeated was added to handle the current mapping for _partition columns partitioned by Binary. Once this mapping is corrected, that arm becomes unnecessary (but harmless to keep as a safety net).

This is a broader change that affects all Binary column reads, not just _partition.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions