Skip to content

refactor: unify ColumnSource::Add and AddStructConstant using Literal #2699

Description

@parthchandra

follow up from #2668

The ColumnSource enum in RecordBatchTransformer currently has two separate variants for constant columns:

Add {
    target_type: DataType,
    value: Option<PrimitiveLiteral>,
},
AddStructConstant {
    fields: Fields,
    child_values: Vec<Option<PrimitiveLiteral>>,
},

AddStructConstant was added in #2668 for the _partition metadata column because Add only supports primitive values. However, iceberg-rust already has Literal::Struct in crates/iceberg/src/spec/values/literal.rs which can represent struct values natively.

Proposed refactoring

  1. Change ColumnSource::Add to use Option instead of Option:
    Add {
    target_type: DataType,
    value: Option,
    },
  2. Remove ColumnSource::AddStructConstant — struct constants become a Literal::Struct inside Add.
  3. Update create_column (and create_struct_column) to handle Literal::Struct by recursively constructing child arrays, similar to how AddStructConstant works today.
  4. This also opens the door for constant_fields: HashMap<i32, Datum> to eventually support struct-valued constants (e.g., if Datum gains a struct variant), which would allow the _partition column to use the same constant_fields path as _file and identity-partitioned columns.

Benefits

  • Eliminates a special-case enum variant
  • Aligns the constant-column infrastructure with iceberg-rust's existing Literal type system
  • Makes future metadata columns or complex-typed constants straightforward to add

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions