Skip to content

Populate FileWriteMetadata.format_metadata with Thrift-serialized Parquet footer for FFI consumers #24010

Description

@qzyu999

Background

PR #23656 added FileWriteMetadata with a format_metadata: Option<Bytes> field designed to carry format-specific metadata (e.g., Thrift-serialized Parquet FileMetaData). The current implementation sets this to None because the typed ParquetMetaData is already accessible via ParquetSink::written() for Rust consumers.

Proposal

When a concrete FFI use case emerges (e.g., Python via PyO3, or C via datafusion-ffi), serialize the Parquet FileMetaData using Thrift Compact Protocol and populate format_metadata so that non-Rust consumers can access per-file column statistics without re-reading file footers.

Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions