Skip to content

[parquet] Consider crate to provide object_store crate integration #10501

Description

@alamb

Is your feature request related to a problem or challenge?

We plan to remove explicit integration with object_store from the parquet crate, see

The core rationale is to

  1. Ease downstream dependency management
  2. The Parquet decoder is fairly well segregated from I/O now (e.g. the PushDecoder) so it is less valuable to have a pre-existing I/O integration
  3. It removes the appearance of favoritism of one I/O library over another (e.g. opendal vs object_store)
  4. The amount of code is relatively small (10s of lines), so it is easy to incorporate into downstream crates and customize as needed

However it might be nicer (we think) if there was an easier way to get a pre-tested integration in a crate

Describe the solution you'd like

The idea would be to add ParquetObjectReader and ParquetObjectWriter (that implement the object_store APIs for parquet reading / writing respectively) into a new crate (e.g. parquet-object-store)

Describe alternatives you've considered

Some options for crate names

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelog

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions