Is your feature request related to a problem or challenge?
We plan to remove explicit integration with object_store from the parquet crate, see
The core rationale is to
- Ease downstream dependency management
- The Parquet decoder is fairly well segregated from I/O now (e.g. the PushDecoder) so it is less valuable to have a pre-existing I/O integration
- It removes the appearance of favoritism of one I/O library over another (e.g. opendal vs object_store)
- The amount of code is relatively small (10s of lines), so it is easy to incorporate into downstream crates and customize as needed
However it might be nicer (we think) if there was an easier way to get a pre-tested integration in a crate
Describe the solution you'd like
The idea would be to add ParquetObjectReader and ParquetObjectWriter (that implement the object_store APIs for parquet reading / writing respectively) into a new crate (e.g. parquet-object-store)
Describe alternatives you've considered
Some options for crate names
Additional context
No response
Is your feature request related to a problem or challenge?
We plan to remove explicit integration with
object_storefrom theparquetcrate, seeobject_storeintegration #10354The core rationale is to
However it might be nicer (we think) if there was an easier way to get a pre-tested integration in a crate
Describe the solution you'd like
The idea would be to add
ParquetObjectReaderandParquetObjectWriter(that implement the object_store APIs for parquet reading / writing respectively) into a new crate (e.g. parquet-object-store)Describe alternatives you've considered
Some options for crate names
parquet-object-store(to match https://crates.io/crates/parquet-opendal maintained by the open DAL project -- thanks @Xuanwo for the link)Additional context
No response