Extract object store related code

[brancz]

We treat object_store as a public API surface of parquet and avro (even if the traits used didn't change eg. from 0.13 to 0.14, presumably because things like how to contruct them has changed).

We should extract all code that uses object_store from the crates in this repository since it prevents us from upgrading object_store, especially when none of those pieces of functionality are being used.

Related to #9878 (comment)

@alamb

[alamb]

I agree - see this past discussion on something similar

• DISCUSS: Should we integrate Apache OpenDAL support in Parquet? #5427

Maybe we can just provide an example of how to use object store (rather than a native integration) -- the core parquet logic is now entirely disconnected from object_store

This would be a breaking change, but I do think it is probably a good time to break the explicit connection

[brancz]

So how do we go about this? Move all code that is currently behind the object_store feature for the avro and parquet crates into two separate repos? Move the code to datafusion (not sure how that solves it, only moves the "public api" to there)?

[brancz]

Oh wait if I'm seeing this right, async reader and writer are already pretty much redundant, so we can just get rid of a lot of that code and just show how to pass object store in via an example?

[alamb]

Oh wait if I'm seeing this right, async reader and writer are already pretty much redundant, so we can just get rid of a lot of that code and just show how to pass object store in via an example?

Yeah that is what I was thinking -- I am not sure about avro, but the object store parquet implementation could be done as an example (make as a dev dependency)

[brancz]

I'll have a look tomorrow, I'd love to get rid of this for the next breaking release.

[brancz]

If #10354 looks good, I can do the same thing for avro (it's even a little bit less code as far as I can tell) either in the same PR or in a follow up PR.