Incremental IPC Stream Writer

[rustyconover]

Hi Everyone,

Would you be interested in a PR for an incremental IPC stream writer?

Thanks,

Rusty

[kylebarron]

Could you lay out a proposed API first?

[rustyconover]

Hi Kyle,

It's mostly just having the ability to write a RecordBatch at a time. This means skipping writing the EOS (end of stream) IPC message. It would also need the have the ability to write dictionary IPC messages too.

I can get something more formal together. I'm trying to evaluate if I can get https://github.com/query-farm/vgi-rpc-python using arro3 in addition to pyarrow as an alternative.

Rusty

[kylebarron]

We could work to emulate the pyarrow API here. Pyarrow seems to have multiple APIs for reading and writing IPC, and I think we'd only want to choose one of them.

I'm open to something like new_stream which can be used as a context manager for streaming output.