Is your feature request related to a problem or challenge?
StreamEncoder::encode currently returns Vec<Buffer>. This keeps the API simple and sans-IO friendly, but it requires the encoder to collect all output buffers for a batch before the caller can forward them to its destination.
Describe the solution you'd like
An alternative API could push encoded pieces directly into a sink-like abstraction, for example:
pub fn encode_to<S: IpcMessageSink>(
&mut self,
batch: &RecordBatch,
sink: &mut S,
) -> Result<(), ArrowError>Describe alternatives you've considered
or make StreamEncoder generic over a sink.
This could reduce intermediate buffering and copying for callers that can consume IPC pieces immediately, such as async writers. The tradeoff is that the encoder starts to look more like a writer, and exposing a sink abstraction may make the public API more complex.
Things to evaluate:
- Whether this should be additive, such as
encode_to, instead of replacing encode
- Whether the sink abstraction should be public
- Whether this preserves the intended sans-IO API boundary
- How it affects copy behavior and memory usage
- Benchmark impact compared with
encode -> Vec<Buffer>
Additional context
Follow-up from #10277 and review discussion:
#10277 (comment)
Is your feature request related to a problem or challenge?
StreamEncoder::encodecurrently returnsVec<Buffer>. This keeps the API simple and sans-IO friendly, but it requires the encoder to collect all output buffers for a batch before the caller can forward them to its destination.Describe the solution you'd like
An alternative API could push encoded pieces directly into a sink-like abstraction, for example:
Describe alternatives you've considered
or make
StreamEncodergeneric over a sink.This could reduce intermediate buffering and copying for callers that can consume IPC pieces immediately, such as async writers. The tradeoff is that the encoder starts to look more like a writer, and exposing a sink abstraction may make the public API more complex.
Things to evaluate:
encode_to, instead of replacingencodeencode -> Vec<Buffer>Additional context
Follow-up from #10277 and review discussion:
#10277 (comment)