feat(arrow-ipc): add sans-IO stream encoder by Phoenix500526 · Pull Request #10277 · apache/arrow-rs

Phoenix500526 · 2026-07-03T08:20:08Z

Which issue does this PR close?

Closes arrow-ipc StreamWriter better ergonomics with async #7812.

Rationale for this change

StreamWriter currently requires a std::io::Write sink, which is awkward for async or chunk-oriented destinations such as object stores. This PR adds a sans-IO IPC stream encoder so callers can encode Arrow IPC stream data into ordered Buffer chunks and send those chunks through their own IO layer.

What changes are included in this PR?

This PR adds StreamEncoder, a stateful IPC stream encoder that:

emits ordered arrow_buffer::Buffer chunks
hides stream lifecycle details such as schema emission and EOS markers
owns stream state such as dictionary tracking and IPC write context
preserves the low-copy path for uncompressed record batch body buffers
supports try_new, try_new_with_options, encode, and finish

Are these changes tested?

Yes.
Added tests compare StreamEncoder output byte-for-byte with StreamWriter output for:
a normal record batch stream
an empty stream
a stream containing dictionary batches

Are there any user-facing changes?

Yes. This adds a new public arrow_ipc::writer::StreamEncoder API.
There are no breaking changes. Existing StreamWriter behavior is unchanged.

Rich-T-kid · 2026-07-05T14:10:24Z

I'll try and give this a review within a few days

Rich-T-kid

I took a brief glance, looks pretty good to me so far. I can take a more in depth look tomorrow. I left a suggestion 🚀

Rich-T-kid

thank you @Phoenix500526, I think there's value in having a sans-I/O stream encoder, but we need to be careful about the duplication we're adding to arrow-ipc. The file is already hard to understand without much context, and adding duplicate logic in two places makes it hard to maintain.
The tests showcase a good way this can be used, but I'd like the implementation to be leaner. I left a few suggestion let me know what you think

Rich-T-kid · 2026-07-17T13:11:32Z

@Phoenix500526 is this good for review?

Phoenix500526 · 2026-07-17T13:26:25Z

@Phoenix500526 is this good for review?

@Rich-T-kid Yes, this is ready for review. Thanks for checking.

Rich-T-kid

I think this looks pretty good. I left a couple of suggestions. Thank you for splitting up the commits into separate independent chunks!

Would be nice to validate that this PR didn't accidentally slow down the Stream writer somehow.

Rich-T-kid · 2026-07-17T17:38:33Z

It would be nice to double check that we didnt accidntly slow down the StreamWriter as well.

@alamb could you please run the arrow-ipc benchmarks? Thank you 👍

Rich-T-kid · 2026-07-18T15:53:14Z

looks to be mostly noise? its within +- 10. can we run it again to check if its reproducible

Rich-T-kid

LGTM @Phoenix500526

@Jefffrey I think this is ready for a review

Jefffrey · 2026-07-24T02:10:01Z

benchmarks seem to suggest a consistent regression 🤔

Jefffrey · 2026-07-24T04:47:55Z

run benchmark ipc_writer

adriangbot · 2026-07-24T04:53:33Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                       issue_7812                             main
-----                                                       ----------                             ----
arrow_ipc_stream_writer/FileWriter/write_10                 1.00     87.9±0.23µs        ? ?/sec    1.06     93.0±2.48µs        ? ?/sec
arrow_ipc_stream_writer/FileWriter/write_10/dict/delta      1.01    150.7±3.58µs        ? ?/sec    1.00    149.1±2.75µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10               1.00     86.4±0.60µs        ? ?/sec    1.02     87.9±0.28µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict          1.02     86.9±5.70µs        ? ?/sec    1.00     85.2±5.41µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict/delta    1.01    147.3±3.64µs        ? ?/sec    1.00    146.4±3.52µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/zstd          1.00      7.2±0.03ms        ? ?/sec    1.00      7.2±0.04ms        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	70.0s
Peak memory	11.9 MiB
Avg memory	9.8 MiB
CPU user	64.8s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	70.0s
Peak memory	12.4 MiB
Avg memory	9.9 MiB
CPU user	64.1s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Jefffrey · 2026-07-24T04:54:59Z

run benchmark ipc_writer

adriangbot · 2026-07-24T04:59:04Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                       issue_7812                             main
-----                                                       ----------                             ----
arrow_ipc_stream_writer/FileWriter/write_10                 1.00     87.8±1.07µs        ? ?/sec    1.03     90.6±0.31µs        ? ?/sec
arrow_ipc_stream_writer/FileWriter/write_10/dict/delta      1.02    148.5±3.23µs        ? ?/sec    1.00    145.3±2.98µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10               1.00     86.2±0.33µs        ? ?/sec    1.02     87.7±0.28µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict          1.03     86.4±5.62µs        ? ?/sec    1.00     83.7±5.55µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict/delta    1.00    144.9±3.52µs        ? ?/sec    1.00    144.4±3.80µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/zstd          1.00      7.2±0.03ms        ? ?/sec    1.00      7.2±0.05ms        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	70.0s
Peak memory	11.9 MiB
Avg memory	9.7 MiB
CPU user	63.9s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	70.0s
Peak memory	12.4 MiB
Avg memory	9.9 MiB
CPU user	64.1s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Jefffrey · 2026-07-24T05:00:31Z

numbers looking better now 👀

Phoenix500526 · 2026-07-24T05:46:18Z

numbers looking better now 👀
Hi, @Jefffrey thanks for the review. And it seems that my change has nothing to do with the failed CI — MIRI(3). Could you rerun it? Thanks 😊

Jefffrey

miri failure unrelated, see

#10415

Jefffrey · 2026-07-24T09:42:46Z

run benchmark ipc_writer

adriangbot · 2026-07-24T09:47:48Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                       issue_7812                             main
-----                                                       ----------                             ----
arrow_ipc_stream_writer/FileWriter/write_10                 1.00     86.4±0.24µs        ? ?/sec    1.05     90.5±0.25µs        ? ?/sec
arrow_ipc_stream_writer/FileWriter/write_10/dict/delta      1.00    149.4±3.20µs        ? ?/sec    1.00    149.7±2.53µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10               1.00     87.0±0.28µs        ? ?/sec    1.00     86.8±0.35µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict          1.02     86.1±5.67µs        ? ?/sec    1.00     84.3±5.42µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict/delta    1.00    145.0±3.59µs        ? ?/sec    1.01    146.6±3.57µs        ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/zstd          1.00      7.2±0.03ms        ? ?/sec    1.00      7.2±0.04ms        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	70.0s
Peak memory	11.9 MiB
Avg memory	9.8 MiB
CPU user	64.9s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	70.0s
Peak memory	12.4 MiB
Avg memory	9.9 MiB
CPU user	64.1s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Jefffrey · 2026-07-24T09:48:22Z

+        Ok(())
+    }
+
+    fn write_record_batch(


do we still need this custom impl, or can rely on the default?

I've tried, but that makes the performance regress again. I extracted a shared metadata_layout method to reduce the duplication in commit 1b502ea

alamb · 2026-07-25T22:14:42Z

MIRI was fixed in #10433, merging up to get fix

alamb

Thank you @Jefffrey and @Phoenix500526 and @Rich-T-kid

I think this is a very nicely written and commented PR. Since most of the PR is internal implementation / not exposed publically most of my comments could be done as a follow on PR

However, I do think it is worth considering a slightly different StreamEncoder API (comments below) that don't force buffering the output buffers into a Vec<Buffer>

alamb · 2026-07-25T23:43:44Z

+}
+
+#[inline]
+fn metadata_layout(metadata_len: usize, write_options: &IpcWriteOptions) -> MetadataLayout {


As a follow on, this might naturally be encapsualted as MetadataLayout::new type constructor

alamb · 2026-07-25T23:44:49Z

+trait IpcMessageSink {
+    fn write_slice(&mut self, bytes: &[u8]) -> Result<(), ArrowError>;
+
+    fn write_vec(&mut self, bytes: Vec<u8>) -> Result<(), ArrowError> {


+1 for taking owned Vec

alamb · 2026-07-25T23:50:02Z

+/// Destination for a complete framed IPC message.
+///
+/// This emits the stream/file framing around the serialized FlatBuffer
+/// [`crate::Message`] metadata plus its optional body buffers.


It might also be worth mentioning to point out that the default implementation will copy data

alamb · 2026-07-25T23:54:36Z

+        options: IpcWriteOptions,
+    ) -> Vec<u8> {
+        let mut encoder = StreamEncoder::try_new_with_options(schema, options).unwrap();
+        let mut bytes = Vec::new();


since Vec implements Write why not just use the Write impl directly?

alamb · 2026-07-26T01:15:45Z

+    /// # Errors
+    ///
+    /// Returns an error if encoding fails.
+    pub fn encode(&mut self, batch: &RecordBatch) -> Result<Vec<Buffer>, ArrowError> {


I think this API is quite good and supports a usecase like async writer quite well.

My only suggestion / feedback is that this API basically requires buffering an entire record batch in RAM and now requies copying into several intermediate Vecs (e.g. for the metadata header, etc)

Another API we might consider would be to make the SteamEncoder templated on IpcMessageSink so it can bass the buffers directly

Something like

struct StreamEncoder<IpcMessageSink>` { ... /// encode the batches, calling methods like `append_vec()`, `append_buffer`, append_slice, etc pub fn encode(&mut self, batch: &RecordBatch) -> Result<(), ArrowError> { }

However, this does still feel like there is "IO" in the encoder 🤔 -- on the other hand you could implement an IpcMessageSync that just buffers the data to get the same effect.

Thanks @alamb , I agree this is worth exploring. I’m hesitant to change this PR to make StreamEncoder generic over a sink because that would be a larger API design change and makes the sans-IO boundary less obvious. I’ll keep the current encode -> Vec<Buffer> API in this PR and open a follow-up issue to track a possible sink-based / encode_to API. I open an issue #10445 to track it.

alamb · 2026-07-26T01:17:44Z

+    }
+
+    /// Writes the IPC continuation marker and metadata length prefix.
+    fn write_continuation(


I found the mixing of methods you are supposed to override (like write_slice and write_vec) and methods that you aren't (like write_continuation) to be somewhat confusing.

It would be easier to read this code I think if we separated out the two types of methods

Perhaps something like this

/// The only things a sink actually needs to customize. trait IpcMessageSink { fn write_slice(&mut self, bytes: &[u8]) -> Result<(), ArrowError>; fn write_vec(&mut self, bytes: Vec<u8>) -> Result<(), ArrowError> { self.write_slice(&bytes) } fn write_encoded_buffer(&mut self, buffer: EncodedBuffer) -> Result<(), ArrowError> { self.write_slice(buffer.as_slice()) } } /// The shared framing code trait IpcMessageSinkExt: IpcMessageSink { fn write_padding(&mut self, len: usize) -> Result<(), ArrowError> { self.write_slice(&PADDING[..len]) } fn write_continuation(&mut self, write_options: &IpcWriteOptions, metadata_len: i32) -> Result<(), ArrowError> { // ... unchanged body ... } fn write_encoded_data(&mut self, encoded: EncodedData, write_options: &IpcWriteOptions) -> Result<(usize, usize), ArrowError> { // ... unchanged body, calling self.write_vec / self.write_padding ... } fn write_record_batch(&mut self, metadata: Vec<u8>, encoded_buffers: Vec<EncodedBuffer>, body_len: usize, tail_pad: usize, write_options: &IpcWriteOptions) -> Result<(usize, usize), ArrowError> { // ... unchanged body ... } fn write_eos(&mut self, write_options: &IpcWriteOptions) -> Result<(), ArrowError> { self.write_continuation(write_options, 0) } } /// Blanket-implemented, so no impl of `IpcMessageSink` can ever override these. impl<T: IpcMessageSink + ?Sized> IpcMessageSinkExt for T {}

Introduce StreamEncoder for IPC streaming without requiring a std::io::Write sink. The encoder owns stream state, emits ordered Buffer chunks, and preserves the low-copy path for uncompressed record batch body buffers. Add byte-for-byte compatibility tests against StreamWriter for normal batches, empty streams, and dictionary batches. Closes apache#7812 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

Exercise StreamEncoder with a Tokio AsyncWrite flow and compare the bytes with StreamWriter. This keeps the PR tied to the async writer use case from the original issue. Refs apache#7812 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

Route writer and buffer output through a shared internal sink. This avoids duplicated IPC framing while preserving Buffer output. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

Avoid erasing W: Write on the writer path while preserving shared IPC framing for buffer output. Refs apache#7812 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

Implement the sink trait for Write directly so the writer path no longer needs a wrapper sink. Consume StreamEncoder in finish to remove the closed-state check. Refs apache#7812 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

Extract the IPC metadata padding calculation so writer and buffer sinks use the same framing layout. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

Separate sink-specific write primitives from shared IPC framing helpers while preserving the record batch writer specialization. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

alamb

Looks good -- thank you @Phoenix500526

alamb · 2026-07-27T15:50:59Z

Thanks also @Jefffrey and @Rich-T-kid for the review

github-actions Bot added the arrow Changes to the arrow crate label Jul 3, 2026

Rich-T-kid suggested changes Jul 13, 2026

View reviewed changes

Comment thread arrow-ipc/src/writer.rs Outdated

Comment thread arrow-ipc/src/writer.rs Outdated

Phoenix500526 force-pushed the issue/7812 branch 2 times, most recently from 234c1f6 to 94eaf13 Compare July 16, 2026 01:16

Rich-T-kid suggested changes Jul 16, 2026

View reviewed changes

Comment thread arrow-ipc/src/writer.rs

Comment thread arrow-ipc/src/writer.rs Outdated

Comment thread arrow-ipc/src/writer.rs

Comment thread arrow-ipc/src/writer.rs Outdated

Comment thread arrow-ipc/src/writer.rs

Comment thread arrow-ipc/src/writer.rs

Phoenix500526 force-pushed the issue/7812 branch 4 times, most recently from 3f24353 to 3a6e3c9 Compare July 17, 2026 06:50

Rich-T-kid reviewed Jul 17, 2026

View reviewed changes

Comment thread arrow-ipc/src/writer.rs

Comment thread arrow-ipc/src/writer.rs

Comment thread arrow-ipc/src/writer.rs Outdated

Comment thread arrow-ipc/src/writer.rs

Comment thread arrow-ipc/src/writer.rs

Comment thread arrow-ipc/src/writer.rs

This comment was marked as outdated.

Sign in to view

This comment was marked as duplicate.

Sign in to view

This comment was marked as outdated.

Sign in to view

Rich-T-kid mentioned this pull request Jul 18, 2026

Add benchmarks to measure Stream Encoder #10373

Closed

Rich-T-kid approved these changes Jul 18, 2026

View reviewed changes

Comment thread arrow-ipc/Cargo.toml

Comment thread arrow-ipc/src/writer.rs Outdated

This comment was marked as outdated.

Sign in to view

This comment was marked as duplicate.

Sign in to view

This comment was marked as outdated.

Sign in to view

Phoenix500526 force-pushed the issue/7812 branch from 5cfcf8a to 84eb07f Compare July 24, 2026 04:24

This comment was marked as duplicate.

Sign in to view

Jefffrey reviewed Jul 24, 2026

View reviewed changes

Comment thread arrow-ipc/src/writer.rs Outdated

Comment thread arrow-ipc/src/writer.rs Outdated

Comment thread arrow-ipc/src/writer.rs Outdated

This comment was marked as duplicate.

Sign in to view

Jefffrey approved these changes Jul 24, 2026

View reviewed changes

alamb reviewed Jul 26, 2026

View reviewed changes

Phoenix500526 added 8 commits July 26, 2026 10:02

test(arrow-ipc): cover async stream encoding

0405bcf

Exercise StreamEncoder with a Tokio AsyncWrite flow and compare the bytes with StreamWriter. This keeps the PR tied to the async writer use case from the original issue. Refs apache#7812 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

refactor(ipc): share message sink

b80b9dc

Route writer and buffer output through a shared internal sink. This avoids duplicated IPC framing while preserving Buffer output. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

docs(ipc): add document comments over sink helpers

2397761

Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

perf(ipc): specialize message sinks

8c53a27

Avoid erasing W: Write on the writer path while preserving shared IPC framing for buffer output. Refs apache#7812 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

refactor(ipc): simplify message sink

8232385

Implement the sink trait for Write directly so the writer path no longer needs a wrapper sink. Consume StreamEncoder in finish to remove the closed-state check. Refs apache#7812 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

refactor(ipc): share metadata layout

45e0543

Extract the IPC metadata padding calculation so writer and buffer sinks use the same framing layout. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

refactor(ipc): split message sink traits

ee44c9c

Separate sink-specific write primitives from shared IPC framing helpers while preserving the record batch writer specialization. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

Phoenix500526 force-pushed the issue/7812 branch from 66a7cb1 to ee44c9c Compare July 26, 2026 03:24

Phoenix500526 mentioned this pull request Jul 27, 2026

Consider sink-based Arrow IPC StreamEncoder API #10445

Open

alamb approved these changes Jul 27, 2026

View reviewed changes

alamb merged commit 1a46dd3 into apache:main Jul 27, 2026
31 checks passed

Rich-T-kid mentioned this pull request Jul 28, 2026

perf(arrow-ipc): Avoid copies and write dictionary batches directly to writers when possible #10128

Open

Jefffrey added the enhancement Any new improvement worthy of a entry in the changelog label Jul 28, 2026

Phoenix500526 deleted the issue/7812 branch July 29, 2026 02:39

Uh oh!

Conversation

Phoenix500526 commented Jul 3, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Rich-T-kid commented Jul 5, 2026

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid commented Jul 17, 2026

Uh oh!

Phoenix500526 commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid commented Jul 17, 2026

Uh oh!

This comment was marked as outdated.

This comment was marked as duplicate.

This comment was marked as outdated.

Rich-T-kid commented Jul 18, 2026

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as outdated.

This comment was marked as outdated.

Jefffrey commented Jul 24, 2026

Uh oh!

Jefffrey commented Jul 24, 2026

Uh oh!

This comment was marked as duplicate.

adriangbot commented Jul 24, 2026

Uh oh!

Jefffrey commented Jul 24, 2026

Uh oh!

This comment was marked as duplicate.

adriangbot commented Jul 24, 2026

Uh oh!

Jefffrey commented Jul 24, 2026

Uh oh!

Phoenix500526 commented Jul 24, 2026

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid left a comment •

edited

Loading

Phoenix500526 commented Jul 17, 2026 •

edited

Loading