Skip to content

fix(js): stream one record batch per result-iterator next() - #42

Open
fornwall wants to merge 1 commit into
mainfrom
fix/js-result-iterator-streaming
Open

fix(js): stream one record batch per result-iterator next()#42
fornwall wants to merge 1 commit into
mainfrom
fix/js-result-iterator-streaming

Conversation

@fornwall

Copy link
Copy Markdown
Owner

Problem

AdbcResultIteratorCore::next (javascript/src/client.rs) buffered the entire result set into a single contiguous IPC buffer on its first call and then marked itself exhausted:

self.exhausted = true;
let mut writer = StreamWriter::try_new(&mut output, &schema)?;
for batch in self.reader.by_ref() {   // drains the ENTIRE reader
    writer.write(&batch?)?;
}

For large result sets this materializes the whole result in memory at once — an OOM risk — and defeats the streaming abstraction the iterator name implies.

Fix

next() now yields the bytes for one record batch per call and returns Ok(None) once the reader is exhausted.

A persistent StreamWriter<Vec<u8>> is created lazily on the first call so the schema message is written exactly once at the front of the first buffer. Each subsequent buffer contains only its record batch; the underlying Vec is drained after each write. An empty result set still emits a schema-only stream (finished with the end-of-stream marker) so the consumer observes the result schema, exactly as before. Error conversion is unchanged: an arrow_schema::ArrowError propagates through ? as ClientError::Arrow.

JS-consumer compatibility (verified)

iteratorToReader in lib/index.ts feeds every per-next() chunk into a single RecordBatchReader.from(asyncIterable), which concatenates the chunks into one logical byte stream. The concatenation of the drained buffers here is byte-for-byte a single valid IPC stream (schema header followed by each batch), so the existing decoder reads every batch — no JS changes required.

This was confirmed empirically against apache-arrow 21:

  • The naive "self-contained IPC stream per batch" variant would stop at the first stream's end-of-stream marker, reading only the first batch (2 of 6 rows in a 3-batch test).
  • The continuous-stream approach used here reads all batches (6 of 6), and an empty-result stream still decodes to a schema with zero rows.

Testing

cd javascript && cargo build passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XNCrC87g9MkppGpL4MDgh5

`AdbcResultIteratorCore::next` previously drained the entire
`RecordBatchReader` into a single contiguous IPC buffer on its first
call and marked itself exhausted. For large result sets this
materialized the whole result in memory at once (OOM risk) and defeated
the streaming abstraction the iterator name implies.

`next()` now yields the bytes for a single record batch per call and
returns `Ok(None)` once the reader is exhausted. A persistent
`StreamWriter<Vec<u8>>` is created lazily on the first call so the schema
message is written exactly once at the front of the first buffer; each
subsequent buffer contains only its record batch, and the underlying
`Vec` is drained (`std::mem::take` on `get_mut`) after each write. An
empty result set still emits a schema-only stream (finished with the
end-of-stream marker) so the consumer observes the result schema, exactly
as before. Errors keep the existing conversion path: an
`arrow_schema::ArrowError` propagates through `?` as `ClientError::Arrow`.

JS-consumer compatibility verified: `iteratorToReader` in lib/index.ts
feeds every per-`next()` chunk into a single
`RecordBatchReader.from(asyncIterable)`, which concatenates the chunks
into one logical byte stream. Because the concatenation of the drained
buffers is byte-for-byte a single valid IPC stream (schema header
followed by each batch), the existing decoder reads every batch with no
JS changes. This was confirmed empirically against apache-arrow 21:
the naive "self-contained stream per batch" variant would have stopped
at the first stream's end-of-stream marker (reading only the first
batch), whereas the continuous-stream approach used here reads all
batches, and the empty-result stream still decodes to a schema with zero
rows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XNCrC87g9MkppGpL4MDgh5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant