feat: add JSON streaming encode support#65
Merged
Conversation
shreyasbhat0
approved these changes
May 19, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked Issue
None
Description
Adds an optional
json_streamfeature to the crate. When enabled you can use serde's streaming deserialization on large JSON objects, concurrently passing chunks of the JSON from theReaderto theWriter. This completely avoids holding the entire input or output in memory and dramatically reduces both memory usage by ~90% and execution time by ~80% (comparison benchmark below).I've found the TOON format is not only better for LLMs, but is superior to both JSON and YAML for storing and transmitting large data sets. I frequently have 300MB+ JSON files and sometimes just converting to TOON can shave off up to 30% before compression; or it can pretty-print in the same size as a compact single-line JSON.
I have used Codex to assist with this PR and have reviewed the code, following design patterns of my own prior work.
Type of Change
Changes Made
json_streamsrc/encode/json_stream.rsmoduleSPEC Compliance
Testing
Output is byte-for-byte identical to the existing code path.
There is one open
cargo clippywarning, but it is unrelated to this PR.Pre-submission Checklist
Breaking Changes
Additional Context
TOON Rust vs TOON Rust Stream
TOON Javascript vs TOON JSON Stream
File Summary
segments.jsonsegments-js.toonsegments-rs.toonsegments-rss.toonNote
segments-rs.toonandsegments-rss.toonare bye-for-byte-identical.The Javascript output is slightly smaller because it omits quotes on some values that Rust keeps quoted. Example: