Skip to content

feat: add JSON streaming encode support#65

Merged
shreyasbhat0 merged 1 commit into
toon-format:mainfrom
VimCommando:json-stream
May 19, 2026
Merged

feat: add JSON streaming encode support#65
shreyasbhat0 merged 1 commit into
toon-format:mainfrom
VimCommando:json-stream

Conversation

@VimCommando

Copy link
Copy Markdown
Contributor

Linked Issue

None

Description

Adds an optional json_stream feature to the crate. When enabled you can use serde's streaming deserialization on large JSON objects, concurrently passing chunks of the JSON from the Reader to the Writer. This completely avoids holding the entire input or output in memory and dramatically reduces both memory usage by ~90% and execution time by ~80% (comparison benchmark below).

I've found the TOON format is not only better for LLMs, but is superior to both JSON and YAML for storing and transmitting large data sets. I frequently have 300MB+ JSON files and sometimes just converting to TOON can shave off up to 30% before compression; or it can pretty-print in the same size as a compact single-line JSON.

I have used Codex to assist with this PR and have reviewed the code, following design patterns of my own prior work.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Changes Made

  • Adds a new crate feature json_stream
  • Adds the src/encode/json_stream.rs module
  • Adds streaming serialization/writer for the TOON output
  • Adds an example to the README.md

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected:
  • Spec version: N/A

Testing

  • All existing tests pass
  • Added new tests for changes
  • Tests cover edge cases and spec compliance

Output is byte-for-byte identical to the existing code path.

There is one open cargo clippy warning, but it is unrelated to this PR.

Pre-submission Checklist

  • My code follows the project's coding standards
  • I have run code formatting/linting tools
  • I have added tests that prove my fix/feature works
  • New and existing tests pass locally
  • I have updated documentation if needed
  • I have reviewed the TOON specification for relevant sections

Breaking Changes

  • No breaking changes
  • Breaking changes (describe migration path below)

Additional Context

TOON Rust vs TOON Rust Stream

Metric TOON RS TOON Stream Change % Change
Real time 76.8s 9.03s -67.8s -88.2%
User time 74.7s 8.37s -66.3s -88.8%
Sys time 1.60s 0.20s -1.40s -87.5%
Max RSS 9,922 MB 36.7 MB -9,885 MB -99.6%
Peak footprint 7,857 MB 28.8 MB -7,828 MB -99.6%

TOON Javascript vs TOON JSON Stream

Metric TOON JS TOON Stream Change % Change
Real time 264.4s 9.03s -255.3s -96.6%
User time 110.9s 8.37s -102.5s -92.5%
Sys time 171.6s 0.20s -171.4s -99.9%
Max RSS 1,325 MB 36.7 MB -1,288 MB -97.2%
Peak footprint 1,353 MB 28.8 MB -1,324 MB 97.9%

File Summary

File Size Bytes Lines Characters
segments.json 388M 406,585,264 12,715,306 406,585,264
segments-js.toon 285M 298,411,834 9,682,960 298,411,834
segments-rs.toon 286M 299,718,398 9,682,960 299,718,398
segments-rss.toon 286M 299,718,398 9,682,960 299,718,398

Note
segments-rs.toon and segments-rss.toon are bye-for-byte-identical.

The Javascript output is slightly smaller because it omits quotes on some values that Rust keeps quoted. Example:

<               version: 9.11.1
---
>               version: "9.11.1"

@VimCommando VimCommando requested review from a team and johannschopplich as code owners April 21, 2026 02:38
@shreyasbhat0 shreyasbhat0 merged commit c7e384e into toon-format:main May 19, 2026
3 checks passed
@github-actions github-actions Bot mentioned this pull request May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants