Skip to content

Add incremental parse API and extract parser engines#5

Merged
default-anton merged 4 commits into
mainfrom
feat/incremental-parse-api
Mar 10, 2026
Merged

Add incremental parse API and extract parser engines#5
default-anton merged 4 commits into
mainfrom
feat/incremental-parse-api

Conversation

@default-anton
Copy link
Copy Markdown
Collaborator

@default-anton default-anton commented Mar 9, 2026

What Changed

JsonCompleter.parse is the new fast path for callers that want Ruby values from partial JSON streams.

On the benchmark in this PR, it is 8.93x faster at p50 than complete + JSON.parse and reduces allocated objects 4.66x (20419154 -> 4381473).

.parse is the primary streaming path: it returns the current Ruby value directly from partial input while maintaining parser state between calls, so each update processes only new data. .complete remains for callers that specifically need completed JSON text.

The implementation is split into:

  • ParserEngine for incremental .parse
  • CompletionEngine for .complete
  • Scanners for shared token scanning

The PR also separates parse/completion specs, adds a benchmark spec, and updates the README to position .parse as the primary streaming API.

Validation

  • bundle exec rubocop
  • bundle exec rspec
  • JSON_COMPLETER_BENCHMARK=1 bundle exec rspec spec/parse_benchmark_spec.rb

Benchmark

Command: for i in $(seq 1 10); do JSON_COMPLETER_BENCHMARK=1 bundle exec rspec spec/parse_benchmark_spec.rb; done

Defaults: iterations=50, chunk_size=8 (roughly 2 streamed tokens per update).

Run Parameters

Payload bytes Prefixes Iterations Chunk size Samples
77690 9712 50 8 10

Runtime Summary

Metric parse complete + JSON.parse
Min total 1.9712s 17.7352s
P50 total 2.0006s 17.8693s
P90 total 2.0251s 18.1113s
P95 total 2.0525s 18.1960s
Max total 2.0799s 18.2806s
Mean total 2.0070s 17.9238s

Allocation and Memory

Strategy Allocated objects Heap growth bytes Heap growth pages Retained objects Retained bytes
parse 4381473 0 0 8 448
complete + JSON.parse 20419154 196608 3 1 160

Comparison

Metric Value
Speedup (min) 8.68x
Speedup (p50) 8.93x
Speedup (p90) 9.10x
Speedup (p95) 9.13x
Speedup (max) 9.16x
Speedup (mean) 8.93x
Object allocation reduction 4.66x

Split the completion and parse engines into dedicated files.
Add parse-focused specs and benchmark coverage, and update the README to position .parse as the primary streaming API.
@default-anton default-anton merged commit d380217 into main Mar 10, 2026
5 checks passed
@default-anton default-anton deleted the feat/incremental-parse-api branch March 10, 2026 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant