[C++] can we use simdjson to replace rapidjson

[wanweiqiangintel]

Describe the enhancement requested

As the performance result mentioned in simdjson community: the simdjson library uses three-quarters less instructions than state-of-the-art parser RapidJSON. And the throughput of simdjson is much higher than that of rapidjson:

So can we replace rapidjson with simdjson to implement json parser?

Component(s)

C++

[kou]

If simdjson is faster than RapidJSON for our use case too, I'm OK with this.

Could you try this and share our benchmark result?
https://github.com/apache/arrow/blob/main/cpp/src/arrow/json/parser_benchmark.cc

[lemire]

We are available to help.

Note that simdjson is used by Apache Doris and ClickHouse.

[kou]

Great!

[mapleFU]

Seems that writer can still use original logic, but parser can make full use of simdjson?

[kou]

Is there any merit to use both RapidJSON and simdjson?

I think that using either RapidJSON or simdjson will reduce our maintenance cost.

[pitrou]

Agreed with @kou , we probably want to avoid depending on two different JSON libraries.

Interested people should try working on a PR.

[pitrou]

I'm skeptical switching to simdjson would improve performance a lot, btw. Parsing is only a small part of the work necessary to convert JSON to Arrow.

[wgtmac]

I just want to revive this discussion again. #45459 will make RapidJson as a required dependency to parquet. However, RapidJson has been poorly maintained since 2016: Tencent/rapidjson#2321. It also seems to be a blocker to support CMake 4: #45985

[kou]

Let's try simdjson for better maintainability!

[paleolimbot]

For what it's worth, I think DuckDB uses yyjson, although I don't know if simdjson was considered. I seem to remember some past issues with homebrew packaging with respect to buildtime vs. runtime SIMD support (although I imagine simdjson has considered this as well!).

[lemire]

I seem to remember some past issues with homebrew packaging with respect to buildtime vs. runtime SIMD support (although I imagine simdjson has considered this as well!).

The simdjson library uses runtime dispatching. It is currently used by Node.js, ClickHouse and other important systems. In turn, Node.js is part of several important systems.

Optionally, we also support fancy C++20 features, and we will be integrating C++26 features (e.g., static reflection and the like) as soon as possible.

Suggested reference:

• On-Demand JSON: A Better Way to Parse Documents? Software: Practice and Experience 54 (6), 2024

Documentation: https://github.com/simdjson/simdjson/blob/master/doc/basics.md

[pitrou]

I agree we need to do something about our RapidJSON dependency. simdjson is a reasonable contender. We should just have to check it doesn't reduce the current platform compatibility (especially when SIMD isn't available/supported by simdjson).