Add cargo-fuzz / OSS-Fuzz coverage for polars-arrow / polars-parquet / polars-json readers

[youichi-uda]

Add cargo-fuzz / OSS-Fuzz coverage for polars-arrow / polars-parquet / polars-json readers

What's missing

I went looking for a fuzz/ directory under crates/polars-arrow, crates/polars-parquet, or anywhere else in the workspace and didn't find one. From a quick search of the issue tracker I also can't find prior discussion about cargo-fuzz / libFuzzer / OSS-Fuzz integration. Apologies if I missed it — please point me at the right thread if so.

In contrast, the equivalent crates upstream (apache/arrow-rs, apache/datafusion) all sit on libraries that have either started or are about to start running under OSS-Fuzz harnesses (apache/arrow-rs#5332, apache/datafusion#21998). So polars's independent reimplementations of arrow / parquet / json readers in crates/polars-arrow, crates/polars-parquet, crates/polars-json are right now the least-fuzzed widely-deployed implementations of those formats in the Rust ecosystem.

Given polars-arrow / polars-parquet sit at the boundary between disk/network bytes and the rest of the polars query engine, they're an attractive target for libFuzzer-style coverage of the deserialization path. A handful of Vec::with_capacity(attacker_controlled_len) style allocations in similar crates have already produced multi-GiB OOM crashes from 4–22 byte inputs — same bug class is plausible here.

Concrete proposal

1. Add per-crate fuzz/ directories with cargo-fuzz harnesses, starting with the IPC reader and growing as we land. Two are already prototyped on my fork at fuzz/initial-harnesses:

   • crates/polars-arrow/fuzz/ipc_stream_reader — drives read_stream_metadata + StreamReader::new over arbitrary &[u8]. 22 s sanity run: 49,208 runs, 193 edges, RSS 108 MiB, no crashes.
   • crates/polars-arrow/fuzz/ipc_file_reader — drives read_file_metadata + FileReader::new. 22 s sanity: 200,000 runs, 82 edges, RSS 71 MiB, no crashes.

   Subsequent PRs would add (in roughly priority order): polars-parquet/fuzz/parquet_reader, polars-json/fuzz/json_deserialize, polars-io/fuzz/csv_reader, possibly polars-arrow/fuzz/avro_reader.

2. Wire ClusterFuzzLite into GitHub Actions for PR-time fuzz + nightly batch + corpus pruning. Catches regressions at PR time instead of relying on me (or anyone) to remember to run cargo-fuzz locally.

3. Open the OSS-Fuzz integration PR (projects/polars/) once the upstream fuzz/ lands and the harnesses are stable. I have a working projects/polars/{project.yaml,Dockerfile,build.sh} set up on my oss-fuzz fork that builds the harnesses cleanly under helper.py build_fuzzers --sanitizer address polars. Coverage target ≥80% on harness-touched modules per the OSS-Fuzz Integration Rewards bar.

Why this is worth doing

• polars's userbase is wide enough (38k stars, py-polars dominant in the Rust+Python data community) that a panic or runaway allocation on malformed input is a real DoS risk for any service that lets a user point polars at attacker-controlled data.
• The format implementations in polars-* are independent of arrow-rs's, so OSS-Fuzz coverage on the apache/arrow-rs side does not cover polars; this is genuinely uncovered surface today.
• Cost is modest: cargo-fuzz harnesses are tens of lines each, and the OSS-Fuzz / ClusterFuzzLite plumbing is a one-time setup.

Two questions for the maintainers

1. Layout preference: per-crate fuzz/ dirs (gitoxide-style — independent buildable units, easy to add per-crate seed corpora) vs. a single top-level fuzz/ workspace? I'd lean per-crate but happy to do whatever you prefer.

2. primary_contact / auto_ccs for OSS-Fuzz project.yaml: OSS-Fuzz wants Google-account emails on file for crash notifications. Who from the maintainer team would want to be on that list? I can default to a single contact if a team-wide alias isn't preferred.

xref apache/arrow-rs#5332 (sibling work for the apache/arrow-rs reader stack) and apache/datafusion#21998 (sibling work for DataFusion's SQL frontend), same author, same OSS-Fuzz integration pattern.

[coastalwhite]

Not the same, but there is #23448.

[youichi-uda]

Thanks @coastalwhite — I read through #23448 and polars-arrow/**/proptest.rs. Agreed it's not the same, and I think the two are complementary rather than redundant:

• proptest / parametric testing in Tracking issue for Parametric Testing Strategies in Rust with proptest #23448 generates valid, typed inputs (random ArrowDataType, random Array, random kernel arguments) and asserts kernel invariants — exactly the right tool for catching correctness bugs in well-formed paths. That's a category libFuzzer is bad at, because it has no notion of structure.

• What I'm proposing in Add cargo-fuzz / OSS-Fuzz coverage for polars-arrow / polars-parquet / polars-json readers #27488 is coverage-guided byte-level fuzzing of the entry points that take attacker-controlled bytes (StreamReader::new(reader, ...), read_metadata(&mut Cursor), infer + deserialize over arbitrary JSON). Different category: it's good at finding panics, OOMs, and overflows triggered by malformed input that proptest never generates because the strategies it draws from are valid by construction.

Concrete examples from the prototype harnesses already on my fork:

• 7-byte input → ~108 GiB Vec::with_capacity(list_size) in polars-parquet/handwritten_thrift::read_thrift_vec (fix(rust): Bound polars-parquet thrift list capacity by remaining input #27490).
• 3.5 KiB JSON → StructArray::new panic from inferred dtype disagreeing with deserialized child dtype (fix(rust): Use try_new in StructArray construction in polars-json #27489).

These are reachable straight from byte slices via the public reader/deserialize APIs and are a different bug class from the kernel-correctness issues #23448 is set up to catch.

I'd love to land them as separate-but-friendly testing infrastructure: proptest for kernel correctness, cargo-fuzz / OSS-Fuzz for byte-level robustness. Happy to fold either into the other if you'd prefer one umbrella, but I think keeping them separate matches the analogous split on the apache/arrow-rs side.