Blog post about changes to write wide columns, large string columns

[alamb]

@adriangb has made some great contributions to the Rust parquet writer to support several usecases where parquet writers typically struggle:

• fix(parquet): bound data page byte size for large variable-width values #9972
  -Pluggable page spilling API for the Parquet ArrowWriter (PageStore) #10020

I think it would be amazing to write a blog post explaining the challenge and the solutions

Challenge 1: Oversized pages for large binary/string columns

• Parquet writer can produce massively oversized data pages for large variable-width values #10061

Challenge 2: RAM buffering requirements when writing large Row Groups

• Bound ArrowWriter peak memory  #10071

The idea is to write a blog in https://arrow.apache.org/blog/ (source in https://github.com/apache/arrow-site) that explains the challenges and how we solved it with Software Engineering rather than a new file format

[alamb]

I made a demo that shows how effective the memory savings from this can be:
https://github.com/alamb/wide-parquet

[adriangb]

15GB to 30MB is crazy! do you have any performance numbers? I wonder if it's faster (because memory) or slower (because more memory fragmentation, etc).

[alamb]

15GB to 30MB is crazy! do you have any performance numbers? I wonder if it's faster (because memory) or slower (because more memory fragmentation, etc).

My very unscientific measurements on https://github.com/alamb/wide-parquet show spilling is slower by around 12%

Total elapsed time             : 1.172 s

vs

Total elapsed time             : 1.336 s

This is almost certainly due to the new small IOs that got inserted. I am sure with some more thoughtful IO optimizations (larger speculative reads and buffering) you could probably minimize almost all the overhead