feat(parquet): precompute `offset_index_disabled` at build-time by HippoBaro · Pull Request #9724 · apache/arrow-rs

HippoBaro · 2026-04-15T06:29:55Z

Which issue does this PR close?

Depends on feat(parquet): add wide-schema writer overhead benchmark #9723
Contributes to Wide schema performance: eliminate quadratic column-count scaling #9722

Rationale for this change

WriterProperties::offset_index_disabled() checked whether any column in the column_properties HashMap has page-level statistics enabled, scanning the entire map on every call. This method is called from GenericColumnWriter::new — once per column per row group. With N columns each having per-column properties, this resulted in quadratic HashMap iterations during row group construction.

What changes are included in this PR?

Move the scan into WriterPropertiesBuilder::build() so it runs once at construction time.

Benchmark results (vs baseline in #9723):

  writer_overhead/1000_cols/per_column_props     2.44 ms  (was   3.25 ms, −25%)
  writer_overhead/5000_cols/per_column_props    13.28 ms  (was  47.45 ms, −72%)
  writer_overhead/10000_cols/per_column_props   27.97 ms  (was 197.97 ms, −86%)

Scaling now linear.

Are these changes tested?

All tests passing.

Are there any user-facing changes?

None.

`WriterProperties::offset_index_disabled()` checked whether any column in the `column_properties` HashMap has page-level statistics enabled, scanning the entire map on every call. This method is called from `GenericColumnWriter::new` — once per column per row group. With N columns each having per-column properties, this resulted in quadratic HashMap iterations during row group construction. Move the scan into `WriterPropertiesBuilder::build()` so it runs once at construction time. Benchmark results (vs baseline): writer_overhead/1000_cols/per_column_props 2.44 ms (was 3.25 ms, −25%) writer_overhead/5000_cols/per_column_props 13.28 ms (was 47.45 ms, −72%) writer_overhead/10000_cols/per_column_props 27.97 ms (was 197.97 ms, −86%) Scaling now linear. Signed-off-by: Hippolyte Barraud <hippolyte.barraud@datadoghq.com>

etseidl

Thanks @HippoBaro, this looks correct to me (and faster 😄🚀). It just took some time for me to get my head around the need for the DisabledOverridden variant. 😅

HippoBaro · 2026-04-15T19:56:34Z

Thank you @etseidl! Let's make sure #9723 is merged first just in case we want the bot to validate the gains 😀

alamb · 2026-04-15T20:57:08Z

Thank you @etseidl! Let's make sure #9723 is merged first just in case we want the bot to validate the gains 😀

I merged this and will kick off benchmark

alamb · 2026-04-15T20:57:18Z

run benchmark writer_overhead

adriangbot · 2026-04-15T20:59:24Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4255418162-1331-4sdr7 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing cheap_offset_index_disabled (2853ffc) to 7a089ad (merge-base) diff
BENCH_NAME=writer_overhead
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench writer_overhead
BENCH_FILTER=
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-15T21:01:10Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                         cheap_offset_index_disabled            main
-----                         ---------------------------            ----
writer_overhead/10000_cols    1.00     34.5±0.28ms        ? ?/sec    4.52    156.2±2.46ms        ? ?/sec
writer_overhead/1000_cols     1.00      3.4±0.01ms        ? ?/sec    1.26      4.2±0.03ms        ? ?/sec
writer_overhead/5000_cols     1.00     19.8±0.21ms        ? ?/sec    2.17     43.1±0.57ms        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	46.2s
Peak memory	2.7 GiB
Avg memory	2.7 GiB
CPU user	44.2s
CPU sys	1.8s
Peak spill	0 B

branch

Metric	Value
Wall time	31.7s
Peak memory	2.7 GiB
Avg memory	2.7 GiB
CPU user	30.1s
CPU sys	1.7s
Peak spill	0 B

File an issue against this benchmark runner

alamb · 2026-04-15T21:11:26Z

Those are some nice benchmark numbers

alamb

❤️ 🚀

…he#9724) # Which issue does this PR close? - Depends on apache#9723 - Contributes to apache#9722 # Rationale for this change `WriterProperties::offset_index_disabled()` checked whether any column in the `column_properties` HashMap has page-level statistics enabled, scanning the entire map on every call. This method is called from `GenericColumnWriter::new` — once per column per row group. With N columns each having per-column properties, this resulted in quadratic HashMap iterations during row group construction. # What changes are included in this PR? Move the scan into `WriterPropertiesBuilder::build()` so it runs once at construction time. Benchmark results (vs baseline in apache#9723): ``` writer_overhead/1000_cols/per_column_props 2.44 ms (was 3.25 ms, −25%) writer_overhead/5000_cols/per_column_props 13.28 ms (was 47.45 ms, −72%) writer_overhead/10000_cols/per_column_props 27.97 ms (was 197.97 ms, −86%) ``` Scaling now linear. # Are these changes tested? All tests passing. # Are there any user-facing changes? None. Signed-off-by: Hippolyte Barraud <hippolyte.barraud@datadoghq.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

github-actions Bot added the parquet Changes to the parquet crate label Apr 15, 2026

HippoBaro changed the title ~~feat(parquet): precompute offset_index_disabled at build-time~~ feat(parquet): precompute offset_index_disabled at build-time Apr 15, 2026

etseidl approved these changes Apr 15, 2026

View reviewed changes

Merge branch 'main' into cheap_offset_index_disabled

2853ffc

alamb approved these changes Apr 15, 2026

View reviewed changes

alamb added the performance label Apr 15, 2026

alamb merged commit 9ed4c6f into apache:main Apr 16, 2026
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(parquet): precompute `offset_index_disabled` at build-time - #9724

feat(parquet): precompute `offset_index_disabled` at build-time#9724
alamb merged 2 commits into
apache:mainfrom
HippoBaro:cheap_offset_index_disabled

HippoBaro commented Apr 15, 2026 •

edited

Loading

Uh oh!

etseidl left a comment

Uh oh!

HippoBaro commented Apr 15, 2026

Uh oh!

alamb commented Apr 15, 2026

Uh oh!

alamb commented Apr 15, 2026

Uh oh!

adriangbot commented Apr 15, 2026

Uh oh!

adriangbot commented Apr 15, 2026

Uh oh!

alamb commented Apr 15, 2026

Uh oh!

alamb left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

HippoBaro commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

HippoBaro commented Apr 15, 2026

Uh oh!

alamb commented Apr 15, 2026

Uh oh!

alamb commented Apr 15, 2026

Uh oh!

adriangbot commented Apr 15, 2026

Uh oh!

adriangbot commented Apr 15, 2026

Uh oh!

alamb commented Apr 15, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HippoBaro commented Apr 15, 2026 •

edited

Loading