[Parquet] Do not panic when trying to skip records in delta encoded files using non-standard block sizes by etseidl · Pull Request #9794 · apache/arrow-rs

etseidl · 2026-04-23T00:09:59Z

Which issue does this PR close?

Closes Parquet DeltaBitPackDecoder::skip could panic on "non-standard" miniblocks #9793.

Rationale for this change

DeltaBitPackDecoder::skip uses some magic numbers when sizing the buffer used for skipping. Files that use non-standard miniblock sizes will cause skip to panic.

What changes are included in this PR?

Check for non-standard miniblock sizes and return an error rather than a panic.

Note: allocating a vec sized with values_per_mini_block resulted in a significant performance regression.

Are these changes tested?

Yes, test with file with non-standard sizes is added to the arrow_reader tests.

Are there any user-facing changes?

No.

etseidl · 2026-04-23T00:11:31Z

-        };
-
-        let mut skip_buffer = vec![T::T::default(); mini_block_batch_size];
+        let mut skip_buffer = vec![T::T::default(); self.values_per_mini_block];


This might panic if values_per_mini_block is crazy. Sadly try_with_capacity is still unstable.

Vec::default() + Vec::try_reserve() + Vec::resize() would be a way to handle allocation errors, but I think enforcing a smaller limit than available memory in set_data could be a good idea. I'm thinking something between 64k to 1million values.

Maybe the skip_buffer could also be stored inside the struct and reused, to avoid allocations per skip.

I just saw someone else already opened a separate issue about moving the skip_buffer (#9792)

That's a good idea @jhorstmann...kill two birds with one stone.

I think enforcing a smaller limit than available memory in set_data could be a good idea. I'm thinking something between 64k to 1million values.

That's probably a reasonable thing to do as well. Let's start with a million.

etseidl · 2026-04-23T00:14:59Z

This also adds a smallish test file...I don't know if it would be better to try adding it to parquet-testing. The file is actually a rewrite of the parquet-testing alltypes_tiny_pages.parquet with a miniblock size of 128.

etseidl · 2026-04-24T17:47:07Z

run benchmark arrow_reader

env:
  BENCH_FILTER: /binary.*skip

adriangbot · 2026-04-24T17:49:58Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4315191571-1825-4gbst 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fix_delta_skip (855e870) to b93240a (merge-base) diff
BENCH_NAME=arrow_reader
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader
BENCH_FILTER=/binary.*skip
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-24T18:10:53Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                                   fix_delta_skip                         main
-----                                                                                   --------------                         ----
arrow_array_reader/INT32/Decimal128Array/binary packed skip increasing value            1.00     49.7±0.35µs        ? ?/sec    1.01     49.9±0.25µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip single value                1.00     45.2±0.22µs        ? ?/sec    1.00     45.4±0.24µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip stepped increasing value    1.05     76.9±0.22µs        ? ?/sec    1.00     73.5±0.14µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs        1.02     90.6±0.22µs        ? ?/sec    1.00     88.7±0.24µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs       1.01     90.2±0.17µs        ? ?/sec    1.00     89.6±0.13µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs         1.03     92.6±0.21µs        ? ?/sec    1.00     90.3±0.27µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip increasing value            1.00     42.6±0.09µs        ? ?/sec    1.02     43.3±0.11µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip single value                1.00     41.0±0.08µs        ? ?/sec    1.02     41.8±0.08µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip stepped increasing value    1.08     72.5±0.11µs        ? ?/sec    1.00     67.1±0.13µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs        1.02     81.3±0.07µs        ? ?/sec    1.00     79.9±0.16µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs       1.01     91.6±0.09µs        ? ?/sec    1.00     90.4±0.11µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs         1.02     83.7±0.09µs        ? ?/sec    1.00     81.9±0.12µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip increasing value                       1.00     32.9±0.31µs        ? ?/sec    1.00     32.8±0.31µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip single value                           1.02     28.7±0.28µs        ? ?/sec    1.00     28.1±0.26µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip stepped increasing value               1.06     59.5±0.16µs        ? ?/sec    1.00     56.2±0.08µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs                   1.03     73.4±0.24µs        ? ?/sec    1.00     71.1±0.14µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs                  1.01     73.4±0.08µs        ? ?/sec    1.00     72.5±0.10µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs                    1.04     76.2±0.23µs        ? ?/sec    1.00     73.3±0.14µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip increasing value                       1.00     31.5±0.25µs        ? ?/sec    1.02     32.1±0.12µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip single value                           1.00     26.9±0.19µs        ? ?/sec    1.00     26.8±0.08µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip stepped increasing value               1.05     57.7±0.13µs        ? ?/sec    1.00     55.2±0.07µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs                   1.03     71.6±0.20µs        ? ?/sec    1.00     69.2±0.25µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs                  1.00     71.7±0.11µs        ? ?/sec    1.00     71.5±0.13µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs                    1.03     74.0±0.18µs        ? ?/sec    1.00     71.6±0.27µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip increasing value                       1.00     24.1±0.09µs        ? ?/sec    1.01     24.4±0.08µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip single value                           1.00     22.5±0.12µs        ? ?/sec    1.01     22.8±0.14µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip stepped increasing value               1.11     53.5±0.08µs        ? ?/sec    1.00     48.1±0.07µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs                   1.03     62.4±0.06µs        ? ?/sec    1.00     60.4±0.51µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs                  1.01     72.4±0.08µs        ? ?/sec    1.00     71.4±0.08µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs                    1.03     64.8±0.08µs        ? ?/sec    1.00     62.8±0.09µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip increasing value                        1.00     32.8±0.33µs        ? ?/sec    1.00     32.6±0.22µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip single value                            1.02     28.5±0.25µs        ? ?/sec    1.00     27.9±0.24µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip stepped increasing value                1.06     59.5±0.14µs        ? ?/sec    1.00     56.0±0.09µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs                    1.02     72.5±0.06µs        ? ?/sec    1.00     71.0±0.09µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs                   1.00     72.9±0.08µs        ? ?/sec    1.00     72.8±0.08µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs                     1.02     74.8±0.09µs        ? ?/sec    1.00     73.3±0.11µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip increasing value                      1.00     32.9±0.32µs        ? ?/sec    1.00     33.0±0.24µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip single value                          1.02     28.6±0.25µs        ? ?/sec    1.00     28.1±0.22µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip stepped increasing value              1.06     59.8±0.16µs        ? ?/sec    1.00     56.2±0.09µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs                  1.03     78.0±0.10µs        ? ?/sec    1.00     75.6±0.10µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs                 1.02     76.4±0.09µs        ? ?/sec    1.00     75.2±0.09µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs                   1.03     80.6±0.09µs        ? ?/sec    1.00     77.9±0.12µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip increasing value                      1.00     31.0±0.29µs        ? ?/sec    1.00     31.0±0.24µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip single value                          1.00     26.4±0.20µs        ? ?/sec    1.00     26.3±0.25µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip stepped increasing value              1.07     58.2±0.15µs        ? ?/sec    1.00     54.4±0.09µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs                  1.04     71.8±0.19µs        ? ?/sec    1.00     69.2±0.23µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs                 1.01     71.9±0.12µs        ? ?/sec    1.00     71.1±0.10µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs                   1.04     74.2±0.17µs        ? ?/sec    1.00     71.4±0.21µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip increasing value                      1.00     24.2±0.08µs        ? ?/sec    1.01     24.3±0.11µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip single value                          1.00     22.5±0.12µs        ? ?/sec    1.01     22.7±0.11µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip stepped increasing value              1.11     53.6±0.08µs        ? ?/sec    1.00     48.0±0.06µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs                  1.04     62.5±0.07µs        ? ?/sec    1.00     60.4±0.11µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs                 1.01     72.3±0.09µs        ? ?/sec    1.00     71.4±0.09µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs                   1.03     64.7±0.07µs        ? ?/sec    1.00     62.9±0.08µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip increasing value                       1.00     32.8±0.35µs        ? ?/sec    1.00     32.6±0.29µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip single value                           1.01     28.5±0.23µs        ? ?/sec    1.00     28.3±0.36µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip stepped increasing value               1.06     59.5±0.12µs        ? ?/sec    1.00     56.0±0.08µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs                   1.02     77.4±0.08µs        ? ?/sec    1.00     75.6±0.08µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs                  1.01     75.2±0.10µs        ? ?/sec    1.00     74.8±0.60µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs                    1.02     79.8±0.06µs        ? ?/sec    1.00     78.1±0.08µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	595.1s
Peak memory	4.2 GiB
Avg memory	4.2 GiB
CPU user	591.0s
CPU sys	1.0s
Peak spill	0 B

branch

Metric	Value
Wall time	595.1s
Peak memory	4.2 GiB
Avg memory	4.2 GiB
CPU user	593.3s
CPU sys	0.2s
Peak spill	0 B

File an issue against this benchmark runner

etseidl · 2026-04-24T18:46:59Z

Looks like all of the benches that use the skip_buffer regressed :( Investigating...

Update:
I think the old code could allocate skip_buffer on the stack rather than the heap, so now I'm not sure how to proceed. We could

take the L on performance for a theoretical safety benefit
check for non-standard sizes in set_data and error there
have two versions of skip depending on miniblock size

etseidl · 2026-04-24T22:56:01Z

run benchmark arrow_reader

env:
  BENCH_FILTER: /binary.*skip

adriangbot · 2026-04-24T22:58:46Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4316895146-1838-pctff 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fix_delta_skip (83dbfcc) to b93240a (merge-base) diff
BENCH_NAME=arrow_reader
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader
BENCH_FILTER=/binary.*skip
Results will be posted here when complete

File an issue against this benchmark runner

etseidl · 2026-04-24T23:00:41Z

I've now opted for erroring in skip, rather than set_data, since this is only an issue on skip. The decoder handles weird block sizes just fine.

@jhorstmann what do you think?

adriangbot · 2026-04-24T23:19:19Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                                   fix_delta_skip                         main
-----                                                                                   --------------                         ----
arrow_array_reader/INT32/Decimal128Array/binary packed skip increasing value            1.00     49.8±0.24µs        ? ?/sec    1.01     50.1±0.33µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip single value                1.00     45.2±0.41µs        ? ?/sec    1.01     45.5±0.15µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip stepped increasing value    1.00     73.5±0.07µs        ? ?/sec    1.00     73.3±0.11µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs        1.01     89.3±0.24µs        ? ?/sec    1.00     88.2±0.15µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs       1.00     89.8±0.18µs        ? ?/sec    1.00     89.9±0.36µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs         1.01     90.7±0.20µs        ? ?/sec    1.00     90.1±0.14µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip increasing value            1.00     43.1±0.08µs        ? ?/sec    1.00     43.0±0.11µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip single value                1.01     41.8±0.10µs        ? ?/sec    1.00     41.5±0.12µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip stepped increasing value    1.01     67.4±0.07µs        ? ?/sec    1.00     66.9±0.07µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs        1.00     79.2±0.06µs        ? ?/sec    1.00     78.9±0.13µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs       1.00     89.7±0.10µs        ? ?/sec    1.00     89.7±0.14µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs         1.00     81.4±0.15µs        ? ?/sec    1.00     81.2±0.11µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip increasing value                       1.01     33.1±0.19µs        ? ?/sec    1.00     32.8±0.26µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip single value                           1.01     28.5±0.30µs        ? ?/sec    1.00     28.3±0.19µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip stepped increasing value               1.00     56.3±0.05µs        ? ?/sec    1.00     56.2±0.10µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs                   1.01     71.5±0.17µs        ? ?/sec    1.00     71.0±0.19µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs                  1.00     72.6±0.07µs        ? ?/sec    1.00     72.4±0.16µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs                    1.00     73.6±0.24µs        ? ?/sec    1.00     73.2±0.16µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip increasing value                       1.01     31.9±0.17µs        ? ?/sec    1.00     31.7±0.15µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip single value                           1.01     26.6±0.16µs        ? ?/sec    1.00     26.5±0.10µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip stepped increasing value               1.00     54.5±0.05µs        ? ?/sec    1.00     54.6±0.09µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs                   1.01     69.6±0.23µs        ? ?/sec    1.00     69.2±0.26µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs                  1.00     71.1±0.11µs        ? ?/sec    1.00     70.9±0.14µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs                    1.01     71.8±0.21µs        ? ?/sec    1.00     71.2±0.27µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip increasing value                       1.00     24.1±0.09µs        ? ?/sec    1.01     24.3±0.09µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip single value                           1.00     22.6±0.10µs        ? ?/sec    1.01     22.8±0.11µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip stepped increasing value               1.00     48.0±0.06µs        ? ?/sec    1.00     48.1±0.06µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs                   1.00     60.0±0.05µs        ? ?/sec    1.00     60.3±0.09µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs                  1.00     71.3±0.08µs        ? ?/sec    1.00     71.3±0.09µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs                    1.00     62.5±0.07µs        ? ?/sec    1.00     62.6±0.08µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip increasing value                        1.02     33.3±0.24µs        ? ?/sec    1.00     32.6±0.30µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip single value                            1.00     28.1±0.26µs        ? ?/sec    1.00     28.1±0.19µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip stepped increasing value                1.01     56.3±0.04µs        ? ?/sec    1.00     56.0±0.11µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs                    1.01     72.1±0.29µs        ? ?/sec    1.00     71.1±0.18µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs                   1.01     73.0±0.08µs        ? ?/sec    1.00     72.3±0.12µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs                     1.02     74.7±0.26µs        ? ?/sec    1.00     73.3±0.11µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip increasing value                      1.02     33.5±0.24µs        ? ?/sec    1.00     32.9±0.27µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip single value                          1.02     28.8±0.15µs        ? ?/sec    1.00     28.1±0.20µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip stepped increasing value              1.00     56.3±0.05µs        ? ?/sec    1.00     56.3±0.13µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs                  1.00     76.0±0.09µs        ? ?/sec    1.00     75.8±0.12µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs                 1.01     75.6±0.07µs        ? ?/sec    1.00     75.1±0.14µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs                   1.01     78.3±0.06µs        ? ?/sec    1.00     77.8±0.15µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip increasing value                      1.01     31.8±0.27µs        ? ?/sec    1.00     31.6±0.22µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip single value                          1.00     26.5±0.25µs        ? ?/sec    1.01     26.8±0.12µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip stepped increasing value              1.00     54.6±0.07µs        ? ?/sec    1.00     54.5±0.10µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs                  1.01     69.9±0.28µs        ? ?/sec    1.00     69.1±0.24µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs                 1.00     70.8±0.09µs        ? ?/sec    1.00     70.9±0.13µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs                   1.01     72.0±0.24µs        ? ?/sec    1.00     71.3±0.25µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip increasing value                      1.00     24.3±0.06µs        ? ?/sec    1.00     24.4±0.11µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip single value                          1.00     22.8±0.09µs        ? ?/sec    1.00     22.8±0.10µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip stepped increasing value              1.00     48.1±0.05µs        ? ?/sec    1.00     48.2±0.08µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs                  1.00     60.4±0.04µs        ? ?/sec    1.01     60.8±0.10µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs                 1.00     71.2±0.13µs        ? ?/sec    1.00     71.3±0.10µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs                   1.00     62.6±0.07µs        ? ?/sec    1.00     62.8±0.11µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip increasing value                       1.02     33.4±0.26µs        ? ?/sec    1.00     32.7±0.25µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip single value                           1.01     28.1±0.30µs        ? ?/sec    1.00     27.8±0.33µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip stepped increasing value               1.00     56.4±0.05µs        ? ?/sec    1.00     56.2±0.09µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs                   1.00     75.2±0.11µs        ? ?/sec    1.00     75.2±0.09µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs                  1.00     74.5±0.12µs        ? ?/sec    1.00     74.5±0.10µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs                    1.00     77.5±0.13µs        ? ?/sec    1.00     77.5±0.07µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	595.1s
Peak memory	4.1 GiB
Avg memory	4.0 GiB
CPU user	591.2s
CPU sys	0.8s
Peak spill	0 B

branch

Metric	Value
Wall time	595.1s
Peak memory	4.0 GiB
Avg memory	4.0 GiB
CPU user	592.4s
CPU sys	0.2s
Peak spill	0 B

File an issue against this benchmark runner

jhorstmann · 2026-04-25T08:55:02Z

@etseidl Looks good, this solves the panic as described in the title. Let's keep #9792 open to maybe further investigate.

etseidl · 2026-04-29T21:46:01Z

Thought it worth mentioning I tried replacing skip_buffer with an array ([T::T::default(); 64]), and that had no measurable impact on skip performance.

alamb

LGTM -- thank you @etseidl and @jhorstmann

alamb · 2026-05-01T12:34:04Z

+        // for the skip_buffer doesn't allow stack allocation and leads to a significant
+        // drop in performance. We'll settle for erroring out here and come up with a
+        // better fix if writers ever start getting creative with block sizes.
+        let mini_block_batch_size = match self.values_per_mini_block {


an error certainly seems better than a panic

alamb · 2026-05-01T12:36:19Z

+        // TODO: uncomment if we ever allow skipping miniblocks > 64 elements
+        //let batch = maybe_batch.expect("skip should succeed");
+        //assert_eq!(batch.num_rows(), 5);
+        assert!(maybe_batch.is_err());


technically this check for is_err is redundant as unwrap_err will panic if maybe_batch is Ok

done in 0f48863

etseidl · 2026-05-01T22:03:12Z

Thanks @alamb and @jhorstmann!

…iles using non-standard block sizes (apache#9794) # Which issue does this PR close? - Closes apache#9793. # Rationale for this change `DeltaBitPackDecoder::skip` uses some magic numbers when sizing the buffer used for skipping. Files that use non-standard miniblock sizes will cause `skip` to panic. # What changes are included in this PR? Check for non-standard miniblock sizes and return an error rather than a panic. Note: allocating a vec sized with `values_per_mini_block` resulted in a significant performance regression. # Are these changes tested? Yes, test with file with non-standard sizes is added to the arrow_reader tests. # Are there any user-facing changes? No.

fix for skipping delta packed files with non-standard block sizes

043fac3

github-actions Bot added the parquet Changes to the parquet crate label Apr 23, 2026

etseidl commented Apr 23, 2026

View reviewed changes

lazy initialize skip_buffer, try to catch errors

855e870

etseidl marked this pull request as draft April 24, 2026 20:17

revert changes and instead error when skipping

83dbfcc

clippy

f497c28

etseidl marked this pull request as ready for review April 29, 2026 15:37

alamb approved these changes May 1, 2026

View reviewed changes

alamb reviewed May 1, 2026

View reviewed changes

Comment thread parquet/tests/arrow_reader/bigdelta.parquet

etseidl added 2 commits May 1, 2026 08:42

enshrinken

56c203d

remove redundant check for err

0f48863

alamb approved these changes May 1, 2026

View reviewed changes

etseidl merged commit d3cad6e into apache:main May 1, 2026
16 checks passed

etseidl deleted the fix_delta_skip branch May 1, 2026 22:03

alamb mentioned this pull request May 7, 2026

Parquet DeltaBitPackDecoder::skip could panic on "non-standard" miniblocks #9793

Closed

Uh oh!

Conversation

etseidl commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

etseidl commented Apr 24, 2026

Uh oh!

adriangbot commented Apr 24, 2026

Uh oh!

adriangbot commented Apr 24, 2026

Uh oh!

etseidl commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

etseidl commented Apr 24, 2026

Uh oh!

adriangbot commented Apr 24, 2026

Uh oh!

etseidl commented Apr 24, 2026

Uh oh!

adriangbot commented Apr 24, 2026

Uh oh!

jhorstmann commented Apr 25, 2026

Uh oh!

etseidl commented Apr 29, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

etseidl commented May 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

etseidl commented Apr 23, 2026 •

edited

Loading

etseidl commented Apr 23, 2026 •

edited

Loading

etseidl commented Apr 24, 2026 •

edited

Loading