Skip to content

[DO NOT MERGE] Add list length min-max aggregate - #8886

Draft
mhk197 wants to merge 1 commit into
developfrom
mk/list-shape-stats
Draft

[DO NOT MERGE] Add list length min-max aggregate#8886
mhk197 wants to merge 1 commit into
developfrom
mk/list-shape-stats

Conversation

@mhk197

@mhk197 mhk197 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Adds an aggregate function ListLengthMinMax to calculate the minimum and maximum length of a List-like array. This is just the current MinMax aggregate composed with the ListLength scalar function.

This will allow us to do several beneficial things if incorporated as a default stat in upcoming pr)

  • Easily detect if variable sized lists have a single shared size and write with upcoming FixedSizeList layout
  • Include as zone map stat to allow for pruning list predicates like list_length (obviously) and others, since we can identify zones with all-empty lists and prune them (e.g. list_sum > k)

What changes are included in this PR?

  • Add ListLengthMinMax for List and FixedSizeList inputs.
  • Return nullable min_length and max_length fields, ignoring null outer lists while allowing valid empty lists to contribute zero.
  • Reuse the numeric min_max implementation over the List sizes child with outer-list validity attached.
  • Short-circuit FixedSizeList inputs from their dtype and outer validity, saturating after the first non-null list.
  • Register the aggregate in the default aggregate-function session.

NOTE: Should come up with framework to do aggregate on derived column instead of just implementing this one-off

@mhk197 mhk197 added the changelog/performance A performance improvement label Jul 21, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 21, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 19.98%

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 1844 untouched benchmarks
🆕 2 new benchmarks
⏩ 46 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation copy_nullable[65536] 578.8 µs 1,033.4 µs -43.99%
Simulation copy_non_nullable[65536] 799 µs 1,112 µs -28.15%
Simulation decode_varbin[(1000, 4)] 34.3 µs 26.9 µs +27.34%
🆕 Simulation add_decimal_i128_nullable N/A 3.9 ms N/A
🆕 Simulation add_decimal_i64_nonnull N/A 1.9 ms N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/list-shape-stats (614492d) with develop (3a098b4)2

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on develop (46b0b2b) during the generation of this report, so 3a098b4 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 45e424f 1 Explore Profiling Data
Previous Runs (2)
Status Commit Job Attempt Link
🟢 Done 4c3a191 1 Explore Profiling Data
🟢 Done dbba71d 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: Likely regression (medium confidence)
Attributed Vortex impact: +11.9%
Engines: DataFusion Likely regression (+21.4%, environment too noisy confidence) · DuckDB No clear signal (+3.2%, low confidence)
Vortex (geomean): 1.094x ➖
Parquet (geomean): 1.035x ➖
Shifts: Parquet (control) +3.5% · Median polish +5.1%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.276x ❌, 0↑ 2↓)
name PR 45e424f (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 🚨 9906785 8197567 1.21
vortex_q01/datafusion:vortex-file-compressed 🚨 6582691 4885078 1.35
datafusion / parquet (1.051x ➖, 0↑ 0↓)
name PR 45e424f (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20931221 20638806 1.01
vortex_q01/datafusion:parquet 4990970 4580085 1.09
duckdb / vortex-file-compressed (1.051x ➖, 0↑ 0↓)
name PR 45e424f (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10561746 10061599 1.05
vortex_q01/duckdb:vortex-file-compressed 6483576 6167184 1.05
duckdb / parquet (1.018x ➖, 0↑ 0↓)
name PR 45e424f (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23699642 23195009 1.02
vortex_q01/duckdb:parquet 9476549 9340266 1.01

No file size changes detected.

@mhk197 mhk197 changed the title Add list shape statistics Add list length min-max statistics Jul 21, 2026
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
mhk197 force-pushed the mk/list-shape-stats branch from 45e424f to 614492d Compare July 23, 2026 14:49
@mhk197 mhk197 changed the title Add list length min-max statistics Add list length min-max aggregate Jul 23, 2026
@mhk197
mhk197 marked this pull request as ready for review July 27, 2026 09:26
@mhk197
mhk197 marked this pull request as draft July 27, 2026 12:30
@mhk197 mhk197 changed the title Add list length min-max aggregate [DO NOT MERGE] Add list length min-max aggregate Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants