Skip to content

Add any_value aggregate function - #23043

Merged
Jefffrey merged 4 commits into
apache:mainfrom
yinli-systems:kevin/add-any-value-aggregate
Jul 18, 2026
Merged

Add any_value aggregate function#23043
Jefffrey merged 4 commits into
apache:mainfrom
yinli-systems:kevin/add-any-value-aggregate

Conversation

@yinli-systems

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

any_value is a common aggregate in SQL engines for queries that need one representative non-null value from each group without imposing an ordering requirement. DataFusion currently has first_value, but that aggregate is order-sensitive, so exposing any_value gives users the intended arbitrary-value semantics directly.

What changes are included in this PR?

  • Adds an any_value(expression) aggregate UDF and registers it with the default aggregate functions.
  • Reuses the existing trivial first-value accumulator with nulls ignored, so evaluation short-circuits after the first non-null value.
  • Marks the aggregate as order-insensitive and preserves the input field metadata/type in the return field.
  • Adds sqllogictest coverage for scalar, grouped, all-null, empty-input, and string return-type cases.

Are these changes tested?

Yes. I ran:

cargo fmt --all
cargo test -p datafusion-functions-aggregate
cargo test -p datafusion-sqllogictest --test sqllogictests -- aggregate_any_value.slt
cargo clippy --all-targets --all-features -- -D warnings

Are there any user-facing changes?

Yes. This adds a new SQL aggregate function, any_value.

I used AI assistance to help inspect the codebase and run validation, and I reviewed the resulting implementation and tests.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Jun 19, 2026

@Jefffrey Jefffrey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +27 to +39
query I
SELECT any_value(column2) FROM any_value_test;
----
10

query IIT rowsort
SELECT column1, any_value(column2), any_value(column3)
FROM any_value_test
GROUP BY column1;
----
1 10 first
2 NULL NULL
3 30 third

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these two tests technically deterministic?

Comment thread datafusion/sqllogictest/test_files/aggregate_any_value.slt
@haohuaijin

Copy link
Copy Markdown
Contributor

Thanks @Kevin-Li-2025 @Jefffrey , LGTM.

One follow-up idea is to impl a native GroupsAccumulator would speed up grouped any_value a lot (it currently falls back to the slower GroupsAccumulatorAdapter).

@yinli-systems

Copy link
Copy Markdown
Contributor Author

Thanks, agreed that a native GroupsAccumulator is the right follow-up for grouped any_value. I’ll keep this PR scoped to adding the function and handle the grouped fast path separately, since it needs type-specific state handling plus targeted grouped-aggregation benchmarks to demonstrate the improvement without delaying this PR.

(3, 30, 'third');

query B
SELECT any_value(column2) IN (10, 20) FROM any_value_test;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
SELECT any_value(column2) IN (10, 20) FROM any_value_test;
SELECT any_value(column2) is not null FROM any_value_test;

or any_value(column2) IN (10, 20, 30) since we have value 30 too

@yinli-systems

Copy link
Copy Markdown
Contributor Author

I implemented the native grouped fast path as the separate follow-up discussed here: #23065. It includes focused state/merge/filter/EmitTo tests and Criterion coverage against GroupsAccumulatorAdapter; I kept it draft until this PR lands so the review stays scoped.

@github-actions github-actions Bot added auto detected api change Auto detected API change and removed auto detected api change Auto detected API change labels Jul 18, 2026
@codecov-commenter

codecov-commenter commented Jul 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.67%. Comparing base (1734d4b) to head (8609d5c).

Files with missing lines Patch % Lines
datafusion/functions-aggregate/src/any_value.rs 87.75% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23043      +/-   ##
==========================================
- Coverage   80.67%   80.67%   -0.01%     
==========================================
  Files        1087     1088       +1     
  Lines      367541   367591      +50     
  Branches   367541   367591      +50     
==========================================
+ Hits       296529   296552      +23     
- Misses      53355    53380      +25     
- Partials    17657    17659       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Generated with dev/update_function_docs.sh output.

Signed-off-by: Yin Li <kxl474@student.bham.ac.uk>
Assisted-by: OpenAI Codex
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 18, 2026
@Jefffrey
Jefffrey added this pull request to the merge queue Jul 18, 2026
@Jefffrey

Copy link
Copy Markdown
Contributor

thanks @yinli-systems & @haohuaijin

Merged via the queue into apache:main with commit 67947b6 Jul 18, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add any_value aggregate function

4 participants