Skip to content

feat: centralizing higher-order list lambda evaluation helpers - #23911

Open
saadtajwar wants to merge 8 commits into
apache:mainfrom
saadtajwar:saadtajwar/abstract-hof-list-lambda
Open

feat: centralizing higher-order list lambda evaluation helpers#23911
saadtajwar wants to merge 8 commits into
apache:mainfrom
saadtajwar:saadtajwar/abstract-hof-list-lambda

Conversation

@saadtajwar

@saadtajwar saadtajwar commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

array_filter, array_any_match, and array_first each implement their own version of the same higher-order list execution pattern:

  • normalize a list-like argument to List / LargeList,
  • extract the flattened child values with slice-aware semantics,
  • evaluate a lambda once over the flattened values,
  • spread captured outer columns to flattened row cardinality with list_values_row_number,
  • map the flattened lambda result back to one output value per input row using adjusted offsets and null-row handling.

This PR introduces shared helpers that allow for one area where the shared logic lives

What changes are included in this PR?

Added a helper in lambda_utils that abstracts shared fields (original list/flat values/evaluated result) and functionality (creating the BooleanArray, adjusting offsets for List/LargeList, extracting list/lambda pairs, etc), and applying to array_filter/array_first/array_any_match

Are these changes tested?

Yes

Are there any user-facing changes?

No - shared helpers added are only visible within in the crate, rest is just rewrites

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Jul 26, 2026
@codecov-commenter

codecov-commenter commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.61905% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.76%. Comparing base (1727b7d) to head (8c36b61).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/functions-nested/src/lambda_utils.rs 84.83% 12 Missing and 15 partials ⚠️
datafusion/functions-nested/src/array_any_match.rs 90.90% 2 Missing and 2 partials ⚠️
datafusion/functions-nested/src/array_filter.rs 92.45% 2 Missing and 2 partials ⚠️
datafusion/functions-nested/src/array_first.rs 90.00% 1 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23911      +/-   ##
==========================================
+ Coverage   80.75%   80.76%   +0.01%     
==========================================
  Files        1096     1096              
  Lines      373413   373634     +221     
  Branches   373413   373634     +221     
==========================================
+ Hits       301537   301762     +225     
+ Misses      53891    53885       -6     
- Partials    17985    17987       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@saadtajwar saadtajwar changed the title feat: centralizing higher-order list lambda evaluation helpers (DRAFT) feat: centralizing higher-order list lambda evaluation helpers Jul 27, 2026
@saadtajwar
saadtajwar marked this pull request as ready for review July 27, 2026 22:25
@saadtajwar

Copy link
Copy Markdown
Contributor Author

@kosiew - PR ready for review! Thanks in advance for your time! 🙌

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saadtajwar

Thanks for working on this.

Looks 👍 to me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Centralize higher-order list lambda evaluation helpers

3 participants