Commit d2fcbbd
[branch-53] bring array_filter support (#131)
* feat(functions-nested): add array_filter higher-order function (apache#21895)
Partially addresses apache#14509 — implements `array_filter` / `list_filter`.
`array_transform` (apache#21679) added the first `HigherOrderUDF`.
`array_filter` is the natural companion: filter array elements with a
boolean lambda, matching Spark `filter` / DuckDB `list_filter`
semantics.
- New `HigherOrderUDF` `ArrayFilter` (`array_filter` / `list_filter`
alias)
- Boolean lambda per element; `true` keeps, `false`/null drops (matches
Spark semantics)
- Handles `List`, `LargeList`, sliced arrays, null sublists
- Scalar predicate short-circuit (`x -> true` / `x -> false`)
- No-copy fast path when nothing is filtered (skips
`arrow::compute::filter`)
- Shared HOF helpers extracted from `array_transform` into a common
module (`value_lambda_pair`, `coerce_single_list_arg`,
`single_list_lambda_parameters`, `extract_list_values`)
- Shared unit test helpers for higher-order function tests
- Unit tests: basic filter, multiple sublists, sliced arrays, null
sublists, all-filtered-out, nothing-filtered (fast path), scalar
true/false predicates
- SQL logic tests in `array_filter.slt`: filter variants, `array_filter`
+ `array_transform` combinations, error cases
Yes — `array_filter(array, lambda)` and alias `list_filter(array,
lambda)` are now available as SQL functions.
(cherry picked from commit d91dcb7)
* trigger CI
---------
Co-authored-by: Lavkesh Lahngir <lavkesh.lahngir@datadoghq.com>1 parent 76da858 commit d2fcbbd
6 files changed
Lines changed: 937 additions & 129 deletions
File tree
- datafusion
- functions-nested/src
- sqllogictest/test_files/array
- docs/source/user-guide/sql
0 commit comments