Skip to content

Commit d2fcbbd

Browse files
[branch-53] bring array_filter support (#131)
* feat(functions-nested): add array_filter higher-order function (apache#21895) Partially addresses apache#14509 — implements `array_filter` / `list_filter`. `array_transform` (apache#21679) added the first `HigherOrderUDF`. `array_filter` is the natural companion: filter array elements with a boolean lambda, matching Spark `filter` / DuckDB `list_filter` semantics. - New `HigherOrderUDF` `ArrayFilter` (`array_filter` / `list_filter` alias) - Boolean lambda per element; `true` keeps, `false`/null drops (matches Spark semantics) - Handles `List`, `LargeList`, sliced arrays, null sublists - Scalar predicate short-circuit (`x -> true` / `x -> false`) - No-copy fast path when nothing is filtered (skips `arrow::compute::filter`) - Shared HOF helpers extracted from `array_transform` into a common module (`value_lambda_pair`, `coerce_single_list_arg`, `single_list_lambda_parameters`, `extract_list_values`) - Shared unit test helpers for higher-order function tests - Unit tests: basic filter, multiple sublists, sliced arrays, null sublists, all-filtered-out, nothing-filtered (fast path), scalar true/false predicates - SQL logic tests in `array_filter.slt`: filter variants, `array_filter` + `array_transform` combinations, error cases Yes — `array_filter(array, lambda)` and alias `list_filter(array, lambda)` are now available as SQL functions. (cherry picked from commit d91dcb7) * trigger CI --------- Co-authored-by: Lavkesh Lahngir <lavkesh.lahngir@datadoghq.com>
1 parent 76da858 commit d2fcbbd

6 files changed

Lines changed: 937 additions & 129 deletions

File tree

0 commit comments

Comments
 (0)