[EPIC] Full lambda support

[gstvg]

Some features haven't been implemented and others pulled out from #18921 #21679 to keep it from getting too large (>5k LOC) and should be implemented in follow on PR's:

Major features:

• Add support for lambda column capture #21323 to support construction like select a, exists(array(1, 2, 3), x-> case when a = 2 then x > 1 else null end)
• Add lambda substrait support #21193
• Add resolve_lambda_variables helper to Expr and LogicalPlan #22101
• Add exact HigherOrderSignature #22326
• Add transform_values UDF #22689
• fix(lambda): only push referenced params into the merged batch #22853
• feat(lambda): support map non empty nulls cleanup for HigherOrderFunctionExpr #22847
• Add protobuf support for lambdas #22362
• refactor: wrap HigherOrderUDFImpl in a concrete HigherOrderUDF struct #22593 (also add alias support)
• add HigherOrderUDF to the public user guide docs
• feat(functions-nested): add array_filter higher-order function #21895
• array_transform index parameter
• add any_match higher-order function #21903
• feat: minor lambda perf improvements #21896
• Add higher-order functions changes to upgrade guide #22107
• minor: make HigherOrderSignature less error-prone #22106
• minor: add higher-order function methods to SessionContext #21950
• Minor: Disallow async function in lambdas #22097

Minor features:

• Named parameters like ScalarUDF already supports: func(a => 3)
• Expr simplifier optimizations for LambdaVariable
• ListView support in remove_list_null_values and HigherOrderUDF::clean_null_values
• Optional ScalarUDFImpl methods not present on HigherOrderUDF(some are unlikely to be used and could be added on demand only):
  • simplify
  • with_updated_config
  • preimage
  • placement
  • evaluate_bounds
  • propagate_constraints
  • output_ordering
  • preserves_lex_ordering

[LiaCastaneda]

@gstvg I think we should also include as a folloup adding HigherOrderUDF to the public user guide docs https://datafusion.apache.org/library-user-guide/functions/adding-udfs.html

[gstvg]

@LiaCastaneda added, thanks

[LiaCastaneda]

Opened a PR to implement any_match - #21903
@ologlogn also opened a PR for array_filter - #21895

[LiaCastaneda]

@gstvg if you don't have a PR/branch already I can open a PR for #21679 (comment), since I think it would simplify the code a bit as well

[gstvg]

@LiaCastaneda Feel free to do it, yes, I haven't done nothing on that

[LiaCastaneda]

@gstvg I opened a small PR #22326 to add the exact signature