What is the problem the feature request solves?
Higher-order functions (exists, forall, transform, filter, aggregate) fall back to Spark because they involve lambda expressions. This causes C2R/R2C transitions even when the rest of the pipeline is native.
Describe the potential solution
Use the JVM UDF bridge (#4170) to evaluate lambdas on the JVM while keeping data in columnar Arrow format. Prototype for exists in #4223.
Additional context
No response
What is the problem the feature request solves?
Higher-order functions (
exists,forall,transform,filter,aggregate) fall back to Spark because they involve lambda expressions. This causes C2R/R2C transitions even when the rest of the pipeline is native.Describe the potential solution
Use the JVM UDF bridge (#4170) to evaluate lambdas on the JVM while keeping data in columnar Arrow format. Prototype for
existsin #4223.Additional context
No response