Rename scalar function strictness contract (#8930)

Changes `is_null_sensitive` to `is_strict` and flips all current
implementations (as `is_strict` implies `!is_null_sensitive`).

A function is strict only when any null input forces a null output and
its return dtype propagates that nullability.

Also fixes `Between` and `Dynamic` scalar functions as they were both
set to strict when they should have been non-strict, so now they are
`is_strict = false`.

## Rationale for this change

Dictionary pushdown rewrites `f(dict(codes, values), constant)` as
`dict(codes, f(values, constant))`. At a null code, the original
evaluates `f(null, constant)`, while the rewritten dictionary is always
null. This is only sound when nulling any one argument forces a null
result while the other arguments remain unchanged.

`is_null_sensitive` documented the unary case but left this
multi-argument contract ambiguous. Kleene `AND`/`OR` show the
difference: masking both inputs produces null, but `false AND null =
false` and `true OR null = true`. That ambiguity caused the bug fixed in
#8929.

Additionally, we have noticed that a large majority of our scalar
functions have strict semantics, and so in the future (hopefully soon)
we can lift out a lot of the shared null logic.

## What changes are included in this PR?

<details>
<summary>Functions that are strict, previously treated as
non-strict</summary>

<br>

These functions are strict, so the old null-sensitivity classification
was overly conservative:

- `ArrayExpr` and `RowIdx` inherited the old conservative default. These
functions take no arguments, so the strictness condition does not apply.
- `Mask` inherited the old conservative default, but a null input
remains null and its mask argument is required to be non-nullable.
- `VariantGet` inherited the old conservative default, but a null
Variant input produces a null output.
- `GetItem` and `Select` preserve the parent struct validity and
propagate its nullability.
- `Merge` only accepts non-nullable inputs, so the strictness law is
vacuously satisfied for well-typed arguments.

</details>

<details>
<summary>Functions that are non-strict, previously treated as
strict</summary>

<br>

These functions are non-strict, so the old classification allowed
unsound dictionary pushdown:

- `Between`: its comparisons are combined with Kleene `AND`; for
example, `10 BETWEEN null AND 5` is `false`, not null.
- `DynamicComparison`: when the dynamic RHS is absent, it returns its
configured default even when the LHS is null.

</details>

<details>
<summary>Remaining non-strict functions</summary>

<br>

- `Binary` for Kleene `AND`/`OR`
- `CaseWhen` and `Zip`
- `Cast`
- `FillNull`, `IsNull`, and `IsNotNull`
- `ForeignScalarFnVTable`
- `ListContains`
- `Pack`
- `RowEncode` and `RowSize`
- `StatFn`

</details>

<details>
<summary>Remaining strict functions</summary>

<br>

- `Binary` for all operators except `AND`/`OR`
- `ByteLength`, `ExtStorage`, `Like`, `ListLength`, `ListSum`, and `Not`
- `Literal`, `Root`, and `RowCount` (nullary)
- `JsonToVariant`
- `GeoContains`, `GeoDistance`, `GeoEnvelope`, and `GeoIntersects`
- `CosineSimilarity`, `InnerProduct`, `L2Denorm`, and `L2Norm`

</details>

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Author: connortsui20

docs/concepts/expressions.md

@@ -7,7 +7,7 @@ to the returned data.
 ## Scalar Functions
 
 Expressions are defined as abstract scalar functions. These vtables define the signature of the function, properties
-such as whether the function is null-sensitive, and the actual logic for executing the function over input arrays.
+such as whether the function is strict, and the actual logic for executing the function over input arrays.
 
 The built-in scalar functions can be found in the `vortex-array::expr` module, with additional use-case specific
 functions provided by integration and plugin crates.

encodings/parquet-variant/src/json_to_variant_tests.rs

@@ -145,7 +145,7 @@ fn converts_json_extension_input() -> VortexResult<()> {
 #[test]
 fn dict_encoded_input_converts_each_row() -> VortexResult<()> {
     // A dictionary-encoded JSON column exercises the dict-pushdown / canonicalization path:
-    // `json_to_variant` is not null-sensitive, so it pushes into the dict values (canonical
+    // `json_to_variant` is strict, so it pushes into the dict values (canonical
     // JSON extension values) where the kernel fires; either way every row must convert correctly.
     let values = json_input(VarBinViewArray::from_iter_str(["1", "2"]).into_array())?;
     let codes = PrimitiveArray::from_iter([0u8, 1, 0, 1, 0]).into_array();

vortex-array/src/arrays/dict/compute/rules.rs

@@ -149,17 +149,18 @@ impl ArrayParentReduceRule<Dict> for DictionaryScalarFnValuesPushDownRule {
             return Ok(None);
         }
 
-        // If the scalar function is null-sensitive, then we cannot push it down to values if
-        // we have any nulls in the codes.
+        // Before this rewrite, a null code supplies null for this argument while the constant
+        // arguments retain their values. After the rewrite, the null code masks the function's
+        // result. Those are equivalent only for a strict function.
         if array.codes().dtype().is_nullable()
             && !matches!(
                 array.codes().validity()?,
                 Validity::NonNullable | Validity::AllValid
             )
-            && sig.is_null_sensitive()
+            && !sig.is_strict()
         {
             tracing::trace!(
-                "Not pushing down null-sensitive scalar function {} over dictionary with null codes {}",
+                "Not pushing down non-strict scalar function {} over dictionary with null codes {}",
                 parent.scalar_fn(),
                 Dict.id(),
             );
@@ -182,11 +183,10 @@ impl ArrayParentReduceRule<Dict> for DictionaryScalarFnValuesPushDownRule {
             .into_array()
             .optimize()?;
 
-        // We can only push down null-sensitive functions when we have all-valid codes.
-        // In these cases, we cannot have the codes influence the nullability of the output DType.
-        // Therefore, we cast the codes to be non-nullable and then cast the dictionary output
-        // back to nullable if needed.
-        if sig.is_null_sensitive() && array.codes().dtype().is_nullable() {
+        // A non-strict function reaches this point only when the codes are all valid, but their
+        // dtype may still be nullable. Remove that declared nullability while rebuilding the
+        // dictionary, then cast its output to the function's declared dtype.
+        if !sig.is_strict() && array.codes().dtype().is_nullable() {
             let new_codes = array.codes().cast(array.codes().dtype().as_nonnullable())?;
             let new_dict = unsafe {
                 DictArray::new_unchecked(new_codes, new_values)

vortex-array/src/arrays/scalar_fn/vtable/mod.rs

@@ -337,4 +337,8 @@ impl scalar_fn::ScalarFnVTable for ArrayExpr {
         let validity_array = options.0.validity()?.to_array(options.0.len());
         Ok(Some(ArrayExpr.new_expr(FakeEq(validity_array), [])))
     }
+
+    fn is_strict(&self, _options: &Self::Options) -> bool {
+        true
+    }
 }

vortex-array/src/expr/analysis/labeling.rs

@@ -10,6 +10,9 @@ use crate::expr::traversal::NodeExt;
 use crate::expr::traversal::NodeVisitor;
 use crate::expr::traversal::TraversalOrder;
 
+/// Boolean labels keyed by each expression node in a tree.
+pub type BooleanLabels<'a> = HashMap<&'a Expression, bool>;
+
 /// Label each node in an expression tree using a bottom-up traversal.
 ///
 /// This function separates tree labeling into two distinct steps:

vortex-array/src/expr/analysis/mod.rs

@@ -5,13 +5,12 @@ pub mod annotation;
 mod fallible;
 pub mod immediate_access;
 mod labeling;
-mod null_sensitive;
 mod referenced_field_paths;
+mod strict;
 
 pub use annotation::*;
 pub use fallible::label_is_fallible;
 pub use immediate_access::*;
 pub use labeling::*;
-pub use null_sensitive::BooleanLabels;
-pub use null_sensitive::label_null_sensitive;
 pub use referenced_field_paths::referenced_field_paths;
+pub use strict::label_strict;

vortex-array/src/expr/analysis/null_sensitive.rs

@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright the Vortex contributors
+
+use super::BooleanLabels;
+use super::labeling::label_tree;
+use crate::expr::Expression;
+
+/// Label each expression with whether its entire subtree is strict.
+///
+/// A subtree is strict only when the node's scalar function and every child subtree are strict.
+/// See [`crate::scalar_fn::ScalarFnVTable::is_strict`] for the scalar-function contract.
+pub fn label_strict(expr: &Expression) -> BooleanLabels<'_> {
+    label_tree(
+        expr,
+        |expr| expr.signature().is_strict(),
+        |acc, &child| acc & child,
+    )
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::expr::col;
+    use crate::expr::eq;
+    use crate::expr::is_null;
+    use crate::expr::lit;
+
+    #[test]
+    fn test_non_strict_with_is_null() {
+        let expr = is_null(col("col1"));
+        let labels = label_strict(&expr);
+
+        assert_eq!(labels.get(&expr), Some(&false));
+    }
+
+    #[test]
+    fn test_strict_expression() {
+        let expr = eq(lit(4), lit(5));
+        let labels = label_strict(&expr);
+
+        assert_eq!(labels.get(&expr), Some(&true));
+    }
+
+    #[test]
+    fn test_non_strict_child_makes_parent_subtree_non_strict() {
+        let left = eq(lit(4), lit(5));
+        let right = is_null(col("col2"));
+        let expr = eq(left.clone(), right.clone());
+
+        let labels = label_strict(&expr);
+
+        assert_eq!(labels.get(&left), Some(&true));
+        assert_eq!(labels.get(&right), Some(&false));
+        assert_eq!(labels.get(&expr), Some(&false));
+    }
+}

vortex-array/src/expr/analysis/strict.rs

@@ -12,7 +12,7 @@
 //!
 //! Each node references a scalar function defined by a
 //! [`ScalarFnVTable`](crate::scalar_fn::ScalarFnVTable). The vtable declares the function signature,
-//! properties such as null-sensitivity, and the logic that executes it over input arrays. Built-in
+//! properties such as strictness, and the logic that executes it over input arrays. Built-in
 //! functions live in [`crate::scalar_fn`]; integration and plugin crates supply additional,
 //! use-case-specific functions.
 //!

vortex-array/src/expr/mod.rs

@@ -306,7 +306,7 @@ impl ScalarFnVTable for Between {
         Ok(Some(and(and(arr, lower), upper)))
     }
 
-    fn is_null_sensitive(&self, _instance: &Self::Options) -> bool {
+    fn is_strict(&self, _options: &Self::Options) -> bool {
         false
     }
 
@@ -343,6 +343,21 @@ mod tests {
 
     static SESSION: LazyLock<VortexSession> = LazyLock::new(crate::array_session);
 
+    #[test]
+    fn is_not_strict() {
+        let expr = between(
+            root(),
+            lit(0),
+            lit(100),
+            BetweenOptions {
+                lower_strict: StrictComparison::NonStrict,
+                upper_strict: StrictComparison::NonStrict,
+            },
+        );
+
+        assert!(!expr.signature().is_strict());
+    }
+
     #[test]
     fn test_display() {
         let expr = between(