refactor: move from 3-variant index search result to single generic variant

[westonpace]

What

Replaces the three-variant enum:

pub enum IndexExprResult {
    Exact(RowAddrMask),
    AtMost(RowAddrMask),   // upper bound on the answer
    AtLeast(RowAddrMask),  // lower bound on the answer
}

with a single struct carrying both bounds:

pub struct IndexExprResult {
    pub lower: RowAddrMask,  // rows definitely in the answer
    pub upper: RowAddrMask,  // rows that may be in the answer
                             // (rows outside `upper` are definitely out)
}

Same change applied to NullableIndexExprResult (the during-evaluation form
that carries SQL 3VL info via NullableRowAddrMask).

The three pre-existing shapes map onto degenerate intervals:

Old variant	New form
Exact(m)	{lower: m, upper: m}
AtMost(m)	{lower: allow_nothing(), upper: m}
AtLeast(m)	{lower: m, upper: all_rows()}

Constructed via IndexExprResult::{exact, at_most, at_least} associated
functions; inspected via is_exact() / is_at_most() / is_at_least()
predicates.

Why

Some indices can return three-way info per row range — "definitely
matches", "maybe matches", "definitely doesn't match" — and the old
representation couldn't express it. The new form makes that fourth shape
representable as a non-degenerate interval (nonempty lower strictly
inside non-universe upper), and the boolean algebra collapses to
elementwise lattice operations on the endpoints:

!{l, u}              = {!u, !l}
{l1, u1} & {l2, u2}  = {l1 & l2, u1 & u2}
{l1, u1} | {l2, u2}  = {l1 | l2, u1 | u2}

The motivating case: a zone-map IS NOT NULL search can now produce
{lower: zones_with_zero_nulls, upper: not_all_null_zones}. The read
planner can both skip the all-null zones entirely (I/O saving on the
50%-clustered-nulls case) and skip the recheck on the
guaranteed-non-null zones (CPU saving). The old algebra had to either
drop the lower bound (resulting in an AtMost that loses the
all-null-skip) or drop the upper bound (resulting in an AtLeast that
loses the recheck-skip).

3VL semantics are preserved: NULL info lives inside each
NullableRowAddrMask (in its nulls field), and NullableRowAddrMask::Not
flips Allow↔Block without touching nulls, so !{l, u} = {!u, !l}
preserves NULL on both endpoints.

Wire format

The INDEX_EXPR_RESULT_SCHEMA columns change from
(result: Binary, discriminant: UInt32, fragments_covered: Binary) to
(lower: Binary, upper: Binary, fragments_covered: Binary). The schema
is internal to the in-process hand-off from ScalarIndexExec to
FilteredReadExec, so this isn't a persisted-format break.

Other touchpoints

• apply_index_to_fragment in filtered_read.rs: switches from a
  3-arm match to a predicate chain. Existing shapes keep their
  existing read-plan semantics. A non-degenerate Refined case is
  handled conservatively for now (read upper, recheck via
  full_filter); a follow-up will split to_read into
  guaranteed-match ranges (skip recheck) and recheck ranges (apply
  full filter) so the new representation actually pays off.
• MaterializeIndexExec in scalar_index.rs uses result.upper
  for the candidate-row set. AtLeast still bails (upper is
  universe); everything else works.
• New NullableRowAddrMask::allow_nothing() / all_rows()
  constructors for the at_most / at_least builders.
• NullableRowAddrMask gains PartialEq so is_exact() can check
  lower == upper.
• IndexExprResult::{from_parts, serialize_to_arrow} (inherent methods
  on the old enum) → index_expr_result_from_parts(lower, upper) /
  serialize_index_expr_result(result, frags) free functions on
  lance-index::scalar::expression.
• Existing certainty-propagation tests in expression.rs rewritten to
  use the new constructors and is_at_most() / is_at_least()
  predicates. All assertions still hold under the new algebra.

Performance

Bench from lance-select/benches/index_expr_result.rs at N=10M rows.
Both runs at criterion --measurement-time 3 --warm-up-time 2 --sample-size 30, same workload (single contiguous run, AllowList with
no nulls), same machine, idle CPU. Median time per op:

Op	Variant pair	Baseline	2-mask	Delta
!	Exact	8 ns	13 ns	+60%
!	AtMost / AtLeast	8 ns	14 ns	+75%
&	Exact_Exact	7.24 µs	13.84 µs	+91%
&	AtMost_AtMost	7.49 µs	7.32 µs	−2%
&	AtLeast_AtLeast	7.22 µs	7.35 µs	+2%
&	Exact_AtMost	7.15 µs	10.16 µs	+42%
&	Exact_AtLeast	1.63 µs	7.19 µs	+341% (was a drop)
&	AtMost_AtLeast	1.66 µs	1.76 µs	+6%
|	Exact_Exact	3.82 µs	7.54 µs	+97%
|	AtMost_AtMost	3.90 µs	3.87 µs	−1%
|	AtLeast_AtLeast	3.82 µs	4.00 µs	+5%
|	Exact_AtMost	3.67 µs	3.75 µs	+2%
|	Exact_AtLeast	3.74 µs	12.49 µs	+234% (was a drop)
|	AtMost_AtLeast	3.04 µs	11.70 µs	+285% (was a drop)

Reading the deltas:

• NOT: a few extra ns. Was one mask flip; now two endpoint flips +
  swap. Irrelevant in absolute terms.
• Same-variant AtMost_AtMost / AtLeast_AtLeast: roughly
  unchanged. The second mask op composes empty-with-empty or
  universe-with-universe and short-circuits in RowAddrMask::BitAnd.
• Exact_Exact: ~1.9-2× slower. Both endpoints carry the full mask
  (the exact() constructor clones), so we genuinely do two mask ops
  instead of one. Expected.
• Cases that the old algebra resolved by dropping a bound
  (Exact_AtLeast for both ops, AtMost_AtLeast for OR): 3-4× slower
  in absolute terms. The old code returned without doing a mask op;
  the new code computes both endpoints and preserves both bounds.
  Follow-up optimization here is to add BlockList(empty) /
  AllowList(empty) fast paths to NullableRowAddrMask::{BitAnd, BitOr}, which would catch the universe/empty operands directly.

Trade-off: a few extra microseconds per binary op on 10M-row masks (no
impact on query latency — millisecond-scale and dominated by I/O) in
exchange for richer interval results. The Refined case unlocks
acceleration paths that weren't reachable before.

Verification

• cargo test -p lance-select --lib: 99 passed
• cargo test -p lance-index --lib: 311 passed
• cargo test -p lance --lib io::exec::filtered_read: 23 passed
• cargo test -p lance --lib io::exec::scalar_index: 2 passed
• cargo clippy --workspace --tests -- -D warnings: clean
• cargo fmt --all --check: clean

Follow-ups

1. Per-range filter pushdown in filtered_read::apply_index_to_fragment
   so the Refined case can split to_read into definitely-match (skip
   recheck) and recheck ranges (apply full filter) within one fragment.
   That's what makes the new representation actually pay off for IS NOT
   NULL.
2. Empty/universe short-circuit in NullableRowAddrMask::{BitAnd, BitOr}. Detect AllowList(empty_set) and BlockList(empty_set)
   operands and bypass the per-row work. Erases the regressions on
   cross-variant cases.
3. Zone map IsNotNull query that produces a Refined result
   directly, instead of going through Not(IsNull).

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[westonpace]

@claude review once

[claude]

⚠️ Code review skipped — your organization has reached its monthly code review spending cap.

An organization admin can view or raise the cap at claude.ai/admin-settings/claude-code. The cap resets at the start of the next billing period.

Once the cap resets or is raised, comment @claude review on this pull request to trigger a review.

[codecov]

Codecov Report

❌ Patch coverage is 85.96838% with 71 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-select/src/result.rs	78.63%	38 Missing and 9 partials ⚠️
rust/lance/src/io/exec/filtered_read.rs	77.41%	8 Missing and 6 partials ⚠️
rust/lance-index/src/scalar/expression.rs	97.72%	1 Missing and 2 partials ⚠️
rust/lance/src/io/exec/utils.rs	0.00%	3 Missing ⚠️
rust/lance/src/dataset/scanner.rs	89.47%	1 Missing and 1 partial ⚠️
rust/lance/src/io/exec/scalar_index.rs	96.87%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Xuanwo]

Mostly LGTM. We need to mark this PR as breaking changes.

[Xuanwo]

Will this affect our running services while doing upgrading?

[westonpace]

Not anymore. I've updated it so that the wire format is still the same as the old wire format. It can't represent all masks though. We will need to choose the newer wire format at some point.

[wjones127]

Agreed it should be marked as breaking if serialization changes.