Regression in DataFusion 54: filtering a full-range column panics / returns an Internal error

[alexschroth]

Describe the bug

Thanks a lot for all the work on DataFusion! While upgrading from 53 to 54 I ran into what looks like a regression.

Filtering a Timestamp(Nanosecond, None) column whose values span the full i64 range (i64::MIN..i64::MAX) fails during query planning:

Panics in dev profile:

thread 'main' (xxx) panicked at /path/to/datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:947:23:
attempt to add with overflow
stack backtrace:

...

     3: datafusion_expr_common::interval_arithmetic::Interval::cardinality::{{closure}}
               at datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:947:23
     4: core::option::Option<T>::map
               at rust/library/core/src/option.rs:1165:29
     5: datafusion_expr_common::interval_arithmetic::Interval::cardinality
               at datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:947:10
     6: datafusion_expr_common::interval_arithmetic::cardinality_ratio
               at datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:1723:59
     7: datafusion_physical_expr::analysis::calculate_selectivity
               at datafusion-physical-expr-54.0.0/src/analysis.rs:331:24
     8: datafusion_physical_expr::analysis::shrink_boundaries
               at datafusion-physical-expr-54.0.0/src/analysis.rs:253:23
     9: datafusion_physical_expr::analysis::analyze
               at datafusion-physical-expr-54.0.0/src/analysis.rs:217:17
    10: datafusion_physical_plan::filter::FilterExec::statistics_helper
               at datafusion-physical-plan-54.0.0/src/filter.rs:365:32
    11: datafusion_physical_plan::filter::FilterExec::compute_properties
               at datafusion-physical-plan-54.0.0/src/filter.rs:401:21
    12: datafusion_physical_plan::filter::FilterExecBuilder::build
               at datafusion-physical-plan-54.0.0/src/filter.rs:201:21

Returns an error in release profile:

Error: Internal("Assertion failed: (0.0..=1.0).contains(&selectivity): Selectivity is out of limit: inf")

I have also seen the same error with NaN instead of inf in the wild:

Error: Internal("Assertion failed: (0.0..=1.0).contains(&selectivity): Selectivity is out of limit: NaN")

but unfortunately I have not been able to put together a minimal reproducer for the NaN variant.

This worked fine in DataFusion 53.

To Reproduce

Cargo.toml:

[dependencies]
datafusion = { version = "54", default-features = false, features = ["parquet"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

src/main.rs:

use datafusion::arrow::array::TimestampNanosecondArray;
use datafusion::arrow::datatypes::{DataType, Field, Schema, TimeUnit};
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::parquet::arrow::ArrowWriter;
use datafusion::prelude::*;
use datafusion::scalar::ScalarValue;
use std::error::Error;
use std::fs::File;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let parquet_file = "test.parquet";
    let schema = Arc::new(Schema::new(vec![Field::new(
        "ts",
        DataType::Timestamp(TimeUnit::Nanosecond, None),
        true,
    )]));
    let batch = RecordBatch::try_new(
        schema.clone(),
        vec![Arc::new(TimestampNanosecondArray::from(vec![
            i64::MIN,
            i64::MAX,
        ]))],
    )?;

    let mut writer = ArrowWriter::try_new(File::create(parquet_file)?, schema, None)?;
    writer.write(&batch)?;
    writer.close()?;

    let ctx = SessionContext::new();
    ctx.register_parquet("t", parquet_file, Default::default())
        .await?;

    let batches = ctx
        .table("t")
        .await?
        .filter(col("ts").lt_eq(lit(ScalarValue::TimestampNanosecond(
            Some(1_000_000_000_000_000_000),
            None,
        ))))?
        .collect()
        .await?;

    println!(
        "collected {} batch(es), {} row(s)",
        batches.len(),
        batches.iter().map(|b| b.num_rows()).sum::<usize>()
    );

    Ok(())
}

Run it:

$ cargo run
thread 'main' panicked at datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:947:23:
attempt to add with overflow

$ cargo run --release
Error: Internal("Assertion failed: (0.0..=1.0).contains(&selectivity): Selectivity is out of limit: inf")

Expected behavior

The query plans and runs successfully and returns the matching rows, as it does on DataFusion 53:

collected 1 batch(es), 1 row(s)

Additional context

• Reproduced identically on macOS 26.5.1 (arm64) and Ubuntu 24.04 (x86_64)
• Versions: datafusion 54.0.0, arrow 58.3.0, rustc 1.96.1 (2026-06-26), Rust edition 2024

[kumarUjjawal]

Thank you @alexschroth I will take a look.

[kumarUjjawal]

I think this is already fixed on main #22309

[alexschroth]

Thanks a lot for taking a look @kumarUjjawal; great to hear this may already be fixed!

In the meantime I found a minimal reproducer for the NaN variant, adding it here in case it's useful as an extra test case. The trigger is a second full-range column that the filter does not reference:

let schema = Arc::new(Schema::new(vec![
    Field::new("ts", DataType::Timestamp(TimeUnit::Nanosecond, None), false),
    // full-range column NOT referenced by the filter
    Field::new("x", DataType::Int64, false),
]));
let batch = RecordBatch::try_new(
    schema.clone(),
    vec![
        Arc::new(TimestampNanosecondArray::from(vec![i64::MIN, i64::MAX])),
        Arc::new(Int64Array::from(vec![i64::MIN, i64::MAX])),
    ],
)?;

Everything else is identical to the repro in the issue description. With this, cargo run --release gives:

Error: Internal("Assertion failed: (0.0..=1.0).contains(&selectivity): Selectivity is out of limit: NaN")

[qazxcdswe123]

Hi @alexschroth , I've confirmed both the NaN case and the original case are all fixed in main now.

cc @kumarUjjawal

[kumarUjjawal]

Thank you @qazxcdswe123 for looking into this. I will close this issue.