Skip to content

fix: generate_series overflow panics at i64 boundary and out-of-range dates - #23723

Open
u70b3 wants to merge 2 commits into
apache:mainfrom
u70b3:fix/generate-series-overflow
Open

fix: generate_series overflow panics at i64 boundary and out-of-range dates#23723
u70b3 wants to merge 2 commits into
apache:mainfrom
u70b3:fix/generate-series-overflow

Conversation

@u70b3

@u70b3 u70b3 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Supersedes #22250 (closed unmerged by its author on 2026-06-04 after the approach was approved). Note: both issues are currently assigned to @xiedeyantu (idle since 2026-05; #22193 was explicitly blocked on #22250). I've left a note on both issues; happy to coordinate if the assignee has work in progress.

Rationale for this change

Two overflow bugs in the generate_series / range table functions, both panicking in debug builds and silently wrapping in release:

  1. panic: generate_series table function overflows when integer step passes i64::MAX #22208 — integer series: SELECT * FROM generate_series(9223372036854775806, 9223372036854775807, 2) panics with attempt to add with overflow when advancing past i64::MAX. PostgreSQL and DuckDB both return one row (9223372036854775806), stopping after the last reachable value.
  2. panic: multiply overflow in generate_series(DATE, DATE, INTERVAL) for dates far from 1970 #22193 — date series: converting Date32 days to timestamp nanoseconds uses an unchecked multiplication, so generate_series(DATE '0001-01-01', ...) panics at planning time. Dates outside the nanosecond timestamp range (1677-09-21 – 2262-04-11) cannot be represented and must error cleanly instead.

What changes are included in this PR?

  • datafusion/functions-table/src/generate_series.rs
    • SeriesValue::advance now takes end: &mut Self; the i64 implementation uses checked_add and, on overflow, clamps end so the series terminates after the last reachable value (the approach approved in fix generate_series table function overflows #22250). The per-batch loop stops right after emitting the final value. TimestampValue::advance already errors on out-of-range interval addition and is unchanged apart from the signature.
    • The Date32→nanoseconds conversion uses checked_mul and returns a planning error naming the offending argument.
  • datafusion/sqllogictest/test_files/table_functions.slt
    • Regression cases: positive/negative step overflow, landing exactly on i64::MAX, range (end-exclusive) overflow, and first/second date argument out of range.

Are these changes tested?

Yes:

  • New sqllogictest cases (exact reproducers from both issues, plus boundary variants); verified the integer-case outputs match PostgreSQL/DuckDB.
  • Verified cargo fmt, workspace clippy (avro,integration-tests,extended_tests, -D warnings), cargo test -p datafusion-functions-table, and the extended workspace suite (ci profile with avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption) — all green.

Are there any user-facing changes?

Only the bug fixes: integer series near i64::MAX/MIN now return the reachable values instead of panicking or wrapping (matching PostgreSQL/DuckDB), and out-of-range dates produce a planning error instead of a panic. No API changes.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Jul 20, 2026
@u70b3
u70b3 force-pushed the fix/generate-series-overflow branch from 00ccd1e to bb38d8e Compare July 21, 2026 01:09
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-functions-table v54.1.0 (current)
       Built [  56.780s] (current)
     Parsing datafusion-functions-table v54.1.0 (current)
      Parsed [   0.013s] (current)
    Building datafusion-functions-table v54.1.0 (baseline)
       Built [  46.344s] (baseline)
     Parsing datafusion-functions-table v54.1.0 (baseline)
      Parsed [   0.013s] (baseline)
    Checking datafusion-functions-table v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.106s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_parameter_count_changed: pub trait method parameter count changed ---

Description:
A trait method now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-item-signature
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/trait_method_parameter_count_changed.ron

Failed in:
  SeriesValue::advance now takes 2 instead of 1 parameters, in file /home/runner/work/datafusion/datafusion/datafusion/functions-table/src/generate_series.rs:88

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 104.683s] datafusion-functions-table
    Building datafusion-sqllogictest v54.1.0 (current)
       Built [ 193.741s] (current)
     Parsing datafusion-sqllogictest v54.1.0 (current)
      Parsed [   0.024s] (current)
    Building datafusion-sqllogictest v54.1.0 (baseline)
       Built [ 195.529s] (baseline)
     Parsing datafusion-sqllogictest v54.1.0 (baseline)
      Parsed [   0.025s] (baseline)
    Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.117s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 392.378s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 21, 2026
@codecov-commenter

codecov-commenter commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.80328% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.69%. Comparing base (1b32517) to head (eb22e6a).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/functions-table/src/generate_series.rs 91.80% 1 Missing and 4 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #23723   +/-   ##
=======================================
  Coverage   80.69%   80.69%           
=======================================
  Files        1095     1095           
  Lines      372623   372678   +55     
  Branches   372623   372678   +55     
=======================================
+ Hits       300695   300740   +45     
- Misses      53979    53981    +2     
- Partials    17949    17957    +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@u70b3
u70b3 force-pushed the fix/generate-series-overflow branch from bb38d8e to 53ef65a Compare July 22, 2026 12:02

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@u70b3
Thanks for working on this. I found one boundary case that still needs to be addressed before this is ready to merge. I also left a suggestion for additional regression coverage.

.current
.should_stop(self.end.clone(), &self.step, false)
{
self.current.advance(&mut self.end, &self.step)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the generator still advances after emitting the inclusive end value. For date and timestamp series near the supported upper bound, this can turn a valid one-row query into an error.

For example:

SELECT * FROM generate_series(
  DATE '2262-04-11',
  DATE '2262-04-11',
  INTERVAL '1' DAY
);

This should return the single reachable row, but it currently fails with Failed to add interval ... to timestamp 9223286400000000000.

Could we apply the same terminal-state rule used for integer overflow and avoid calling advance once the current value is the last value to emit? One option would be to have advance report whether a next value exists. Another would be to break before advancing when the exclusive stop condition has already been reached.

----
9223372036854775806

# Regression test for https://github.com/apache/datafusion/issues/22193

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be helpful to add positive boundary tests for the maximum representable date and timestamp, alongside the existing out-of-range error cases. That would lock in the expected behaviour of returning all reachable values at the edge of the nanosecond range.

@u70b3

u70b3 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @kosiew! Fixed the terminal boundary handling so generate_series no longer advances after emitting the inclusive end value, and added regression tests for the maximum representable date and nanosecond timestamp.

u70b3 added 2 commits July 28, 2026 19:24
… dates

Two overflow fixes in the generate_series/range table functions:

- Integer series: advancing past i64::MAX (or i64::MIN for negative
  steps) panicked in debug builds and silently wrapped in release.
  advance() now uses checked_add and clamps the series end so iteration
  stops after the last reachable value, matching PostgreSQL/DuckDB
  (e.g. generate_series(9223372036854775806, 9223372036854775807, 2)
  returns one row 9223372036854775806).

- Date series: converting Date32 days to timestamp nanoseconds used an
  unchecked multiplication, panicking at planning time for dates outside
  the nanosecond timestamp range. It now returns a planning error.

Supersedes apache#22250 (closed unmerged); the integer-side approach follows
the one approved there.

Closes apache#22208, closes apache#22193.
@u70b3
u70b3 force-pushed the fix/generate-series-overflow branch from 1ad77f4 to eb22e6a Compare July 28, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

3 participants