Add interval mul/div support for Float64 - #10409
Conversation
|
Thanks - I'll try and find time for this but I need to focus on performance and bugfixes first |
TIA. If you need any help on other "performance and bugfixes" issue, please let me know, I would like to take a stab at them 😄 |
We are always looking for help reviewing other PRs (ensure test coverage, make sure you can unerstand the change, see if you can find any logic holes, etc) 🙏 |
| factor_scalar, | ||
| { | ||
| // Float scaling can create components that narrower interval units cannot store. | ||
| let interval = T::to_month_day_nano(interval); |
There was a problem hiding this comment.
i dont think we should have this conversion logic in arrow-rs; we should probably support only IntervalMonthDay explicitly and expect callers (e.g. datafusion) to coerce to this type themselves if they want to use this kernel. i think that has been the precedent for other kernels in arrow-rs
There was a problem hiding this comment.
Thanks. I agree with caller-side coercion for the numeric operand: callers can coerce multiplication and division factors to Int64 or Float64, so arrow-arith does not need to support every numeric type.
I’m less clear that the same precedent requires coercing the interval representation. #10336 directly supports Interval(YearMonth), Interval(DayTime), and Interval(MonthDayNano) for multiplication by Int64; this PR follows the same input coverage. For Float64, the result must be MonthDayNano because fractional months and days can cascade into smaller components, and widening YearMonth/DayTime to MonthDayNano is lossless.
If callers should also normalize the interval representation, should the same rule apply to #10336, or is the distinction that integer multiplication preserves the input representation while floating-point arithmetic widens it?
Could you also point me to a specific kernel or PR establishing this precedent for coercion between related Arrow types? The precedent discussed in #6906 appears to concern heterogeneous numeric operands; limiting factors to Int64 or Float64 already follows that boundary.
There was a problem hiding this comment.
since i64 multiplication preserves input type, its not doing any coercion so its fine as is. f64 requires converting to monthdaynano so this is something the caller should cast to instead of us casting inside an arithmetic kernel
Could you also point me to a specific kernel or PR establishing this precedent for coercion between related Arrow types? The precedent discussed in #6906 (comment) appears to concern heterogeneous numeric operands; limiting factors to
Int64orFloat64already follows that boundary.
the precedent is just the existing codebase as a whole; im not aware of any arithmetic kernels that cast the input types (apart from things like decimal widening, etc.) though i may be wrong
just in general, this type of input casting is best left to the caller to opt in to, and at the arrow-rs level its simpler to just support the exact types we actually support (in this cast f64 * monthdaynano)
There was a problem hiding this comment.
since i64 multiplication preserves input type, its not doing any coercion so its fine as is. f64 requires converting to monthdaynano so this is something the caller should cast to instead of us casting inside an arithmetic kernel
just in general, this type of input casting is best left to the caller to opt in to, and at the arrow-rs level its simpler to just support the exact types we actually support (in this cast f64 * monthdaynano)
sounds fair, let me remove the conversion and only keep the Interval(MonthDayNano) * f64 input. Thanks for the explaination!
Jefffrey
left a comment
There was a problem hiding this comment.
could we keep the PR body up to date as well; it still refers to supporting yearmonth/daytime
| )) | ||
| } | ||
|
|
||
| /// Multiplies using `IntervalMonthDayNano` as the common interval representation, |
There was a problem hiding this comment.
this docstring is out of date; no need to mention "common interval representation" since its only defined for monthdaynano intervals
| // DuckDB defines interval division as multiplication by the reciprocal: | ||
| // https://github.com/duckdb/duckdb/blob/21aca0424f1faf78b593b1e6fbfdd4846624c987/src/function/scalar/operator/arithmetic.cpp#L1102-L1110 | ||
| Op::Div => interval_mul_f64(interval, 1. / factor), | ||
| _ => unreachable!(), |
There was a problem hiding this comment.
this unreachable!() seems reachable in normal usage (if we use add(mdn, f64) for example)
There was a problem hiding this comment.
changed to throw exception!
| if !months_product.is_finite() | ||
| || months_product < f64::from(i32::MIN) | ||
| || months_product > f64::from(i32::MAX) | ||
| { | ||
| return Err(overflow("months")); | ||
| } | ||
| let months = months_product.to_i32().ok_or_else(|| overflow("months"))?; |
There was a problem hiding this comment.
does to_i32() not check the range and non-finite numbers when converting? trying to understand why we have two checks here
There was a problem hiding this comment.
If only relying on to_i32(), i32::MIN * 1.000_000_000_4 truncates to valid i32::MIN, so it no longer overflow.
arrow-rs/arrow-arith/src/numeric.rs
Lines 1931 to 1940 in 8393e14
| } | ||
| let mut days = days_product.to_i32().ok_or_else(|| overflow("days"))?; | ||
|
|
||
| let month_remainder = timestamp_round((months_product - f64::from(months)) * DAYS_PER_MONTH); |
There was a problem hiding this comment.
could we use f64::fract()` here instead?
There was a problem hiding this comment.
good catch, makes it cleaner!
Thanks for the heads-up! |
Which issue does this PR close?
Rationale for this change
#10336 added interval multiplication by
Int64. Floating-point multiplication and division require preserving calendar components rather than flattening the entire interval into a duration.This PR adapts DuckDB's
INTERVAL * DOUBLEbehavior. Whole months and days remain calendar components, while fractional months cascade to days using 30 days per month and fractional days cascade to nanoseconds using 24 hours per day.DuckDB explicitly attributes its implementation to PostgreSQL's
interval_mul:What changes are included in this PR?
Add checked arithmetic for:
Interval(MonthDayNano) * Float64(communicative)Interval(MonthDayNano) / Float64All results use
Interval(MonthDayNano).Are these changes tested?
Float64factors, preserving exact nanosecondsInterval / Int64,Float64 / Interval, wrapping multiplication, and additional numeric factor types are not added.Are these changes tested?
Yes. Tests cover all three interval units, both multiplication operand orders, division, array and scalar operands, null propagation, component cascading, integral-factor precision, negative values, ties-to-even rounding, non-finite values, division by zero, and overflow.
cargo fmt --all -- --check cargo test -p arrow-arith cargo clippy -p arrow-arith --all-targets --all-features -- -D warningsAre there any user-facing changes?
Yes. The checked
mulanddivkernels now accept the combinations listed above. Floating-point interval arithmetic always returnsInterval(MonthDayNano).There are no public API signature changes.