feat: support interval types and make_ym_interval / make_dt_interval

[andygrove]

Which issue does this PR close?

Part of #4540.

Rationale for this change

Comet had no support for Spark's interval data types, so every expression producing or consuming an interval forced a fallback to Spark, and the JVM codegen dispatcher (CometCodegenDispatch) could not cover them either because CometBatchKernelCodegen.isSupportedDataType rejected interval output. This PR adds the foundational type support and uses the existing codegen-dispatch mechanism to make the interval constructors run natively while matching Spark exactly.

What changes are included in this PR?

Type support for YearMonthIntervalType and DayTimeIntervalType:

• native/proto/src/proto/types.proto: add YEAR_MONTH_INTERVAL (18) and DAY_TIME_INTERVAL (19) to DataTypeId.
• native/core/src/execution/serde.rs: map them to Arrow Interval(YearMonth) and Duration(Microsecond).
• Utils.toArrowType / fromArrowType: same mapping on the JVM side.
• QueryPlanSerde.serializeDataType: emit the new type ids.
• CometBatchKernelCodegen: accept both interval types in isSupportedDataType, resolve IntervalYearVector / DurationVector, and emit primitive set() writes.

Representation note: Spark stores DayTimeIntervalType as microseconds in an int64, which round-trips faithfully through Arrow Duration(Microsecond). Arrow Interval(DayTime) uses a {days, millis} layout that would lose microsecond precision, so it is intentionally not used.

Expressions (via CometCodegenDispatch, registered in temporalExpressions):

• make_ym_interval -> CometMakeYMInterval
• make_dt_interval -> CometMakeDTInterval

Out of scope (follow-ups tracked in #4540): CalendarIntervalType, and interval arithmetic / multiply_*_interval / extract of interval fields.

How are these changes tested?

SQL file tests under spark/src/test/resources/sql-tests/expressions/datetime/ (make_ym_interval.sql, make_dt_interval.sql) run by CometSqlFileTestSuite. The query mode uses checkSparkAnswerAndOperator, asserting both answer parity with Spark and native execution (a fallback fails the test). Cases cover column and literal inputs, default arguments, negatives, and nulls. The full SQL suite shows no new regressions.

[mbutrovich]

Mostly just a test request, this is looking good @andygrove!

[mbutrovich]

Verified correct, no change needed for this PR. The representation choices match Spark's internal storage exactly: YearMonthIntervalType is int32 months, which maps cleanly to Interval(YearMonth); DayTimeIntervalType is an int64 microsecond count, which round-trips faithfully through Duration(Microsecond). The comment correctly notes that Interval(DayTime)'s {days, millis} layout would lose microsecond precision, so avoiding it is right.

One forward-looking note, not a blocker. Both mappings drop Spark's interval field qualifiers. YearMonthIntervalType(YEAR, YEAR) and YearMonthIntervalType(YEAR, MONTH) both serialize to the same Interval(YearMonth), and on the way back fromArrowType produces the default YearMonthIntervalType() (YEAR TO MONTH); same story for DayTimeIntervalType start/end fields. This is harmless here because the only expressions wired are make_ym_interval and make_dt_interval, both of which Spark types with the default fields (YearMonthIntervalType(), DayTimeIntervalType()), and interval columns are not scanned yet. But once #4540 adds interval scans or field-qualified interval handling, this normalization will change a column's declared type. Worth a tracking note so it is not lost.

[mbutrovich]

The coverage of column, literal, default, negative, and null inputs is good, and using checkSparkAnswerAndOperator (the default query mode) correctly asserts native execution and exercises the interval read-back path.

One gap worth filling: overflow. make_ym_interval computes years * 12 + months through IntervalUtils.makeYearMonthInterval, which uses exact arithmetic and throws ARITHMETIC_OVERFLOW unconditionally (not ANSI-gated); make_dt_interval overflows its int64 microsecond total similarly. Since these route through the codegen dispatcher (Spark's own doGenCode), an overflow case is a good way to confirm the dispatched path propagates Spark's exception identically rather than diverging. Something like:

query expect_error(ARITHMETIC_OVERFLOW)
SELECT make_ym_interval(178956971, 0)

Please confirm the exact error token the suite matches. The value 178956971 * 12 exceeds Int.MaxValue, so both engines should raise the same overflow.

[andygrove]

Thanks @mbutrovich. I added overflow tests

[mbutrovich]

Observation, no change requested. fromArrowType now maps both Interval(DayTime) (line 110) and Duration(Microsecond) (line 111) to DayTimeIntervalType, while toArrowType only ever emits Duration(Microsecond). So the Interval(DayTime) reverse case is dead for Comet-produced data and is presumably kept for externally-produced Arrow. That is fine and consistent with the precision rationale, just flagging the asymmetry in case it was unintentional.

[mbutrovich]

Thanks @andygrove!

[andygrove]

Merged. Thanks @mbutrovich

[peterxcli]

@andygrove I notice that shuffle and array or nested support for YearMonthInterval and DayTimeInterval hasn't done yet.

26/07/20 17:26:13 WARN CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
CollectLimit
+- Project
   +-  Exchange [COMET: unsupported shuffle data type YearMonthIntervalType(0,1) for input i#11, Comet columnar shuffle not enabled]
      +- CometProject
         +- CometNativeScan parquet

+-  LocalTableScan [COMET: Unsupported array element of type YearMonthIntervalType(0,1)]