Hashing a CalendarInterval value fails with "Unsupported data type in hasher: Interval(MonthDayNano)"

[andygrove]

Describe the bug

Hashing a CalendarIntervalType value fails the query with a native error instead of either working or falling back to Spark:

org.apache.comet.CometNativeException: Unsupported data type in hasher: Interval(MonthDayNano).
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:155)
	...

#4898 added CalendarIntervalType to QueryPlanSerde.supportedDataType:

datafusion-comet/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala

Lines 530 to 534 in 1d3044f

	_: DecimalType | _: DateType | _: BooleanType | _: NullType | CalendarIntervalType =>
	true
	case dt if isTimeType(dt) =>
	true
	case s: StructType if allowComplex =>

That is the gate hash.scala consults when deciding whether a hash expression's child type is supported:

datafusion-comet/spark/src/main/scala/org/apache/comet/serde/hash.scala

Lines 136 to 147 in 1d3044f

	private def unsupportedReasonFor(dt: DataType): Option[String] = dt match {
	case d: DecimalType if d.precision > 18 => Some(unsupportedDecimalReason)
	case s: StructType =>
	s.fields.iterator.flatMap(f => unsupportedReasonFor(f.dataType).iterator).toSeq.headOption
	case a: ArrayType => unsupportedReasonFor(a.elementType)
	case m: MapType =>
	unsupportedReasonFor(m.keyType).orElse(unsupportedReasonFor(m.valueType))
	case dt if isTimeType(dt) => Some(unsupportedTimeTypeReason)
	case _ if !supportedDataType(dt, allowComplex = true) =>
	Some(s"Unsupported child data type: $dt")
	case _ => None
	}

So Murmur3Hash / XxHash64 over a CalendarInterval child now reports Compatible and is serialized to native, but native/spark-expr/src/hash_funcs/utils.rs has no branch for Interval(MonthDayNano) and falls through to the catch-all DataFusionError::Internal("Unsupported data type in hasher: ...").

The ANSI interval types are not in supportedDataType, so hash(dt_interval) and hash(ym_interval) correctly fall back to Spark. Only CalendarIntervalType reaches native and fails.

Steps to reproduce

Against a Parquet-backed table so the expression is not constant-folded away:

CREATE TABLE t(d int, h int, y int, m int) USING parquet;
INSERT INTO t VALUES (1, 2, 1, 2), (0, 0, 0, 3);

SELECT hash(make_interval(y, m, 0, d, h, 0, 0)) FROM t;

Equivalently, as a sql-file test under spark/src/test/resources/sql-tests/expressions/datetime/.

A OneRowRelation form reproduces it too:

SELECT hash(make_interval(1, 2, 3, 4, 5, 6, 7.008009));

Physical plan at failure time (the whole projection is native):

CometNativeColumnarToRow
+- CometProject [hash(make_interval(1, 2, 3, 4, 5, 6, cast(7.008009 as decimal(18,6)), false), 42) ...]
   +- CometSparkRowToColumnar
      +- *(1) Scan OneRowRelation[]

Expected behavior

The query should return the same values as Spark. Failing that, it should fall back to Spark rather than aborting the stage.

Two possible fixes:

1. Implement Spark-compatible hashing for Interval(MonthDayNano) (and Interval(YearMonth) / Duration(Microsecond) while we are there) in native/spark-expr/src/hash_funcs/utils.rs. Spark's HashExpression hashes a CalendarInterval as its months, days and microseconds fields, so the native side needs to match that field order and width exactly.
2. Or narrow the Scala gate so CalendarIntervalType is rejected for hash expressions, matching how the ANSI interval types behave today, and leave native hashing for the follow-up that implements it properly.

Either way this needs a regression test covering hash and xxhash64 over all three interval types.

Additional context

• Found while auditing interval support after feat: add CalendarIntervalType support #4898 and feat: support interval codegen dispatch for nested values and native shuffle #4976 landed. Part of [EPIC] Support Spark interval types (CalendarInterval / YearMonthInterval / DayTimeInterval) and interval expressions #4540.
• Reproduced on 1d3044f35 with the Spark 4.1 profile.
• QueryPlanSerde.supportedDataType is also consulted by CometScalarSubquery and CometSink, so it is worth checking whether a CalendarInterval-typed scalar subquery hits a similar unhandled-type path in the planner.
• Related gap worth noting separately: native shuffle accepts all three interval types after feat: support interval codegen dispatch for nested values and native shuffle #4976, but supportedHashPartitioningDataType in CometShuffleExchangeExec rejects them, so hash partitioning on an interval key still falls back. That is a fallback rather than a failure, so it is not part of this issue.

[andygrove]

cc @peterxcli

[peterxcli]

take

[peterxcli]

@andygrove Thanks for raising the issue!