Skip to content

Commit 666bec2

Browse files
committed
Decouple PPL logical plans from UDT temporal types
Logical RelNode/RexNode trees now use standard Calcite DATE/TIME(9)/ TIMESTAMP(9) types for date columns. A new TemporalUdtRewriteShuttle runs at the prepare-statement boundary (OpenSearchCalcitePreparingStmt .implement and OpenSearchRelRunners.run) and rewrites the standard temporal types back to UDTs (ExprDateType/ExprTimeType/ExprTimeStampType) just before physical execution, so Linq4j keeps receiving the VARCHAR-backed representation. This commit folds in the prior "Decouple Calcite PPL planning from ExprType" change as well: Calcite-side PPL planning (RelNode/RexNode code, coercion, type checking, and UDF implementations) operates on RelDataType throughout instead of bouncing through ExprType. Type system: - OpenSearchTypeFactory.convertExprTypeToRelDataType returns standard TIMESTAMP(9)/DATE/TIME(9) for the corresponding ExprCoreType. IP and BINARY remain UDT. - New helper isStandardTemporalType. - PPLOperandTypes constants renamed DATE_UDT/TIME_UDT/TIMESTAMP_UDT to DATE_T/TIME_T/TIMESTAMP_T. - UserDefinedFunctionUtils NULLABLE_*_UDT renamed to NULLABLE_*_T and repointed to standard temporal RelDataTypes (IP and BINARY constants unchanged). - UDTs are recognised via instanceof rather than getExprType(); cloneWith preserves UDT identity through createTypeWithNullability. Cast emission and pushdown: - CalciteRexNodeVisitor.visitCast emits standard temporal types for AST DATE/TIME/TIMESTAMP cast targets. - ExtendedRexBuilder.makeCast routes IP separately; standard temporal targets dispatch to PPLBuiltinOperators.DATE/TIME/TIMESTAMP with the standard target type so the call's RelDataType stays standard. - PredicateAnalyzer.isTimestamp/isDate accept both standard SqlTypeName and UDT, and read literal values via getValueAs(String.class) so TimestampString/DateString round-trip without ClassCastException. Coercion + type-checker: - RelDataType-typed common-type resolver with a CoercionTag widening DAG. - CoercionUtils' DATE+TIME -> TIMESTAMP resolver emits standard TIMESTAMP(9). - PPLTypeChecker gains a temporalKind helper used in typesMatch and isComparable so standard and UDT temporal pairs match across forms. - getRelDataTypes(family) returns standard temporal types for DATETIME/TIMESTAMP/DATE/TIME families (BINARY family stays UDT). - UDF return-type inference (AddSubDate/Weekday/LastDay/TimestampDiff /Format/TimestampAdd/Extract/Span/WidthBucket) recognise both standard and UDT temporal operand types. Shuttle implementation: - Atomic rebuild path for Project/Filter/Calc/Aggregate/Values nodes (Calcite's default copy-then-validate path doesn't work because each half of the rewrite would briefly violate row-type consistency). - Filter rebuild preserves variablesSet so correlated subqueries keep their CorrelationId binding. - TemporalSchemaRewritable marker interface lets OpenSearch table scans rewrite their schema in place rather than wrap in a LogicalProject(CAST), so Calcite Linq4j codegen reads String values matching what OpenSearchExprValueFactory delivers at runtime. Other: - DatetimeUdtNormalizeRule (api/datetime extension) recognises both UDT and standard temporal RexCalls and forces precision to MAX so unified-API consumers see TIMESTAMP(9). - Calcite IT golden files updated to reflect logical plans now showing TIMESTAMP(9) instead of EXPR_TIMESTAMP VARCHAR (physical plans still show UDT post-shuttle). Tests: - New unit: TemporalUdtRewriteShuttleTest, CalcitePPLLogicalPlanStandardTemporalTest. - Updated CoercionUtilsTest, OpenSearchTypeFactoryTest. All module unit tests pass. Full Calcite IT suite (including ExplainIT) green. Signed-off-by: Peng Huo <penghuo@gmail.com>
1 parent 3d4938a commit 666bec2

56 files changed

Lines changed: 1791 additions & 1003 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

api/src/main/java/org/opensearch/sql/api/spec/UnifiedPplSpec.java

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@
1010
import lombok.NoArgsConstructor;
1111
import org.apache.calcite.sql.parser.SqlParser;
1212
import org.apache.calcite.sql.validate.SqlValidator;
13-
import org.opensearch.sql.api.spec.datetime.DatetimeExtension;
1413

1514
/**
1615
* PPL language specification.
@@ -38,6 +37,6 @@ public SqlValidator.Config validatorConfig() {
3837

3938
@Override
4039
public List<LanguageExtension> extensions() {
41-
return List.of(new DatetimeExtension());
40+
return List.of();
4241
}
4342
}

api/src/main/java/org/opensearch/sql/api/spec/UnifiedSqlSpec.java

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@
1616
import org.apache.calcite.sql.parser.babel.SqlBabelParserImpl;
1717
import org.apache.calcite.sql.validate.SqlConformanceEnum;
1818
import org.apache.calcite.sql.validate.SqlValidator;
19-
import org.opensearch.sql.api.spec.datetime.DatetimeExtension;
2019

2120
/**
2221
* SQL language specification. Configures Calcite's parser, validator, and composable extensions for
@@ -52,7 +51,7 @@ public static UnifiedSqlSpec extended() {
5251
SqlConformanceEnum.BABEL,
5352
// CoreExtension and SearchExtension were for Calcite standard SQL pipeline and are
5453
// unused in the V2 ANTLR parser path (CalciteRelNodeVisitor resolves functions directly)
55-
List.of(/* new CoreExtension(), new SearchExtension(), */ new DatetimeExtension()));
54+
List.of(/* new CoreExtension(), new SearchExtension() */ ));
5655
}
5756

5857
@Override

api/src/main/java/org/opensearch/sql/api/spec/datetime/DatetimeExtension.java

Lines changed: 0 additions & 49 deletions
This file was deleted.

api/src/main/java/org/opensearch/sql/api/spec/datetime/DatetimeUdtNormalizeRule.java

Lines changed: 0 additions & 58 deletions
This file was deleted.

api/src/test/java/org/opensearch/sql/api/UnifiedQueryPlannerTest.java

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -171,19 +171,6 @@ public void unsupportedWindowFunctionIsRethrownAsSemanticCheckException() {
171171
.assertErrorMessageContains("Unexpected window function: rank");
172172
}
173173

174-
@Test
175-
public void assertionErrorIsWrappedAsSemanticCheckException() {
176-
// Remove when the underlying Calcite assertion is fixed.
177-
givenInvalidQuery(
178-
"""
179-
source = catalog.employees
180-
| eval ts = timestamp('2024-01-01')
181-
| stats max(ts)
182-
""")
183-
.assertErrorType(SemanticCheckException.class)
184-
.assertErrorMessageEquals("Failed to plan query: invalid plan structure")
185-
.assertCauseType(AssertionError.class);
186-
}
187174

188175
/**
189176
* Without the {@code PATTERN_*} defaults in {@link UnifiedQueryContext}, a bare {@code patterns

api/src/test/java/org/opensearch/sql/api/spec/datetime/DatetimeExtensionSqlTest.java

Lines changed: 0 additions & 90 deletions
This file was deleted.

0 commit comments

Comments
 (0)