diff --git a/release-notes/opensearch-sql.release-notes-3.8.0.0.md b/release-notes/opensearch-sql.release-notes-3.8.0.0.md new file mode 100644 index 00000000000..0ac9430465e --- /dev/null +++ b/release-notes/opensearch-sql.release-notes-3.8.0.0.md @@ -0,0 +1,103 @@ +## Version 3.8.0 Release Notes + +Compatible with OpenSearch and OpenSearch Dashboards version 3.8.0 + +### Features + +* Add explain to AnalyticsExecutionEngine ([#5442](https://github.com/opensearch-project/sql/pull/5442)) +* DATE/TIME label and rendering for date-only / time-only fields ([#5521](https://github.com/opensearch-project/sql/pull/5521)) +* Support PPL timewrap command ([#5241](https://github.com/opensearch-project/sql/pull/5241)) +* Support PPL foreach command ([#5613](https://github.com/opensearch-project/sql/pull/5613)) +* Add PPL `makeresults` command ([#5622](https://github.com/opensearch-project/sql/pull/5622)) +* Xyseries command implementation ([#5343](https://github.com/opensearch-project/sql/pull/5343)) + +### Enhancements + +* Support SQL query profiling with the analytics engine ([#5475](https://github.com/opensearch-project/sql/pull/5475)) +* fix(core): make appendcol row ordering deterministic on parallel engines ([#5474](https://github.com/opensearch-project/sql/pull/5474)) +* Wires `EngineContextProvider` (analytics-engine API) into the SQL plugin for schema resolution ([#5486](https://github.com/opensearch-project/sql/pull/5486)) +* Make addcoltotals/addtotals summary row backend-deterministic ([#5477](https://github.com/opensearch-project/sql/pull/5477)) +* Add `UNION ALL` to the unified query SQL path ([#5506](https://github.com/opensearch-project/sql/pull/5506)) +* Branch percentile and sum-null IT expectations for the analytics-engine route ([#5522](https://github.com/opensearch-project/sql/pull/5522)) +* Fix CalcitePPLAggregationIT on the analytics-engine route (parquet testSimpleCount0 + APPROX_COUNT_DISTINCT name) ([#5525](https://github.com/opensearch-project/sql/pull/5525)) +* Parquet-back raw-PUT test indices on the analytics-engine route ([#5529](https://github.com/opensearch-project/sql/pull/5529)) +* Stabilize order-dependent PPL ITs with explicit sort for multi-shard analytics runs ([#5537](https://github.com/opensearch-project/sql/pull/5537)) +* Support bare-field join criteria `join on ` shorthand ([#5517](https://github.com/opensearch-project/sql/pull/5517)) +* fix(integ): align DateTimeComparisonIT today's date to UTC ([#5543](https://github.com/opensearch-project/sql/pull/5543)) +* Bring CalcitePPLBasicIT to parity on the analytics-engine route ([#5542](https://github.com/opensearch-project/sql/pull/5542)) +* Bring CalciteWhereCommandIT to parity on the analytics-engine route ([#5546](https://github.com/opensearch-project/sql/pull/5546)) +* Classify unsupported-feature errors as client errors (4xx) on the SQL path ([#5569](https://github.com/opensearch-project/sql/pull/5569)) +* Reject unsupported output formats on the analytics-engine route with a 4xx ([#5570](https://github.com/opensearch-project/sql/pull/5570)) +* Fix NPE when rex sits inside appendcol subsearch for Analytic Engine ([#5574](https://github.com/opensearch-project/sql/pull/5574)) +* Fix NPE on case() with incompatible branch types ([#5575](https://github.com/opensearch-project/sql/pull/5575)) +* Add configurable expression depth limit during AST building ([#5602](https://github.com/opensearch-project/sql/pull/5602)) +* feat: add json_tree explain format ([#5576](https://github.com/opensearch-project/sql/pull/5576)) +* Widen narrow integer operands in PPL +/-/* to prevent overflow ([#5603](https://github.com/opensearch-project/sql/pull/5603)) +* Onboard new backport-pr re-usable github workflow (sql) ([#5586](https://github.com/opensearch-project/sql/pull/5586)) +* Support constant_keyword field type in PPL ([#5639](https://github.com/opensearch-project/sql/pull/5639)) +* Decouple Calcite PPL planning from ExprType ([#5633](https://github.com/opensearch-project/sql/pull/5633)) +* Anonymize xyseries command and mark it experimental in docs ([#5643](https://github.com/opensearch-project/sql/pull/5643)) + +### Bug Fixes + +* Publish failure metrics for SQL analytics engine execution path ([#5487](https://github.com/opensearch-project/sql/pull/5487)) +* Exclude AnalyticsEngineCompatIT from the main integTest task ([#5501](https://github.com/opensearch-project/sql/pull/5501)) +* Preserve aggregate aliases for unified query SQL ([#5509](https://github.com/opensearch-project/sql/pull/5509)) +* Skip AnalyticsEngineCompatIT when analytics-engine plugin is absent ([#5510](https://github.com/opensearch-project/sql/pull/5510)) +* Coerce temporal operands in IN and BETWEEN when leastRestrictive finds no common type ([#5513](https://github.com/opensearch-project/sql/pull/5513)) +* Restore dedup pushdown when combined with WHERE clause (#5482) ([#5488](https://github.com/opensearch-project/sql/pull/5488)) +* Key date_time pushdown on field type, not literal UDT (#5481) ([#5515](https://github.com/opensearch-project/sql/pull/5515)) +* Fix SQL aggregate window with ORDER BY defaulting to whole partition ([#5526](https://github.com/opensearch-project/sql/pull/5526)) +* Fix `FILTER(WHERE)` dropped on aggregates in unified SQL path ([#5523](https://github.com/opensearch-project/sql/pull/5523)) +* Fix `SHOW`/`DESCRIBE` statement routing under `cluster.pluggable.dataformat` setting ([#5528](https://github.com/opensearch-project/sql/pull/5528)) +* Fix invalid field or index error misclassified as internal failures ([#5532](https://github.com/opensearch-project/sql/pull/5532)) +* Handle opaque NullPointerException for unresolvable alias-type field path ([#5536](https://github.com/opensearch-project/sql/pull/5536)) +* fix(integ): repair two pre-existing IT failures on main (QueryValidationIT error type, NoMv explain flake) ([#5545](https://github.com/opensearch-project/sql/pull/5545)) +* Strip AE-unsupported fields from test data and exclude the ITs doomed by it ([#5541](https://github.com/opensearch-project/sql/pull/5541)) +* Fix `GROUP BY` expression resolution in `SELECT`/`HAVING`/`ORDER BY` ([#5548](https://github.com/opensearch-project/sql/pull/5548)) +* Propagate request-task cancellation into the analytics PPL route ([#5563](https://github.com/opensearch-project/sql/pull/5563)) +* Return all columns (struct and nested fields) listed when using head ([#5518](https://github.com/opensearch-project/sql/pull/5518)) +* Preserve SQL-layer profiling alongside the analytics-engine profile ([#5571](https://github.com/opensearch-project/sql/pull/5571)) +* allow partial pushdown for semi-scripted predicates ([#5565](https://github.com/opensearch-project/sql/pull/5565)) +* Fix ClassCastException in PPL multisearch on indexes with @timestamp alias field ([#5577](https://github.com/opensearch-project/sql/pull/5577)) +* Return 4xx instead of 500 for unsupported window functions ([#5587](https://github.com/opensearch-project/sql/pull/5587)) +* Fix SQL window functions with `ORDER BY`/`LIMIT` on unified query path ([#5592](https://github.com/opensearch-project/sql/pull/5592)) +* Fix AE `isPluggableDataformatIndex` for multi index PPL queries ([#5581](https://github.com/opensearch-project/sql/pull/5581)) +* fix: Gracefully handle malformed documents in result scanning ([#5618](https://github.com/opensearch-project/sql/pull/5618)) +* fix(dedup): use Map> for fieldNameMapping to handle alias collision ([#5593](https://github.com/opensearch-project/sql/pull/5593)) +* Detect long (BIGINT) arithmetic overflow instead of silently wrapping (#5164) ([#5604](https://github.com/opensearch-project/sql/pull/5604)) +* Fix PPL foreach JSON array type coercion ([#5637](https://github.com/opensearch-project/sql/pull/5637)) + +### Infrastructure + +* Exclude full-text search-filter operator tests on the analytics-engine route ([#5527](https://github.com/opensearch-project/sql/pull/5527)) +* feat(test): add WHERE-prefix view infrastructure for cross-command testing (#5505) ([#5508](https://github.com/opensearch-project/sql/pull/5508)) +* update datetime tests to stay within ae bounds ([#5534](https://github.com/opensearch-project/sql/pull/5534)) +* case test patches ([#5531](https://github.com/opensearch-project/sql/pull/5531)) +* Fix doctest job-scheduler dependency resolution for 3.8.0 ([#5540](https://github.com/opensearch-project/sql/pull/5540)) +* Bring CalciteBinCommandIT and CalciteMultisearchCommandIT to parity on the analytics-engine route ([#5551](https://github.com/opensearch-project/sql/pull/5551)) +* Bring CalcitePPLJoinIT to parity on the analytics-engine route ([#5554](https://github.com/opensearch-project/sql/pull/5554)) +* Bring CalcitePPLEnhancedCoalesceIT to parity on the analytics-engine route ([#5552](https://github.com/opensearch-project/sql/pull/5552)) +* fix(integ): use engine-zone today in DateTimeFunctionIT now()-based assertions ([#5553](https://github.com/opensearch-project/sql/pull/5553)) +* Stabilize subquery PPL ITs on the analytics-engine route ([#5555](https://github.com/opensearch-project/sql/pull/5555)) +* Stabilize CalcitePPLConditionBuiltinFunctionIT on the analytics-engine route ([#5556](https://github.com/opensearch-project/sql/pull/5556)) +* Recover ANY-typed column schema on the analytics route (fixes eval max/min) ([#5557](https://github.com/opensearch-project/sql/pull/5557)) +* Decouple IT from execution backend with capability-based gating ([#5560](https://github.com/opensearch-project/sql/pull/5560)) +* Stabilize PPL ITs on the analytics-engine route (case/string/full-text/like/appendpipe/datatype) ([#5561](https://github.com/opensearch-project/sql/pull/5561)) +* Stabilize PPL ITs on the analytics-engine route (array/map-path/datatype/basic) ([#5562](https://github.com/opensearch-project/sql/pull/5562)) +* Stabilize PPL ITs on the analytics-engine route (percentile/float/datetime/json/dedup/union/rename/chart) ([#5564](https://github.com/opensearch-project/sql/pull/5564)) +* fix: Honor PPL fetch_size on the analytics-engine route ([#5567](https://github.com/opensearch-project/sql/pull/5567)) +* Stabilize more PPL ITs on the analytics-engine route (sort/streamstats/IP-UDT/metadata/strip-verifier) ([#5566](https://github.com/opensearch-project/sql/pull/5566)) +* Stabilize CalciteStreamstatsCommandIT on the analytics-engine route ([#5582](https://github.com/opensearch-project/sql/pull/5582)) +* Bump get-ci-image-tag.yml ref to SHA-pinned opensearch-build commit to unblock CI ([#5583](https://github.com/opensearch-project/sql/pull/5583)) +* Fix SQL IT test queries, assertions, and data for engine-agnostic compatibility ([#5584](https://github.com/opensearch-project/sql/pull/5584)) +* Gate analytics engine incompatible IT tests with capability matrix ([#5585](https://github.com/opensearch-project/sql/pull/5585)) +* Bump Apache Calcite 1.41.0 -> 1.42.0 (CVE-2026-46718) ([#5619](https://github.com/opensearch-project/sql/pull/5619)) +* Make HTTP status assertion protocol-independent ([#5623](https://github.com/opensearch-project/sql/pull/5623)) + +### Maintenance + +* Fix transpose VARCHAR length drift in unpivot/pivot lowering ([#5479](https://github.com/opensearch-project/sql/pull/5479)) +* Increment version to 3.8.0-SNAPSHOT ([#5498](https://github.com/opensearch-project/sql/pull/5498)) +* fix lychee ([#5451](https://github.com/opensearch-project/sql/pull/5451)) +* Fix flaky TPC-H Q15 floating-point assertion ([#5629](https://github.com/opensearch-project/sql/pull/5629))