Skip to content

fix: accept LargeUtf8 and Utf8View patterns in SIMILAR TO planning - #23735

Merged
adriangb merged 3 commits into
apache:mainfrom
u70b3:similar-to-pattern-types
Jul 29, 2026
Merged

fix: accept LargeUtf8 and Utf8View patterns in SIMILAR TO planning#23735
adriangb merged 3 commits into
apache:mainfrom
u70b3:similar-to-pattern-types

Conversation

@u70b3

@u70b3 u70b3 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Follow-up to #23704, which made the TypeCoercion analyzer coerce SIMILAR TO operands to a common string type. The SQL planner still rejects LargeUtf8 and Utf8View patterns before the analyzer ever runs:

SELECT s FROM test, patterns WHERE s SIMILAR TO arrow_cast(pat, 'Utf8View');
-- error: Invalid pattern in SIMILAR TO expression

Now that the analyzer coerces both operands, this planner restriction is unnecessary (and inconsistent with LIKE, whose planner path has no such check).

What changes are included in this PR?

The planner relaxation and tests were originally part of #22887; this PR lands the remaining piece of it.

Are these changes tested?

Yes — new sqllogictest cases in type_coercion.slt. The existing planner rejection of non-string patterns (SELECT 'a' SIMILAR TO 1) still errors as before.

Are there any user-facing changes?

Yes: SIMILAR TO queries with non-literal LargeUtf8 / Utf8View patterns that previously failed during SQL planning with Invalid pattern in SIMILAR TO expression now plan and execute successfully. No breaking API changes.

@codecov-commenter

codecov-commenter commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.69%. Comparing base (994fc81) to head (9e61c37).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23735      +/-   ##
==========================================
- Coverage   80.69%   80.69%   -0.01%     
==========================================
  Files        1095     1095              
  Lines      372626   372622       -4     
  Branches   372626   372622       -4     
==========================================
- Hits       300700   300690      -10     
- Misses      53978    53979       +1     
- Partials    17948    17953       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@u70b3
u70b3 force-pushed the similar-to-pattern-types branch from f9061a3 to 69f531e Compare July 21, 2026 06:09
SELECT t.s NOT SIMILAR TO arrow_cast(p.pat, 'Utf8View') FROM t CROSS JOIN p;
----
false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add tests for teh null case?

@u70b3 u70b3 Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in add3f8d — added a literal NULL pattern in the table context and a non-scalar Null-typed pattern column (CREATE TABLE pn AS SELECT NULL AS pat), both evaluate to NULL.

@u70b3
u70b3 force-pushed the similar-to-pattern-types branch 2 times, most recently from 7738741 to 21f48ec Compare July 22, 2026 12:02

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@u70b3
Thanks for working on this.
This is moving in the right direction, but I think there is still one blocking issue before we can remove the planner side type gate for SIMILAR TO.

Comment thread datafusion/sql/src/expr/mod.rs Outdated
// The analyzer coerces both operands to a common string type, so any
// string-typed (or untyped NULL) pattern is accepted here.
if !matches!(
pattern_type,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for expanding this to LargeUtf8 and Utf8View. I think we can go a bit further here though.

The planner still rejects dictionary encoded string patterns, so we still have the same duplicated planner and analyzer type gate that this PR is trying to remove. regex_coercion already handles dictionary string operands via dictionary_coercion, and it also owns the broader string-like coercion behaviour, such as REE when constructible.

Unlike SIMILAR TO, LIKE does not have an equivalent planner side check. As a result, this still fails during planning:

SELECT t.s SIMILAR TO arrow_cast(p.pat, 'Dictionary(Int32, Utf8)')
FROM t CROSS JOIN p;

with Invalid pattern in SIMILAR TO expression.

Could we let the analyzer own SIMILAR TO operand coercion and validation entirely, or otherwise accept all of the string-like types that it already supports? It would also be great to add a regression test for a dictionary encoded pattern.

----
true

# non-scalar Utf8View / LargeUtf8 patterns are accepted by the SQL planner and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small suggestion. Could we move the issue URL into the section heading instead of only this subsection comment? The existing Utf8View and LargeUtf8 value cases and these new pattern cases are all exercising the same SIMILAR TO coercion invariant, so it seems nice to have the issue reference apply to the whole section.

@u70b3

u70b3 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review! I've removed the planner-side type gate so the analyzer now owns SIMILAR TO coercion and validation, added dictionary-pattern coverage, and moved the issue reference to the section heading. Fixed in a209834.

@u70b3
u70b3 force-pushed the similar-to-pattern-types branch from a209834 to b065774 Compare July 28, 2026 11:24
u70b3 added 3 commits July 28, 2026 22:28
Closes apache#23732.

Follow-up to apache#23704: now that the TypeCoercion analyzer coerces
SIMILAR TO operands to a common string type, the SQL planner no
longer needs to reject LargeUtf8/Utf8View patterns (consistent
with LIKE, which has no such check).
@u70b3
u70b3 force-pushed the similar-to-pattern-types branch from b065774 to 9e61c37 Compare July 28, 2026 14:28

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@u70b3
Thanks for the iteration.

Looks 👍 to me

@adriangb
adriangb added this pull request to the merge queue Jul 29, 2026
@adriangb

Copy link
Copy Markdown
Contributor

thanks folks!

Merged via the queue into apache:main with commit 5f7feda Jul 29, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sql SQL Planner sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SIMILAR TO: SQL planner rejects non-literal LargeUtf8 / Utf8View patterns

4 participants