Skip to content

[ES|QL] Multi-column IN subquery (behind snapshot)#155011

Draft
fang-xing-esql wants to merge 1 commit into
elastic:mainfrom
fang-xing-esql:in-subquery-multi-column
Draft

[ES|QL] Multi-column IN subquery (behind snapshot)#155011
fang-xing-esql wants to merge 1 commit into
elastic:mainfrom
fang-xing-esql:in-subquery-multi-column

Conversation

@fang-xing-esql

Copy link
Copy Markdown
Member

Resolves: https://github.com/elastic/esql-planning/issues/1116

Extends WHERE ... IN (subquery) to support matching against a tuple of columns:

FROM employees
| WHERE (emp_no, salary) IN (FROM employees | SORT emp_no ASC | LIMIT 3 | KEEP emp_no, salary)

FROM employees
| WHERE (first_name, last_name) NOT IN (FROM employees | SORT emp_no ASC | LIMIT 3 | KEEP first_name, last_name)

The subquery must return exactly as many columns as there are values on the left-hand side, and types must be compatible.

Major Changes

Parsing: LogicalPlanBuilder produces a MultiColumnInSubquery expression node (parallel to the existing InSubquery for single-column).

Resolution. InSubqueryResolver rewrites MultiColumnInSubquery into the same join shapes as single-column:

  • Top-level AND conjunct → SemiJoin (IN) or AntiJoin (NOT IN)
  • Inside an OR / NOT-above-OR → MarkJoin with a synthetic boolean mark attribute

Execution. AbstractSubqueryJoin.inlineDataMultiColumn handles the multi-column path once the subquery result is available as a LocalRelation:

  • Filter path (small right side): builds an OR-of-AND-Equals expression covering all deduplicated right-side tuples and rewrites to Filter.
  • Hash-join path (large right side): builds a multi-column LEFT join with per-field MvSingleValueOrNull guards and a sentinel column.
  • For ANTI with any NULL-containing right tuples, the filter path is forced to rely on Equals three-valued semantics rather than the hash-join path (which can't safely handle NULLs in multi-column equality matching).
  • NULL / MV semantics mirror single-column IN: multi-valued positions on the right are collapsed to NULL via per-column BlockHash dedup; the all-NULL short-circuit and ANTI-on-any-NULL short-circuit both apply.

@github-actions

Copy link
Copy Markdown
Contributor

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

@github-actions

Copy link
Copy Markdown
Contributor

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant