ESQL: gate heterogeneous FROM behind a feature flag

[quackaplop]

What this is about

FROM currently lets one query read a dataset and an index-like target (index, alias, data stream) together — a heterogeneous FROM, whether written explicitly (FROM my_index, my_dataset) or produced when a wildcard spans both (FROM logs_*). We want that off in released builds while the shape is still being stabilized, without disturbing queries that read only datasets or only indices.

What this PR does

Adds a feature flag, esql_heterogeneous_from (off by default in released builds, on in snapshot/dev). When it's off, the rule is simply:

A FROM fails if — for what the caller is authorized to read — it resolves to both a dataset and an index-like target. Resolving to only datasets works; resolving to only index-likes works.

• Expression form is never a factor. FROM ds*, exclusions, exact names — all fine. What matters is only what the expression resolves to.
• Any count on either side: one dataset or many, one index or many — a dataset next to any index-like is the mix, and it's rejected.
• The check keys off the caller's authorized view. On a secured cluster, FROM data* where the caller can read the dataset but not an index the wildcard also matched is not a mix — it reads the dataset, isn't rejected, and no error names the unreadable index. (The un-narrowed index branch is still built and then pruned downstream by field-caps security, exactly as the flag-on path does — no index data is read.)
• Data streams count as index-like: the resolve request includes them in its security narrowing, so a dataset mixed with a data stream is rejected too.

When the flag is on (snapshot/dev), heterogeneous FROM behaves exactly as before.

The decision runs in DatasetRewriter.rewriteOne, during dataset resolution — before index field-caps and before any external-source schema discovery. So a query that will be rejected never resolves a dataset schema, and a query that resolves to only indices never builds a dataset branch: no wasted dataset schema I/O in either case.

Why detect against the authorized set

An earlier version keyed the "index" side off the un-narrowed pattern. Adversarial review found that over-rejected on secured clusters: FROM data* where the caller could read only the dataset was rejected, and the 400 named the unreadable index — a false-reject plus an existence-oracle. Detecting the index-like side from the security-narrowed request indices removes both: a name the caller can't see isn't counted and is never named.

Does it work?

• Unit tests in DatasetRewriterTests cover the flag-off paths: a wildcard resolving to both is rejected; a wildcard resolving to only datasets reads them; dataset-only and multi-dataset FROMs resolve; an explicit dataset + index is rejected; on a secured cluster a wildcard spanning a dataset + an unauthorized index is not rejected and leaks nothing; a genuine mix names only the authorized index; and the flag defaults off in release builds.
• The dataset integration suites (FromDatasetIT, FromDatasetSubqueryIT, the External*ITs) run with the flag enabled — the same way the datasets feature flag itself is enabled for tests in build.gradle — so the on behavior is exercised in both snapshot and release-tests.
• Green locally: DatasetRewriterTests, DatasetResolverTests, the flag-on ITs, spotlessApply, and forbiddenApis (main / test / internalClusterTest).

What's in the diff

• DatasetRewriter: the flag constant, the authorized-mix gate in rewriteOne, an authorizedNonDatasetNames field on the resolution computed from the security-narrowed request indices, and package-private overloads that thread the enabled state through for tests.
• EsqlResolveDatasetAction: carry the authorized non-dataset set on the (local-only) response; include data streams in the request's security narrowing.
• DatasetResolver / DatasetRewriterTests / DatasetResolverTests: thread the field and cover the behavior.
• x-pack/plugin/esql/build.gradle: enable the flag for the esql test tasks (alongside the existing dataset flags), so the on behavior is exercised in release-tests.

No wire/transport-version change — the resolve response is local-only.

What's out of scope

A cross-relation mix written as a subquery (FROM idx, (FROM ds)) still runs — those are separate relations, each homogeneous — and the cross-project (CPS) preserved-wildcard path, which is serverless-only and not reachable on a stateful release build. Neither is the wildcard-spanning-both case this targets.

To be backported to 9.5.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[github-actions]

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[julian-elastic]

Looks good