[WIP] EQL source command in ES|QL — rework of abatsakis#1#2
Closed
quackaplop wants to merge 3679 commits into
Closed
[WIP] EQL source command in ES|QL — rework of abatsakis#1#2quackaplop wants to merge 3679 commits into
quackaplop wants to merge 3679 commits into
Conversation
Fix one cause of losing esql's warnings by preserving thread context when we shift to a new thread.
## Problem When a KNN query filters on a `constant_keyword` field (or any field whose query rewrites to `MatchAllDocsQuery`), the filter was being materialized as a full-index `FixedBitSet` via `CachingEnableFilterQuery(ConstantScore(*:*))`. Root cause is `buildFilterQuery` added every filter `Query` to a `BooleanQuery(FILTER)` without checking whether any of them was a `MatchAllDocsQuery`. Lucene's `BooleanQuery.rewrite()` then collapsed the single-clause bool to `ConstantScoreQuery(MatchAllDocsQuery)`, which `CachingEnableFilterQuery.rewrite()` does not strip (it only strips bare `MatchAllDocsQuery`). ## Fix Skip `MatchAllDocsQuery` instances in `buildFilterQuery`. They contribute no selectivity in a conjunction and forcing their materialization is pure overhead. ## Side note While investigating we found that `SemanticMatchQueryRewriteInterceptor` wraps **all** `match` queries in `InterceptedQueryBuilderWrapper` unconditionally, which silently defeats `instanceof MatchNoneQueryBuilder` short-circuit checks in `KnnSearchBuilder`, `BoolQueryBuilder`, and others. Results remain correct but early-exit optimizations are bypassed. Not addressed here, but will look at creating a separate issue to check on the implementation. These `instanceof` checks feel quite brittle, but in this case, we're able to add sufficient test coverage that I feel comfortable adding this optimization. ## Test `AbstractKnnVectorQueryBuilderTestCase.testMatchAllFilterIsDropped`: asserts that adding a `MatchAllQueryBuilder` as the sole filter produces an identical Lucene query to having no filter, across all index types, as well as a couple other cases.
These issues are closed, but the mutes remain. So remove the mutes.
This fixes an issue where, specifying the "format" parameter, but suppliing an (outter) compressed file would result in a failure, caused by the resolver providing an uncompressed reader.
…-datasource-* tests (elastic#153518) Converts 72 leaf test classes in the x-pack ESQL ecosystem (esql/compute, esql-datasource-csv/grpc/gcs/ndjson/orc/parquet/parquet-rs/s3) from JUnit3-style setUp()/tearDown() overrides to @Before/@after annotations, dropping the now-redundant super calls. Part of the broader effort to make ESTestCase.setUp()/tearDown() final.
… test {csv-spec:spatial_shapes.convertFromStringParseError} elastic#154157
* Fix BUCKET with bucket count > MAX_INT * Update docs/changelog/153392.yaml * testcase with MAX_LONG buckets * improve code * extra testcases * [CI] Auto commit changes from spotless * fix * Fix slow bucket counting across time zones * [CI] Auto commit changes from spotless * add test that's slow in main * heuristic to prevent too many iterations * prevent overflows * add missing capabilities to tests * fix heuristic to not overshoot * lower fudge factor to 30 days + improve comments * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
…c#154123) ## What this is about elastic#153923 merged to main carrying only the 9.6 id (`9466000`) for the `declared_read_spec_provenance` transport version. It is being backported to 9.5, which needs a 9.5-line id registered on main too. ## Why on main, not just the backport The `supports()` gate resolves the TV id from each node's own definition. If main's definition stayed single-id `9466000` while the 9.5 branch gained a 9.5-line id, a 9.5↔9.6 mixed cluster would disagree on the gate — the 9.5 node writes the provenance field, the 9.6 node does not read it — and corrupt the stream. Registering the 9.5 id on main lets a 9.6 node gate correctly against a 9.5 peer. ## What's in the diff Two transport-version registry files, generated by `./gradlew generateTransportVersion --name=declared_read_spec_provenance --backport-branches=9.5` — no code: - `definitions/referable/declared_read_spec_provenance.csv`: `9466000` → `9466000,9458002` - `upper_bounds/9.5.csv`: bumped to `declared_read_spec_provenance,9458002` Merge this before the 9.5 backport (elastic/elasticsearch backport of elastic#153923).
`FromDatasetIT#testExplicitFormatOverGzipCsvReads` was calling `assertStrictGzippedTextReads` with 4 arguments; the helper needs 6 (`tsPath`, `notePath`).
This broadens the skipping of the test on network failure, when fetching test data in ParquetTestingIT.
This test failure was resolved by commit 090341e. Specifically, the threadpool assertion in tryPrefetch needed to include the SHARD_READ_THREAD_POOL thread pool. Closes elastic#150028
* ES|QL HIGHLIGHT: full-text query semantics This PR evolves ES|QL's snapshot-only `HIGHLIGHT` command from a simple bag-of-words highlighter into something closer to Query DSL behavior. It adds support for: - Lucene `query_string` parsing for literal queries: quoted phrases, `AND`/`OR`/`NOT`, prefix/leading/multi wildcards, `AUTO` fuzzy `~`, case-sensitive regex, field-qualified terms, ranges, and parenthesized grouping. - Full-text function expressions as the query: `MATCH`, `MATCH_PHRASE`, `QSTR`, `KQL`, and `:`, combined with `AND`/`OR`/`NOT`. - Query DSL options supported by those functions, including `boost`, `operator`, `fuzziness`, `prefix_length`, `max_expansions`, `fuzzy_transpositions`, `minimum_should_match`, `slop`, and `default_field`. - Per-field highlighting using the `ON` field names, with `require_field_match` seman
… mismatch (elastic#153523) Fixes four mis-pruning shapes in ParquetPushedExpressions
Now that elastic/kibana#278580 is merged, it's safe to remove Kibana's access to the old prefix. Devs will be able to resolve permission issues by pulling latest ES and Kibana.
Point folks in the right direction when editing capabilities.
This renames an attribute on various recovery-related metrics from `primary` to `es_is_primary`, to follow ES OTel naming conventions. (N.B. The validation requires at least two underscores, so `es_primary` is not allowed. The use of `is` is intended to indicate that it is binary.) See also elastic#153526 which did the same for `recovery_type`. Note that this PR does _not_ change the name of the `primary` attribute on metrics related to blob cache prewarming, which would require more work on the dashboards and is out of scope for this change. As as result, we no longer share code between the two — but this ends up making the code simpler on both the recovery and prewarming sides. Relates elastic/elasticsearch-team#4256
…4140) Extracts the ShardRecoveryCancellation inner record from CancelRecoveriesAction into a standalone top-level class in org.elasticsearch.indices.recovery. No behavioral change. The motivation is that upcoming master-side wiring will need to reference ShardRecoveryCancellation from allocator-layer code. Keeping it as an inner class of the action would create an awkward dependency in the code. Relates to elastic/elasticsearch-team#2859
…lastic#154113) Add SystemDataStreamDescriptor for .kibana_change_history in KibanaPlugin so it is a first-class system data stream on serverless, and tighten KIBANA_INDEX_DESCRIPTOR with a complement pattern (.kibana_~(change_history*)) so the two descriptors do not overlap. --------- Co-authored-by: Mary Gouseti <mary.gouseti@elastic.co>
* Increase metrics publish cadence in EstimatedHeapUsageAllocationDeciderIT * Fix
…lastic#153903) * Lazily allocate uncompressedBlock in AbstractTSDBDocValuesProducer * Allocate based on the current block size
`CancelRecoveriesAction.Response` returned a `Set<String>` of allocation IDs for recoveries cancelled out of the throttling queue. This is insufficient for the master to act on the response. It had the allocation ID but not the shardId, so it would have needed an extra cluster state lookup to submit a `FailedShardEntry`. This PR switches the response to be a `Set<CancelledInQueue>`, where each record pairs the allocationId with the `ShardId`. The master can then directly submit a `FailedShardEntry` when it receives the response, without any additional lookup. No transport version is needed, since the action is not wired up on the master side, so there are no mixed-cluster concerns for the response format. Relates to elastic/elasticsearch-team#2859
…st {p0=search.vectors/41_knn_search_byte_quantized_bfloat16/KNN Vector similarity search only} elastic#154197
…ocators (elastic#153867) Close the def-dispatch and statically-typed gaps where an allocation- annotated allowlist method escaped charging.
…anningIndexAndDatasetSucceeds elastic#154200
Introduces SliceableColumn and wires it through EscfColumn and all subtypes. Var-width columns switch from raw int[] offsets to IntsRef so windowing adjusts offset/length without copying. EscfBatch.slice() delegates to per-column sliceInternal and avoids re-serialisation on full-range slices.
…c#154431) Use Clock.systemUTC() with a small tolerance when asserting next scheduled time so test bounds match the scheduler's MonotonicClock source.
…elastic#154262) (elastic#154429) Guarantee distinct ML node sizes in heterogeneous-memory scenarios so currentPerNodeMemoryBytes() is always asserted as zero deterministically.
…ensStayInvalidatedInUpgradedCluster elastic#154627
## Summary Migrates the `elasticsearch-agentic-workflow` Buildkite pipeline from the `pi-agent` CLI to its successor **archimedes**, and adds ES delivery stats credentials for CI test-history access and observability. ## Changes ### `pre-command` hook - Renames the bootstrap block from `pi-agent` to `archimedes`. - Replaces the manual tarball-download bootstrap with a smarter installer: - Resolves the latest release via `gh api .../releases/latest` (falls back to `sort_by(.created_at)`). - Compares installed version against latest and only re-bootstraps when behind. - Runs the upstream `bootstrap.sh` from `elastic/elasticsearch-infra` (`add-archimedes` ref). - Switches from `gh_token` to `gh_admin_token` (SAML-authorized for the elastic org) and exports it as `GH_TOKEN` before bootstrap. - Adds two new Vault secrets for ES delivery stats: - `ES_DELIVERY_STATS_URL` - `ES_DELIVERY_STATS_API_KEY` - Removes the old in-process nono CLI note (sandbox is now built into the agentic tool). ### `agentic-workflow.yml` pipeline - Renames all labels, keys, and env vars (`USE_PI_AGENT` -> `USE_ARCHIMEDES`). - Removes the manual `input` step — input validation is now handled by `archimedes ci`. - Replaces the inline `case` command with a single `archimedes ci` entry point. - Right-sizes the agent from `n4-standard-16` to `n4-standard-4` (the orchestrator is I/O-bound; reproduction runs on its own `n4-standard-64` job).
…rhoodGraphSearchFindsSelf elastic#154633
…estStressWithRelocationOrObjectStoreFailures elastic#154655
…tLargeRequestIsNeverDispatched elastic#154657
…T testPreFlushWaitsForOngoingFlushes elastic#154658
* Add OperationLocation to track rowIndex in a batch Remove the `batchRowIndex` from Location and track it using an additional abstraction called `OperationLocation` used only in the Live version map in `IndexVersionValue` Since this is just a refactoring, existing tests cover regressions and no new tests are needed * Fix NPE and add a unit test
…eIT test {yaml=indices.create/10_basic/Create index with write aliases} elastic#154659
Assisted by Cursor/Claude
…ic#153670) Follows: elastic#149553 When the desired balance changes and an initializing shard is no longer desired, we previously waited for the recovery to complete before. That can leave unnecessary recoveries running or queued for longer than needed. This PR adds a best-effort master -> node direct cancellation path of started or queued recoveries. When the desired balance changes, the master now computes direct recovery cancellation candidates for initializing shards that no longer match that desired balance. For each of those candidates, the master sends a CancelRecoveriesAction request directly to the relevant data node. Queued cancellations reported back by the data node are failed immediately through a dedicated shard-failed task queue. Started recovery cancellations go through the normal failShard path.
Cross-project ML datafeeds can search indices in multiple linked projects whose mappings disagree on field types. Today that surfaces as silent partial extraction or hard-to-diagnose failures. This change adds actionable audit messages for two failure modes: optional-field type conflicts and time-field / required-field conflicts. For optional-field type conflicts, when CrossClusterSearchStats confirms a project link or unlink, the datafeed re-checks field capabilities on the confirmed scope and emits a deduplicated WARNING audit entry (JOB_AUDIT_DATAFEED_FIELD_TYPE_CONFLICT) when an optional field has incompatible types across projects. Detection uses pinned compatibility groups aligned with ExtractionMethodDetector (integral, floating, date/date_nanos, keyword, text, boolean, ip, geo, object). Extraction is unchanged; the message is report-only so operators can align mappings or narrow project_routing without stopping the job. For time-field / required-field conflicts, incompatible cross-project types on the configured time field fail fast at datafeed creation (400 + DATAFEED_TIME_FIELD_TYPE_CONFLICT, with date + date_nanos allowed). If a scope change introduces a new conflict mid-run, the conflicting project is excluded via excludedProjects (narrowed routing) and an audit ERROR is recorded (JOB_AUDIT_DATAFEED_PROJECT_EXCLUDED_FIELD_CONFLICT) while the datafeed continues on the remaining scope. This check is event-driven only (no periodic field_caps recheck). Unit tests cover the compatibility predicate matrix, tracker dedup/unlink resolution, and project-exclusion routing.
Introduce a snapshot-only `EQL "<query>" WITH {...}` source command that
runs an EQL query and returns its results as an ES|QL table. Rather than
re-implementing EQL, the command delegates execution to the EQL engine
(EqlSearchAction) and exposes the results under a fixed schema resolved at
planning time.
The schema is chosen from the query's mode, determined by a shallow parse:
event queries produce `_index, _id, _source`; sequence and sample queries
are unnested to one row per event as `_seq, _position, join_keys, _index,
_id, _source`, so the schema is fixed regardless of stage count. The event
payload is exposed as an opaque `_source` column, so no field-caps
resolution is required.
Execution is coordinator-local: an async source operator issues a single
EqlSearchAction and converts the bounded response into one page. To share
class identity across the esql->eql plugin boundary, esql extends the eql
plugin. This required de-duplicating the mapper-version jar that both
esql-core and ql bundled (now consumed as a shared extended module) and
making mapper-version and eql extensible plugins.
Adds unit, internalCluster and csv-spec tests, plus user documentation.
The EQL command passes its indices option through to EqlSearchAction, so a remote-cluster pattern is resolved by EQL cross-cluster search. Add CrossClusterEqlCommandIT verifying a local query reads events from a remote cluster, and correct the docs to state that CCS is supported.
Add a mapping section to the EQL command reference showing how EQL search API requests translate to the ES|QL EQL command and the table each produces, covering event and sequence-with-join-keys queries plus sample, missing events and cross-cluster search.
Assert every WITH option maps to the EqlSearchRequest (adds fetch_size, tiebreaker_field, result_position, previously unasserted), and add a csv-spec test exercising a non-default timestamp_field and a tiebreaker_field against a real cluster. Document which EQL search API options have ES|QL equivalents vs are unsupported.
- Pin allow_partial_search_results/allow_partial_sequence_results to false so a shard failure errors instead of returning a silently incomplete table (a security detection command must not present partial results as complete). - EqlSourceOperator: add a closed flag with synchronized release so a response arriving after the driver closes the operator (cancellation or sibling-operator failure) releases its blocks instead of leaking breaker-accounted memory. - CsvIT: skip EQL specs (they delegate to _search over real indices and cannot run on CSV-loaded data), mirroring the EXTERNAL skip.
Owner
Author
|
Superseded by elastic#154780. This fork-internal PR was mis-based against a 3739-commit-stale fork main and showed a bogus 15,879-file diff; the real change is 56 files. Fork main has been synced; the correct draft PR is now against elastic:main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Work-in-progress rework of the
EQLsource command, grafted onto this fork from@abatsakis's PR (
abatsakis/elasticsearch#1) to continue it. Same foundation — theEQLcommand delegates an EQL query to
EqlSearchActionand folds the results into an ES|QLtable, rather than re-implementing EQL semantics. This is a draft: the correctness
blockers are applied and compile-green; the syntax/schema/dedup/unmapped rework is
designed and specced, implementation in progress.
Applied in this branch (on top of Alex's PR)
EqlRequests): a shard failure now errors instead ofreturning a silently incomplete table. The EQL cluster default is
allow_partial=true, sothe command previously could present partial detection results as complete.
EqlSourceOperator): aclosedflag + synchronizedrelease, so a response arriving after the driver closes the operator (cancellation /
sibling-operator failure) releases its blocks instead of leaking breaker-accounted memory.
_searchover real indices andcannot run on CSV-loaded data), mirroring the EXTERNAL skip.
Reworked design — specced, not yet coded
EQL <indexPattern> "<query>" [METADATA _index,_id,_source] [WITH {tuning}].Index pattern first-class like
FROM(CCS rewriting, quoting, Verifier-time requiredcheck);
WITHcarries optional tuning only;sizeisLIMIT-driven for event queries witha truncation warning. Replaces the current
WITH {"indices": …}(which overloadsWITHandclones the deprecated EXTERNAL arm, and defaults
sizeto 10 — a silentSTATStrap).IndexResolver/mappingAsAttributespath — no parallel mechanism) plus_sequence/_sequence_stageandjoin-key columns typed from
Sequence.joinKeys(). Replaces the opaque_sourceblob, soSTATS … BY host.idworks withoutJSON_EXTRACT.unmapped_fields(DEFAULT/NULLIFY/LOAD) mechanismby making
EqlRelationaResolveUnmappedparticipant — consistent with the rest of ES|QL,no bespoke path.
fetch injected into EQL. EQL-side surface is additive and gated; non-injected EQL is
byte-identical; falls back to today's behavior on any doubt (CCS, index filter).
Status
Draft / WIP. Blockers compile-verified (
:x-pack:plugin:esql:compileJavagreen). The fullrework is designed and validated (proposal + a field-caps-dedup spec + an Alex-vs-rework
architecture narrative); staged, compiling increments to follow. Not merge-ready.
Builds on @abatsakis's EQL source command PR; his factoring (the delegate seam, the
sequence-unnesting shape, the
EqlQueryModeencapsulation) carries through.🤖 Generated with Claude Code