Draft for CI#155020
Draft
GalLalouche wants to merge 44 commits into
Draft
Conversation
- lookupJoinDoesNotContributeToUnmappedFields: verifies LOOKUP EsRelation is skipped by DetermineUnmappedFieldsToKeep; _unmapped_fields reflects only the main index's unmapped source fields - tsStatsDropsUnmappedFields: verifies LOAD_ALL works with TS queries; _unmapped_fields annotated on the TIME_SERIES EsRelation is dropped by STATS aggregation before it reaches the output
Add lookupJoinColumnShadowsUnmappedField (gap test: LOOKUP
non-key column that shares a name with an unmapped source field
should exclude that field from _unmapped_fields — not yet
implemented) and tsWithoutStatsHasUnmappedFields (passes: raw
TS rows carry _unmapped_fields = {} when all fields are mapped).
New message_language_code_lookup index fixtures support the
shadowing test.
Add ts_window_partial_mapping TSDB fixture (dynamic:false, stored
source) and two new csv-spec tests:
- tsWithoutStatsAllMappedFields: TS on fully-mapped ts_window shows
_unmapped_fields = {} per row (passes)
- tsWithoutStatsHasUnmappedFields: gap test — TSDB does not surface
dynamic:false unmapped fields via _source to UnmappedFieldsBlockLoader,
so extra_label does not appear in _unmapped_fields; documents the
intended correct behaviour until the gap is fixed
Also exclude the two new test indices from broad FROM * patterns in
lookup-join.csv-spec (5 occurrences) and enrich.csv-spec (1 occurrence)
to prevent type conflicts and extra rows from breaking unrelated tests.
Add fromSubquery and viewHasUnmappedFields gap tests documenting that DetermineUnmappedFieldsToKeep does not yet traverse Subquery or ViewUnionAll/NamedSubquery plan nodes to annotate the inner EsRelation. - fromSubquery (unmapped-load-all.csv-spec): FROM (FROM partial_mapping_sample_data) should carry _unmapped_fields through the subquery boundary - viewHasUnmappedFields (views.csv-spec): FROM view_partial_mapping_sample_data (backed by partial_mapping_sample_data) should produce the same result Supporting infrastructure: add view_partial_mapping_sample_data.esql, register the view in CsvTestsDataLoader.VIEW_CONFIGS, and extend CsvIT loadViews() to also activate views for the unmapped-load-all test group.
Add synthetic_source_partial_mapping index (same data and mapping as partial_mapping_sample_data but with _source.mode: synthetic) and a fromSyntheticSource test in unmapped-load-all.csv-spec. The test may be a gap: with synthetic source and dynamic:false, unmapped fields land in _ignored_source via addIgnoredField(), but whether SyntheticSourceLoader merges them into the _source returned to UnmappedFieldsBlockLoader is uncertain. Also extend the FROM * exclusion lists in lookup-join.csv-spec (5) and enrich.csv-spec (1) to cover the new index.
Positive csv-spec tests (unmapped-load-all.csv-spec): - dropMappedColumn: DROP mapped column leaves _unmapped_fields unchanged - whereReducesDocumentSet: WHERE filters rows; block loader receives fewer docs - renameMappedField: RENAME mapped column; _unmapped_fields unaffected - fromStatsDropsUnmappedFields: FROM-based STATS drops _unmapped_fields - complexDropRenameEval: combined EVAL+DROP+RENAME on mapped columns Gap csv-spec tests: - dropUnmappedField: DROP unmapped_message should remove it from _unmapped_fields JSON (DetermineUnmappedFieldsToKeep gap) - dropUnmappedFieldWildcard: DROP *messag* should remove matched unmapped fields from _unmapped_fields JSON Analyzer error unit tests (AnalyzerUnmappedTests.java): - testDropUnmappedFieldsColumn: DROP _unmapped_fields must be rejected - testRenameUnmappedFieldsColumn: RENAME _unmapped_fields must be rejected
- InlineStats: exclude aggregate output names so INLINESTATS-added columns don't appear in _unmapped_fields - patternForKeep: translate _unmapped_fields in KEEP to '*' so the block loader includes all source fields - annotate(): propagate _unmapped_fields through Fork (UnionAll, ViewUnionAll) fixed output boundary - LookupJoin: exclude lookup non-key column names from unmapped fields pattern
- Move synthetic source config from deprecated mapping-level _source.mode to index settings, eliminating a deprecation warning that cascaded into EsqlSpecIT test failures - Update fromSyntheticSource expected output: synthetic source reconstructs dot-notation fields as nested objects - Add LOAD_ALL_EXPAND validation to QuerySettingsTests - Document LOAD_ALL_EXPAND in generated docs
Three bugs fixed: 1. patternForKeep (Analyzer): pass child.output() to distinguish mapped vs. unmapped projected fields. Named fields absent from child output are demand-loaded from source, so they go into includes; named mapped fields do not constrain includes at all (they default to ["*"]). This restores keepUnmappedFieldByName and keepUnmappedFieldByNameExpand semantics. 2. UnmappedFieldsBlockLoader: new two-arg constructor accepts mappedFieldNames from the shard's live MappingLookup as extra excludes. EsPhysicalOperationProviders now passes those names when creating the loader, so counter-type metrics in TS queries (absent from field-caps due to the +dimension filter) are never surfaced in _unmapped_fields. 3. ExpandUnmappedFieldsPostProcessor: call page.releaseBlocks() after building each expanded page to release the original _unmapped_fields block from the circuit breaker. CSV-spec: fix JSON escaping for tsWithoutStatsHasUnmappedFields.
We originally had two load-all modes: LOAD_ALL, which dumped all unmapped source fields into one _unmapped_fields JSON column, and LOAD_ALL_EXPAND, which expanded each unmapped field into its own keyword column. Only the expanding behavior is wanted, so it is kept and renamed to LOAD_ALL. - Remove the LOAD_ALL_EXPAND enum and OPTIONAL_FIELDS_LOAD_ALL_EXPAND capability; LOAD_ALL now always expands. - Delete the old JSON-column spec and rename the expand spec onto it. - Drop tests asserting the dead _unmapped_fields JSON output (keepUnmappedFieldsColumnExpand, viewHasUnmappedFields). - Regenerate the unmapped_fields docs.
For the MVP, unmapped_fields="LOAD_ALL" only supports FROM, KEEP, DROP, RENAME, EVAL, WHERE, SORT and LIMIT. Any other command (STATS, JOIN, FORK, ENRICH, views, ...) now fails analysis with a temporary verification error, since their interaction with the expanded _unmapped_fields column is not yet designed. - Add checkLoadAllModeSupportedCommands to the Verifier. - Remove csv-spec tests that combined LOAD_ALL with STATS, INLINE STATS, LOOKUP JOIN and TS. - Remove the fromTwoViews LOAD_ALL-through-views test.
Centralize the unmappedFieldsToKeep computation in the analyzer's DetermineUnmappedFieldsToKeep rule and drop the scattered per-node overrides (LogicalPlan, UnaryPlan, EsRelation, Eval, LookupJoin, InlineStats). Have UnmappedFieldsBlockLoader take an UnmappedFieldsPattern directly (combined with the mapped field names in EsPhysicalOperationProviders) and add UnmappedFieldsPattern#includesAllFields. Make the analyzer pattern helpers and ResolvingProject#create private, drop redundant javadocs, and tidy the unmapped-load-all csv-spec.
Record the review feedback on UnmappedFieldsBlockLoader as a "known limitations" note: it applies no field-level security, re-reads _source per document without a shared cached parse, and does no circuit-breaker accounting. Rework the complexDropRenameEval csv-spec test so its EVAL and RENAME targets clash with unmapped source field names, exercising shadowing rather than introducing only fresh names.
UnmappedFieldsAttribute now extends TypedAttribute (MetadataAttribute re-finalized) and InsertFieldExtraction recognizes it. Simplify the Analyzer pattern helpers, UnmappedFieldsPattern.matches and the block loader with streams. Remove Result.toExpand and always expand via ExpandUnmappedFieldsPostProcessor (now split into helpers). Move pattern tests into DetermineUnmappedFieldsToKeepTests and @ignore the known gaps; extend csv-spec EVAL/KEEP/DROP coverage, documenting wildcard KEEP/DROP over unmapped fields as gaps.
Integrates upstream/main (~2300 commits). Brings in PR elastic#153137's <no-fields> sentinel fix, which makes the LOAD_ALL allFieldsUnmapped case work, and main's ignoreUnmatchedPatterns DROP handling, which fixes dropUnmappedByPattern; both are promoted out of the unmapped-load-all csv-spec "gap" section. Key conflict resolutions: - Analyzer: resolveKeep/resolveDrop combine main's VirtualAttribute provenance with UnmappedFieldsPattern threading; resolveFork switch handles LOAD_ALL. - QuerySettings: adopt main's fluent UNMAPPED_FIELDS builder with a snapshot-gating validator for LOAD_ALL. - Verifier: keep the LOAD_ALL command guardrail; drop the now-unused disallowed-functions check. - EsqlCapabilities: keep both branches' new capability entries. - EsqlSession/TransportEsqlQueryAction: adopt main's resolvedSettings and partial-marking paths. Post-merge fixups: add missing EsqlCapabilities import in AnalyzerUnmappedTestBase and regenerate the unmapped_fields settings docs.
Under SET unmapped_fields="LOAD_ALL": - KEEP listing only mapped columns now keeps no unmapped source field (empty include list -> NONE instead of "*"). - A KEEP wildcard matching only unmapped source fields is tolerated instead of failing analysis with "No matches found"; the pattern is captured and applied during _unmapped_fields expansion. - Explicit references to the synthetic _unmapped_fields column via KEEP/DROP/RENAME/functions now fail as "Unknown column". UnmappedFieldsPattern becomes a NamedWriteable and UnmappedFieldsAttribute serializes through it.
Contributor
🔍 Preview links for changed docsThis PR only changes snippets or data files. No direct page links are available. |
Contributor
ℹ️ Important: Docs version tagging👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version. We use applies_to tags to mark version-specific features and changes. Expand for a quick overviewWhen to use applies_to tags:✅ At the page level to indicate which products/deployments the content applies to (mandatory) What NOT to do:❌ Don't remove or replace information that applies to an older version 🤔 Need help?
|
The new message_language_code_lookup index types language_code as keyword, so the enrich.fieldsInOtherIndicesBug FROM * query saw it as type-conflicted (UNSUPPORTED). Exclude it like lookup-join.csv-spec.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.