[PPL Lint] Context-fed rules, hover card fixes, and a headless lint API for cross-repo CI - #12464
[PPL Lint] Context-fed rules, hover card fixes, and a headless lint API for cross-repo CI#12464Hanyu-W wants to merge 30 commits into
Conversation
Add the tree-only `invalid-capture-group-name` lint rule for the `rex`/`parse`/`grok` extraction commands. It flags two cases the engine rejects at runtime (RegexCommonUtils.isValidJavaRegexGroupName): - Names that do not match `^[A-Za-z][A-Za-z0-9]*$` (underscores, leading digits, punctuation). Offers a quick fix that strips the disallowed characters when a valid non-empty name remains. - The Python/PCRE `(?P<name>` opener, which is invalid in Java regex. Offers a quick fix that deletes the single `P`, targeting the `P` span rather than the squiggled name. The rule ships with its deterministic fix rendered via the existing code-action provider + fix registry. Also restore the fix-range shift in `remapPipeFirstColumns`: an explicit `fix.range` (the Python-opener case) now gets the same pipe-first column offset as the diagnostic range, so the edit lands on the intended `P` instead of nine columns to its right on a pipe-first query. Registration: append the rule to the catalog, detector registry, doc-links snapshot (exact quality → rex#parameters), the `PPL_LINT_RULE_DEFAULTS` uiSetting (in catalog order), and an engine-outcomes hover entry. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…cript rules Add the explain-plan-backed performance lint layer under `ppl/lint/explain/`. Two advisory rules read the Calcite physical plan returned by `_explain` (never a parse tree): - `operation-not-pushed` — a filter/aggregation/sort the optimizer left running on the coordinator after a full fetch. - `operation-pushed-as-script` — a filter/sort pushed into OpenSearch as a per-document Painless script rather than a native query/sort. Both are Calcite-only (minVersion 3.3.0), default-disabled, and tagged `needsExplain` so the synchronous tree pass skips them. They run through a separate `runExplainLint` pass and its own explain-detector registry, emitting whole-query-range diagnostics with an `explainTarget` hint and a `hoverFacts.operation` clause label. This ships the diagnostics-only layer: plan normalization, outcome detection, the two detectors, the async runner, and the `_explain` fixtures + tests. Command-range narrowing and the deterministic filter-inversion quick fix (the attribution layer) are deferred to a follow-up, as is wiring `runExplainLint` into the runtime editor path. Core types gain two optional fields to carry the explain hints (`Diagnostic.explainTarget`, `HoverFacts.operation`) and a `slow-path` failure class for the hover card. Registration mirrors the existing rule lists: catalog (2 rows), doc-links snapshot (gap → limitation page), and the `PPL_LINT_RULE_DEFAULTS` uiSetting, all in catalog order. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Make the `operation-not-pushed` / `operation-pushed-as-script` rules
functional in the Monaco editor by running the explain pass in
`processLintHighlighting` (the only place that renders lint markers and
holds the full `PPLLintContext`, including the http client).
Phased render: the static tree diagnostics are applied immediately so
squiggles never wait on a network round-trip; the explain pass then runs
asynchronously and re-renders with its diagnostics merged after the static
ones. This lands entirely in `language.ts` — the bridge contract
(`PPLLintBridgeRequest`) is unchanged, so no shared API is widened.
The `_explain` round-trip fires only when an http client is present and a
Calcite-applicable explain rule is enabled (`hasExplainRules`), so a query
with no explain rule active pays zero network cost. Results are cached
per (dataSourceId, query) with in-flight dedup (`explain_cache.ts`);
transient errors are returned but not cached.
No clean-parse guard: a half-typed or unparseable query makes `_explain`
return a non-Calcite / error body, which `toExplainPlan` maps to
`{ isCalcite: false }`, and every explain detector no-ops on that — so an
unusable plan yields no diagnostics rather than a wrong one, and static
field-validation on partially-invalid queries is never suppressed. Stale
responses are dropped via the existing generation + content guard.
Both rules remain default-disabled; the wiring makes them work the moment
a user enables them. Command-range narrowing and the deterministic
filter-inversion quick fix (the attribution layer) are still deferred.
Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Fixes five correctness bugs found in review of the capture-group and explain-backed rules: - invalid-capture-group-name: the opener regex matched Java lookbehind assertions ((?<=, (?<!), whose bodies the [^>]* name capture then swallowed, so a valid pattern like "(?<=id: )(?<val>...)" was flagged as an invalid group name at the default-on ERROR severity. The bare (?< branch now excludes lookbehinds via (?![=!]); the (?P< branch is left unexcluded since P always marks a group opener. - explain outcomes: residual-filter detection treated any rel carrying a "condition" key as a coordinator filter, so join rels (whose join predicate serializes as "condition") false-fired operation-not-pushed. Detection now requires a Calc/Filter-family relOp suffix first. - explain outcomes (legacy text path): filter/aggregation/sort push signals were matched plan-wide, so a pushed filter on the scan masked a residual filter on a downstream Calc (partial pushdown) - a false negative on the string-plan path current clusters use. The legacy path now evaluates the shared signal cascade per plan line, matching the tree path's relation-locality. - explain outcomes: coordinator aggregation/sort matched rel names by substring, so EnumerableSortMergeJoin read as a coordinator sort and EnumerableSortedAggregate was missed. Matching is now anchored on the relOp suffix (endsWith 'Aggregate' / 'Sort'). - explain cache: clear() did not cancel in-flight requests, so a response computed before the clear could repopulate the cleared cache with stale data. An epoch counter now gates cache writes. Adds regression tests for each and bumps the outcome detector version. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Flip operation-not-pushed and operation-pushed-as-script from disabled to enabled in the shipped catalog and the query_enhancements PPL lint rules default. Both advisory rules already require Calcite 3.3+ and gate their network round-trip behind data-source version and engine applicability, so they stay inert on clusters that cannot serve _explain. Update the explain-layer and run_explain_lint test comments that described the shipped default as disabled, and pin the no-network-path test to an explicit disable override instead of relying on the old default. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
The diagnostic message spelled out the raw Java regex constraint (^[A-Za-z][A-Za-z0-9]*$), which is meaningless to a user who does not read regex. Replace it with 'Use only letters and numbers, and start with a letter.' and reword the hover card's engine-behavior line to match, naming underscores/hyphens and leading digits as the rejected cases instead of citing the pattern. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…fast/thorough)
The explain-backed performance rules (operation-not-pushed,
operation-pushed-as-script) flagged the whole query. This ports the explain
range-narrowing subsystem from poc-ppl-linter-v3 so a finding lands on the
specific where / stats / sort command instead, matching the per-token range
the sibling invalid-capture-group-name rule already produces.
Because the PR branch drives explain from language.ts (which owns explainCache
and reaches the worker), the whole subsystem lives inside @osd/monaco rather
than the data plugin the POC used:
- Restore DiagnosticAttribution / attribution? / DiagnosticFix.expectedText? on
the diagnostic model.
- Port the attribution snapshot builder (candidates), snapshot validator,
source-position mapper, probe-set builder, the range resolver, and the
additive-inversion quick fix, all verbatim from the POC (tests included).
- Build the source-candidate snapshot at parse time in the analyzer's new
analyzeLint(), reachable over the worker via new analyzeLint /
validateLintQueries proxy methods.
- Extend explainCache with a probe partition (versioned by the outcome
detector) and an abort signal, so probe requests never collide with the
baseline plan and can be cancelled on staleness.
- Port the probe orchestration (createExplainAttributionState /
runExplainIsolation) into @osd/monaco, driven by lintContext.http +
explainCache + the worker query validator.
- Wire the resolve + fast/thorough switch into layerExplainDiagnostics: a
single-candidate finding narrows with no extra network; ambiguous findings
are dropped in fast mode or disambiguated with bounded control/treatment
probes in thorough mode.
The mode is a global behavior, so the query:enhancements:pplLint:rules setting
migrates from a bare array to { mode, rules } with mode defaulting to
"thorough". A shared readRulesSetting normalizer keeps both client readers
working and treats a legacy stored array as thorough, so upgrading installs get
probe-backed narrowing automatically.
Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…ngine Scope the rule to what the connected cluster actually validates at runtime, judged solely against OpenSearch behavior (not abstract Java-regex semantics): - Scope to rex extract mode and parse. Drop grokCommand (a different dialect that never reaches the name validator) and skip rex mode=sed via an AST check of rexOption's SED terminal (the sed pattern is a substitution whose text is never read as capture-group names). These were the only true false positives. - Mirror the engine's own lexerless scan (RegexCommonUtils.ANY_NAMED_GROUP_PATTERN, unchanged 3.4 to main): remove the (?![=!]) lookbehind exclusion so the naive opener scan agrees byte-for-byte with the engine, including on escaped openers, character classes, \Q...\E, and lookbehind — all of which OpenSearch rejects on 3.4+ with the same phantom name the scan produces. - Version-gate at minVersion 3.4.0 (the opensearch-project#4434 throw). Pre-3.4 clusters treat an invalid name as a silent no-op, so firing there would be a false positive. - Remove the unsafe sanitizing rename fix (silent field renames, collisions). Keep only the collision-safe Python-opener conversion ((?P<name> -> (?<name>), offered only when the resulting name is valid and unique. Emit one combined diagnostic for an invalid Python name. - Update comments and hover facts to state the rule mirrors the engine's validation for rex-extract and parse, not correct Java-regex parsing. Verified live on two clusters (OpenSearch/SQL 3.8-local with Calcite on, and stock 2.19.0): identical queries 400 on 3.8 (invalid/escaped/lookbehind names) and 200 silent no-op on 2.19, confirming the version split. In-browser Discover PPL checks confirm each diagnostic and the quick-fix round-trip. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…d query, abort + skip-incomplete Addresses three of Eric's review comments on opensearch-project#12411 (the typeMap/settings rebase onto opensearch-project#12394 is deferred until that PR merges): - Engine-gate the explain rules (was version-only). deriveIsCalcite now takes the engine type and returns false for Open Distro engines (Elasticsearch), which coerce to >= 3.3.0 by version and were misread as Calcite — causing the explain rules to hit /_plugins/_ppl/_explain on an endpoint Open Distro does not expose. lint_context_builder passes engineType to both deriveIsCalcite and shouldUseRuntimeGrammar (the latter matching the two sibling context builders). - Explain the query the host actually runs, not raw editor text. A new prepareExplainQuery callback on PPLLintContext returns { query, cacheKey }: query is source-prepended + dashboard/time filters (so a leading-pipe query no longer explains with no source, and the plan reflects the applied filters); cacheKey omits the volatile time clause so the cached plan is reused across time-picker moves. The preparer is registered from query_enhancements (which owns the interceptor, filter formatters, and filter state) into a data-side singleton — data never imports query_enhancements, and @osd/monaco only gains an optional cacheKey. When unregistered (e.g. explore), behavior is unchanged. - Skip incomplete queries and abort superseded requests. The explain pass now parses first and issues _explain only when the attribution snapshot is present (a complete parse), so a half-typed query costs no network. A per-model AbortController cancels the prior in-flight request on a new pass, dispose, or language switch. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 7294d5a. ⛔ Hard block: Issues at High severity or above will block this PR from merging.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
✅ All unit and integration tests passing
|
`unsupported` means the `_explain` response carried no Calcite plan, which is a property of the cluster's engine settings rather than of the query. It was cached like a real plan, and nothing in production invalidates this cache: the Calcite settings cache stops fetching after its first successful fill, emits no event when an administrator later toggles the engine, and `explainCache.clear()` has no production caller. So a verdict read once became permanent for the session, and enabling Calcite mid-session left the explain rules silently inert. Cache only a real plan. In-flight deduplication is untouched, so repeated lint passes over the same text within one window still collapse to a single request — the cost is one request per pass afterwards, on an already-debounced path. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…-rules read from throwing
Two review follow-ups that were reported as done but are not in the code.
Explain attribution still defaulted to "thorough" everywhere: the registered
uiSetting value, `readRulesSetting`'s three fallbacks (unset, garbage, and the
legacy top-level array), and `language.ts`'s own `?? 'thorough'`. Thorough fires
a baseline plan plus up to four probes per debounced pause, which is more network
than a first release should take without being asked. Default to fast in all four
places; thorough now requires an explicit opt-in, and an unrecognized stored mode
falls back to fast rather than to the heavier behaviour.
Reading the lint-rules uiSetting also threw when queryEnhancements was disabled
but explore was not. `readRulesSetting` passed `undefined` as the default, which
is identical to passing nothing, so an undeclared key reached core's throw — and
this runs while the query editor builds its lint context, with no capability
check, so it broke the editor's mount even with lint off. Guard with `isDeclared`
rather than a `get()` default: the setting is registered `type: 'json'`, so a
default is substituted for the registered value and then JSON-parsed, and passing
`{}` would throw on every deployment where the key IS registered and the user has
not customised it.
The test fake could not reproduce either failure, since it returned the default
for any key and never threw. It now models `isDeclared`, and a second fake covers
the undeclared-key case.
Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…s Calcite capability Review follow-ups for opensearch-project#12411 (items 1 and 5): - The lint-rules uiSetting schema now accepts both the current { mode, rules } object and the bare rules array earlier releases persisted. UiSettingsClient.onReadHook validates saved user values against this schema on every read and silently drops failures, so rejecting the legacy array wiped an upgrading user's saved rule customizations. The client normalizer already handles both shapes, and since the saved object itself is never rewritten, existing installs get their customizations back on upgrade. - lint_context_builder now folds the already-fetched calciteEnabled capability into isCalcite: version+engine derivation cannot see an administratively disabled Calcite on a >= 3.3.0 cluster, so the explain-backed rules issued a wasted _explain on every idle pause there (the unsupported verdict is deliberately uncached). The route fails open, so unknown state keeps the version-derived answer. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…efcount explain aborts Review follow-ups for opensearch-project#12411 (items 2, 3, 4, and 7): - Thorough-mode probes are built by editing the raw editor text but were posted to _explain unprepared, so a leading-pipe query's probes had no source clause: the control failed, every ambiguous finding was dropped, and (errors being uncached) the doomed probe re-fired on every pause. Sourced queries probed without the injected time/dashboard filters, so a verdict could contradict the baseline plan it explains. ProbeBudget now runs each probe through the same host preparer as the baseline and keys the probe cache on the time-stripped variant. - The preparer reports how many where commands it injected. When that is non-zero, a filter outcome is never pinned on the user's only where with unique-source confidence (the injected clause may be the culprit); the finding instead goes through a causal probe, which the prepared control now resolves correctly in both directions. - prependSource returned already-sourced queries unchanged, dropping the backtick normalization the host applies before executing an INDEXES/INDEX_PATTERN query (addPPLSourceClause, sql#4444/opensearch-project#4445), so the explained text could differ from the executed one and fail _explain for source names that need quoting. The hasSource branch is mirrored here (copied, not imported: query_enhancements cannot depend on the explore plugin). - The explain cache handed one in-flight promise to every caller of a key but wired only the first caller's abort signal into the fetch, so one model's supersession destroyed a co-subscribed model's diagnostics (e.g. an editor remount joining the request before the old model's dispose aborts). The cache now owns one controller per pending entry and refcounts subscribers: the fetch is cancelled only when the last live subscriber aborts, and an aborting caller gets an immediate, never-cached error resolution. The stale class doc claiming the cache never needs to abort is updated to match. Hover: both explain-backed rules set hoverFacts.operation and three doc comments promised the hover card would name the flagged clause, but renderFactsLine had no branch for it, so the line never rendered (item 6). The branch is added last so a filter finding whose quick fix resolved concrete field/literal facts keeps the richer line. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Rebase PR opensearch-project#12411 onto main after opensearch-project#12394 merged. Resolved conflicts: - lint_context_builder.ts / ppl_grammar_cache.ts: took opensearch-project#12394's measured-Calcite derivation (deriveIsCalcite gains measuredCalciteEnabled and withholds true on >=3.3.0 until the cluster reports it), which subsumes the earlier calciteEnabled-only override; kept engineType on the context. - lint_overrides.ts: dropped opensearch-project#12394's duplicate readRulesSetting; kept opensearch-project#12411's { mode, rules } normalizer that accepts the legacy array shape. - register_ppl_lint.ts: combined both improvements — always return a disposer that disables the engine on teardown (opensearch-project#12394) AND register the explain-query preparer (opensearch-project#12411). - Additive lists (detector_registry, rules_catalog.json, ui_settings.ts, catalog.ts, doc_links snapshot): unioned to 15 rules. - ppl.worker.ts / language.ts: kept selectedSourcePattern + engineType in the run-context so source-scoped rules keep firing. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…lidation Extract the runtime-bundle deserializer and the parse-and-lint core into Node-safe modules so the opensearch-project/sql repo's CI can run OSD's production lint detectors against a candidate grammar bundle exported by a SQL PR — without launching OSD, Monaco, a browser, or an HTTP server. - ppl_grammar_deserialize.ts: ATN_DESERIALIZE_OPTIONS, bundle shape validation, vocabulary/ATN deserialization and the CachedGrammar interface, lifted out of the browser-coupled PPLGrammarCache.doFetch. The cache now calls the shared deserializer; behavior-preserving. - headless_ppl_lint.ts: buildRuntimeTree / lintWithGrammar / deserializeBundleOrThrow / lintQueryWithBundle. Always stamps grammarSurface 'runtime-bundle' so runtimeOnly rules are not skipped, which is the anti-vacuous guard the validation exists to provide. runtime_lint.ts now imports that shared core instead of keeping its own identical copies, so the browser fallback and the CI runner cannot drift into different parse trees. Two fixes over the prototype: - lintWithGrammar passes isPipeFirst into the run context. The runner reads it to classify the query's top-level source and defaults it to false, so omitting it silently disabled source-scoped suppression on every pipe-first query. - The de-duplication the original change described but did not perform is actually done here; buildRuntimeTree and lintWithGrammar existed twice. No new @osd/monaco export is needed: all five imports already exist on the ppl-lint barrel. Consumers must run 'yarn osd bootstrap' first, since the module resolves through @osd/monaco/ppl-lint into packages/osd-monaco/target. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…ch rules Two diagnostics for cases where OpenSearch fails silently, plus the host metadata probes that feed them. Both rules self-suppress when their context input is absent, so neither can false-fire before its probe resolves. enabled-false-object (warning) flags a reference into an object mapped `enabled: false`. Verified live on 3.7: the field is stored but not indexed, so the reference returns null with schema type undefined and HTTP 200 — no error anywhere. This is invisible without tooling, because _field_caps strips the attribute, which is also why typeMap cannot detect it. The probe walks the existing read-only DSL mapping route's response client-side; no new endpoint. wildcard-source-zero-match (info) flags a `source=` wildcard matching none of the indices visible to the user, and offers up to five prefix-matched "did you mean" candidates. Its presentation layer already shipped: HoverFacts.pattern / totalIndices / candidateIndices and renderFactsLine's zero-match branch were on main with no rule producing them. The probe reuses the existing resolve_index route, capped at 5000 indices so a very large cluster leaves the rule dormant rather than shipping a huge name set to the engine on every keystroke. Notes on the implementation: - The wildcard rule routes through classifyTopLevelSource rather than walking fromClause itself. An unquoted `source=logs-*` parses as a searchFieldComparison + searchLiteral, not a tableSource, so a fromClause-only walk missed the common unquoted case entirely and also left backticks in the reported pattern. The shared classifier covers both grammar surfaces and normalizes quoting. - wildcard-source-zero-match is deliberately NOT sourceScoped: it reads the cluster-wide index list, not the selected dataset's field metadata, so it must still fire when the query's source differs from the active dataset. - visibleIndices is likewise not gated on the dataset-identity check that guards the field metadata, for the same reason. - No new worker or context plumbing: both context slots are already declared, serialized, and rehydrated on main. Host wiring is ~12 lines per editor host, inside the loadFields effect that already loads field metadata, so there is no new effect, ref, or subscription. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Three follow-ups to the lint hover surface.
1. The hover card no longer vanishes the instant the pointer leaves it. The
query editors run with fixedOverflowWidgets, so the card renders outside the
editor's bounding rect; Monaco's own mouseleave handler then hides it
unconditionally as soon as the pointer is outside that rect, which the
ordinary 'move onto the card, then toward its doc link' gesture satisfies.
Neither hover.sticky nor hover.hidingDelay affects that path. A capture-phase
guard replaces the instant hide with a 600ms grace timer, cancelled by
re-entering the card, and replays the event afterwards so Monaco still
performs the real hide. No private Monaco API is used.
The guard identifies the card as 'the wrapper that CONTAINS .monaco-hover',
not by matching .monaco-hover itself: Monaco listens on
ResizableContentWidget's domNode, a bare div with no className, and
.monaco-hover is its descendant. Since mouseleave does not bubble, a
closest('.monaco-hover') check would never match and the guard would silently
do nothing. The test fixture reproduces that DOM shape, and fails on the
naive selector, so it cannot pass vacuously.
2. HoverFacts.operation was written by both explain-backed performance rules but
had no branch in renderFactsLine, so their per-query line rendered empty.
3. Hoist one exported ruleIdOf into diagnostic_to_marker, beside the code that
writes the marker's code field, and delete the two private copies in the
code-action and hover providers. Its tests round-trip against
diagnosticToMarker so the decoder cannot drift from the encoder.
The hover-persistence hook attaches through the existing PPLDetachRefs bundle,
so both editor hosts pick it up and tear it down with the other lint contexts.
Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Ports the last rule that existed on the poc-v3 branch but not upstream. `rex`/`parse`/`grok` run their pattern over a source field for every input row; when that field is a `text` mapping the engine loads `_source`, decompresses, and evaluates per document (the dominant cost in the pattern-prefilter RFC, opensearch-project/sql#5608). This surfaces that cost. Advisory by construction: `info` severity, shipped disabled, and no quick-fix. Extraction is row-preserving — planned as a Calcite Project, not a Filter — so a non-matching document is kept with a null extracted value. A `where` inserted before the extraction would delete those rows, which is a result change rather than an optimization. The linter therefore only educates; the message states that caveat inline, and the enriched form repeats it. When the pattern begins with a clean literal token the message names it plus both filter forms and their opposite failure directions (`like` is an exact substring but is not pushed down on `text`; `match_phrase` is pushed down but matches whole analyzer tokens, so it misses values where the token is glued into a larger one). `leadingLiteralToken` returns a token only when it is provably a substring of every string the pattern can match, and bails to no hint otherwise: on a top-level alternation, a leading anchor/metaclass/group, an underscore-glued or non-ASCII token, a stopword, a pure-numeric token, or a token below the length floor. Bailing is always the safe default. `decodePatternLiteral` and `leadingLiteralToken` join `findPatternLiteral` in pattern_literal.ts, shared with invalid-capture-group-name. Registered in all four parity points (catalog, detector_registry, ui_settings, doc_links snapshot) plus hover engine_outcomes, so the card renders like its siblings rather than message-only. Verified beyond the ported tests: for 15 patterns, every string the regex matches contains the emitted token, and the 10 hazardous shapes bail rather than passing vacuously (5 emit a token, 10 emit none). Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
2051748 to
7294d5a
Compare
PR Reviewer Guide 🔍(Review updated until commit 54b3dd8)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 54b3dd8
Previous suggestionsSuggestions up to commit 7294d5a
|
…is PR's new rules Two follow-throughs from opensearch-project#12394's review that its own rules got but the rules added here did not. 1. Eric's source-match finding (lint_runner.ts:40). Rules that read the selected dataset's metadata must self-suppress when the query's explicit `source=` names a different index, or they lint against the wrong mapping. opensearch-project#12394 added `sourceScoped` and set it on all four of its context-fed rules; `enabled-false-object` (reads `disabledObjectFields`) and `rex-scan-cost` (reads `typeMap`) have the same exposure and now carry it too. `wildcard-source-zero-match` is deliberately left unscoped: it reads the cluster-wide `visibleIndices` list, not dataset metadata, and suppressing it on a mismatch would silence it in exactly the case it exists to catch. That exemption is now pinned by a test so it is not "fixed" into a bug later. Extends source_mismatch_suppression.test.ts rather than adding a suite: suppressed-on-mismatch and still-fires-on-match for both rules. Verified the suppression assertions fail without the catalog flags. 2. Joshua's test-layout ask (agg_on_text.test.ts:6). opensearch-project#12394 moved its rule tests beside their sources; the three rule tests added here went to `rules/__tests__/`. Moved to match, leaving the pre-existing `__tests__` files alone as that reply said. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
CI's "Lint and validate" failed on jest/no-identical-title: two identically titled `an undeclared lint-rules setting (queryEnhancements disabled)` blocks. The duplicate arises where main and opensearch-project#12411 both added the same test. I had removed it once, but that fix lived in a hand-rolled merge commit which was discarded when this branch was rebased onto opensearch-project#12411's own main merge, so it came back. Keeping the first block, which is the superset: it also asserts readExplainMode. Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
|
Persistent review updated to latest commit 54b3dd8 |
Description
1. Node-safe headless lint API — extracts the bundle deserializer and the parse-and-lint core so the
opensearch-project/sqlCI can run OSD's production detectors against a candidate grammar bundle, without launching OSD, Monaco, or a browser.runtime_lint.tsnow imports that shared core instead of keeping its own duplicate copies. Needs no new@osd/monacoexport.2. Two context-fed rules + their host probes
enabled-false-object(warning) — a reference into an object mappedenabled: false. Verified on 3.7: returnsnullwith HTTP 200 and no error. Invisible without tooling, since_field_capsstrips the attribute (which is also whytypeMapcan't detect it).wildcard-source-zero-match(info) — asource=wildcard matching no visible index, with up to five "did you mean" candidates. Its hover rendering already shipped onmainwith no rule producing it.Both probes reuse existing read-only routes — no new endpoint. Both rules self-suppress when their input is absent. No new worker or context plumbing; both slots are already declared, serialized, and rehydrated on
main.3. Hover card fixes
fixedOverflowWidgetsthe card renders outside the editor's rect, and Monaco'smouseleavehandler then hides it unconditionally — a path neitherhover.stickynorhidingDelayaffects. A capture-phase guard swaps the instant hide for a 600 ms grace timer, then replays the event so Monaco still does the real hide. No private Monaco API used.ruleIdOfreplaces two private copies in the code-action and hover providers.The
HoverFacts.operationrendering fix that was originally here now arrives via #12411, so it is no longer part of this diff.4.
rex-scan-costadvisory rule —rex/parse/grokrun their pattern over a source field for every input row; on atextmapping the engine loads_source, decompresses, and evaluates per document (the dominant cost in the pattern-prefilter RFC, opensearch-project/sql#5608). This surfaces that cost.Advisory by construction:
infoseverity, shipped disabled, no quick-fix. Extraction is row-preserving — planned as a Calcite Project, not a Filter — so a non-matching document is kept with a null extracted value, and awhereinserted before the extraction would delete those rows. That is a result change, not an optimization, so the rule only educates and states the caveat inline.When the pattern begins with a clean literal token, the message names it plus both filter forms and their opposite failure directions (
likeis an exact substring but is not pushed down ontext;match_phraseis pushed down but matches whole analyzer tokens, so it misses values where the token is glued into a larger one). The token is emitted only when it is provably a substring of every string the pattern can match; a top-level alternation, a leading anchor/metaclass/group, an underscore-glued or non-ASCII token, a stopword, a pure-numeric token, or a token below the length floor all bail to no hint.decodePatternLiteralandleadingLiteralTokenjoinfindPatternLiteralinpattern_literal.ts, shared withinvalid-capture-group-name.Issues Resolved
N/A — follow-up to #12394 and #12411.
Screenshot
Captured live in Discover's PPL editor against a local 3.8 cluster, on an index whose mapping has an object mapped
enabled: false.1.
enabled-false-object— the hover names the field and its enclosing object.2.
wildcard-source-zero-match— "matched 0 of 7 visible indices."3. Same rule with candidates — "Did you mean one of:
screenshot-logs-2026.04.05,screenshot-logs-2026.04.06?"4. The
operationline (context, not this PR) — "Your query — Affects thefilterin this query." Captured before the rebase, when therenderFactsLinebranch forHoverFacts.operationwas part of this diff; that fix now comes from #12411, and the wording there reads "The flagged operation is afilter."The hover-card reachability fix is not screenshotted: it only manifests with
fixedOverflowWidgets, which Explore's query panel sets and Discover's editor does not, so the guard correctly no-ops on the surface above. Its unit test reproduces Monaco's real DOM and fails against the naive selector.Testing the changes
2125 tests pass across 160 suites;
yarn typecheckadds no new errors (verified withtsc -b --force, since the cached run reports success without compiling).Manual (needs a mapping with an
enabled: falseobject):enabled: falseobject.source=<index> | where <thatObject>.<field> = 1→ warning, hover names the field and its enclosing object.source=no-such-prefix-*→ info, hover reads "matched 0 of N visible indices".rex-scan-costships disabled; enable it in Advanced Settings, thensource=<index> | rex field=<textField> "error (?<code>[0-9]+)"→ info, naming theerrortoken and both filter forms.Check List
yarn test:jestyarn test:jest_integration