Skip to content

[PPL Lint] Context-fed rules, hover card fixes, and a headless lint API for cross-repo CI - #12464

Draft
Hanyu-W wants to merge 30 commits into
opensearch-project:mainfrom
Hanyu-W:ppl-lint-pr1-nonai
Draft

[PPL Lint] Context-fed rules, hover card fixes, and a headless lint API for cross-repo CI#12464
Hanyu-W wants to merge 30 commits into
opensearch-project:mainfrom
Hanyu-W:ppl-lint-pr1-nonai

Conversation

@Hanyu-W

@Hanyu-W Hanyu-W commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Description

1. Node-safe headless lint API — extracts the bundle deserializer and the parse-and-lint core so the opensearch-project/sql CI can run OSD's production detectors against a candidate grammar bundle, without launching OSD, Monaco, or a browser. runtime_lint.ts now imports that shared core instead of keeping its own duplicate copies. Needs no new @osd/monaco export.

2. Two context-fed rules + their host probes

  • enabled-false-object (warning) — a reference into an object mapped enabled: false. Verified on 3.7: returns null with HTTP 200 and no error. Invisible without tooling, since _field_caps strips the attribute (which is also why typeMap can't detect it).
  • wildcard-source-zero-match (info) — a source= wildcard matching no visible index, with up to five "did you mean" candidates. Its hover rendering already shipped on main with no rule producing it.

Both probes reuse existing read-only routes — no new endpoint. Both rules self-suppress when their input is absent. No new worker or context plumbing; both slots are already declared, serialized, and rehydrated on main.

3. Hover card fixes

  • The card no longer vanishes the instant the pointer leaves it, so its doc link is reachable. With fixedOverflowWidgets the card renders outside the editor's rect, and Monaco's mouseleave handler then hides it unconditionally — a path neither hover.sticky nor hidingDelay affects. A capture-phase guard swaps the instant hide for a 600 ms grace timer, then replays the event so Monaco still does the real hide. No private Monaco API used.
  • One exported ruleIdOf replaces two private copies in the code-action and hover providers.

The HoverFacts.operation rendering fix that was originally here now arrives via #12411, so it is no longer part of this diff.

4. rex-scan-cost advisory rulerex/parse/grok run their pattern over a source field for every input row; on a text mapping the engine loads _source, decompresses, and evaluates per document (the dominant cost in the pattern-prefilter RFC, opensearch-project/sql#5608). This surfaces that cost.

Advisory by construction: info severity, shipped disabled, no quick-fix. Extraction is row-preserving — planned as a Calcite Project, not a Filter — so a non-matching document is kept with a null extracted value, and a where inserted before the extraction would delete those rows. That is a result change, not an optimization, so the rule only educates and states the caveat inline.

When the pattern begins with a clean literal token, the message names it plus both filter forms and their opposite failure directions (like is an exact substring but is not pushed down on text; match_phrase is pushed down but matches whole analyzer tokens, so it misses values where the token is glued into a larger one). The token is emitted only when it is provably a substring of every string the pattern can match; a top-level alternation, a leading anchor/metaclass/group, an underscore-glued or non-ASCII token, a stopword, a pure-numeric token, or a token below the length floor all bail to no hint. decodePatternLiteral and leadingLiteralToken join findPatternLiteral in pattern_literal.ts, shared with invalid-capture-group-name.

Issues Resolved

N/A — follow-up to #12394 and #12411.

Screenshot

Captured live in Discover's PPL editor against a local 3.8 cluster, on an index whose mapping has an object mapped enabled: false.

1. enabled-false-object — the hover names the field and its enclosing object.

enabled false object

2. wildcard-source-zero-match — "matched 0 of 7 visible indices."

wildcard source zero match

3. Same rule with candidates — "Did you mean one of: screenshot-logs-2026.04.05, screenshot-logs-2026.04.06?"

wildcard did you mean

4. The operation line (context, not this PR) — "Your query — Affects the filter in this query." Captured before the rebase, when the renderFactsLine branch for HoverFacts.operation was part of this diff; that fix now comes from #12411, and the wording there reads "The flagged operation is a filter."

operation line fix

The hover-card reachability fix is not screenshotted: it only manifests with fixedOverflowWidgets, which Explore's query panel sets and Discover's editor does not, so the guard correctly no-ops on the surface above. Its unit test reproduces Monaco's real DOM and fails against the naive selector.

Testing the changes

yarn test:jest packages/osd-monaco/src/ppl src/plugins/data/public/ppl_lint \
  src/plugins/data/public/antlr/opensearch_ppl src/plugins/data/public/ui/query_editor \
  src/plugins/explore/public/components/query_panel src/plugins/query_enhancements

2125 tests pass across 160 suites; yarn typecheck adds no new errors (verified with tsc -b --force, since the cached run reports success without compiling).

Manual (needs a mapping with an enabled: false object):

  1. Discover or Explore, PPL selected, dataset whose mapping has an enabled: false object.
  2. source=<index> | where <thatObject>.<field> = 1 → warning, hover names the field and its enclosing object.
  3. source=no-such-prefix-* → info, hover reads "matched 0 of N visible indices".
  4. Hover any marker, move onto the card, then toward "Learn more" — it should stay open long enough to click.
  5. rex-scan-cost ships disabled; enable it in Advanced Settings, then source=<index> | rex field=<textField> "error (?<code>[0-9]+)" → info, naming the error token and both filter forms.

Check List

  • All tests pass
    • yarn test:jest
    • yarn test:jest_integration
  • New functionality includes testing.
  • New functionality has been documented.
  • Commits are signed per the DCO using --signoff

Hanyu Wei and others added 17 commits July 17, 2026 16:07
Add the tree-only `invalid-capture-group-name` lint rule for the
`rex`/`parse`/`grok` extraction commands. It flags two cases the engine
rejects at runtime (RegexCommonUtils.isValidJavaRegexGroupName):

- Names that do not match `^[A-Za-z][A-Za-z0-9]*$` (underscores, leading
  digits, punctuation). Offers a quick fix that strips the disallowed
  characters when a valid non-empty name remains.
- The Python/PCRE `(?P<name>` opener, which is invalid in Java regex.
  Offers a quick fix that deletes the single `P`, targeting the `P` span
  rather than the squiggled name.

The rule ships with its deterministic fix rendered via the existing
code-action provider + fix registry.

Also restore the fix-range shift in `remapPipeFirstColumns`: an explicit
`fix.range` (the Python-opener case) now gets the same pipe-first column
offset as the diagnostic range, so the edit lands on the intended `P`
instead of nine columns to its right on a pipe-first query.

Registration: append the rule to the catalog, detector registry,
doc-links snapshot (exact quality → rex#parameters), the
`PPL_LINT_RULE_DEFAULTS` uiSetting (in catalog order), and an
engine-outcomes hover entry.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…cript rules

Add the explain-plan-backed performance lint layer under
`ppl/lint/explain/`. Two advisory rules read the Calcite physical plan
returned by `_explain` (never a parse tree):

- `operation-not-pushed` — a filter/aggregation/sort the optimizer left
  running on the coordinator after a full fetch.
- `operation-pushed-as-script` — a filter/sort pushed into OpenSearch as a
  per-document Painless script rather than a native query/sort.

Both are Calcite-only (minVersion 3.3.0), default-disabled, and tagged
`needsExplain` so the synchronous tree pass skips them. They run through a
separate `runExplainLint` pass and its own explain-detector registry,
emitting whole-query-range diagnostics with an `explainTarget` hint and a
`hoverFacts.operation` clause label.

This ships the diagnostics-only layer: plan normalization, outcome
detection, the two detectors, the async runner, and the `_explain`
fixtures + tests. Command-range narrowing and the deterministic
filter-inversion quick fix (the attribution layer) are deferred to a
follow-up, as is wiring `runExplainLint` into the runtime editor path.

Core types gain two optional fields to carry the explain hints
(`Diagnostic.explainTarget`, `HoverFacts.operation`) and a `slow-path`
failure class for the hover card. Registration mirrors the existing rule
lists: catalog (2 rows), doc-links snapshot (gap → limitation page), and
the `PPL_LINT_RULE_DEFAULTS` uiSetting, all in catalog order.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Make the `operation-not-pushed` / `operation-pushed-as-script` rules
functional in the Monaco editor by running the explain pass in
`processLintHighlighting` (the only place that renders lint markers and
holds the full `PPLLintContext`, including the http client).

Phased render: the static tree diagnostics are applied immediately so
squiggles never wait on a network round-trip; the explain pass then runs
asynchronously and re-renders with its diagnostics merged after the static
ones. This lands entirely in `language.ts` — the bridge contract
(`PPLLintBridgeRequest`) is unchanged, so no shared API is widened.

The `_explain` round-trip fires only when an http client is present and a
Calcite-applicable explain rule is enabled (`hasExplainRules`), so a query
with no explain rule active pays zero network cost. Results are cached
per (dataSourceId, query) with in-flight dedup (`explain_cache.ts`);
transient errors are returned but not cached.

No clean-parse guard: a half-typed or unparseable query makes `_explain`
return a non-Calcite / error body, which `toExplainPlan` maps to
`{ isCalcite: false }`, and every explain detector no-ops on that — so an
unusable plan yields no diagnostics rather than a wrong one, and static
field-validation on partially-invalid queries is never suppressed. Stale
responses are dropped via the existing generation + content guard.

Both rules remain default-disabled; the wiring makes them work the moment
a user enables them. Command-range narrowing and the deterministic
filter-inversion quick fix (the attribution layer) are still deferred.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Fixes five correctness bugs found in review of the capture-group and
explain-backed rules:

- invalid-capture-group-name: the opener regex matched Java lookbehind
  assertions ((?<=, (?<!), whose bodies the [^>]* name capture then
  swallowed, so a valid pattern like "(?<=id: )(?<val>...)" was flagged
  as an invalid group name at the default-on ERROR severity. The bare
  (?< branch now excludes lookbehinds via (?![=!]); the (?P< branch is
  left unexcluded since P always marks a group opener.

- explain outcomes: residual-filter detection treated any rel carrying a
  "condition" key as a coordinator filter, so join rels (whose join
  predicate serializes as "condition") false-fired operation-not-pushed.
  Detection now requires a Calc/Filter-family relOp suffix first.

- explain outcomes (legacy text path): filter/aggregation/sort push
  signals were matched plan-wide, so a pushed filter on the scan masked
  a residual filter on a downstream Calc (partial pushdown) - a false
  negative on the string-plan path current clusters use. The legacy path
  now evaluates the shared signal cascade per plan line, matching the
  tree path's relation-locality.

- explain outcomes: coordinator aggregation/sort matched rel names by
  substring, so EnumerableSortMergeJoin read as a coordinator sort and
  EnumerableSortedAggregate was missed. Matching is now anchored on the
  relOp suffix (endsWith 'Aggregate' / 'Sort').

- explain cache: clear() did not cancel in-flight requests, so a
  response computed before the clear could repopulate the cleared cache
  with stale data. An epoch counter now gates cache writes.

Adds regression tests for each and bumps the outcome detector version.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Flip operation-not-pushed and operation-pushed-as-script from
disabled to enabled in the shipped catalog and the query_enhancements
PPL lint rules default. Both advisory rules already require Calcite
3.3+ and gate their network round-trip behind data-source version and
engine applicability, so they stay inert on clusters that cannot serve
_explain.

Update the explain-layer and run_explain_lint test comments that
described the shipped default as disabled, and pin the no-network-path
test to an explicit disable override instead of relying on the old
default.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
The diagnostic message spelled out the raw Java regex constraint
(^[A-Za-z][A-Za-z0-9]*$), which is meaningless to a user who does not
read regex. Replace it with 'Use only letters and numbers, and start
with a letter.' and reword the hover card's engine-behavior line to
match, naming underscores/hyphens and leading digits as the rejected
cases instead of citing the pattern.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…fast/thorough)

The explain-backed performance rules (operation-not-pushed,
operation-pushed-as-script) flagged the whole query. This ports the explain
range-narrowing subsystem from poc-ppl-linter-v3 so a finding lands on the
specific where / stats / sort command instead, matching the per-token range
the sibling invalid-capture-group-name rule already produces.

Because the PR branch drives explain from language.ts (which owns explainCache
and reaches the worker), the whole subsystem lives inside @osd/monaco rather
than the data plugin the POC used:

- Restore DiagnosticAttribution / attribution? / DiagnosticFix.expectedText? on
  the diagnostic model.
- Port the attribution snapshot builder (candidates), snapshot validator,
  source-position mapper, probe-set builder, the range resolver, and the
  additive-inversion quick fix, all verbatim from the POC (tests included).
- Build the source-candidate snapshot at parse time in the analyzer's new
  analyzeLint(), reachable over the worker via new analyzeLint /
  validateLintQueries proxy methods.
- Extend explainCache with a probe partition (versioned by the outcome
  detector) and an abort signal, so probe requests never collide with the
  baseline plan and can be cancelled on staleness.
- Port the probe orchestration (createExplainAttributionState /
  runExplainIsolation) into @osd/monaco, driven by lintContext.http +
  explainCache + the worker query validator.
- Wire the resolve + fast/thorough switch into layerExplainDiagnostics: a
  single-candidate finding narrows with no extra network; ambiguous findings
  are dropped in fast mode or disambiguated with bounded control/treatment
  probes in thorough mode.

The mode is a global behavior, so the query:enhancements:pplLint:rules setting
migrates from a bare array to { mode, rules } with mode defaulting to
"thorough". A shared readRulesSetting normalizer keeps both client readers
working and treats a legacy stored array as thorough, so upgrading installs get
probe-backed narrowing automatically.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…ngine

Scope the rule to what the connected cluster actually validates at runtime,
judged solely against OpenSearch behavior (not abstract Java-regex semantics):

- Scope to rex extract mode and parse. Drop grokCommand (a different dialect
  that never reaches the name validator) and skip rex mode=sed via an AST check
  of rexOption's SED terminal (the sed pattern is a substitution whose text is
  never read as capture-group names). These were the only true false positives.
- Mirror the engine's own lexerless scan (RegexCommonUtils.ANY_NAMED_GROUP_PATTERN,
  unchanged 3.4 to main): remove the (?![=!]) lookbehind exclusion so the naive
  opener scan agrees byte-for-byte with the engine, including on escaped openers,
  character classes, \Q...\E, and lookbehind — all of which OpenSearch rejects on
  3.4+ with the same phantom name the scan produces.
- Version-gate at minVersion 3.4.0 (the opensearch-project#4434 throw). Pre-3.4 clusters treat an
  invalid name as a silent no-op, so firing there would be a false positive.
- Remove the unsafe sanitizing rename fix (silent field renames, collisions).
  Keep only the collision-safe Python-opener conversion ((?P<name> -> (?<name>),
  offered only when the resulting name is valid and unique. Emit one combined
  diagnostic for an invalid Python name.
- Update comments and hover facts to state the rule mirrors the engine's
  validation for rex-extract and parse, not correct Java-regex parsing.

Verified live on two clusters (OpenSearch/SQL 3.8-local with Calcite on, and
stock 2.19.0): identical queries 400 on 3.8 (invalid/escaped/lookbehind names)
and 200 silent no-op on 2.19, confirming the version split. In-browser Discover
PPL checks confirm each diagnostic and the quick-fix round-trip.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…d query, abort + skip-incomplete

Addresses three of Eric's review comments on opensearch-project#12411 (the typeMap/settings
rebase onto opensearch-project#12394 is deferred until that PR merges):

- Engine-gate the explain rules (was version-only). deriveIsCalcite now takes
  the engine type and returns false for Open Distro engines (Elasticsearch),
  which coerce to >= 3.3.0 by version and were misread as Calcite — causing the
  explain rules to hit /_plugins/_ppl/_explain on an endpoint Open Distro does
  not expose. lint_context_builder passes engineType to both deriveIsCalcite and
  shouldUseRuntimeGrammar (the latter matching the two sibling context builders).

- Explain the query the host actually runs, not raw editor text. A new
  prepareExplainQuery callback on PPLLintContext returns { query, cacheKey }:
  query is source-prepended + dashboard/time filters (so a leading-pipe query no
  longer explains with no source, and the plan reflects the applied filters);
  cacheKey omits the volatile time clause so the cached plan is reused across
  time-picker moves. The preparer is registered from query_enhancements (which
  owns the interceptor, filter formatters, and filter state) into a data-side
  singleton — data never imports query_enhancements, and @osd/monaco only gains
  an optional cacheKey. When unregistered (e.g. explore), behavior is unchanged.

- Skip incomplete queries and abort superseded requests. The explain pass now
  parses first and issues _explain only when the attribution snapshot is present
  (a complete parse), so a half-typed query costs no network. A per-model
  AbortController cancels the prior in-flight request on a new pass, dispose, or
  language switch.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
@github-actions

github-actions Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 7294d5a.

Hard block: Issues at High severity or above will block this PR from merging.

PathLineSeverityDescription
nulllowBase64-encoded strings appear in the `sourceBuilder` fields of Calcite physical plan fixtures. These are serialized Calcite query expressions captured from real explain output, not payloads. Context (field names, surrounding structure, companion fixture deep_pipe.json) confirms legitimacy. Flagged because base64-encoded blobs in committed test data warrant maintainer awareness.

The table above displays the top 10 most important findings.

Total: 1 | Critical: 0 | High: 0 | Medium: 0 | Low: 1


Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.


⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

Hanyu-W added a commit to Hanyu-W/OpenSearch-Dashboards that referenced this pull request Jul 26, 2026
Hanyu-W added a commit to Hanyu-W/OpenSearch-Dashboards that referenced this pull request Jul 26, 2026
Hanyu-W added a commit to Hanyu-W/OpenSearch-Dashboards that referenced this pull request Jul 26, 2026
Hanyu-W added a commit to Hanyu-W/OpenSearch-Dashboards that referenced this pull request Jul 26, 2026
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

✅ All unit and integration tests passing

🔗 Workflow run · commit 54b3dd83710f0b98b8d861f9d2c4cf3074718a4e

Hanyu Wei and others added 5 commits July 27, 2026 15:40
`unsupported` means the `_explain` response carried no Calcite plan, which is a
property of the cluster's engine settings rather than of the query. It was cached
like a real plan, and nothing in production invalidates this cache: the Calcite
settings cache stops fetching after its first successful fill, emits no event
when an administrator later toggles the engine, and `explainCache.clear()` has no
production caller. So a verdict read once became permanent for the session, and
enabling Calcite mid-session left the explain rules silently inert.

Cache only a real plan. In-flight deduplication is untouched, so repeated lint
passes over the same text within one window still collapse to a single request —
the cost is one request per pass afterwards, on an already-debounced path.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…-rules read from throwing

Two review follow-ups that were reported as done but are not in the code.

Explain attribution still defaulted to "thorough" everywhere: the registered
uiSetting value, `readRulesSetting`'s three fallbacks (unset, garbage, and the
legacy top-level array), and `language.ts`'s own `?? 'thorough'`. Thorough fires
a baseline plan plus up to four probes per debounced pause, which is more network
than a first release should take without being asked. Default to fast in all four
places; thorough now requires an explicit opt-in, and an unrecognized stored mode
falls back to fast rather than to the heavier behaviour.

Reading the lint-rules uiSetting also threw when queryEnhancements was disabled
but explore was not. `readRulesSetting` passed `undefined` as the default, which
is identical to passing nothing, so an undeclared key reached core's throw — and
this runs while the query editor builds its lint context, with no capability
check, so it broke the editor's mount even with lint off. Guard with `isDeclared`
rather than a `get()` default: the setting is registered `type: 'json'`, so a
default is substituted for the registered value and then JSON-parsed, and passing
`{}` would throw on every deployment where the key IS registered and the user has
not customised it.

The test fake could not reproduce either failure, since it returned the default
for any key and never threw. It now models `isDeclared`, and a second fake covers
the undeclared-key case.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…s Calcite capability

Review follow-ups for opensearch-project#12411 (items 1 and 5):

- The lint-rules uiSetting schema now accepts both the current
  { mode, rules } object and the bare rules array earlier releases
  persisted. UiSettingsClient.onReadHook validates saved user values
  against this schema on every read and silently drops failures, so
  rejecting the legacy array wiped an upgrading user's saved rule
  customizations. The client normalizer already handles both shapes,
  and since the saved object itself is never rewritten, existing
  installs get their customizations back on upgrade.

- lint_context_builder now folds the already-fetched calciteEnabled
  capability into isCalcite: version+engine derivation cannot see an
  administratively disabled Calcite on a >= 3.3.0 cluster, so the
  explain-backed rules issued a wasted _explain on every idle pause
  there (the unsupported verdict is deliberately uncached). The route
  fails open, so unknown state keeps the version-derived answer.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…efcount explain aborts

Review follow-ups for opensearch-project#12411 (items 2, 3, 4, and 7):

- Thorough-mode probes are built by editing the raw editor text but were
  posted to _explain unprepared, so a leading-pipe query's probes had no
  source clause: the control failed, every ambiguous finding was dropped,
  and (errors being uncached) the doomed probe re-fired on every pause.
  Sourced queries probed without the injected time/dashboard filters, so
  a verdict could contradict the baseline plan it explains. ProbeBudget
  now runs each probe through the same host preparer as the baseline and
  keys the probe cache on the time-stripped variant.

- The preparer reports how many where commands it injected. When that is
  non-zero, a filter outcome is never pinned on the user's only where
  with unique-source confidence (the injected clause may be the culprit);
  the finding instead goes through a causal probe, which the prepared
  control now resolves correctly in both directions.

- prependSource returned already-sourced queries unchanged, dropping the
  backtick normalization the host applies before executing an
  INDEXES/INDEX_PATTERN query (addPPLSourceClause, sql#4444/opensearch-project#4445), so
  the explained text could differ from the executed one and fail
  _explain for source names that need quoting. The hasSource branch is
  mirrored here (copied, not imported: query_enhancements cannot depend
  on the explore plugin).

- The explain cache handed one in-flight promise to every caller of a key
  but wired only the first caller's abort signal into the fetch, so one
  model's supersession destroyed a co-subscribed model's diagnostics
  (e.g. an editor remount joining the request before the old model's
  dispose aborts). The cache now owns one controller per pending entry
  and refcounts subscribers: the fetch is cancelled only when the last
  live subscriber aborts, and an aborting caller gets an immediate,
  never-cached error resolution. The stale class doc claiming the cache
  never needs to abort is updated to match.

Hover: both explain-backed rules set hoverFacts.operation and three doc
comments promised the hover card would name the flagged clause, but
renderFactsLine had no branch for it, so the line never rendered (item
6). The branch is added last so a filter finding whose quick fix
resolved concrete field/literal facts keeps the richer line.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Hanyu Wei added 5 commits July 28, 2026 15:59
Rebase PR opensearch-project#12411 onto main after opensearch-project#12394 merged. Resolved conflicts:
- lint_context_builder.ts / ppl_grammar_cache.ts: took opensearch-project#12394's
  measured-Calcite derivation (deriveIsCalcite gains measuredCalciteEnabled
  and withholds true on >=3.3.0 until the cluster reports it), which
  subsumes the earlier calciteEnabled-only override; kept engineType on the
  context.
- lint_overrides.ts: dropped opensearch-project#12394's duplicate readRulesSetting; kept
  opensearch-project#12411's { mode, rules } normalizer that accepts the legacy array shape.
- register_ppl_lint.ts: combined both improvements — always return a disposer
  that disables the engine on teardown (opensearch-project#12394) AND register the
  explain-query preparer (opensearch-project#12411).
- Additive lists (detector_registry, rules_catalog.json, ui_settings.ts,
  catalog.ts, doc_links snapshot): unioned to 15 rules.
- ppl.worker.ts / language.ts: kept selectedSourcePattern + engineType in the
  run-context so source-scoped rules keep firing.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…lidation

Extract the runtime-bundle deserializer and the parse-and-lint core into
Node-safe modules so the opensearch-project/sql repo's CI can run OSD's
production lint detectors against a candidate grammar bundle exported by a
SQL PR — without launching OSD, Monaco, a browser, or an HTTP server.

- ppl_grammar_deserialize.ts: ATN_DESERIALIZE_OPTIONS, bundle shape
  validation, vocabulary/ATN deserialization and the CachedGrammar interface,
  lifted out of the browser-coupled PPLGrammarCache.doFetch. The cache now
  calls the shared deserializer; behavior-preserving.
- headless_ppl_lint.ts: buildRuntimeTree / lintWithGrammar /
  deserializeBundleOrThrow / lintQueryWithBundle. Always stamps
  grammarSurface 'runtime-bundle' so runtimeOnly rules are not skipped, which
  is the anti-vacuous guard the validation exists to provide.

runtime_lint.ts now imports that shared core instead of keeping its own
identical copies, so the browser fallback and the CI runner cannot drift into
different parse trees.

Two fixes over the prototype:
- lintWithGrammar passes isPipeFirst into the run context. The runner reads it
  to classify the query's top-level source and defaults it to false, so
  omitting it silently disabled source-scoped suppression on every pipe-first
  query.
- The de-duplication the original change described but did not perform is
  actually done here; buildRuntimeTree and lintWithGrammar existed twice.

No new @osd/monaco export is needed: all five imports already exist on the
ppl-lint barrel. Consumers must run 'yarn osd bootstrap' first, since the
module resolves through @osd/monaco/ppl-lint into packages/osd-monaco/target.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
…ch rules

Two diagnostics for cases where OpenSearch fails silently, plus the host
metadata probes that feed them. Both rules self-suppress when their context
input is absent, so neither can false-fire before its probe resolves.

enabled-false-object (warning) flags a reference into an object mapped
`enabled: false`. Verified live on 3.7: the field is stored but not indexed, so
the reference returns null with schema type undefined and HTTP 200 — no error
anywhere. This is invisible without tooling, because _field_caps strips the
attribute, which is also why typeMap cannot detect it. The probe walks the
existing read-only DSL mapping route's response client-side; no new endpoint.

wildcard-source-zero-match (info) flags a `source=` wildcard matching none of
the indices visible to the user, and offers up to five prefix-matched "did you
mean" candidates. Its presentation layer already shipped: HoverFacts.pattern /
totalIndices / candidateIndices and renderFactsLine's zero-match branch were on
main with no rule producing them.

The probe reuses the existing resolve_index route, capped at 5000 indices so a
very large cluster leaves the rule dormant rather than shipping a huge name set
to the engine on every keystroke.

Notes on the implementation:
- The wildcard rule routes through classifyTopLevelSource rather than walking
  fromClause itself. An unquoted `source=logs-*` parses as a
  searchFieldComparison + searchLiteral, not a tableSource, so a fromClause-only
  walk missed the common unquoted case entirely and also left backticks in the
  reported pattern. The shared classifier covers both grammar surfaces and
  normalizes quoting.
- wildcard-source-zero-match is deliberately NOT sourceScoped: it reads the
  cluster-wide index list, not the selected dataset's field metadata, so it must
  still fire when the query's source differs from the active dataset.
- visibleIndices is likewise not gated on the dataset-identity check that
  guards the field metadata, for the same reason.
- No new worker or context plumbing: both context slots are already declared,
  serialized, and rehydrated on main.

Host wiring is ~12 lines per editor host, inside the loadFields effect that
already loads field metadata, so there is no new effect, ref, or subscription.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Three follow-ups to the lint hover surface.

1. The hover card no longer vanishes the instant the pointer leaves it. The
   query editors run with fixedOverflowWidgets, so the card renders outside the
   editor's bounding rect; Monaco's own mouseleave handler then hides it
   unconditionally as soon as the pointer is outside that rect, which the
   ordinary 'move onto the card, then toward its doc link' gesture satisfies.
   Neither hover.sticky nor hover.hidingDelay affects that path. A capture-phase
   guard replaces the instant hide with a 600ms grace timer, cancelled by
   re-entering the card, and replays the event afterwards so Monaco still
   performs the real hide. No private Monaco API is used.

   The guard identifies the card as 'the wrapper that CONTAINS .monaco-hover',
   not by matching .monaco-hover itself: Monaco listens on
   ResizableContentWidget's domNode, a bare div with no className, and
   .monaco-hover is its descendant. Since mouseleave does not bubble, a
   closest('.monaco-hover') check would never match and the guard would silently
   do nothing. The test fixture reproduces that DOM shape, and fails on the
   naive selector, so it cannot pass vacuously.

2. HoverFacts.operation was written by both explain-backed performance rules but
   had no branch in renderFactsLine, so their per-query line rendered empty.

3. Hoist one exported ruleIdOf into diagnostic_to_marker, beside the code that
   writes the marker's code field, and delete the two private copies in the
   code-action and hover providers. Its tests round-trip against
   diagnosticToMarker so the decoder cannot drift from the encoder.

The hover-persistence hook attaches through the existing PPLDetachRefs bundle,
so both editor hosts pick it up and tear it down with the other lint contexts.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
Ports the last rule that existed on the poc-v3 branch but not upstream.
`rex`/`parse`/`grok` run their pattern over a source field for every input
row; when that field is a `text` mapping the engine loads `_source`,
decompresses, and evaluates per document (the dominant cost in the
pattern-prefilter RFC, opensearch-project/sql#5608). This surfaces that cost.

Advisory by construction: `info` severity, shipped disabled, and no quick-fix.
Extraction is row-preserving — planned as a Calcite Project, not a Filter — so
a non-matching document is kept with a null extracted value. A `where` inserted
before the extraction would delete those rows, which is a result change rather
than an optimization. The linter therefore only educates; the message states
that caveat inline, and the enriched form repeats it.

When the pattern begins with a clean literal token the message names it plus
both filter forms and their opposite failure directions (`like` is an exact
substring but is not pushed down on `text`; `match_phrase` is pushed down but
matches whole analyzer tokens, so it misses values where the token is glued
into a larger one). `leadingLiteralToken` returns a token only when it is
provably a substring of every string the pattern can match, and bails to no
hint otherwise: on a top-level alternation, a leading anchor/metaclass/group,
an underscore-glued or non-ASCII token, a stopword, a pure-numeric token, or a
token below the length floor. Bailing is always the safe default.

`decodePatternLiteral` and `leadingLiteralToken` join `findPatternLiteral` in
pattern_literal.ts, shared with invalid-capture-group-name.

Registered in all four parity points (catalog, detector_registry, ui_settings,
doc_links snapshot) plus hover engine_outcomes, so the card renders like its
siblings rather than message-only.

Verified beyond the ported tests: for 15 patterns, every string the regex
matches contains the emitted token, and the 10 hazardous shapes bail rather
than passing vacuously (5 emit a token, 10 emit none).

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
@Hanyu-W
Hanyu-W force-pushed the ppl-lint-pr1-nonai branch from 2051748 to 7294d5a Compare July 28, 2026 23:15
@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

(Review updated until commit 54b3dd8)

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 Multiple PR themes

Sub-PR theme: Explain-backed lint layer and attribution

Relevant files:

  • packages/osd-monaco/src/ppl/lint/explain/explain_attribution.ts
  • packages/osd-monaco/src/ppl/lint/explain/attribution/candidates.ts
  • packages/osd-monaco/src/ppl/lint/explain/explain_cache.ts
  • packages/osd-monaco/src/ppl/lint/explain/tests/explain_attribution.test.ts
  • packages/osd-monaco/src/ppl/lint/explain/tests/explain_cache.test.ts
  • packages/osd-monaco/src/ppl/lint/explain/tests/explain_detectors.test.ts
  • packages/osd-monaco/src/ppl/language.ts
  • packages/osd-monaco/src/ppl/language_explain_layer.test.ts

Sub-PR theme: Pattern-literal rules (rex-scan-cost, invalid-capture-group-name)

Relevant files:

  • packages/osd-monaco/src/ppl/lint/pattern_literal.ts
  • packages/osd-monaco/src/ppl/lint/rules/rex_scan_cost.ts
  • packages/osd-monaco/src/ppl/lint/rules/tests/invalid_capture_group_name.test.ts

Sub-PR theme: Headless lint API extraction and plugin wiring

Relevant files:

  • src/plugins/data/public/antlr/opensearch_ppl/runtime_lint.ts
  • src/plugins/query_enhancements/public/plugin.tsx

⚡ Recommended focus areas for review

Possible race on abort refcount

When a signalled subscriber's abort listener fires after the shared fetch has already resolved (or is about to), liveSubscribers is decremented but settled may still be false briefly, or the promise chain has resolved but the abort listener races the .then that removes the listener. The aborting subscriber's outer promise is resolved with an error even though the shared fetch produced a valid result — this is documented behavior, but note that if signal fires after settled = true, controller?.abort() will still be called (harmless), and the caller receives an error resolution while co-subscribers get the plan. Verify this is intentional for callers that abort as a cleanup after they've already received data.

return new Promise<ExplainResolution>((resolve) => {
  const onAbort = () => {
    liveSubscribers--;
    if (liveSubscribers === 0 && !settled) {
      controller?.abort();
    }
    // This caller is done regardless of what the shared fetch does next;
    // co-subscribers keep their own pending resolution.
    resolve({ status: 'error', error: new Error('explain request aborted') });
  };
  signal.addEventListener('abort', onAbort, { once: true });
  promise.then((resolution) => {
    signal.removeEventListener('abort', onAbort);
    resolve(resolution);
  });
});
Timeout leak on race resolution

In ProbeBudget.explain, when the timeoutResult promise wins the Promise.race, the underlying explainCache.resolveResult promise is not awaited or cancelled here (the AbortController is aborted, but the pending promise from resolveResult may resolve later). The clearTimeout call also only runs after Promise.race resolves. If the cache request never settles (e.g., subscribers refcount keeps it alive), no leak occurs, but the deadline branch relies on controller.abort() propagating to the shared fetch — which it does not when other subscribers still hold the pending entry. In that case the probe returns an error but the underlying request continues, which is acceptable but worth confirming.

  const remainingMs = Math.max(1, this.deadline - Date.now());
  let timeout: ReturnType<typeof setTimeout> | undefined;
  const timeoutResult = new Promise<ExplainResolution>((resolve) => {
    timeout = setTimeout(() => {
      controller?.abort();
      resolve({ status: 'error', error: new Error('probe wall-clock budget exceeded') });
    }, remainingMs);
  });
  // Prepare the raw probe text the same way the baseline was prepared (source
  // prepend + injected filters) so its plan is comparable to the baseline's;
  // key the cache on the time-stripped variant, exactly like the baseline.
  const prepared = this.inputs.prepareExplainQuery?.(query) ?? {
    query,
    cacheKey: query,
  };
  const resolution = await Promise.race([
    explainCache.resolveResult(this.inputs.http, prepared.query, this.inputs.dataSourceId, {
      partition: 'probe',
      signal: controller?.signal,
      cacheKey: prepared.cacheKey,
    }),
    timeoutResult,
  ]);
  if (timeout) {
    clearTimeout(timeout);
  }
  return this.inputs.isCurrent()
    ? resolution
    : { status: 'error', error: new Error('stale lint generation') };
}
Stale generation check race

layerExplainDiagnostics calls renderLintMarkers(model, [...staticDiagnostics, ...state.immediateDiagnostics]) after isLintPassStale was checked prior to the _explain call but not immediately before this render. A model edit or new lint pass that occurs between the resolution check and this render could cause immediate diagnostics to be rendered on a stale generation, overwriting a newer pass's markers. Consider re-checking isLintPassStale immediately before each renderLintMarkers call in this async path.

};
const state = createExplainAttributionState(attributionInputs);

// Fast, and the shared first pass for Thorough: render the uniquely-sourced
// findings immediately (fixes withheld until a probe confirms them).
renderLintMarkers(model, [...staticDiagnostics, ...state.immediateDiagnostics]);

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

PR Code Suggestions ✨

Latest suggestions up to 54b3dd8
Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
Possible issue
Abort fetch when first caller pre-aborted

When every subscriber uses an already-aborted signal, subscribe returns error
resolutions without incrementing liveSubscribers, leaving pending populated and the
underlying fetch running forever (no abort ever triggers controller.abort(), and the
.then finalizer only removes the entry after the fetch settles). Explicitly abort
the controller when the initial subscribe call cannot join with any signal
(all-aborted first-caller case), or count the aborted signal as a transient
subscriber that immediately decrements to trigger cleanup.

packages/osd-monaco/src/ppl/lint/explain/explain_cache.ts [211-225]

 let liveSubscribers = 0;
 const subscribe = (signal?: AbortSignal): Promise<ExplainResolution> => {
   if (!signal) {
     liveSubscribers++;
     return promise;
   }
   if (signal.aborted) {
-    // Never joined: no refcount change, and the caller gets the same
-    // error-shaped resolution an abort mid-flight would have produced.
+    if (liveSubscribers === 0 && !settled) {
+      controller?.abort();
+      pending.delete(k);
+    }
     return Promise.resolve({
       status: 'error',
       error: new Error('explain request aborted'),
     } as ExplainResolution);
   }
   liveSubscribers++;
Suggestion importance[1-10]: 7

__

Why: Valid edge case: if the very first (and only) subscriber uses an already-aborted signal, liveSubscribers stays 0, the fetch runs to completion but nobody cares, and pending isn't cleared until the fetch settles. The suggestion identifies a real resource leak, though the impact is bounded (fetch will eventually settle and clean up).

Medium
Fix malformed MDS URL construction

The MDS route segment appears malformed: appending /dataSourceMDSId= to the base URL
produces a path segment like .../indices.getFieldMapping/dataSourceMDSId=abc rather
than the expected query-param or route-parameter form. Verify against the actual
proxy route contract; the current construction likely 404s on any MDS request,
silently masking disabled-object detection.

src/plugins/data/public/ppl_lint/disabled_object_fields.ts [84-88]

 const mdsId = indexPattern.dataSourceRef?.id;
-const url = mdsId
-  ? `${DSL_MAPPING_URL}/dataSourceMDSId=${encodeURIComponent(mdsId)}`
-  : DSL_MAPPING_URL;
-const resp = await http.get(url, { query: { index: pattern } });
+const query: Record<string, string> = { index: pattern };
+if (mdsId) {
+  query.dataSourceMDSId = mdsId;
+}
+const resp = await http.get(DSL_MAPPING_URL, { query });
Suggestion importance[1-10]: 6

__

Why: The suggestion raises a legitimate concern about the URL construction using /dataSourceMDSId=<id> as a path segment, which looks unusual. However, without access to the actual proxy route contract, it's unclear whether this is a bug or an intentional convention used elsewhere in the codebase. The verification-style nature limits the score.

Low
General
Unquote backtick-quoted field identifiers

bareFieldSource returns the field name via getText(), which for a backtick-quoted
identifier like raw log returns the quoted form. Downstream typeMap.get(field) is
presumably keyed by the unquoted name, so a quoted field will fail lookup and
silently suppress the finding. Strip surrounding backticks (as
candidates.ts#fieldName does) before returning.

packages/osd-monaco/src/ppl/lint/rules/rex_scan_cost.ts [85-97]

 function bareFieldSource(
   command: ParserRuleContext,
   ruleNameToIndex: RuleNameToIndex
 ): string | undefined {
   const expression = findChildByRule(command, ruleNameToIndex, 'expression');
   if (!expression) {
     return undefined;
   }
   const fieldExprs = findAllDescendantsByRule(expression, ruleNameToIndex, 'fieldExpression');
   if (fieldExprs.length !== 1 || fieldExprs[0].getText() !== expression.getText()) {
     return undefined;
   }
-  return fieldExprs[0].getText();
+  const text = fieldExprs[0].getText();
+  return text.startsWith('`') && text.endsWith('`') ? text.slice(1, -1) : text;
 }
Suggestion importance[1-10]: 6

__

Why: Legitimate concern: if a user writes parse `raw log` "...", the returned field name would include backticks and typeMap.get() would miss it, silently suppressing the finding. The candidates.ts file already handles this via fieldName(), so consistency argues for the fix.

Low
Skip backtick-quoted regions when splitting pipe

Splitting on the first | character is unsafe: a source name (or any content before
the first pipe) could contain a | inside a quoted string or backticked identifier.
Additionally, the comment claims field comparisons like | where source=prod are
untouched, but if the query has no pipe at all yet contains source= in an unusual
position, behavior may still be surprising. Consider ensuring the pipe search skips
over backtick-quoted regions.

src/plugins/query_enhancements/public/ppl_lint/explain_query_preparer.ts [39-40]

 function backtickWrapSource(queryString: string): string {
-  const pipeIndex = queryString.indexOf('|');
+  // Find first pipe that isn't inside backticks
+  let pipeIndex = -1;
+  let inBackticks = false;
+  for (let i = 0; i < queryString.length; i++) {
+    if (queryString[i] === '`') inBackticks = !inBackticks;
+    else if (queryString[i] === '|' && !inBackticks) { pipeIndex = i; break; }
+  }
   const prefix = pipeIndex === -1 ? queryString : queryString.slice(0, pipeIndex);
   const suffix = pipeIndex === -1 ? '' : queryString.slice(pipeIndex);
-  const updatedPrefix = prefix.replace(
-    /(\bsource\s*=\s*)(`[^`]+`|[^\s|,]+(?:\s*,\s*[^\s|,]+)*,?)/i,
-    (_match, srcPrefix, sourceValue) => {
+  // ... rest unchanged
+}
Suggestion importance[1-10]: 5

__

Why: A valid edge case: a backticked identifier containing | before the first real pipe would be split incorrectly. However, this is an unusual case for a source= clause and the existing tests do not exercise it, so impact is moderate.

Low
Avoid shared regex state across calls

CAPTURE_GROUP_OPENER is a module-level regex with the /g flag, so its lastIndex
state is shared across all invocations of extractGroups. Although extractGroups
resets lastIndex = 0 at the start, this is not thread/reentrancy safe and can cause
subtle bugs if extractGroups is ever called from within itself (or during
iteration). Prefer constructing a new RegExp per call, or use matchAll which does
not carry state.

packages/osd-monaco/src/ppl/lint/rules/invalid_capture_group_name.ts [29]

-const CAPTURE_GROUP_OPENER = /\(\?(P?)<([^>]*)>/g;
+function extractGroups(literalRaw: string): ExtractedGroup[] {
+  const groups: ExtractedGroup[] = [];
+  const re = /\(\?(P?)<([^>]*)>/g;
+  let match: RegExpExecArray | null;
+  while ((match = re.exec(literalRaw)) !== null) {
+    // ... existing body
+  }
+  return groups;
+}
Suggestion importance[1-10]: 4

__

Why: The concern is legitimate for shared regex state with /g flag, but extractGroups explicitly resets lastIndex = 0 and JavaScript is single-threaded, making reentrancy the only real concern which is unlikely here. Minor maintainability improvement.

Low
Tighten residual-filter rel matching

*The comment claims suffix-matching is anchored and cannot collide with names that
merely contain the operator, but endsWith('Filter') still matches operators like
BloomFilter or any future Filter rel, and endsWith('Calc') matches any rel ending
in "Calc". Consider matching against an explicit allowlist of rel operator names (or
a stricter pattern) to avoid mis-classifying unrelated rels as residual-filter
carriers.

packages/osd-monaco/src/ppl/lint/explain/explain_tree_utils.ts [79-90]

-const RESIDUAL_FILTER_REL_SUFFIXES = ['Calc', 'Filter'];
+const RESIDUAL_FILTER_REL_OPS = new Set([
+  'EnumerableCalc',
+  'LogicalCalc',
+  'EnumerableFilter',
+  'LogicalFilter',
+]);
 
-/**
- * True when a rel operator name belongs to the Calc/Filter family — the only
- * rels that can carry a residual (coordinator-side) filter condition. Join rels
- * also serialize a `condition` attribute (their join predicate), but a join
- * condition always evaluates at the coordinator by design and must not be
- * reported as a filter that failed to push down.
- */
 export function isResidualFilterRelOp(relOp: string): boolean {
-  return RESIDUAL_FILTER_REL_SUFFIXES.some((suffix) => relOp.endsWith(suffix));
+  return RESIDUAL_FILTER_REL_OPS.has(relOp);
 }
Suggestion importance[1-10]: 4

__

Why: The suggestion correctly notes that endsWith('Filter') could match unrelated rels like BloomFilter. However, the existing comment acknowledges the suffix approach and Calcite rel naming conventions make this a low-probability issue in practice. An explicit allowlist would be more robust but is a minor improvement.

Low
Prevent cache key collisions from delimiter chars

The cache key concatenates dataSourceId and query with ::, which can collide when a
data source id or query text contains ::. Use a delimiter or length-prefixing that
cannot appear in the inputs, or JSON-encode the components, to avoid cross-key
collisions where different (dataSourceId, query) pairs hash to the same key.

packages/osd-monaco/src/ppl/lint/explain/explain_cache.ts [124-134]

 private key(
   query: string,
   dataSourceId: string | undefined,
   partition: 'baseline' | 'probe'
 ): string {
-  // Probe verdicts depend on the outcome detector; version the key so a
-  // detector change never reuses a stale probe result. Baseline plans are the
-  // raw `_explain`, independent of the detector, so they carry no version.
-  const version = partition === 'probe' ? `::outcomes-${EXPLAIN_OUTCOME_DETECTOR_VERSION}` : '';
-  return `${dataSourceId ?? '__local__'}${version}::${query}`;
+  const version = partition === 'probe' ? `outcomes-${EXPLAIN_OUTCOME_DETECTOR_VERSION}` : '';
+  return JSON.stringify([dataSourceId ?? '__local__', version, query]);
 }
Suggestion importance[1-10]: 3

__

Why: The :: collision is theoretically possible but extremely unlikely in practice for dataSourceId (typically UUIDs/simple ids) and query text collisions would require crafted inputs; the risk is minor and the JSON.stringify alternative adds overhead.

Low
Guard against fused reserved keywords in field pattern

The ADDITIVE_RE is constructed once at module load but extractGroups in another file
uses a global regex with lastIndex. Since ADDITIVE_RE here is used only with exec on
a fresh string (not stateful via /g), that's fine — but ensure the pattern's FIELD
group cannot match a leading operator keyword like NOT. The current FIELD pattern
[A-Za-z_][\w.]* will match NOTage as a single token if the caller ever passes text
with fused keywords. Consider tightening by rejecting names that begin with reserved
keywords, or documenting that the caller must pass tokenized text only.

packages/osd-monaco/src/ppl/lint/explain/explain_quick_fix.ts [65]

-const ADDITIVE_RE = new RegExp(`^${FIELD}\\s*([+-])\\s*${UINT}\\s*${CMP}\\s*${SINT}$`);
+// Field pattern: rejects fused reserved keywords (NOT, AND, OR) at the start
+const FIELD_RESERVED = '(?!(?:NOT|AND|OR)(?![\\w.]))';
+const ADDITIVE_RE = new RegExp(`^${FIELD_RESERVED}${FIELD}\\s*([+-])\\s*${UINT}\\s*${CMP}\\s*${SINT}$`, 'i');
Suggestion importance[1-10]: 3

__

Why: The PR's own comments and tests (NOTage-2>30) explicitly state that the caller passes only single comparison node text without fused keywords, and a test verifies this case returns undefined because NOTage is not a mapped field. The suggestion adds defense-in-depth but is largely redundant given the type-map lookup guard.

Low
Preserve no-context lint call semantics

Previously, lint() short-circuited to this.analyzer.lint(content) with no context to
preserve default behavior. Now it always passes rebuildRunContext(context), which
returns undefined when context is missing — this changes the arity of the call.
Confirm analyzer.lint(content, undefined) behaves identically to
analyzer.lint(content); otherwise consumers not sending context will see behavior
changes.

packages/osd-monaco/src/ppl/worker/ppl.worker.ts [37-41]

-async lint(...): Promise<LintResult> {
+async lint(
+    content: string,
+    context?: SerializableLintContext
+  ): Promise<LintResult> {
     if (!this.analyzer) {
       this.analyzer = getPPLLanguageAnalyzer();
     }
-    return this.analyzer.lint(content, rebuildRunContext(context));
+    const runContext = rebuildRunContext(context);
+    return runContext
+      ? this.analyzer.lint(content, runContext)
+      : this.analyzer.lint(content);
   }
Suggestion importance[1-10]: 3

__

Why: Since context is typed as optional (context?: SerializableLintContext), calling analyzer.lint(content, undefined) should be semantically equivalent to analyzer.lint(content) in TypeScript. The suggestion is a verification request with limited actual impact.

Low
Ensure timeout timer is cleared reliably

validate races the validator against a timeout but does not clear the timeout when
the validator rejects. Because the catch block returns undefined without going
through the finally clearTimeout path when the timeout was already assigned but the
race resolves via rejection before timing out, the pending timer can still fire and
hold the event loop. Ensure the timeout is always cleared regardless of outcome (the
current finally handles the resolve/timeout paths, but the rejection path in
Promise.race can leave the timer pending until the deadline).

packages/osd-monaco/src/ppl/lint/explain/explain_attribution.ts [239-252]

 async validate(queries: string[]): Promise<boolean[] | undefined> {
   if (
     queries.length === 0 ||
     queries.some((query) => !query) ||
     !this.inputs.isCurrent() ||
     Date.now() >= this.deadline
   ) {
     return undefined;
   }
 
   let timeout: ReturnType<typeof setTimeout> | undefined;
   const timeoutResult = new Promise<undefined>((resolve) => {
-    timeout = setTimeout(() => resolve(undefined), Math.max(1, this.deadline - Date.now()));
+    timeout = setTimeout(() => {
+      timeout = undefined;
+      resolve(undefined);
+    }, Math.max(1, this.deadline - Date.now()));
   });
Suggestion importance[1-10]: 2

__

Why: The existing finally block already calls clearTimeout(timeout) regardless of resolve/reject outcome, so the timer is cleared correctly. The suggestion's reasoning about the rejection path is incorrect.

Low
Respect caller-provided pipe-first flag

lintWithGrammar spreads context into the runLint options and then overrides
grammarSurface, grammarHash, and isPipeFirst. If a caller passes a context that
already contains grammarSurface (e.g. 'compiled'), the spread order correctly
overrides it, but if the caller passes isPipeFirst explicitly, it is also overridden
by the locally-computed value. This may be intentional, but consider preferring the
caller's isPipeFirst when provided, since the caller may have more accurate
information (e.g. after column remapping in a pipeline).

src/plugins/data/public/antlr/opensearch_ppl/headless_ppl_lint.ts [178-183]

+context: {
+  ...context,
+  grammarSurface: 'runtime-bundle',
+  grammarHash: grammar.grammarHash,
+  isPipeFirst: context?.isPipeFirst ?? isPipeFirst,
+},
 
-
Suggestion importance[1-10]: 2

__

Why: The locally-computed isPipeFirst is derived deterministically from the query text and is the authoritative value for this parse. Preferring a caller-provided value could introduce inconsistency with how the query was actually parsed.

Low

Previous suggestions

Suggestions up to commit 7294d5a
CategorySuggestion                                                                                                                                    Impact
Possible issue
Fix malformed data source URL construction

The MDS id is appended as a path segment after a slash but uses = as if it were a
query parameter, producing a malformed URL like
/api/directquery/dsl/indices.getFieldMapping/dataSourceMDSId=. Either append it as a
proper query parameter or use the correct path convention expected by the route.

src/plugins/data/public/ppl_lint/disabled_object_fields.ts [85-88]

-const url = mdsId
-  ? `${DSL_MAPPING_URL}/dataSourceMDSId=${encodeURIComponent(mdsId)}`
-  : DSL_MAPPING_URL;
+const url = DSL_MAPPING_URL;
+const query: Record<string, string> = { index: pattern };
+if (mdsId) {
+  query.dataSourceMDSId = mdsId;
+}
+const resp = await http.get(url, { query });
Suggestion importance[1-10]: 8

__

Why: The URL construction ${DSL_MAPPING_URL}/dataSourceMDSId=${...} uses = after a slash which looks malformed. However, this may be the actual convention used by the directquery route; without seeing that route's definition, the concern is plausible but not definitively confirmed. Still, it flags a likely real bug worth reviewing.

Medium
Guard against missing dot in path

collectDottedPathNodes may yield non-dotted names too (bare fields, per the test
"does not flag a bare field with no dotted path"). When path contains no .,
indexOf('.') returns -1 and path.slice(0, -1) produces the field name minus its last
character — which could coincidentally match a disabled root and false-fire. Guard
on the presence of a dot before computing root.

packages/osd-monaco/src/ppl/lint/rules/enabled_false_object.ts [32-37]

 for (const node of collectDottedPathNodes(tree, ruleNameToIndex)) {
   const path = node.getText();
-  const root = path.slice(0, path.indexOf('.'));
+  const dotIndex = path.indexOf('.');
+  if (dotIndex < 0) {
+    continue;
+  }
+  const root = path.slice(0, dotIndex);
   if (!disabled.has(root)) {
     continue;
   }
Suggestion importance[1-10]: 7

__

Why: Valid concern: if collectDottedPathNodes returns non-dotted names, path.slice(0, -1) would truncate the last character which could accidentally match. The test "does not flag a bare field" passes only if the collector filters or by accident; adding an explicit guard is defensive and correct.

Medium
General
Prevent unhandled rejection when timeout wins race

When validateGeneratedQueries rejects, Promise.race rejects and the finally block
runs, but the timeout still fires later since it is only cleared in finally (which
does clear it) — however, if the timeout wins the race first,
validateGeneratedQueries's eventual rejection becomes an unhandled promise
rejection. Attach a .catch(() => undefined) to the validator promise to suppress
this.

packages/osd-monaco/src/ppl/lint/explain/explain_attribution.ts [249-257]

 let timeout: ReturnType<typeof setTimeout> | undefined;
 const timeoutResult = new Promise<undefined>((resolve) => {
   timeout = setTimeout(() => resolve(undefined), Math.max(1, this.deadline - Date.now()));
 });
 try {
   const result = await Promise.race([
-    this.inputs.validateGeneratedQueries(queries),
+    this.inputs.validateGeneratedQueries(queries).catch(() => undefined),
     timeoutResult,
   ]);
Suggestion importance[1-10]: 6

__

Why: Valid concern: if the timeout wins and validateGeneratedQueries later rejects, it becomes an unhandled rejection. The try/catch around the race only catches the winner. Adding .catch on the validator promise is a reasonable defensive fix.

Low
Track probe request cancellation for supersession

The abort controller for the baseline explain request is stored on
lintAbortControllers, but the isolation probes inside runExplainIsolation create
their own controllers internally that aren't tracked here. If a new lint pass starts
during the probe phase, abortInFlightExplain only aborts the baseline controller
(which is already settled by then), leaving probe requests to complete uselessly.
Consider passing an outer signal into the isolation pass or tracking probe
controllers as well.

packages/osd-monaco/src/ppl/language.ts [482-491]

+const controller = typeof AbortController === 'undefined' ? undefined : new AbortController();
+if (controller) {
+  lintAbortControllers.set(model.id, controller);
+}
+const resolution = await explainCache.resolveResult(
+  http,
+  prepared.query,
+  lintContext?.dataSourceId,
+  { cacheKey: prepared.cacheKey, signal: controller?.signal }
+);
 
-
Suggestion importance[1-10]: 5

__

Why: Valid observation: probe requests inside runExplainIsolation are not tracked by lintAbortControllers, so supersession relies only on isCurrent() guards rather than actual request cancellation. However, isCurrent guards within the probes do abandon stale work, mitigating the impact.

Low
Remove dead fallback after shape validation

isValidBundleShape already asserts literalNames and symbolicNames are present
arrays, so the || [] fallback is dead code that also weakens the type narrowing.
More importantly, buildSymbolicNameToTokenType(bundle.symbolicNames) below is called
with the original (unmapped) array — if a symbolic name is the empty string, it will
be skipped by the if (name) check there, which is fine, but the two symbolicNames
values are now inconsistent (one has nulls, one has empty strings). This is harmless
today but a subtle footgun.

src/plugins/data/public/antlr/opensearch_ppl/ppl_grammar_deserialize.ts [174-175]

-const literalNames = (bundle.literalNames || []).map((n) => (n === '' ? null : n));
-const symbolicNames = (bundle.symbolicNames || []).map((n) => (n === '' ? null : n));
+const literalNames = bundle.literalNames.map((n) => (n === '' ? null : n));
+const symbolicNames = bundle.symbolicNames.map((n) => (n === '' ? null : n));
Suggestion importance[1-10]: 4

__

Why: Valid observation: since isValidBundleShape already validates these arrays, the || [] fallback is redundant. Minor code cleanliness improvement.

Low
Deduplicate names before capping

Deduplicate names before sorting: an index can appear both as itself and via an
alias/data-stream entry in the resolve_index response, which would inflate the count
against the 5000-cap and produce duplicate entries the rule then treats as distinct
visible indices.

src/plugins/data/public/ppl_lint/visible_indices.ts [50-53]

-if (names.length > MAX_VISIBLE_INDICES) {
+const unique = Array.from(new Set(names));
+if (unique.length > MAX_VISIBLE_INDICES) {
   return [];
 }
-return names.sort();
+return unique.sort();
Suggestion importance[1-10]: 4

__

Why: Deduplication is a reasonable minor improvement, but the resolve_index API typically returns disjoint buckets for indices/aliases/data_streams, so duplicates are unlikely in practice. Marginal impact.

Low
Decrement refcount when signalless subscriber settles

A signal-less subscriber increments liveSubscribers but never decrements it, which
is intentional to pin the request open. However, since the promise is returned
directly without any .finally handler to decrement on settlement, the counter is
fine — but this behavior effectively means once any signal-less caller subscribes,
no combination of aborts can cancel the fetch. Document or consider decrementing on
promise settlement to allow cancellation when the pinning caller is done.

packages/osd-monaco/src/ppl/lint/explain/explain_cache.ts [211-216]

 let liveSubscribers = 0;
 const subscribe = (signal?: AbortSignal): Promise<ExplainResolution> => {
   if (!signal) {
     liveSubscribers++;
-    return promise;
+    return promise.finally(() => { liveSubscribers--; });
   }
Suggestion importance[1-10]: 3

__

Why: The signal-less pinning behavior is intentional and documented; changing it could break the design's cancellation semantics. The suggestion itself acknowledges this is intentional, making it low-impact.

Low
Use the default constant for unknown modes

The mode normalization silently coerces any non-'thorough' value (including a valid
'fast' string, but also typos like 'Thorough' with different casing) to 'fast'.
However, since DEFAULT_EXPLAIN_MODE is 'fast', the fallback is consistent — but the
docstring says "Defaults to DEFAULT_EXPLAIN_MODE when unset or malformed". Consider
explicitly comparing against both known modes and defaulting via the constant to
keep behavior aligned if DEFAULT_EXPLAIN_MODE ever changes.

src/plugins/data/public/ppl_lint/lint_overrides.ts [75-79]

 if (
   stored &&
   typeof stored === 'object' &&
   Array.isArray((stored as { rules?: unknown }).rules)
 ) {
   const rawMode = (stored as { mode?: unknown }).mode;
+  const mode: ExplainMode =
+    rawMode === 'thorough' || rawMode === 'fast' ? rawMode : DEFAULT_EXPLAIN_MODE;
   return {
-    mode: rawMode === 'thorough' ? 'thorough' : 'fast',
+    mode,
     rules: (stored as { rules: StoredRule[] }).rules,
   };
 }
Suggestion importance[1-10]: 3

__

Why: Minor consistency improvement; current behavior is functionally equivalent since DEFAULT_EXPLAIN_MODE is 'fast', but explicit validation against known modes is slightly more robust to future changes.

Low
Avoid shared regex lastIndex state

CAPTURE_GROUP_OPENER is a module-level regex with the g flag, and while lastIndex is
reset at the start of extractGroups, this function is invoked per parse-tree command
in a loop. If the code path ever becomes reentrant (e.g. via a rule that recurses or
is called concurrently in workers sharing module state), the shared lastIndex could
cause skipped matches. Consider creating a fresh regex per call or using matchAll to
avoid the shared-state hazard.

packages/osd-monaco/src/ppl/lint/rules/invalid_capture_group_name.ts [47-68]

-while ((match = CAPTURE_GROUP_OPENER.exec(literalRaw)) !== null) {
+const opener = /\(\?(P?)<([^>]*)>/g;
+let match: RegExpExecArray | null;
+while ((match = opener.exec(literalRaw)) !== null) {
   const isPythonOpener = match[1] === 'P';
   const name = match[2];
Suggestion importance[1-10]: 3

__

Why: The concern is theoretical: lastIndex is reset at the start of extractGroups, and JavaScript is single-threaded, so reentrancy is unlikely. A local regex would be slightly cleaner but not a real bug.

Low
Reduce querySelector calls on hot event path

node.querySelector(HOVER_BODY_SELECTOR) runs on every mouseleave and mouseenter
event dispatched to the overflow container, which can include unrelated widgets.
Since mouseleave/enter events fire frequently during pointer movement across many
child overflow widgets, calling querySelector on each event target on the capture
path may become a hot path. Consider caching the current hover widget node (updated
via a subtree observer or Monaco widget-visibility hook) or checking a cheaper
condition first.

src/plugins/data/public/ui/query_editor/lint_hover_persistence.ts [88-96]

 const hoverWidgetOf = (node: EventTarget | null): Element | null => {
   if (!(node instanceof Element)) {
     return null;
+  }
+  // Cheap check first: descendant hover body implies this is the widget wrapper.
+  const body = node.firstElementChild;
+  if (body && body.classList?.contains('monaco-hover')) {
+    return node;
   }
   if (node.querySelector(HOVER_BODY_SELECTOR)) {
     return node;
   }
   return node.closest(HOVER_BODY_SELECTOR);
 };
Suggestion importance[1-10]: 3

__

Why: The performance concern is speculative; querySelector on a small overflow-widget subtree is unlikely to be a real hot path, and the suggested optimization adds complexity for negligible gain.

Low
Prevent infinite recursion on cyclic arrays

valueText recurses into arrays without cycle protection, and safeStringify on
objects with cyclic references will throw and return '' — losing information. Since
rel nodes can reference other rels via inputs, consider tracking visited nodes to
avoid infinite recursion should an array contain a self-reference.

packages/osd-monaco/src/ppl/lint/explain/explain_tree_utils.ts [16-27]

-export function valueText(value: unknown): string {
+export function valueText(value: unknown, seen: WeakSet<object> = new WeakSet()): string {
   if (typeof value === 'string') {
     return value;
   }
   if (Array.isArray(value)) {
-    return value.map(valueText).join('\n');
+    if (seen.has(value)) return '';
+    seen.add(value);
+    return value.map((v) => valueText(v, seen)).join('\n');
   }
   if (value && typeof value === 'object') {
     return safeStringify(value);
   }
   return value == null ? '' : String(value);
 }
Suggestion importance[1-10]: 2

__

Why: Explain plan responses are JSON-serialized from the server, so they cannot contain cyclic references. The cycle protection is unnecessary defensive coding.

Low

Hanyu Wei added 2 commits July 28, 2026 16:36
…is PR's new rules

Two follow-throughs from opensearch-project#12394's review that its own rules got but the rules
added here did not.

1. Eric's source-match finding (lint_runner.ts:40). Rules that read the selected
   dataset's metadata must self-suppress when the query's explicit `source=`
   names a different index, or they lint against the wrong mapping. opensearch-project#12394 added
   `sourceScoped` and set it on all four of its context-fed rules;
   `enabled-false-object` (reads `disabledObjectFields`) and `rex-scan-cost`
   (reads `typeMap`) have the same exposure and now carry it too.

   `wildcard-source-zero-match` is deliberately left unscoped: it reads the
   cluster-wide `visibleIndices` list, not dataset metadata, and suppressing it
   on a mismatch would silence it in exactly the case it exists to catch. That
   exemption is now pinned by a test so it is not "fixed" into a bug later.

   Extends source_mismatch_suppression.test.ts rather than adding a suite:
   suppressed-on-mismatch and still-fires-on-match for both rules. Verified the
   suppression assertions fail without the catalog flags.

2. Joshua's test-layout ask (agg_on_text.test.ts:6). opensearch-project#12394 moved its rule tests
   beside their sources; the three rule tests added here went to
   `rules/__tests__/`. Moved to match, leaving the pre-existing `__tests__` files
   alone as that reply said.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
CI's "Lint and validate" failed on jest/no-identical-title: two identically
titled `an undeclared lint-rules setting (queryEnhancements disabled)` blocks.

The duplicate arises where main and opensearch-project#12411 both added the same test. I had
removed it once, but that fix lived in a hand-rolled merge commit which was
discarded when this branch was rebased onto opensearch-project#12411's own main merge, so it
came back. Keeping the first block, which is the superset: it also asserts
readExplainMode.

Signed-off-by: Hanyu Wei <weihanyu@amazon.com>
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 54b3dd8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant