diff --git a/docs/implementation/cli-commands.md b/docs/implementation/cli-commands.md index dd7d0ee8..ea4b6aee 100644 --- a/docs/implementation/cli-commands.md +++ b/docs/implementation/cli-commands.md @@ -10,8 +10,8 @@ The CLI exposes four primary top-level commands: `example`, `languages`, and `fe |---|---|---|---| | `init` | — | `-y, --yes`, `--skip-login` | Authenticate and set up MCP server for coding agents | | `example ` | `-l, --lang ` | `--license `, `--explain`, `--json` | Search for code examples | -| `search ` | `--in ` | `--source `, `--kind `, `--category `, `--path-prefix `, `--intent `, `--public`, `--name `, `--lang `, `--limit `, `--offset `, `--wait `, `--json` | Unified indexed search across dependency/repository code, docs, and symbols | -| `search-status ` | `` | `--json` | Check progress or fetch final results for a prior unified search | +| `search ` | `--in ` | `--source `, `--kind `, `--category `, `--path-prefix `, `--intent `, `--public`, `--name `, `--lang `, `--allow-partial`, `--limit `, `--offset `, `--wait `, `--json` | Unified indexed search across dependency/repository code, docs, and symbols | +| `search-status ` | `` | `--json` | Check progress, fetch partial hits, or fetch final results for a prior unified search | | `languages [query]` | — | `--json` | List or filter supported languages | | `feedback ` | `--accept` or `--reject` | `-m, --message `, `--json` | Submit feedback on a search result | | `code search [query]` | package spec | `--keywords`, `--keyword`, `--match-mode`, `--category`, `--kind`, `--file`, `--intent`, `--limit`, `--wait`, `--json` | Search indexed dependency source code | @@ -21,7 +21,7 @@ The CLI exposes four primary top-level commands: `example`, `languages`, and `fe | `pkg changelog [spec]` | package spec OR `--repo-url` | `--from`, `--to`, `--limit`, `--git-ref`, `--no-body`, `--verbose`, `--json` | Release notes / changelog entries for a package or GitHub repo (GitHub Releases, CHANGELOG.md, or HexDocs). Default shows each entry with a 10-line body preview; `--verbose` uncaps, `--no-body` drops. | | `code files [spec] [path-prefix]` | package spec OR `--repo-url` + `--git-ref`; optional `[path-prefix]` | `--limit`, `--wait`, `--verbose`, `--json` | List files in an indexed dependency. `[path-prefix]` is a literal directory prefix (not a glob). Plain output is one path per line; `--verbose` adds language / type / size annotations. Indexing-retry via `--wait` or the `availableVersions` hint in the error envelope. | | `code read ` | package spec OR `--repo-url` + `--git-ref`; plus `` | `--lines`, `--start`, `--end`, `--wait`, `--verbose`, `--json` | Read a file's contents. Plain output is the raw file bytes (pipe-friendly); `--verbose` adds a header and a line-number gutter. `--lines 10-40` concise form; `--start`/`--end` equivalent. Binary files show a sentinel line. | -| `code grep [spec] [path-prefix]` | package spec OR `--repo-url` + `--git-ref`; plus `` and optional `[path-prefix]` | `--path`, repeatable `--glob`, repeatable `--ext`, `--regex`, `--case-sensitive`, `-C/-A/-B`, `--exclude-docs`, `--exclude-tests`, `--limit`, `--per-file-limit`, `--cursor`, `--wait`, `--verbose`, `--json` | Deterministic text grep over indexed dependency or repository source. Defaults to whole-target, literal, case-insensitive matching; narrow with `[path-prefix]`, `--path`, `--glob`, or `--ext`. Plain output is `file:line:text`; `--verbose` groups matches by file. | +| `code grep [spec] [path-prefix]` | package spec OR `--repo-url` + `--git-ref`; plus `` and optional `[path-prefix]` | `--path`, repeatable `--glob`, repeatable `--ext`, `--regex`, `--case-sensitive`, `-C/-A/-B`, `--exclude-docs`, `--exclude-tests`, `--limit`, `--per-file-limit`, `--cursor`, `--symbol-field`, `--wait`, `--verbose`, `--json` | Deterministic text grep over indexed dependency or repository source. Defaults to whole-target, literal, ASCII case-insensitive matching; non-ASCII letters match case-sensitively. Narrow with `[path-prefix]`, `--path`, `--glob`, or `--ext`. Plain output is `file:line:text`; `--verbose` groups matches by file. | ### `githits init` @@ -54,13 +54,13 @@ Default output is markdown (the API response). With `--explain`, an AI-generated ``` githits search "router middleware" --in npm:express -githits search "handler" --in npm:express --kind function --path-prefix src/ githits search '"body parser" OR multer' --in npm:express --source docs -githits search "retry logic" --in npm:got --in npm:ky --source code -githits search "createServer" --in npm:@types/node --name createServer --lang typescript --json +githits search "compose" --in npm:lodash --source code --kind function +githits search "debounce" --in npm:lodash --source symbol +githits search "composeArgs" --in npm:lodash --name composeArgs --json ``` -Unified search spans indexed dependency and repository code, docs, and explicit symbols. Structured flags are the primary UX. They are compiled together with the raw query using `AND` semantics before the request reaches the backend. +Unified search spans indexed dependency and repository code, docs, and explicit symbols. The positional query is the backend discovery syntax, not a raw pass-through to a per-source search engine. It supports implicit `AND`, uppercase `OR`, parentheses, unary `-`, quoted phrases, semantic qualifiers (`kind:`, `category:`, `path:`, `lang:`, `name:`, `intent:`), and routing qualifiers (`registry:`, `package:`, `version:`, `repo:`). Structured flags are compiled together with the query using `AND` semantics before the request reaches the backend. **Decision guide.** Use `githits example` for canonical cross-project examples. Use `githits search` for indexed dependency/repository search. Use `githits search --source symbol` when you want symbol-shaped unified search without dropping to the older dedicated `code search` surface. @@ -68,9 +68,9 @@ Unified search spans indexed dependency and repository code, docs, and explicit **Sources and filters.** `--source docs|code|symbol` is repeatable; omitting it delegates source selection to backend AUTO. Use `--source symbol` when you want symbol-shaped search results without dropping down to the older `code search` surface. `--category` is the broad filter (`callable`, `type`, `module`, `data`, `documentation`); `--kind` is the precise taxonomy. `--path-prefix`, `--intent`, and `--public` narrow the result set further. `--name` and `--lang` compile into query qualifiers instead of becoming separate backend fields. -**Production intent by default.** When `--intent` is omitted, unified search defaults to `production` intent. This removes test / benchmark / example noise where the backend supports the filter. Some sources can still ignore `fileIntent`; when they do, the JSON `sourceStatus` block and terminal notes report that explicitly. +**Production intent by default.** When `--intent` is omitted, unified search defaults to `production` intent for AUTO, code, and symbol searches. This removes test / benchmark / example noise where the backend supports the filter. Explicit docs-only searches do not get a file-intent default because docs do not support that filter. Some sources can still ignore `fileIntent`; when they do, the JSON `sourceStatus` block and terminal notes report that explicitly. -**Complete-by-default results.** The CLI always forces `allowPartialResults: false`. If indexing does not complete within the wait window, the command returns a `searchRef` and progress summary instead of partial hits. `--wait` is in seconds (0-60, default 20). +**Complete-by-default results.** The CLI sends `allowPartialResults: false` unless `--allow-partial` is passed. If indexing does not complete within the wait window, the default behavior returns a `searchRef` and progress summary instead of partial hits. With `--allow-partial`, available hits are included while remaining sources continue indexing. `--wait` is in seconds (0-60, default 20). **Output.** Plain output preserves backend ranking order. It starts with a lightweight per-type count summary, then shows one result per block. The header line is optimized for scanning and copy-paste follow-up: `target path:range [type] - title`. For file-backed hits, that header can be turned directly into a `githits code read` call because `code read` accepts `path:start-end` suffixes. Summaries are rendered verbatim from the backend response. Labels are: `docs page` (hosted package docs), `repo doc` (documentation-like block from a repository file), `repo code` (code block from a repository file), and `repo symbol` (explicit symbol hit from the repository index). `--json` emits the shared success/error envelope used by the MCP `search` tool, including a full `query` echo for initial searches. @@ -83,7 +83,7 @@ githits search-status ref_abc123 githits search-status ref_abc123 --json ``` -Follow-up for a prior unified search. Use the `searchRef` returned by `githits search` when the initial request could not complete inside the wait window. +Follow-up for a prior unified search. Use the `searchRef` returned by `githits search` when the initial request could not complete inside the wait window. If the original request used `--allow-partial`, `search-status` can return updated partial hits before final completion. `search-status` deliberately does **not** reconstruct the original structured request echo. The backend status API exposes progress, final results, and the backend-normalized query string, but it does not expose the original target/filter/defaulting inputs. The JSON payload therefore contains only fields the follow-up endpoint can actually know: `{completed, searchRef?, progress?, result?}`. @@ -322,7 +322,7 @@ githits code grep --repo-url https://github.com/expressjs/express --git-ref main githits code grep npm:express middleware --path lib/express.js --json ``` -Deterministic text grep over indexed dependency or repository source. Defaults to case-insensitive literal matching across the whole target. Pass `[path-prefix]`, `--path`, `--glob`, or `--ext` to narrow scope. `--regex` switches to RE2 regex mode. Max pattern 200 chars. For discovery and ranking, use top-level `githits search` instead. +Deterministic text grep over indexed dependency or repository source. Defaults to ASCII case-insensitive literal matching across the whole target; non-ASCII letters match case-sensitively. Pass `[path-prefix]`, `--path`, `--glob`, or `--ext` to narrow scope. `--regex` switches to RE2 regex mode. Whole-target regexes must include at least one literal substring the index can use for pre-filtering. Max pattern 200 UTF-8 bytes. For discovery and ranking, use top-level `githits search` instead. Repeat `--symbol-field` to hydrate enclosing symbol metadata; hints appear under each `--verbose` match, full payload in `--json`. **Plain output (default).** One `file:line:text` record per match on stdout, pipe-friendly and deterministic. `-C/--context`, `-A/--after-context`, and `-B/--before-context` add surrounding lines. Distinct match groups are separated by `--`. diff --git a/docs/implementation/mcp-cli-parity.md b/docs/implementation/mcp-cli-parity.md index 6748f9d1..8b00f724 100644 --- a/docs/implementation/mcp-cli-parity.md +++ b/docs/implementation/mcp-cli-parity.md @@ -37,6 +37,8 @@ the test suite anchors the doc. - **MCP arguments** use `snake_case`. They are the wire contract agents see; the JSON-schema description is the primary UX. - **CLI flags** use `--kebab-case`. They are the user-facing surface. + `allow_partial_results` maps to CLI `--allow-partial` because the CLI + name reads better as a command flag while preserving the same behavior. - **Public enum values** are lowercase strings on both surfaces (`production`, `test`, `summary`, `all`). - **Service coercion** from lowercase enum values to the internal @@ -92,6 +94,10 @@ the test suite anchors the doc. echo. Follow-up `search_status` responses intentionally omit that echo and return only backend-known fields: `{completed, searchRef?, progress?, result?}`. +- Unified `search` is complete-by-default (`allowPartialResults: false`). + `allow_partial_results` / `--allow-partial` opt into backend partial + payloads while indexing continues; incomplete JSON envelopes may then + carry non-empty `results` plus the `searchRef`. ### `PARITY-ERROR-ENVELOPE` @@ -438,8 +444,14 @@ so envelope-drift surfaces in the test rather than at an agent. The shared request builder compiles `path`, `path_prefix`, and `globs` into backend `pathSelectors`, keeps `allowUnscoped` internal-only, and defaults grep to whole-target, literal, - case-insensitive matching. The shared response builder keeps CLI - `--json` and MCP payloads byte-identical for equivalent inputs. + ASCII case-insensitive matching; non-ASCII letters match + case-sensitively. Whole-target regexes must include at least one + literal substring the backend index can use for pre-filtering. + `symbol_fields` / `--symbol-field` passes backend symbol hydration + field names through to `symbolFields` and the response envelope + carries `matches[].symbol` when the backend hydrates it. The shared + response builder keeps CLI `--json` and MCP payloads byte-identical + for equivalent inputs. - **Parity assertion policy** (coded in the three parity tests): diff --git a/docs/implementation/tools.md b/docs/implementation/tools.md index eedfe9ed..9ed73ab7 100644 --- a/docs/implementation/tools.md +++ b/docs/implementation/tools.md @@ -22,8 +22,8 @@ Both expose the same tools with identical names, parameters, and descriptions. T | `get_example` | `query`, `language`, `license_mode?` | Search for canonical code examples. Returns markdown-formatted results. | | `search_language` | `query` | Find supported programming language names before searching. | | `feedback` | `solution_id`, `accepted`, `feedback_text?` | Submit feedback on a search result to improve quality. | -| `search` | `query`, `target?`, `targets?`, `sources?`, `category?`, `kind?`, `path_prefix?`, `file_intent?`, `public_only?`, `name?`, `language?`, `limit?`, `offset?`, `wait_timeout_ms?` | Unified indexed dependency/repository search across code, docs, and symbols. Defaults `file_intent` to production when omitted. Prefer `sources:["symbol"]` for symbol-shaped unified search. Returns complete results or a `searchRef` follow-up state. | -| `search_status` | `search_ref` | Check progress or fetch final results for a prior unified search. | +| `search` | `query`, `target?`, `targets?`, `sources?`, `category?`, `kind?`, `path_prefix?`, `file_intent?`, `public_only?`, `name?`, `language?`, `allow_partial_results?`, `limit?`, `offset?`, `wait_timeout_ms?` | Unified indexed dependency/repository discovery search across code, docs, and symbols. Defaults `file_intent` to production for AUTO/code/symbol searches when omitted; explicit docs-only searches do not get a file-intent default. Prefer `sources:["symbol"]` for symbol-shaped unified search. Complete-by-default; set `allow_partial_results: true` to receive available partial hits while indexing continues. | +| `search_status` | `search_ref` | Check progress, fetch partial hits when the original request used `allow_partial_results: true`, or fetch final results for a prior unified search. | | `search_symbols` | `target`, `query?`, `keywords?`, `match_mode?`, `category?`, `kind?`, `file_path?`, `limit?`, `file_intent?`, `wait_timeout_ms?` | Capability-gated code navigation search over indexed dependency source. | | `package_summary` | `registry`, `package_name` | Package overview: latest version, license, description, repository, downloads, GitHub metadata, install command, and known vulnerabilities. Always returns the latest published version. | | `package_vulnerabilities` | `registry`, `package_name`, `version?`, `min_severity?`, `include_withdrawn?` | Known vulnerabilities for a package on npm, PyPI, Hex, or Crates. Count summary, per-advisory OSV ID + severity + affected/fix ranges, and upgrade paths. Malware is surfaced in a disjoint bucket. | @@ -31,12 +31,14 @@ Both expose the same tools with identical names, parameters, and descriptions. T | `package_changelog` | `registry?`, `package_name?`, `repo_url?`, `from_version?`, `to_version?`, `limit?`, `git_ref?`, `include_bodies?` | Release notes or changelog entries for a package or GitHub repo. Default latest mode returns the ten most recent entries; `from_version` switches to range mode (no count cap). Dual addressing (spec vs repo URL) mutually exclusive. Response always includes `source` (`"releases"` / `"changelog_file"` / `"hexdocs"`), `mode` (`"latest"` / `"range"`), and `entries: { count, items }` with full markdown bodies by default; set `include_bodies: false` for a lean version / date / URL timeline. | | `list_files` | `target`, `path_prefix?`, `limit?`, `wait_timeout_ms?` | List files in an indexed dependency. Returns `{total, hasMore, files: [{path, name, language, fileType, byteSize}], resolution, indexedVersion}`. Dual addressing via `target.registry + target.package_name` (spec) or `target.repo_url + target.git_ref` (repo). `path_prefix` is a literal directory prefix — NOT a glob (`*.ts` won't match); glob / pattern filtering is an upstream enhancement. Emits an `INDEXING` error envelope when the dependency is being indexed on-demand — retry with a longer `wait_timeout_ms` or pick a version from `details.availableVersions`. | | `read_file` | `target`, `path`, `start_line?`, `end_line?`, `wait_timeout_ms?` | Read a file from an indexed dependency. Default full file; use `start_line` / `end_line` for a bounded range. Binary files set `isBinary: true` and omit `content` — agents branch on the flag. On `NOT_FOUND` / `FILE_NOT_FOUND` call `list_files` to discover the actual path. | -| `grep_repo` | `target`, `pattern`, `path?`, `path_prefix?`, `globs?`, `extensions?`, `pattern_type?`, `case_sensitive?`, `exclude_doc_files?`, `exclude_test_files?`, `context_lines?`, `context_lines_before?`, `context_lines_after?`, `max_matches?`, `max_matches_per_file?`, `cursor?`, `wait_timeout_ms?` | Deterministic text grep over indexed dependency or repository source. Defaults to literal, case-insensitive matching across the whole target; narrow with `path`, `path_prefix`, `globs`, or `extensions`. `pattern_type: "regex"` uses RE2 syntax. Returns matches plus pagination and scan counters. | +| `grep_repo` | `target`, `pattern`, `path?`, `path_prefix?`, `globs?`, `extensions?`, `pattern_type?`, `case_sensitive?`, `exclude_doc_files?`, `exclude_test_files?`, `context_lines?`, `context_lines_before?`, `context_lines_after?`, `max_matches?`, `max_matches_per_file?`, `cursor?`, `symbol_fields?`, `wait_timeout_ms?` | Deterministic text grep over indexed dependency or repository source. Defaults to literal, ASCII case-insensitive matching across the whole target; non-ASCII letters match case-sensitively. Narrow with `path`, `path_prefix`, `globs`, or `extensions`. `pattern_type: "regex"` uses RE2 syntax; whole-target regexes must include at least one literal substring for index pre-filtering. Returns matches plus pagination and scan counters; `symbol_fields` hydrates enclosing symbol metadata on each match. | `search`, `search_status`, `search_symbols`, `package_summary`, `package_vulnerabilities`, `package_dependencies`, `package_changelog`, `list_files`, `read_file`, and `grep_repo` are only registered when package/source access is open for the current session. The package/source service URL can be overridden via `GITHITS_CODE_NAV_URL` for local development. `search_symbols` shares request-construction, error classification, and JSON-payload shape with the CLI `githits code search` command via shared helpers under `src/shared/`. The parity rules are codified in [`mcp-cli-parity.md`](./mcp-cli-parity.md); the parity test (`src/tools/search-symbols-parity.test.ts`) asserts that both surfaces emit identical JSON for equivalent inputs. +**Unified `search` query syntax.** The `search.query` field is the backend discovery query syntax, not a raw pass-through to a per-source search engine. It supports implicit `AND`, uppercase `OR`, parentheses, unary `-`, quoted phrases, semantic qualifiers (`kind:`, `category:`, `path:`, `lang:`, `name:`, `intent:`), and routing qualifiers (`registry:`, `package:`, `version:`, `repo:`). The backend parses the query once and compiles it per source. Structured `name` and `language` inputs are compiled into `name:` / `lang:` qualifiers and AND-ed with the query before sending. Per-source support, ignored features, and incompatibilities are reported in `sourceStatus`. + **Response shape.** `search_symbols` always requests `mode: DETAILED` and always selects the `code`, `resolution`, `kind`, and `category` fields. Responses include each match's full source code, precise symbol kind (from the unified symbol taxonomy), broad symbol category, and line range. The tool does not expose `mode` or `verbose` inputs — the service layer makes the choice once so both surfaces get the richest response without callers juggling the knobs. The legacy `chunkType` field is no longer selected or surfaced client-side; `kind` is the single source of truth for taxonomy. The text payload is always valid JSON (whether the result is success or error), so MCP clients can parse `content[0].text` without branching on `isError`. **Filter parameters.** `category` is the preferred surface for filtering (`callable`, `type`, `module`, `data`, `documentation`); `kind` is for the "I want this specific construct" case (27-value taxonomy). Both filters may be combined; both route through the shared `buildSearchSymbolsParams` helper. diff --git a/src/cli.ts b/src/cli.ts index 99074d46..89b5dc9f 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -82,18 +82,29 @@ registerMcpCommand(program); registerExampleCommand(program); registerLanguagesCommand(program); registerFeedbackCommand(program); -await withTelemetrySpan("cli.register.search", () => - registerUnifiedSearchCommands(program), -); const argv = process.argv.slice(2); +const helpInvocation = isHelpInvocation(argv); +const helpRegistrationOptions = helpInvocation + ? { + capability: "enabled" as const, + expiredStoredAuth: false, + envTokenPresent: false, + } + : undefined; + +if (shouldEagerLoadSearchCommands(argv)) { + await withTelemetrySpan("cli.register.search", () => + registerUnifiedSearchCommands(program, helpRegistrationOptions), + ); +} if (shouldEagerLoadGatedCommandGroup(argv, "code")) { await withTelemetrySpan("cli.register.code-group", () => - registerCodeCommandGroup(program), + registerCodeCommandGroup(program, helpRegistrationOptions), ); } if (shouldEagerLoadGatedCommandGroup(argv, "pkg")) { await withTelemetrySpan("cli.register.pkg-group", () => - registerPkgCommandGroup(program), + registerPkgCommandGroup(program, helpRegistrationOptions), ); } @@ -127,6 +138,27 @@ function shouldEagerLoadGatedCommandGroup( ); } +function shouldEagerLoadSearchCommands(args: string[]): boolean { + const [firstArg] = args; + return ( + firstArg === "search" || + firstArg === "search-status" || + firstArg === "help" || + firstArg === "--help" || + firstArg === "-h" || + args.length === 0 + ); +} + +function isHelpInvocation(args: string[]): boolean { + return ( + args.length === 0 || + args[0] === "help" || + args.includes("--help") || + args.includes("-h") + ); +} + function getTelemetryCommandName(command: Command): string { const names: string[] = []; let current: Command | null = command; diff --git a/src/commands/code/code-nav-cli-helpers.ts b/src/commands/code/code-nav-cli-helpers.ts index cd428ef9..b729828a 100644 --- a/src/commands/code/code-nav-cli-helpers.ts +++ b/src/commands/code/code-nav-cli-helpers.ts @@ -163,6 +163,12 @@ export function formatFileErrorWithFilesHint(mapped: MappedError): string { " Retry with `--wait 60000`, use an already-indexed version/ref, or try again later.", ].join("\n"); } + if (mapped.code === "BACKEND_ERROR") { + const retry = mapped.retryable + ? "Retry in a moment; if it persists, narrow the target or file an issue." + : "Narrow the target (path, path-prefix, glob) and retry; if it persists, file an issue."; + return `${mapped.message}\n ${retry}`; + } return formatIndexingError(mapped); } diff --git a/src/commands/code/grep.test.ts b/src/commands/code/grep.test.ts index aaf6bcae..147a3415 100644 --- a/src/commands/code/grep.test.ts +++ b/src/commands/code/grep.test.ts @@ -77,6 +77,8 @@ describe("pkgGrepAction", () => { value: true, configurable: true, }); + const noColor = process.env.NO_COLOR; + delete process.env.NO_COLOR; await pkgGrepAction( "npm:express", @@ -95,6 +97,11 @@ describe("pkgGrepAction", () => { "src/index.js:4:module.exports = require('./lib/express');", ); writeSpy.mockRestore(); + if (noColor === undefined) { + delete process.env.NO_COLOR; + } else { + process.env.NO_COLOR = noColor; + } if (stdoutDescriptor) { Object.defineProperty(process.stdout, "isTTY", stdoutDescriptor); } @@ -308,6 +315,7 @@ describe("pkgGrepAction", () => { excludeDocs: true, excludeTests: true, cursor: "cursor-123", + symbolField: ["name", "qualified_path"], }, createDeps({ codeNavigationService: createMockCodeNavigationService({ grepRepo }), @@ -327,6 +335,7 @@ describe("pkgGrepAction", () => { excludeDocFiles?: boolean; excludeTestFiles?: boolean; cursor?: string; + symbolFields?: string[]; }, ] >; @@ -341,6 +350,7 @@ describe("pkgGrepAction", () => { excludeDocFiles: true, excludeTestFiles: true, cursor: "cursor-123", + symbolFields: ["name", "qualified_path"], }); expect(calls[0]?.[0]?.pathSelectors).toEqual([ { kind: "EXACT", value: "src/index.js" }, diff --git a/src/commands/code/grep.ts b/src/commands/code/grep.ts index b1e7e5a3..0bb1abf9 100644 --- a/src/commands/code/grep.ts +++ b/src/commands/code/grep.ts @@ -1,4 +1,4 @@ -import { type Command, Option } from "commander"; +import type { Command } from "commander"; import type { CodeNavigationService } from "../../services/index.js"; import { DEFAULT_WAIT_TIMEOUT_MS, @@ -10,6 +10,9 @@ import { buildGrepRepoSuccessPayload, formatGrepRepoTerminal, GREP_REPO_PATTERN_NOTE, + GREP_REPO_SYMBOL_FIELDS_NOTE, + type GrepRepoRequestBuildResult, + type GrepRepoRequestInput, InvalidPackageSpecError, requireAuth, } from "../../shared/index.js"; @@ -37,6 +40,7 @@ export interface PkgGrepCommandOptions { excludeDocs?: boolean; excludeTests?: boolean; cursor?: string; + symbolField?: string[]; wait?: string; verbose?: boolean; json?: boolean; @@ -106,7 +110,7 @@ export async function pkgGrepAction( MAX_WAIT_TIMEOUT_MS, ); - const build = buildGrepRepoParams({ + const build = buildCliGrepParams({ target, pattern, path: options.path, @@ -123,6 +127,7 @@ export async function pkgGrepAction( maxMatches, maxMatchesPerFile, cursor: options.cursor, + symbolFields: options.symbolField, waitTimeoutMs: wait, }); const result = await deps.codeNavigationService.grepRepo(build.params); @@ -147,6 +152,7 @@ export async function pkgGrepAction( maxMatches: build.params.maxMatches ?? 50, maxMatchesPerFile: build.params.maxMatchesPerFile, cursor: options.cursor, + symbolFields: build.params.symbolFields, excludeDocFiles: build.params.excludeDocFiles, excludeTestFiles: build.params.excludeTestFiles, explicit: build.explicit, @@ -210,19 +216,53 @@ function collectRepeatable(value: string, previous: string[] = []): string[] { return [...previous, value]; } +function buildCliGrepParams( + input: GrepRepoRequestInput, +): GrepRepoRequestBuildResult { + try { + return buildGrepRepoParams(input); + } catch (error) { + if (!(error instanceof InvalidPackageSpecError)) throw error; + const rewritten = error.message + .replace(/^`symbol_fields`/, "`--symbol-field`") + .replace(/`symbol_fields` value/g, "`--symbol-field` value"); + if (rewritten === error.message) throw error; + throw new InvalidPackageSpecError(rewritten); + } +} + +const CLI_GREP_PATTERN_NOTE = GREP_REPO_PATTERN_NOTE.replace( + "with no path, path_prefix, or glob", + "with no --path, [path-prefix], or --glob", +) + .replace("pass case_sensitive: true", "pass --case-sensitive") + .replace( + "(`path`, `path_prefix`, `globs`)", + "(--path, [path-prefix], --glob)", + ) + .replace( + "Use `extensions` to intersect further.", + "Use --ext to intersect further.", + ); + const PKG_GREP_DESCRIPTION = `Deterministic text grep over indexed dependency and repository source files. -${GREP_REPO_PATTERN_NOTE} +${CLI_GREP_PATTERN_NOTE} Use \`githits search\` for discovery; use \`githits code grep\` when you know the text or regex to match. Addressing: (registry:name[@version]) OR --repo-url --git-ref . In spec mode pass [path-prefix]; in repo-URL mode pass only [path-prefix]. -[path-prefix] matches the same literal prefix semantics as \`githits code files\`. Use --path for one exact file, -repeatable --glob for glob narrowing, and repeatable --ext for extension filtering. +[path-prefix] matches the same literal prefix semantics as \`githits code files\`. +Use --path for one exact file, repeatable --glob for glob narrowing, and +repeatable --ext for extension filtering. When [path-prefix], --path, and +--glob are combined they are unioned — a file matches if any selector matches; +use --ext to narrow further (intersection). -Default output is \`file:line:text\`, pipe-friendly like grep. Use -C/ -A / -B for context, --verbose for grouped output, -and --cursor to continue a paginated grep run.`; +Default output is \`file:line:text\`, pipe-friendly like grep. Use -C / -A / -B +for context, --verbose for grouped output, and --cursor to continue a paginated +grep run. --symbol-field hydrates enclosing symbol metadata (appears under each +match in --verbose output; full payload in --json).`; export function registerCodeGrepCommand(pkgCommand: Command): Command { return pkgCommand @@ -287,6 +327,12 @@ export function registerCodeGrepCommand(pkgCommand: Command): Command { "--cursor ", "Opaque nextCursor from a previous grep result", ) + .option( + "--symbol-field ", + `Repeatable; surfaces in --json and under each --verbose match. ${GREP_REPO_SYMBOL_FIELDS_NOTE}`, + collectRepeatable, + [] as string[], + ) .option( "--wait ", `Indexing wait timeout (0-${MAX_WAIT_TIMEOUT_MS}, default ${DEFAULT_WAIT_TIMEOUT_MS})`, diff --git a/src/commands/mcp-instructions.test.ts b/src/commands/mcp-instructions.test.ts index 9a49447a..dfc4e725 100644 --- a/src/commands/mcp-instructions.test.ts +++ b/src/commands/mcp-instructions.test.ts @@ -133,6 +133,8 @@ describe("buildMcpInstructions", () => { expect(instructions).toContain("`package_changelog`"); expect(instructions).toContain("`search`"); expect(instructions).toContain("`search_status`"); + expect(instructions).toContain("allow_partial_results"); + expect(instructions).toContain("partial hits"); }); it("keeps the core block first when both sections are present", () => { diff --git a/src/commands/mcp-instructions.ts b/src/commands/mcp-instructions.ts index 77ce823c..cf3937d1 100644 --- a/src/commands/mcp-instructions.ts +++ b/src/commands/mcp-instructions.ts @@ -36,10 +36,10 @@ const PACKAGE_CHANGELOG_BULLET = "- `package_changelog` — release notes for a package or GitHub repo, newest-first. Default latest mode returns the 10 most recent entries with full markdown bodies; `from_version` switches to range mode between two versions. Addressable via `registry` + `package_name` or `repo_url`. Set `include_bodies: false` for a version / date / URL timeline when bodies aren't needed."; const SEARCH_BULLET = - "- `search` — unified search across indexed dependency code, docs, and explicit symbols. Structured fields are the primary UX; omit `sources` for AUTO. Production file intent is applied by default where supported, but some sources may ignore that filter and report it in `sourceStatus`. Returns only trustworthy complete results by default. If indexing is still in progress, the response carries a `searchRef` instead of partial hits."; + "- `search` — unified search across indexed dependency code, docs, and explicit symbols. Structured fields are the primary UX; omit `sources` for AUTO. Production file intent is applied by default where supported, but some sources may ignore that filter and report it in `sourceStatus`. Returns only trustworthy complete results by default; opt into partial hits with `allow_partial_results: true`. If indexing is still in progress, the response carries a `searchRef`."; const SEARCH_STATUS_BULLET = - "- `search_status` — follow up a prior unified search by `searchRef`. Use it after `search` returns incomplete state to check progress or fetch final results."; + "- `search_status` — follow up a prior unified search by `searchRef`. Use it after `search` returns incomplete state to check progress, fetch partial hits when the original request used `allow_partial_results: true`, or fetch final results."; const LIST_FILES_BULLET = "- `list_files` — discover what files a dependency ships. Use `path_prefix` to scope to a subdirectory; the response includes each file's language, type, and byte size. Returned `path` values feed directly into `read_file` and help scope `grep_repo`."; diff --git a/src/commands/search.test.ts b/src/commands/search.test.ts index 61e709db..04dff76c 100644 --- a/src/commands/search.test.ts +++ b/src/commands/search.test.ts @@ -40,33 +40,6 @@ describe("searchAction", () => { }; } - function createIncompleteOutcomeWithProgress( - status: UnifiedSearchSessionStatus, - searchRef: string, - progressOverrides: Partial = {}, - ): UnifiedSearchIncomplete { - const baseProgress: UnifiedSearchProgress = { - searchRef, - status, - targetsTotal: 1, - targetsReady: 0, - elapsedMs: 100, - query: "router", - queryWarnings: [], - sources: ["CODE"], - }; - - return { - state: "incomplete", - completed: false, - searchRef, - progress: { - ...baseProgress, - ...progressOverrides, - }, - }; - } - function createDeps( overrides: Partial = {}, ): SearchDependencies { @@ -92,6 +65,7 @@ describe("searchAction", () => { in: ["npm:express", "npm:koa"], kind: "function", lang: "typescript", + allowPartial: true, }, deps, ); @@ -103,6 +77,7 @@ describe("searchAction", () => { { registry: "NPM", packageName: "koa", version: undefined }, ], query: "(router middleware) AND (lang:typescript)", + allowPartialResults: true, filters: expect.objectContaining({ kind: "FUNCTION" }), }), ); @@ -204,7 +179,7 @@ describe("searchAction", () => { consoleSpy.mockRestore(); }); - it("prints source-status notes when the backend ignored filters", async () => { + it("prints source-status notes when the backend ignored an explicitly-set filter", async () => { const consoleSpy = spyOn(console, "log").mockImplementation(() => {}); if (defaultUnifiedSearchOutcome.state !== "completed") { @@ -234,7 +209,7 @@ describe("searchAction", () => { await searchAction( "router middleware", - { in: ["npm:express"] }, + { in: ["npm:express"], intent: "production" }, createDeps({ codeNavigationService: createMockCodeNavigationService({ search: mock(() => Promise.resolve(outcomeWithIgnoredFilters)), @@ -249,6 +224,97 @@ describe("searchAction", () => { consoleSpy.mockRestore(); }); + it("suppresses ignored-filter notes for auto-defaulted filters", async () => { + const consoleSpy = spyOn(console, "log").mockImplementation(() => {}); + + if (defaultUnifiedSearchOutcome.state !== "completed") { + throw new Error("expected completed outcome fixture"); + } + const completedOutcome = defaultUnifiedSearchOutcome; + const outcomeWithIgnoredFilters: UnifiedSearchOutcome = { + ...completedOutcome, + result: { + ...completedOutcome.result, + sourceStatus: [ + { + source: "DOCS", + targetLabel: "npm:express@4.18.2", + indexingStatus: "INDEXED", + resultCount: 1, + ignoredFilters: ["fileIntent"], + incompatibleFilters: [], + appliedFilters: [], + appliedQueryFeatures: [], + ignoredQueryFeatures: [], + incompatibleQueryFeatures: [], + }, + ], + }, + }; + + await searchAction( + "router middleware", + { in: ["npm:express"] }, + createDeps({ + codeNavigationService: createMockCodeNavigationService({ + search: mock(() => Promise.resolve(outcomeWithIgnoredFilters)), + }), + }), + ); + + const output = String(consoleSpy.mock.calls[0]?.[0]); + expect(output).not.toContain("ignored filters: fileIntent"); + consoleSpy.mockRestore(); + }); + + it("renders ignored and incompatible query-feature notes", async () => { + const consoleSpy = spyOn(console, "log").mockImplementation(() => {}); + + if (defaultUnifiedSearchOutcome.state !== "completed") { + throw new Error("expected completed outcome fixture"); + } + const completedOutcome = defaultUnifiedSearchOutcome; + const outcomeWithQueryFeatures: UnifiedSearchOutcome = { + ...completedOutcome, + result: { + ...completedOutcome.result, + sourceStatus: [ + { + source: "DOCS", + targetLabel: "npm:express@4.18.2", + indexingStatus: "INDEXED", + resultCount: 1, + ignoredFilters: [], + incompatibleFilters: [], + appliedFilters: [], + appliedQueryFeatures: [], + ignoredQueryFeatures: ["kind"], + incompatibleQueryFeatures: ["name"], + }, + ], + }, + }; + + await searchAction( + "router middleware", + { in: ["npm:express"] }, + createDeps({ + codeNavigationService: createMockCodeNavigationService({ + search: mock(() => Promise.resolve(outcomeWithQueryFeatures)), + }), + }), + ); + + const output = String(consoleSpy.mock.calls[0]?.[0]); + expect(output).toContain( + "Note: docs on npm:express@4.18.2 ignored query features: kind", + ); + expect(output).toContain( + "Note: docs on npm:express@4.18.2 incompatible query features: name", + ); + consoleSpy.mockRestore(); + }); + it("renders backend summaries verbatim in terminal output", async () => { const consoleSpy = spyOn(console, "log").mockImplementation(() => {}); @@ -390,7 +456,7 @@ describe("searchAction", () => { } }); - it("prints compact docs hint when full doc fetch is unavailable in CLI", async () => { + it("omits doc-fetch placeholder line for documentation pages", async () => { const consoleSpy = spyOn(console, "log").mockImplementation(() => {}); if (defaultUnifiedSearchOutcome.state !== "completed") { @@ -432,9 +498,8 @@ describe("searchAction", () => { expect(output).toContain( "npm:express@4.18.2 [docs page] - Using Express middleware", ); - expect(output).toContain( - "Full doc fetch not exposed in CLI yet (pageId=docs-123)", - ); + expect(output).not.toContain("Full doc fetch"); + expect(output).not.toContain("pageId="); consoleSpy.mockRestore(); }); }); diff --git a/src/commands/search.ts b/src/commands/search.ts index eb5d2b48..ac462588 100644 --- a/src/commands/search.ts +++ b/src/commands/search.ts @@ -14,7 +14,6 @@ import { buildUnifiedSearchParams, buildUnifiedSearchStatusPayload, buildUnifiedSearchSuccessPayload, - colorize, dim, highlight, highlightRanges, @@ -27,6 +26,8 @@ import { toSearchSymbolsFileIntent, toSearchSymbolsKind, toSymbolCategory, + type UnifiedSearchStatusIncompletePayload, + type UnifiedSearchStatusResultPayload, } from "../shared/index.js"; export interface SearchCommandOptions { @@ -39,6 +40,7 @@ export interface SearchCommandOptions { public?: boolean; name?: string; lang?: string; + allowPartial?: boolean; limit?: string; offset?: string; wait?: string; @@ -87,6 +89,7 @@ export async function searchAction( publicOnly: options.public, name: options.name, language: options.lang, + allowPartialResults: options.allowPartial, limit: parseOptionalInt(options.limit, "--limit", 1, 100), offset: parseOptionalInt(options.offset, "--offset", 0), waitTimeoutMs: parseWaitMs(options.wait), @@ -130,24 +133,42 @@ export async function searchStatusAction( } if (!payload.completed) { - console.log(formatSearchStatusTerminal(payload)); + if (payload.result) { + console.log( + formatSearchStatusPartialTerminal({ + ...payload, + result: payload.result, + }), + ); + } else { + console.log(formatSearchStatusTerminal(payload)); + } return; } console.log(formatSearchStatusCompletedTerminal(payload)); } catch (error) { - handleSearchError(error, options.json ?? false); + handleSearchError(error, options.json ?? false, "status"); } } const SEARCH_DESCRIPTION = `Search code, docs, and symbols across indexed dependencies and repositories. -Use repeatable --in targets in package form (npm:express[@version]) or repo form -(https://github.com/org/repo[#ref]). Structured flags are the primary UX; they are -compiled with AND semantics against the raw query. Unified search defaults to -production file intent where the backend supports that filter. Results are -complete-by-default: if indexing is still in progress, search returns a -searchRef instead of partial hits. +Use repeatable --in targets in package form (npm:express[@version]) or repo +form (https://github.com/org/repo[#ref]). Structured flags are AND-combined with +the discovery query. Unified search defaults to production file intent where the +backend supports that filter. Results are complete-by-default: if indexing is +still in progress, search returns a searchRef instead of partial hits unless +--allow-partial is passed. + +Query syntax: + implicit AND foo bar + OR foo OR bar (must be uppercase) + grouping (foo OR bar) baz + exclude foo -bar + phrase "exact phrase" + qualifiers kind: category: path: lang: name: intent: + routing registry: package: version: repo: Decision guide: githits example ... canonical cross-project examples @@ -162,18 +183,16 @@ Plain output labels: Examples: githits search "router middleware" --in npm:express - githits search "handler" --in npm:express --kind function --path-prefix src/ githits search '"body parser" OR multer' --in npm:express --source docs - githits search "retry logic" --in npm:got --in npm:ky --source code - githits search "createServer" --in npm:@types/node --name createServer --lang typescript`; + githits search "compose" --in npm:lodash --source code --kind function + githits search "debounce" --in npm:lodash --source symbol + githits search "composeArgs" --in npm:lodash --name composeArgs`; const SEARCH_STATUS_DESCRIPTION = `Check the status of a unified search started earlier. -Pass the searchRef returned by \ - - githits search ... - -when the initial request could not complete within the wait window.`; +Pass the searchRef returned by githits search when the initial request could +not complete within the wait window. This can return progress, partial hits when +the original request used --allow-partial, or final results.`; export function registerSearchCommand(program: Command) { program @@ -213,7 +232,7 @@ export function registerSearchCommand(program: Command) { .addOption( new Option( "--intent ", - "File intent filter (default: production)", + "File intent filter (default: production for AUTO/code/symbol, omitted for docs-only)", ).choices([ "production", "test", @@ -228,6 +247,10 @@ export function registerSearchCommand(program: Command) { .option("--public", "Filter to public symbols when supported") .option("--name ", "Structured name qualifier") .option("--lang ", "Structured language qualifier") + .option( + "--allow-partial", + "Include hits already available while indexing continues; a searchRef is still returned so search-status can fetch the rest", + ) .option("--limit ", "Max results (1-100)") .option("--offset ", "Result offset") .option( @@ -313,9 +336,22 @@ function parseTargetSpecs(specs: string[] | undefined) { if (!specs || specs.length === 0) { throw new InvalidArgumentError("Provide at least one --in target."); } + for (const spec of specs) { + warnIfUnprefixedTargetSpec(spec); + } return specs.map(parseUnifiedSearchTargetSpec); } +function warnIfUnprefixedTargetSpec(spec: string): void { + const trimmed = spec.trim(); + if (trimmed.length === 0) return; + if (trimmed.startsWith("http://") || trimmed.startsWith("https://")) return; + if (trimmed.includes(":")) return; + console.error( + `Warning: --in '${trimmed}' has no registry prefix; treating as 'npm:${trimmed}'. Pass 'npm:${trimmed}' explicitly to suppress this warning.`, + ); +} + function parseSources( values: string[] | undefined, ): UnifiedSearchSource[] | undefined { @@ -366,17 +402,31 @@ function collectRepeatable(value: string, previous: string[]): string[] { return [...previous, value]; } -function handleSearchError(error: unknown, json: boolean): never { +function handleSearchError( + error: unknown, + json: boolean, + context: "search" | "status" = "search", +): never { const payload = buildUnifiedSearchErrorPayload(error); if (json) { console.error(JSON.stringify(payload)); } else { - console.error(payload.error); + console.error(formatSearchErrorTerminal(payload, context)); } process.exit(1); } +function formatSearchErrorTerminal( + payload: { error: string; code: string }, + context: "search" | "status", +): string { + if (context === "status" && payload.code === "NOT_FOUND") { + return `${payload.error}\n Search sessions expire; run \`githits search ...\` to start a new one.`; + } + return payload.error; +} + function formatUnifiedSearchTerminal(payload: { completed: boolean; returnedCount: number; @@ -405,14 +455,8 @@ function formatUnifiedSearchTerminal(payload: { }>; searchRef?: string; progress?: { targetsReady?: number; targetsTotal?: number }; - query: { warnings?: string[] }; - sourceStatus?: Array<{ - source: string; - targetLabel: string; - ignoredFilters: string[]; - incompatibleFilters: string[]; - note?: string; - }>; + query: { warnings?: string[]; defaulted?: ReadonlyArray }; + sourceStatus?: SourceStatusEntry[]; }): string { const lines: string[] = []; const useColors = shouldUseColors(); @@ -425,25 +469,49 @@ function formatUnifiedSearchTerminal(payload: { } if (!payload.completed) { - return formatSearchStatusTerminal({ + const statusText = formatSearchStatusTerminal({ completed: false, searchRef: payload.searchRef ?? "", progress: payload.progress, }); + if (payload.results.length === 0) { + return statusText; + } + lines.push(statusText); + lines.push(""); + lines.push("Partial results:"); } + const sourceStatusNotes = formatSourceStatusNotes( + payload.sourceStatus, + payload.query.defaulted, + ); + if (payload.results.length === 0) { lines.push("No results."); - return lines.join("\n"); + if (sourceStatusNotes.length > 0) { + lines.push(""); + lines.push(...sourceStatusNotes); + } + return lines.join("\n").trimEnd(); } + const { display, duplicatesFolded } = dedupeSearchResultsForDisplay( + payload.results, + ); + + const baseCount = `${display.length} result(s)`; + const countSuffix = [ + payload.hasMore ? " (more available)" : "", + duplicatesFolded > 0 ? ` (+${duplicatesFolded} near-duplicate folded)` : "", + ].join(""); lines.push( - `${highlight(`${payload.returnedCount} result(s)`, useColors)}${payload.hasMore ? dim(" (more available)", useColors) : ""}`, + `${highlight(baseCount, useColors)}${dim(countSuffix, useColors)}`, ); - lines.push(dim(formatUnifiedSearchTypeSummary(payload.results), useColors)); + lines.push(dim(formatUnifiedSearchTypeSummary(display), useColors)); lines.push(""); - for (const entry of payload.results) { + for (const entry of display) { const location = formatUnifiedSearchLocation(entry.locator); const header = formatUnifiedSearchHeader(entry, useColors, location); lines.push(header); @@ -456,10 +524,6 @@ function formatUnifiedSearchTerminal(payload: { ), ); } - const detailLine = formatUnifiedSearchDetailLine(entry, useColors); - if (detailLine) { - lines.push(detailLine); - } lines.push(""); } @@ -467,7 +531,6 @@ function formatUnifiedSearchTerminal(payload: { lines.push(dim(`Next offset: ${payload.nextOffset}`, useColors)); } - const sourceStatusNotes = formatSourceStatusNotes(payload.sourceStatus); if (sourceStatusNotes.length > 0) { lines.push(""); lines.push(...sourceStatusNotes); @@ -536,13 +599,7 @@ function formatSearchStatusCompletedTerminal(payload: { returnedCount: number; hasMore: boolean; nextOffset?: number; - sourceStatus?: Array<{ - source: string; - targetLabel: string; - ignoredFilters: string[]; - incompatibleFilters: string[]; - note?: string; - }>; + sourceStatus?: SourceStatusEntry[]; results: Array<{ type: string; target: string; @@ -569,43 +626,90 @@ function formatSearchStatusCompletedTerminal(payload: { }); } +function formatSearchStatusPartialTerminal( + payload: UnifiedSearchStatusIncompletePayload & { + result: UnifiedSearchStatusResultPayload; + }, +): string { + return formatUnifiedSearchTerminal({ + completed: false, + returnedCount: payload.result.returnedCount, + hasMore: payload.result.hasMore, + nextOffset: payload.result.nextOffset, + results: payload.result.results, + searchRef: payload.searchRef, + progress: payload.progress, + query: { warnings: payload.result.queryWarnings }, + sourceStatus: payload.result.sourceStatus, + }); +} + +interface SourceStatusEntry { + source: string; + targetLabel: string; + ignoredFilters: string[]; + incompatibleFilters: string[]; + ignoredQueryFeatures?: string[]; + incompatibleQueryFeatures?: string[]; + note?: string; +} + function formatSourceStatusNotes( - sourceStatus: - | Array<{ - source: string; - targetLabel: string; - ignoredFilters: string[]; - incompatibleFilters: string[]; - note?: string; - }> - | undefined, + sourceStatus: SourceStatusEntry[] | undefined, + defaulted: ReadonlyArray | undefined, ): string[] { const useColors = shouldUseColors(); if (!sourceStatus) { return []; } + const defaultedSet = new Set(defaulted ?? []); const lines: string[] = []; for (const entry of sourceStatus) { const label = `${entry.source.toLowerCase()} on ${entry.targetLabel}`; - if (entry.ignoredFilters.length > 0) { + const ignoredFilters = entry.ignoredFilters.filter( + (name) => !defaultedSet.has(name), + ); + if (ignoredFilters.length > 0) { lines.push( dim( - `Note: ${label} ignored filters: ${entry.ignoredFilters.join(", ")}`, + `Note: ${label} ignored filters: ${ignoredFilters.join(", ")}`, useColors, ), ); } - if (entry.incompatibleFilters.length > 0) { + const incompatibleFilters = entry.incompatibleFilters.filter( + (name) => !defaultedSet.has(name), + ); + if (incompatibleFilters.length > 0) { lines.push( dim( - `Note: ${label} incompatible filters: ${entry.incompatibleFilters.join( + `Note: ${label} incompatible filters: ${incompatibleFilters.join( ", ", )}`, useColors, ), ); } + if (entry.ignoredQueryFeatures && entry.ignoredQueryFeatures.length > 0) { + lines.push( + dim( + `Note: ${label} ignored query features: ${entry.ignoredQueryFeatures.join(", ")}`, + useColors, + ), + ); + } + if ( + entry.incompatibleQueryFeatures && + entry.incompatibleQueryFeatures.length > 0 + ) { + lines.push( + dim( + `Note: ${label} incompatible query features: ${entry.incompatibleQueryFeatures.join(", ")}`, + useColors, + ), + ); + } if (entry.note) { lines.push(dim(`Note: ${label}: ${entry.note}`, useColors)); } @@ -614,6 +718,34 @@ function formatSourceStatusNotes( return lines; } +function dedupeSearchResultsForDisplay< + T extends { + type: string; + target: string; + title?: string; + summary?: string; + }, +>(results: T[]): { display: T[]; duplicatesFolded: number } { + const seen = new Set(); + const display: T[] = []; + let duplicatesFolded = 0; + for (const entry of results) { + const key = [ + entry.type, + entry.target, + entry.title ?? "", + (entry.summary ?? "").slice(0, 120), + ].join(""); + if (seen.has(key)) { + duplicatesFolded += 1; + continue; + } + seen.add(key); + display.push(entry); + } + return { display, duplicatesFolded }; +} + function formatUnifiedSearchTypeSummary( results: Array<{ type: string }>, ): string { @@ -722,34 +854,3 @@ function formatUnifiedSearchHeader( : undefined; return `${highlight(primary, useColors)} ${dim(badge, useColors)}${title ? ` - ${title}` : ""}`; } - -function formatUnifiedSearchDetailLine( - entry: { - type: string; - target: string; - locator: { - registry?: string; - packageName?: string; - version?: string; - repoUrl?: string; - gitRef?: string; - pageId?: string; - filePath?: string; - startLine?: number; - endLine?: number; - }; - }, - useColors: boolean, -): string | undefined { - if (entry.type === "documentation_page") { - const docHint = entry.locator.pageId - ? `pageId=${entry.locator.pageId}` - : entry.target; - return dim( - ` Full doc fetch not exposed in CLI yet (${docHint})`, - useColors, - ); - } - - return undefined; -} diff --git a/src/services/code-navigation-service.test.ts b/src/services/code-navigation-service.test.ts index 0e831d07..ca9faf5a 100644 --- a/src/services/code-navigation-service.test.ts +++ b/src/services/code-navigation-service.test.ts @@ -1036,6 +1036,12 @@ describe("CodeNavigationServiceImpl", () => { contextAfter: ["", "app.get();"], fileContentHash: "abc123", fileIntent: "production", + symbolRowId: "42", + symbol: { + name: "createRouter", + qualifiedPath: "express.createRouter", + kind: "function", + }, }, ], nextCursor: null, @@ -1069,16 +1075,22 @@ describe("CodeNavigationServiceImpl", () => { target: { registry: "NPM", packageName: "express" }, pattern: "middleware", pathSelectors: [{ kind: "PREFIX", value: "src/" }], + symbolFields: ["name", "qualified_path", "kind"], }); expect(result.matches.length).toBe(1); expect(result.matches[0]?.line).toBe(10); + expect(result.matches[0]?.symbol).toMatchObject({ + name: "createRouter", + qualifiedPath: "express.createRouter", + kind: "function", + }); expect(result.totalMatches).toBe(1); expect(result.routeTaken).toBe("CONTENT_INDEX"); expect(result.resolution?.resolvedRef).toBe("v5.2.1"); }); it("normalises unified search highlight spans", async () => { - mockFetch(() => + const fn = mockFetch(() => Promise.resolve( new Response( JSON.stringify({ @@ -1140,6 +1152,7 @@ describe("CodeNavigationServiceImpl", () => { const result = await service.search({ targets: [{ registry: "NPM", packageName: "express" }], query: "router middleware", + allowPartialResults: true, }); expect(result.state).toBe("completed"); @@ -1150,6 +1163,9 @@ describe("CodeNavigationServiceImpl", () => { title: [[7, 17]], summary: [[9, 15]], }); + const [, init] = fn.mock.calls[0] as unknown as [string, RequestInit]; + const body = JSON.parse(init.body as string); + expect(body.variables.allowPartialResults).toBe(true); }); it("throws CodeNavigationIndexingError for data-path INDEXING sentinel on grepRepo", async () => { @@ -1282,6 +1298,7 @@ describe("CodeNavigationServiceImpl", () => { maxMatches: 100, maxMatchesPerFile: 3, cursor: "cursor-123", + symbolFields: ["name", "qualified_path", "kind"], waitTimeoutMs: 5000, }); const [, init] = fn.mock.calls[0] as unknown as [string, RequestInit]; @@ -1305,8 +1322,14 @@ describe("CodeNavigationServiceImpl", () => { maxMatches: 100, maxMatchesPerFile: 3, cursor: "cursor-123", + symbolFields: ["name", "qualified_path", "kind"], waitTimeoutMs: 5000, }); + expect(body.query).toContain("symbol {"); + expect(body.query).toContain("name"); + expect(body.query).toContain("qualifiedPath"); + expect(body.query).toContain("kind"); + expect(body.query).not.toContain("symbolRef"); }); it("sends GraphQL variables with the correct listRepoFiles shape", async () => { diff --git a/src/services/code-navigation-service.ts b/src/services/code-navigation-service.ts index 3b582ebd..ee2371e0 100644 --- a/src/services/code-navigation-service.ts +++ b/src/services/code-navigation-service.ts @@ -193,6 +193,7 @@ export interface UnifiedSearchParams { query: string; sources?: UnifiedSearchSource[]; filters?: UnifiedSearchFilters; + allowPartialResults?: boolean; limit?: number; offset?: number; waitTimeoutMs?: number; @@ -286,6 +287,7 @@ export interface UnifiedSearchIncomplete { state: "incomplete"; completed: false; searchRef: string; + result?: UnifiedSearchResult; progress?: UnifiedSearchProgress; } @@ -370,9 +372,28 @@ export interface GrepRepoParams { maxMatches?: number; maxMatchesPerFile?: number; cursor?: string; + symbolFields?: string[]; waitTimeoutMs?: number; } +export interface NavigationSymbol { + symbolRef?: string; + name?: string; + qualifiedPath?: string; + kind?: string; + category?: string; + arity?: number; + isPublic?: boolean; + filePath?: string; + startLine?: number; + endLine?: number; + code?: string; + callerCount?: number; + contentHash?: string; + parentSymbolRef?: string; + parentPath?: string; +} + export interface GrepRepoMatch { filePath: string; line: number; @@ -383,6 +404,8 @@ export interface GrepRepoMatch { contextAfter?: string[]; fileContentHash?: string; fileIntent?: string; + symbolRowId?: string; + symbol?: NavigationSymbol; } export type GrepRouteTaken = "SINGLE_FILE" | "CONTENT_INDEX"; @@ -1154,6 +1177,27 @@ const grepRepoMatchSchema = z.object({ contextAfter: z.array(z.string()).nullable().optional(), fileContentHash: z.string().nullable().optional(), fileIntent: z.string().nullable().optional(), + symbolRowId: z.string().nullable().optional(), + symbol: z + .object({ + symbolRef: z.string().optional(), + name: z.string().optional(), + qualifiedPath: z.string().nullable().optional(), + kind: z.string().nullable().optional(), + category: z.string().nullable().optional(), + arity: z.number().int().nullable().optional(), + isPublic: z.boolean().nullable().optional(), + filePath: z.string().nullable().optional(), + startLine: z.number().int().nullable().optional(), + endLine: z.number().int().nullable().optional(), + code: z.string().nullable().optional(), + callerCount: z.number().int().nullable().optional(), + contentHash: z.string().nullable().optional(), + parentSymbolRef: z.string().nullable().optional(), + parentPath: z.string().nullable().optional(), + }) + .nullable() + .optional(), }); const grepRepoResponseSchema = z.object({ @@ -1190,7 +1234,38 @@ const grepRepoGraphQLResponseSchema = z.object({ errors: z.array(graphQLErrorSchema).optional(), }); -const GREP_REPO_QUERY = ` +const GREP_REPO_SYMBOL_SELECTIONS: Record = { + symbol_ref: "symbolRef", + name: "name", + qualified_path: "qualifiedPath", + kind: "kind", + category: "category", + arity: "arity", + is_public: "isPublic", + file_path: "filePath", + start_line: "startLine", + end_line: "endLine", + code: "code", + caller_count: "callerCount", + content_hash: "contentHash", + parent_symbol_ref: "parentSymbolRef", + parent_path: "parentPath", +}; + +function buildGrepRepoQuery( + symbolFields: readonly string[] | undefined, +): string { + const symbolSelection = (symbolFields ?? []) + .map((field) => GREP_REPO_SYMBOL_SELECTIONS[field]) + .filter((field): field is string => Boolean(field)) + .filter((field, index, fields) => fields.indexOf(field) === index) + .join("\n "); + const symbolBlock = + symbolSelection.length > 0 + ? `\n symbol {\n ${symbolSelection}\n }` + : ""; + + return ` query GrepRepo( $registry: Registry $packageName: String @@ -1211,6 +1286,7 @@ query GrepRepo( $maxMatches: Int $maxMatchesPerFile: Int $cursor: String + $symbolFields: [String!] ) { grepRepo( registry: $registry @@ -1232,6 +1308,7 @@ query GrepRepo( maxMatches: $maxMatches maxMatchesPerFile: $maxMatchesPerFile cursor: $cursor + symbolFields: $symbolFields ) { matches { filePath @@ -1243,6 +1320,7 @@ query GrepRepo( contextAfter fileContentHash fileIntent + symbolRowId${symbolBlock} } nextCursor totalMatches @@ -1269,6 +1347,7 @@ query GrepRepo( } } }`; +} // `data` may be null (seen live for unknown packages that also carry // `errors`), and `searchSymbols` may be null even when `data` is present. @@ -1369,7 +1448,7 @@ export class CodeNavigationServiceImpl implements CodeNavigationService { query: params.query, sources: params.sources, filters: params.filters, - allowPartialResults: false, + allowPartialResults: params.allowPartialResults ?? false, limit: params.limit, offset: params.offset, waitTimeoutMs: params.waitTimeoutMs, @@ -1482,6 +1561,7 @@ export class CodeNavigationServiceImpl implements CodeNavigationService { state: "incomplete", completed: false, searchRef: progress.searchRef, + result, progress, }; } @@ -1811,10 +1891,15 @@ export class CodeNavigationServiceImpl implements CodeNavigationService { ); } + const result = data.result + ? this.normaliseUnifiedSearchResult(data.result) + : undefined; + return { state: "incomplete", completed: false, searchRef, + result, progress, }; } @@ -2125,7 +2210,7 @@ export class CodeNavigationServiceImpl implements CodeNavigationService { response = await postPkgseerGraphql({ endpointUrl: this.codeNavigationUrl, token, - query: GREP_REPO_QUERY, + query: buildGrepRepoQuery(params.symbolFields), variables: { registry: params.target.registry, packageName: params.target.packageName, @@ -2149,6 +2234,7 @@ export class CodeNavigationServiceImpl implements CodeNavigationService { maxMatches: params.maxMatches, maxMatchesPerFile: params.maxMatchesPerFile, cursor: params.cursor, + symbolFields: params.symbolFields, }, fetchFn: this.fetchFn, }); @@ -2197,6 +2283,26 @@ export class CodeNavigationServiceImpl implements CodeNavigationService { contextAfter: entry.contextAfter ?? undefined, fileContentHash: entry.fileContentHash ?? undefined, fileIntent: entry.fileIntent ?? undefined, + symbolRowId: entry.symbolRowId ?? undefined, + symbol: entry.symbol + ? { + symbolRef: entry.symbol.symbolRef, + name: entry.symbol.name, + qualifiedPath: entry.symbol.qualifiedPath ?? undefined, + kind: entry.symbol.kind ?? undefined, + category: entry.symbol.category ?? undefined, + arity: entry.symbol.arity ?? undefined, + isPublic: entry.symbol.isPublic ?? undefined, + filePath: entry.symbol.filePath ?? undefined, + startLine: entry.symbol.startLine ?? undefined, + endLine: entry.symbol.endLine ?? undefined, + code: entry.symbol.code ?? undefined, + callerCount: entry.symbol.callerCount ?? undefined, + contentHash: entry.symbol.contentHash ?? undefined, + parentSymbolRef: entry.symbol.parentSymbolRef ?? undefined, + parentPath: entry.symbol.parentPath ?? undefined, + } + : undefined, })), nextCursor: data.nextCursor ?? undefined, hasMore: data.hasMore, diff --git a/src/services/index.ts b/src/services/index.ts index b520890a..19d39878 100644 --- a/src/services/index.ts +++ b/src/services/index.ts @@ -39,6 +39,7 @@ export type { GrepTruncatedReason, ListFilesParams, ListFilesResult, + NavigationSymbol, ReadFileParams, ReadFileResult, RepoFileEntry, diff --git a/src/shared/code-navigation-error-map.ts b/src/shared/code-navigation-error-map.ts index df8f9736..d248c160 100644 --- a/src/shared/code-navigation-error-map.ts +++ b/src/shared/code-navigation-error-map.ts @@ -170,7 +170,7 @@ function classify(error: unknown): MappedError { if (error instanceof CodeNavigationValidationError) { return { code: "INVALID_ARGUMENT", - message: error.message, + message: normalizeBackendMessage(error.message), retryable: false, }; } @@ -218,9 +218,11 @@ function classifyBackendError(error: CodeNavigationBackendError): MappedError { if (typeof error.status === "number") details.status = error.status; if (error.graphqlCode) details.graphqlCode = error.graphqlCode; + const message = normalizeBackendMessage(error.message); + const build = (code: MappedErrorCode, defaultRetryable: boolean) => ({ code, - message: error.message, + message, retryable: error.retryable ?? defaultRetryable, details: Object.keys(details).length > 0 ? details : undefined, }); @@ -232,13 +234,23 @@ function classifyBackendError(error: CodeNavigationBackendError): MappedError { return build("RATE_LIMITED", true); case "UPSTREAM_ERROR": return build("BACKEND_ERROR", true); - case "INTERNAL_ERROR": - case "UNKNOWN_ERROR": default: return build("BACKEND_ERROR", false); } } +/** + * Align backend wording with the CLI/MCP docs. The backend labels the + * required regex pre-filter term as a "literal anchor"; our surfaces + * call it a "literal substring" to avoid anchor/^$ confusion. + */ +function normalizeBackendMessage(message: string): string { + return message + .replace(/extractable literal anchor/g, "extractable literal substring") + .replace(/at least one literal anchor/g, "at least one literal substring") + .replace(/literal prefix/g, "literal substring"); +} + /** * Any Error whose `name` starts with `Invalid` or `Unsupported` is * treated as an invalid-argument case. Covers parser errors from diff --git a/src/shared/grep-repo-request.test.ts b/src/shared/grep-repo-request.test.ts index cff5739a..9a298004 100644 --- a/src/shared/grep-repo-request.test.ts +++ b/src/shared/grep-repo-request.test.ts @@ -22,6 +22,7 @@ describe("buildGrepRepoParams", () => { expect(params.contextLinesAfter).toBe(0); expect(params.maxMatches).toBe(50); expect(params.waitTimeoutMs).toBe(20000); + expect(params.symbolFields).toBeUndefined(); }); it("compiles path, pathPrefix, and globs into pathSelectors", () => { @@ -60,6 +61,27 @@ describe("buildGrepRepoParams", () => { expect(asymmetric.params.contextLinesAfter).toBe(5); }); + it("passes symbol fields through when requested", () => { + const { params, explicit } = buildGrepRepoParams({ + target, + pattern: "middleware", + symbolFields: ["name", "qualified_path", "kind"], + }); + + expect(params.symbolFields).toEqual(["name", "qualified_path", "kind"]); + expect(explicit.symbolFields).toBe(true); + }); + + it("rejects unknown symbol fields", () => { + expect(() => + buildGrepRepoParams({ + target, + pattern: "middleware", + symbolFields: ["name", "qualifiedPath"], + }), + ).toThrow(/symbol_fields.*qualifiedPath/); + }); + it("rejects leading dots in extensions", () => { expect(() => buildGrepRepoParams({ @@ -87,10 +109,11 @@ describe("buildGrepRepoParams", () => { }); describe("GREP_REPO_PATTERN_NOTE", () => { - it("mentions literal, regex, RE2, and byte limit", () => { + it("mentions literal, regex, RE2, byte limit, and whole-target regex planning", () => { expect(GREP_REPO_PATTERN_NOTE).toMatch(/literal/i); expect(GREP_REPO_PATTERN_NOTE).toMatch(/regex/i); expect(GREP_REPO_PATTERN_NOTE).toMatch(/RE2/i); expect(GREP_REPO_PATTERN_NOTE).toMatch(/200/i); + expect(GREP_REPO_PATTERN_NOTE).toMatch(/literal substring/i); }); }); diff --git a/src/shared/grep-repo-request.ts b/src/shared/grep-repo-request.ts index 568dbbcd..14656131 100644 --- a/src/shared/grep-repo-request.ts +++ b/src/shared/grep-repo-request.ts @@ -17,8 +17,30 @@ const LIMIT_MAX = 1000; const LIMIT_DEFAULT = 50; const WAIT_MIN = 0; +export const GREP_REPO_SYMBOL_FIELDS = [ + "symbol_ref", + "name", + "qualified_path", + "kind", + "category", + "arity", + "is_public", + "file_path", + "start_line", + "end_line", + "code", + "caller_count", + "content_hash", + "parent_symbol_ref", + "parent_path", +] as const; + +export type GrepRepoSymbolField = (typeof GREP_REPO_SYMBOL_FIELDS)[number]; + +export const GREP_REPO_SYMBOL_FIELDS_NOTE = `Hydrate these enclosing-symbol fields on each match; omit for no symbol hydration. Valid values: ${GREP_REPO_SYMBOL_FIELDS.join(", ")}.`; + export const GREP_REPO_PATTERN_NOTE = - "Text grep over indexed source files. `literal` (default) does substring matching; `regex` uses RE2 syntax (no lookaround, no backreferences). Pattern max 200 UTF-8 bytes."; + "Text grep over indexed source files. `literal` (default) does substring matching. `regex` uses RE2 syntax (no lookaround, no backreferences); when scoping the whole target with no path, path_prefix, or glob, the regex must include at least one literal substring the index can use for pre-filtering. Pattern max 200 UTF-8 bytes. Matching is ASCII case-insensitive by default: non-ASCII letters match case-sensitively; pass case_sensitive: true for exact casing. When multiple selectors (`path`, `path_prefix`, `globs`) are combined, they are unioned — a file matches if any selector matches. Use `extensions` to intersect further."; export interface GrepRepoRequestPathSelectorInput { kind: "exact" | "prefix" | "glob"; @@ -42,6 +64,7 @@ export interface GrepRepoRequestInput { maxMatches?: number; maxMatchesPerFile?: number; cursor?: string; + symbolFields?: readonly string[]; waitTimeoutMs?: number; } @@ -62,6 +85,7 @@ export interface GrepRepoRequestBuildResult { maxMatches: boolean; maxMatchesPerFile: boolean; cursor: boolean; + symbolFields: boolean; }; } @@ -107,6 +131,7 @@ export function buildGrepRepoParams( const maxMatchesPerFile = normalizeMaxMatchesPerFile(input.maxMatchesPerFile); const waitTimeoutMs = normalizeWaitTimeoutMs(input.waitTimeoutMs); const cursor = normalizeOptionalNonEmpty(input.cursor, "cursor"); + const symbolFields = normalizeSymbolFields(input.symbolFields); const pathSelectors = buildPathSelectors({ path, pathPrefix, globs }); const hasPathSelectors = (pathSelectors?.length ?? 0) > 0; @@ -132,6 +157,7 @@ export function buildGrepRepoParams( maxMatches, maxMatchesPerFile, cursor, + symbolFields: symbolFields.length > 0 ? symbolFields : undefined, waitTimeoutMs, }, explicit: { @@ -149,6 +175,7 @@ export function buildGrepRepoParams( maxMatches: input.maxMatches !== undefined, maxMatchesPerFile: input.maxMatchesPerFile !== undefined, cursor: cursor !== undefined, + symbolFields: symbolFields.length > 0, }, }; } @@ -201,6 +228,20 @@ function normalizeStringList( return out; } +function normalizeSymbolFields( + values: readonly string[] | undefined, +): GrepRepoSymbolField[] { + const out = normalizeStringList([...(values ?? [])], "symbol_fields"); + for (const value of out) { + if (!GREP_REPO_SYMBOL_FIELDS.includes(value as GrepRepoSymbolField)) { + throw new InvalidPackageSpecError( + `\`symbol_fields\` value must be one of: ${GREP_REPO_SYMBOL_FIELDS.join(", ")}. Got: ${value}.`, + ); + } + } + return out as GrepRepoSymbolField[]; +} + function normalizeExtensions(values: string[] | undefined): string[] { const out = normalizeStringList(values, "extensions"); for (const value of out) { diff --git a/src/shared/grep-repo-response.test.ts b/src/shared/grep-repo-response.test.ts index 59dfb380..3e08c66f 100644 --- a/src/shared/grep-repo-response.test.ts +++ b/src/shared/grep-repo-response.test.ts @@ -17,6 +17,13 @@ const baseResult: GrepRepoResult = { contextAfter: ["", "app.get('/', …);"], fileContentHash: "abc123", fileIntent: "production", + symbolRowId: "42", + symbol: { + symbolRef: "npm:express:4.18.2:42", + name: "createRouter", + qualifiedPath: "express.createRouter", + kind: "function", + }, }, ], nextCursor: undefined, @@ -52,6 +59,7 @@ const baseOptions = { maxMatches: 50, maxMatchesPerFile: 3, cursor: undefined, + symbolFields: ["name", "qualified_path", "kind"], excludeDocFiles: true, excludeTestFiles: true, explicit: { @@ -69,6 +77,7 @@ const baseOptions = { maxMatches: true, maxMatchesPerFile: true, cursor: false, + symbolFields: true, }, }; @@ -79,6 +88,10 @@ describe("buildGrepRepoSuccessPayload", () => { expect(envelope.patternType).toBe("literal"); expect(envelope.totalMatches).toBe(1); expect(envelope.matches[0]?.filePath).toBe("src/index.js"); + expect(envelope.matches[0]?.symbol).toMatchObject({ + name: "createRouter", + qualifiedPath: "express.createRouter", + }); expect(envelope.routeTaken).toBe("content_index"); expect(envelope.filter).toEqual({ pathPrefix: "src/", @@ -89,6 +102,7 @@ describe("buildGrepRepoSuccessPayload", () => { contextLines: 2, maxMatches: 50, maxMatchesPerFile: 3, + symbolFields: ["name", "qualified_path", "kind"], }); }); }); diff --git a/src/shared/grep-repo-response.ts b/src/shared/grep-repo-response.ts index d3344f50..e5e8f57d 100644 --- a/src/shared/grep-repo-response.ts +++ b/src/shared/grep-repo-response.ts @@ -11,6 +11,24 @@ export interface LeanGrepRepoMatch { contextAfter?: string[]; fileContentHash?: string; fileIntent?: string; + symbolRowId?: string; + symbol?: { + symbolRef?: string; + name?: string; + qualifiedPath?: string; + kind?: string; + category?: string; + arity?: number; + isPublic?: boolean; + filePath?: string; + startLine?: number; + endLine?: number; + code?: string; + callerCount?: number; + contentHash?: string; + parentSymbolRef?: string; + parentPath?: string; + }; } export interface LeanGrepRepoResolution { @@ -35,6 +53,7 @@ export interface LeanGrepRepoFilter { maxMatches?: number; maxMatchesPerFile?: number; cursor?: string; + symbolFields?: string[]; } export interface LeanGrepRepoEnvelope { @@ -79,6 +98,7 @@ export interface BuildGrepRepoPayloadOptions { maxMatches: number; maxMatchesPerFile?: number; cursor?: string; + symbolFields?: string[]; excludeDocFiles?: boolean; excludeTestFiles?: boolean; explicit: { @@ -96,6 +116,7 @@ export interface BuildGrepRepoPayloadOptions { maxMatches: boolean; maxMatchesPerFile: boolean; cursor: boolean; + symbolFields: boolean; }; } @@ -145,6 +166,8 @@ function projectMatch(match: GrepRepoMatch): LeanGrepRepoMatch { contextAfter: match.contextAfter, fileContentHash: match.fileContentHash, fileIntent: match.fileIntent, + symbolRowId: match.symbolRowId, + symbol: match.symbol, }; } @@ -206,6 +229,13 @@ function buildFilterBlock( filter.maxMatchesPerFile = options.maxMatchesPerFile; } if (options.explicit.cursor && options.cursor) filter.cursor = options.cursor; + if ( + options.explicit.symbolFields && + options.symbolFields && + options.symbolFields.length > 0 + ) { + filter.symbolFields = options.symbolFields; + } return Object.keys(filter).length > 0 ? filter : undefined; } @@ -226,6 +256,7 @@ interface RenderLine { content: string; isMatch: boolean; highlightRanges?: Array; + symbolHint?: string; } interface RenderBlock { @@ -363,7 +394,7 @@ function buildRenderBlocks(matches: LeanGrepRepoMatch[]): RenderBlock[] { } const existingMatch = lineMap.get(match.line); - if (existingMatch && existingMatch.isMatch) { + if (existingMatch?.isMatch) { existingMatch.highlightRanges = mergeRanges( existingMatch.highlightRanges, [ @@ -384,6 +415,7 @@ function buildRenderBlocks(matches: LeanGrepRepoMatch[]): RenderBlock[] { clampCharacterOffset(match.lineContent, match.matchEndByte), ], ], + symbolHint: formatSymbolHint(match.symbol), }); } @@ -479,12 +511,35 @@ function renderVerboseLine( ): string { const gutter = padLeft(String(line.lineNumber), gutterWidth); if (line.isMatch) { - return `${colorize(">", "bold", useColors)} ${gutter} ${highlightRanges(line.content, line.highlightRanges, useColors)}`; + const matchRow = `${colorize(">", "bold", useColors)} ${gutter} ${highlightRanges(line.content, line.highlightRanges, useColors)}`; + if (line.symbolHint) { + const hintIndent = " ".repeat(2 + gutterWidth + 2); + return `${matchRow}\n${hintIndent}${dim(`↳ in: ${line.symbolHint}`, useColors)}`; + } + return matchRow; } return ` ${dim(gutter, useColors)} ${dim(line.content, useColors)}`; } +function formatSymbolHint( + symbol: LeanGrepRepoMatch["symbol"], +): string | undefined { + if (!symbol) return undefined; + const primary = symbol.qualifiedPath ?? symbol.name; + const parts: string[] = []; + if (primary) parts.push(primary); + if (symbol.kind) parts.push(`(${symbol.kind})`); + if (symbol.isPublic === true) parts.push("public"); + if (symbol.arity !== undefined) parts.push(`arity=${symbol.arity}`); + if (symbol.callerCount !== undefined) + parts.push(`callers=${symbol.callerCount}`); + if (symbol.startLine !== undefined && symbol.endLine !== undefined) { + parts.push(`L${symbol.startLine}-${symbol.endLine}`); + } + return parts.length > 0 ? parts.join(" ") : undefined; +} + function widestLineNumberInBlocks(blocks: RenderBlock[]): number { let maxWidth = 1; for (const block of blocks) { diff --git a/src/shared/index.ts b/src/shared/index.ts index 69d8018e..b46a9852 100644 --- a/src/shared/index.ts +++ b/src/shared/index.ts @@ -37,8 +37,11 @@ export { debugLog } from "./debug-log.js"; export { buildGrepRepoParams, GREP_REPO_PATTERN_NOTE, + GREP_REPO_SYMBOL_FIELDS, + GREP_REPO_SYMBOL_FIELDS_NOTE, type GrepRepoRequestBuildResult, type GrepRepoRequestInput, + type GrepRepoSymbolField, } from "./grep-repo-request.js"; export { type BuildGrepRepoPayloadOptions, diff --git a/src/shared/unified-search-request.test.ts b/src/shared/unified-search-request.test.ts index 6fed76ae..449a8cad 100644 --- a/src/shared/unified-search-request.test.ts +++ b/src/shared/unified-search-request.test.ts @@ -33,6 +33,28 @@ describe("buildUnifiedSearchParams", () => { expect(built.defaulted).not.toContain("fileIntent"); }); + it("does not default fileIntent for explicit docs-only searches", () => { + const built = buildUnifiedSearchParams({ + target: { registry: "NPM", packageName: "express" }, + query: "routing", + sources: ["DOCS"], + }); + + expect(built.params.filters).toBeUndefined(); + expect(built.defaulted).not.toContain("fileIntent"); + }); + + it("defaults fileIntent when selected sources include code search", () => { + const built = buildUnifiedSearchParams({ + target: { registry: "NPM", packageName: "express" }, + query: "routing", + sources: ["DOCS", "CODE"], + }); + + expect(built.params.filters).toEqual({ fileIntent: "PRODUCTION" }); + expect(built.defaulted).toContain("fileIntent"); + }); + it("compiles structured name and language into AND-ed query qualifiers", () => { const built = buildUnifiedSearchParams({ target: { registry: "NPM", packageName: "express" }, @@ -78,6 +100,21 @@ describe("buildUnifiedSearchParams", () => { }); }); + it("passes through allowPartialResults without changing the default", () => { + const defaulted = buildUnifiedSearchParams({ + target: { registry: "NPM", packageName: "express" }, + query: "router", + }); + expect(defaulted.params.allowPartialResults).toBeUndefined(); + + const explicit = buildUnifiedSearchParams({ + target: { registry: "NPM", packageName: "express" }, + query: "router", + allowPartialResults: true, + }); + expect(explicit.params.allowPartialResults).toBe(true); + }); + it("dedupes exact duplicate targets while preserving order", () => { const built = buildUnifiedSearchParams({ targets: [ diff --git a/src/shared/unified-search-request.ts b/src/shared/unified-search-request.ts index 8e1bb876..2a1bd6fb 100644 --- a/src/shared/unified-search-request.ts +++ b/src/shared/unified-search-request.ts @@ -25,6 +25,7 @@ export interface UnifiedSearchRequestInput { publicOnly?: boolean; name?: string; language?: string; + allowPartialResults?: boolean; limit?: number; offset?: number; waitTimeoutMs?: number; @@ -64,7 +65,7 @@ export function buildUnifiedSearchParams( kind: input.kind, category: input.category, pathPrefix: input.pathPrefix, - fileIntent: resolveFileIntent(input.fileIntent, defaulted), + fileIntent: resolveFileIntent(input.fileIntent, input.sources, defaulted), publicOnly: input.publicOnly, }); @@ -74,6 +75,7 @@ export function buildUnifiedSearchParams( query: compiledQuery, sources: input.sources, filters, + allowPartialResults: input.allowPartialResults, limit, offset, waitTimeoutMs, @@ -142,9 +144,17 @@ function resolveNumber( function resolveFileIntent( value: SearchSymbolsFileIntent | undefined, + sources: UnifiedSearchSource[] | undefined, defaulted: Array<"fileIntent" | "limit" | "offset" | "waitTimeoutMs">, -): SearchSymbolsFileIntent { +): SearchSymbolsFileIntent | undefined { if (value === undefined) { + if ( + sources && + sources.length > 0 && + sources.every((entry) => entry === "DOCS") + ) { + return undefined; + } defaulted.push("fileIntent"); return SEARCH_SYMBOLS_DEFAULT_FILE_INTENT; } diff --git a/src/shared/unified-search-response.test.ts b/src/shared/unified-search-response.test.ts index 5f717096..089ca46b 100644 --- a/src/shared/unified-search-response.test.ts +++ b/src/shared/unified-search-response.test.ts @@ -81,6 +81,46 @@ describe("buildUnifiedSearchSuccessPayload", () => { progress: expect.objectContaining({ status: "INDEXING" }), }); }); + + it("normalises incomplete outcomes with opt-in partial results", () => { + if (defaultUnifiedSearchOutcome.state !== "completed") { + throw new Error("expected completed outcome fixture"); + } + + const payload = buildUnifiedSearchSuccessPayload( + { ...params, allowPartialResults: true }, + "router middleware", + "router middleware", + ["fileIntent", "limit", "offset", "waitTimeoutMs"], + { + state: "incomplete", + completed: false, + searchRef: "search-ref-123", + result: { + ...defaultUnifiedSearchOutcome.result, + partialResults: true, + }, + progress: { + searchRef: "search-ref-123", + status: "INDEXING", + targetsTotal: 2, + targetsReady: 1, + elapsedMs: 200, + query: "router middleware", + queryWarnings: [], + sources: ["CODE"], + }, + }, + ); + + expect(payload.completed).toBe(false); + expect(payload.query.allowPartialResults).toBe(true); + expect(payload.returnedCount).toBe(1); + expect(payload.results[0]?.target).toBe("npm:express@4.18.2"); + expect(payload.sourceStatus).toEqual( + defaultUnifiedSearchOutcome.result.sourceStatus, + ); + }); }); describe("buildUnifiedSearchErrorPayload", () => { diff --git a/src/shared/unified-search-response.ts b/src/shared/unified-search-response.ts index 090875fa..57d86e49 100644 --- a/src/shared/unified-search-response.ts +++ b/src/shared/unified-search-response.ts @@ -20,6 +20,7 @@ export interface UnifiedSearchQueryEcho { fileIntent?: string; publicOnly?: boolean; }; + allowPartialResults: boolean; limit: number; offset: number; waitTimeoutMs: number; @@ -70,11 +71,13 @@ export interface UnifiedSearchCompletedPayload { export interface UnifiedSearchIncompletePayload { query: UnifiedSearchQueryEcho; completed: false; - returnedCount: 0; - hasMore: false; - results: []; + returnedCount: number; + hasMore: boolean; + nextOffset?: number; + results: UnifiedSearchHitPayload[]; searchRef: string; progress?: UnifiedSearchIncomplete["progress"]; + sourceStatus?: UnifiedSearchCompleted["result"]["sourceStatus"]; } export interface UnifiedSearchErrorPayload { @@ -106,6 +109,7 @@ export interface UnifiedSearchStatusIncompletePayload { completed: false; searchRef: string; progress?: UnifiedSearchIncomplete["progress"]; + result?: UnifiedSearchStatusResultPayload; } export function buildUnifiedSearchSuccessPayload( @@ -118,7 +122,9 @@ export function buildUnifiedSearchSuccessPayload( const warnings = outcome.state === "completed" ? outcome.result.queryWarnings - : (outcome.progress?.queryWarnings ?? []); + : (outcome.result?.queryWarnings ?? + outcome.progress?.queryWarnings ?? + []); const query = buildQueryEcho( params, rawQuery, @@ -128,15 +134,23 @@ export function buildUnifiedSearchSuccessPayload( ); if (outcome.state === "incomplete") { - return { + const result = outcome.result; + const payload: UnifiedSearchIncompletePayload = { query, completed: false, - returnedCount: 0, - hasMore: false, - results: [], + returnedCount: result?.results.length ?? 0, + hasMore: result?.page.hasMore ?? false, + results: result?.results.map(buildHitPayload) ?? [], searchRef: outcome.searchRef, progress: outcome.progress, }; + if (result?.page.hasMore === true) { + payload.nextOffset = result.page.offset + result.page.returned; + } + if (result) { + payload.sourceStatus = result.sourceStatus; + } + return payload; } return { @@ -175,29 +189,39 @@ export function buildUnifiedSearchStatusPayload( outcome: UnifiedSearchOutcome, ): UnifiedSearchStatusCompletedPayload | UnifiedSearchStatusIncompletePayload { if (outcome.state === "incomplete") { - return { + const payload: UnifiedSearchStatusIncompletePayload = { completed: false, searchRef: outcome.searchRef, progress: outcome.progress, }; + if (outcome.result) { + payload.result = buildUnifiedSearchStatusResultPayload(outcome.result); + } + return payload; } return { completed: true, searchRef: outcome.searchRef, progress: outcome.progress, - result: { - query: outcome.result.query, - queryWarnings: outcome.result.queryWarnings, - sources: outcome.result.sources.map((entry) => entry.toLowerCase()), - returnedCount: outcome.result.results.length, - hasMore: outcome.result.page.hasMore, - nextOffset: outcome.result.page.hasMore - ? outcome.result.page.offset + outcome.result.page.returned - : undefined, - results: outcome.result.results.map(buildHitPayload), - sourceStatus: outcome.result.sourceStatus, - }, + result: buildUnifiedSearchStatusResultPayload(outcome.result), + }; +} + +function buildUnifiedSearchStatusResultPayload( + result: UnifiedSearchCompleted["result"], +): UnifiedSearchStatusResultPayload { + return { + query: result.query, + queryWarnings: result.queryWarnings, + sources: result.sources.map((entry) => entry.toLowerCase()), + returnedCount: result.results.length, + hasMore: result.page.hasMore, + nextOffset: result.page.hasMore + ? result.page.offset + result.page.returned + : undefined, + results: result.results.map(buildHitPayload), + sourceStatus: result.sourceStatus, }; } @@ -223,6 +247,7 @@ function buildQueryEcho( publicOnly: params.filters.publicOnly, } : undefined, + allowPartialResults: params.allowPartialResults ?? false, limit: params.limit ?? 0, offset: params.offset ?? 0, waitTimeoutMs: params.waitTimeoutMs ?? 0, diff --git a/src/tools/grep-repo.test.ts b/src/tools/grep-repo.test.ts index b1915007..a0908828 100644 --- a/src/tools/grep-repo.test.ts +++ b/src/tools/grep-repo.test.ts @@ -34,9 +34,11 @@ describe("createGrepRepoTool — metadata", () => { "path_prefix", "pattern", "pattern_type", + "symbol_fields", "target", "wait_timeout_ms", ]); + expect(tool.schema.symbol_fields.description).toContain("parent_path"); expect(tool.annotations?.readOnlyHint).toBe(true); }); }); @@ -73,6 +75,7 @@ describe("createGrepRepoTool — happy path", () => { caseSensitive?: boolean; contextLinesBefore?: number; contextLinesAfter?: number; + symbolFields?: string[]; }, ] >; @@ -87,6 +90,28 @@ describe("createGrepRepoTool — happy path", () => { ]); }); + it("passes symbol field hydration through to grepRepo", async () => { + const grepRepo = mock(() => Promise.resolve(defaultGrepRepoResult)); + const tool = createGrepRepoTool( + createMockCodeNavigationService({ grepRepo }), + ); + + await tool.handler( + { + target: { registry: "npm", package_name: "express" }, + pattern: "middleware", + symbol_fields: ["name", "qualified_path", "kind"], + }, + {}, + ); + + expect(grepRepo).toHaveBeenCalledWith( + expect.objectContaining({ + symbolFields: ["name", "qualified_path", "kind"], + }), + ); + }); + it("emits the new envelope shape", async () => { const tool = createGrepRepoTool(createMockCodeNavigationService()); const result = await tool.handler( diff --git a/src/tools/grep-repo.ts b/src/tools/grep-repo.ts index f2e67cc0..2caa5fdf 100644 --- a/src/tools/grep-repo.ts +++ b/src/tools/grep-repo.ts @@ -5,6 +5,9 @@ import { buildGrepRepoParams, buildGrepRepoSuccessPayload, GREP_REPO_PATTERN_NOTE, + GREP_REPO_SYMBOL_FIELDS, + GREP_REPO_SYMBOL_FIELDS_NOTE, + type GrepRepoSymbolField, } from "../shared/index.js"; import { toPkgseerRegistryLowercase } from "../shared/pkgseer-registry.js"; import { @@ -31,6 +34,7 @@ export interface GrepRepoArgs { max_matches?: number; max_matches_per_file?: number; cursor?: string; + symbol_fields?: GrepRepoSymbolField[]; wait_timeout_ms?: number; } @@ -74,6 +78,10 @@ const schema = { max_matches: z.number().optional(), max_matches_per_file: z.number().optional(), cursor: z.string().optional(), + symbol_fields: z + .array(z.enum(GREP_REPO_SYMBOL_FIELDS)) + .optional() + .describe(GREP_REPO_SYMBOL_FIELDS_NOTE), wait_timeout_ms: z.number().optional(), }; @@ -113,6 +121,7 @@ export function createGrepRepoTool( maxMatches: args.max_matches, maxMatchesPerFile: args.max_matches_per_file, cursor: args.cursor, + symbolFields: args.symbol_fields, waitTimeoutMs: args.wait_timeout_ms, }); const result = await service.grepRepo(build.params); @@ -137,6 +146,7 @@ export function createGrepRepoTool( maxMatches: build.params.maxMatches ?? 50, maxMatchesPerFile: build.params.maxMatchesPerFile, cursor: args.cursor, + symbolFields: build.params.symbolFields, excludeDocFiles: build.params.excludeDocFiles, excludeTestFiles: build.params.excludeTestFiles, explicit: build.explicit, diff --git a/src/tools/search-status.test.ts b/src/tools/search-status.test.ts index a86ae35c..05d5ab9d 100644 --- a/src/tools/search-status.test.ts +++ b/src/tools/search-status.test.ts @@ -55,6 +55,13 @@ describe("searchStatusTool", () => { }); }); + it("describes partial-result follow-up behavior", () => { + const tool = createSearchStatusTool(createMockCodeNavigationService()); + + expect(tool.description).toContain("partial hits"); + expect(tool.description).toContain("allow_partial_results"); + }); + it("returns completed payload without fabricating initial query echo", async () => { const tool = createSearchStatusTool(createMockCodeNavigationService()); diff --git a/src/tools/search-status.ts b/src/tools/search-status.ts index 3592bbaf..d964700a 100644 --- a/src/tools/search-status.ts +++ b/src/tools/search-status.ts @@ -18,7 +18,7 @@ const schema = { }; const DESCRIPTION = - "Check progress or fetch final results for a prior unified search. " + + "Check progress, fetch partial hits when the original request used allow_partial_results: true, or fetch final results for a prior unified search. " + "Pass the search_ref returned by `search` when the original request did not complete within the wait window."; export function createSearchStatusTool( diff --git a/src/tools/search.test.ts b/src/tools/search.test.ts index 5c3d94cb..13d411c4 100644 --- a/src/tools/search.test.ts +++ b/src/tools/search.test.ts @@ -33,6 +33,7 @@ describe("searchTool", () => { target: { registry: "npm", package_name: "express" }, kind: "function", language: "typescript", + allow_partial_results: true, }, {}, ); @@ -40,6 +41,7 @@ describe("searchTool", () => { expect(search).toHaveBeenCalledWith( expect.objectContaining({ query: "(handler) AND (lang:typescript)", + allowPartialResults: true, filters: expect.objectContaining({ kind: "FUNCTION" }), }), ); diff --git a/src/tools/search.ts b/src/tools/search.ts index 1fbb132d..eb865f3a 100644 --- a/src/tools/search.ts +++ b/src/tools/search.ts @@ -68,13 +68,19 @@ export interface SearchArgs { public_only?: boolean; name?: string; language?: string; + allow_partial_results?: boolean; limit?: number; offset?: number; wait_timeout_ms?: number; } const schema = { - query: z.string().min(1).describe("Search query string."), + query: z + .string() + .min(1) + .describe( + "Discovery query string. Supports implicit AND, uppercase OR, parentheses, unary -, quoted phrases, semantic qualifiers (kind:, category:, path:, lang:, name:, intent:), and routing qualifiers (registry:, package:, version:, repo:). Parsed once and compiled per source; it is not forwarded as a raw backend query.", + ), target: codeTargetSchema.optional(), targets: z.array(codeTargetSchema).max(20).optional(), sources: z @@ -128,10 +134,19 @@ const schema = { "build", "vendor", ]) - .optional(), + .optional() + .describe( + "Optional file-intent filter. When omitted, AUTO/code/symbol searches default to production; explicit docs-only searches omit the filter because docs do not support it.", + ), public_only: z.boolean().optional(), name: z.string().optional(), language: z.string().optional(), + allow_partial_results: z + .boolean() + .optional() + .describe( + "Default false waits for all sources; if the wait window expires, returns only searchRef/progress. When true, includes hits from sources that finished so far and still returns searchRef for continuation. Partial payloads support normal pagination via nextOffset.", + ), limit: z.coerce.number().int().min(1).max(100).optional(), offset: z.coerce.number().int().min(0).optional(), wait_timeout_ms: z.coerce.number().int().min(0).max(60000).optional(), @@ -139,9 +154,10 @@ const schema = { const DESCRIPTION = "Search indexed dependency and repository code, docs, and explicit symbols. " + - "Structured parameters are the primary UX and combine with the raw query using AND semantics. " + + "The query field uses GitHits discovery syntax (AND/OR/parens/qualifiers; see the parameter description). " + + "Structured parameters combine with that query using AND semantics. " + "Provide either `target` for one target or `targets` for many. Omit `sources` to use backend AUTO. " + - "Results are trustworthy by default: if indexing is still in progress, this tool returns a `searchRef` state instead of partial hits. " + + "Results are complete by default; set `allow_partial_results: true` to include available hits while indexing continues. " + "Use `search_status` with that ref to continue."; export function createSearchTool( @@ -187,6 +203,7 @@ export function createSearchTool( publicOnly: args.public_only, name: args.name, language: args.language, + allowPartialResults: args.allow_partial_results, limit: args.limit, offset: args.offset, waitTimeoutMs: args.wait_timeout_ms,