Add agentic eval harness

.gitignore

@@ -13,6 +13,7 @@ coverage
 
 # Agents
 .claude
+.agent-eval/
 
 # logs
 logs

AGENTS.md

@@ -36,6 +36,7 @@ Philosophy: "If it is not tested, it is likely broken"
 
 - Use `bun test` for running tests
 - Use `bun run smoke:mcp` and `bun run smoke:cli` when changing MCP tools, CLI commands, shared formatters, auth/error envelopes, or MCP/CLI parity behavior. These are live smoke suites, not the normal unit suite; they must pass unauthenticated by validating auth handling, and provide deeper coverage when authenticated.
+- Use `bun run agent:e2e` when changing MCP instructions, tool descriptions, or agent-facing tool behavior. This is a human/agent-driven qualitative eval, not a deterministic CI gate. Pick targeted workloads from `eval/agentic/README.md`; run both Claude and Codex for broad instruction changes when practical. Inspect `tool-calls.json` and `final.json` for actual tool use, `toolIssues`, `instructionIssues`, and usefulness, not just harness pass/fail.
 - Maintain smoke coverage when adding or changing user-facing tools/commands. Prefer structural UX assertions over brittle snapshots, and keep MCP `format: "json"` and CLI `--json` behavior aligned.
 - Keep tests async and isolated
 - Mock services at the interface level using factory functions

eval/agentic/README.md

@@ -0,0 +1,155 @@
+# Agentic Eval Harness
+
+This harness runs real coding agents against the real GitHits MCP server and
+records whether the agent can use GitHits tools effectively from the MCP
+server's own instructions and tool descriptions.
+
+It is not a smoke test. Smoke tests exercise CLI and MCP contracts directly.
+Agentic evals exercise agent behavior end-to-end.
+
+This harness is intentionally human/agent-driven, not CI. Use it to understand
+how MCP instruction or tool-description changes affect real agent behavior. Do
+not treat a live agent pass/fail result as a deterministic regression test: model
+behavior, backend indexing state, auth state, network conditions, and package
+data can all change. The useful output is the artifact set, especially
+`tool-calls.json`, `final.json`, `toolIssues`, `instructionIssues`, and the
+agent's usefulness assessment.
+
+## What Is Under Test
+
+- Local mode starts the MCP server from this checkout with
+  `bun run --cwd <repo> dev mcp start`.
+- Published mode starts the MCP server with `npx -y githits@latest mcp start`
+  by default.
+- To evaluate MCP instruction changes, change branch/source and run local mode.
+
+Smoke tests are the right fit for CI gating. Agentic evals are the right fit for
+qualitative review before/after instruction, tool-description, and agent-facing
+UX changes.
+
+The harness must not add GitHits usage guidance through agent system prompts,
+append prompts, alternate MCP instruction files, project instructions, or plugin
+commands. Workload prompts may ask the agent to report what happened, but must
+not tell the agent how to use GitHits.
+
+## Isolation
+
+Runs execute agents from an empty temporary workspace so repository-local files
+such as `AGENTS.md`, `.mcp.json`, commands, skills, and plugin payloads do not
+contaminate results. The harness keeps the user's normal Claude/GitHits auth
+environment so human-driven keychain/OAuth sessions continue to work.
+
+GitHits authentication follows normal local behavior. Keychain-backed human
+login should work by default. Automation can use `GITHITS_API_TOKEN`.
+
+## Usage
+
+```bash
+bun run agent:e2e --server local --workload eval/agentic/workloads/express-router.md
+bun run agent:e2e --server published --workload eval/agentic/workloads/express-router.md
+bun run agent:e2e --agent codex --server local --workload eval/agentic/workloads/express-router.md
+```
+
+Useful options:
+
+```bash
+--agent <claude|codex>          Agent to run, default `claude`
+--dry-run                       Generate artifacts without invoking the agent
+--out <dir>                     Output directory, default `.agent-eval/runs/<timestamp>`
+--timeout <seconds>             Per-workload timeout, default 300
+--published-package <spec>      Published package spec, default `githits@latest`
+--workload <path>               Repeatable workload path
+```
+
+Normal GitHits backend overrides are passed through when set:
+
+- `GITHITS_API_URL`
+- `GITHITS_MCP_URL`
+- `GITHITS_CODE_NAV_URL`
+- `PKGSEER_URL`
+- `GITHITS_API_TOKEN`
+- `GITHITS_AUTH_STORAGE`
+
+Secret-like values are redacted in run metadata.
+
+## Workloads
+
+Workloads are Markdown prompts. They should contain:
+
+- The task.
+
+The harness appends `eval/agentic/workloads/REPORTING.md` to every workload so
+all agents return the same structured report. Workload files should not repeat
+that reporting contract.
+
+They should not contain instructions such as "call `search` first" or "use
+`code_read` after `search`". That guidance must come from the MCP server.
+
+### Workload Selection
+
+Use targeted workloads when a change affects a specific tool family. Use both
+Claude and Codex for instruction/tool-description changes when practical; use at
+least one agent for quick iteration.
+
+| Affected Area | Workload |
+|---|---|
+| Core global examples, `get_example`, `search_language`, `feedback` | `global-example.md` |
+| Unified `search` / `search_status` behavior | `unified-search-investigation.md` |
+| Package overview or vulnerability UX, `pkg_info`, `pkg_vulns` | `package-overview-vulnerabilities.md` |
+| Dependency graph UX, `pkg_deps` | `package-dependencies.md` |
+| Release notes UX, `pkg_changelog` | `package-changelog.md` |
+| Documentation browsing, `docs_list`, `docs_read` | `docs-discovery.md` |
+| File listing / file read UX, `code_files`, `code_read` | `code-file-navigation.md` |
+| Deterministic source search UX, `code_grep` | `code-grep-investigation.md` |
+| Multi-tool code navigation strategy and MCP instructions | `express-router.md` |
+
+For broad MCP instruction edits, run at least:
+
+```bash
+bun run agent:e2e --agent claude --server local --workload eval/agentic/workloads/express-router.md
+bun run agent:e2e --agent codex --server local --workload eval/agentic/workloads/express-router.md
+```
+
+For tool-specific edits, add the workload from the table. Compare
+`tool-calls.json` plus the final JSON's `toolIssues`, `instructionIssues`, and
+`githitsUsefulnessReason` across branches or against a published run.
+
+### Current Baseline Observations
+
+The initial local baseline ran all workloads against Claude and Codex. Expected
+tool families were exercised after tightening the shared reporting contract.
+Notable findings to keep in mind when evaluating future changes:
+
+- `global-example.md` exercises `get_example`; agents may combine it with docs,
+  source, or package metadata when they need stronger canonical evidence.
+- `code-grep-investigation.md` surfaced a real `code_grep` regex limitation for
+  short/stopword-heavy patterns; literal grep is the reliable path for that
+  workload.
+- `unified-search-investigation.md` intentionally exposes `search` warnings and
+  follow-up needs. Agents should inspect warnings and use `code_read`,
+  `docs_read`, or `code_grep` when top search hits are incomplete/noisy.
+- Codex sometimes reports a tool as unavailable until it performs additional
+  tool discovery. Use `tool-calls.json` to distinguish actual unavailable tools
+  from delayed discovery.
+- `tool-calls.json` is the source of truth for tool usage. The final JSON is for
+  the agent's assessment of clarity, issues, and usefulness.
+
+## Artifacts
+
+Each run writes:
+
+- `run.json` with command, git, environment, and timing metadata.
+- One workload directory per workload with `prompt.md`, `mcp.json`,
+  `stdout.json`, `stderr.txt`, `tool-calls.json`, and `final.json` when parsing
+  succeeds.
+
+Claude is launched with `--permission-mode bypassPermissions` so non-interactive
+evals can exercise configured MCP tools without a human approval prompt, and
+`--disable-slash-commands` to reduce plugin/skill contamination while preserving
+normal human auth. Codex is launched with per-run `-c` MCP config overrides,
+`--ignore-rules`, and a read-only sandbox so it can use normal human auth
+without mutating global MCP configuration.
+
+Malformed final JSON, schema mismatches, Claude failures, and timeouts are
+harness failures. Raw stdout and stderr are preserved for diagnosis with known
+secret values redacted.

eval/agentic/result.schema.json

@@ -0,0 +1,54 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "status",
+    "answer",
+    "toolIssues",
+    "expectedToolUse",
+    "unexpectedToolUse",
+    "instructionIssues",
+    "githitsUsefulness",
+    "githitsUsefulnessReason",
+    "confidence"
+  ],
+  "properties": {
+    "status": {
+      "type": "string",
+      "enum": ["success", "failure", "inconclusive"]
+    },
+    "answer": {
+      "type": "string"
+    },
+    "toolIssues": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "expectedToolUse": {
+      "type": "array",
+      "items": { "type": "string" }
+    },
+    "unexpectedToolUse": {
+      "type": "array",
+      "items": { "type": "string" }
+    },
+    "instructionIssues": {
+      "type": "array",
+      "items": { "type": "string" }
+    },
+    "githitsUsefulness": {
+      "type": "string",
+      "enum": ["helped", "hurt", "unused", "unclear"]
+    },
+    "githitsUsefulnessReason": {
+      "type": "string"
+    },
+    "confidence": {
+      "type": "string",
+      "enum": ["high", "medium", "low"]
+    }
+  }
+}

eval/agentic/workloads/REPORTING.md

@@ -0,0 +1,16 @@
+Return only valid JSON matching the provided schema. Do not include markdown,
+prose, code fences, or any text outside the JSON object. Include:
+
+- `status`: `success`, `failure`, or `inconclusive`.
+- `answer`: your final answer with evidence.
+- `toolIssues`: string descriptions of GitHits tool calls that failed or
+  returned unclear output, if any.
+- `expectedToolUse`: string list of GitHits tools you expected to use for this
+  workload. Use `[]` if none.
+- `unexpectedToolUse`: string list of tools or fallback paths you used but did
+  not expect to need. Use `[]` if none.
+- `instructionIssues`: MCP guidance that was unclear, missing, or contradicted
+  observed tool behavior, if any.
+- `githitsUsefulness`: `helped`, `hurt`, `unused`, or `unclear`.
+- `githitsUsefulnessReason`: why you chose that usefulness value.
+- `confidence`: `high`, `medium`, or `low`.

eval/agentic/workloads/code-file-navigation.md

@@ -0,0 +1,5 @@
+# Workload: Source File Navigation
+
+You are debugging how `npm:express` creates an application object. Identify the
+source file that exports the application factory, read the relevant section, and
+explain how request and response prototypes are attached.

eval/agentic/workloads/code-grep-investigation.md

@@ -0,0 +1,5 @@
+# Workload: Deterministic Source Grep
+
+You are investigating where Express imports the standalone `router` package.
+Find the exact files and lines in `npm:express` that import it, then summarize
+what each import site does.

eval/agentic/workloads/docs-discovery.md

@@ -0,0 +1,5 @@
+# Workload: Documentation Discovery
+
+You need to learn how Express documents routing behavior. Find relevant package
+documentation for `npm:express`, read enough to summarize how routes are defined,
+and include source references for the documentation you relied on.

eval/agentic/workloads/express-router.md

@@ -0,0 +1,5 @@
+# Workload: Express Router Investigation
+
+You are investigating how Express implements routing. Determine where the main
+Router implementation lives in the Express package and summarize the evidence
+you found.

eval/agentic/workloads/global-example.md

@@ -0,0 +1,5 @@
+# Workload: Canonical Example Discovery
+
+You need a current, canonical TypeScript example for validating user input with
+Zod and returning typed parsed data. Find a suitable example or pattern from
+open source and explain how you would adapt it in a small CLI command.

eval/agentic/workloads/package-changelog.md

@@ -0,0 +1,6 @@
+# Workload: Release History Assessment
+
+You are assessing recent `npm:express` releases before upgrading. Summarize the
+most recent release notes and call out anything that looks security-relevant,
+breaking, or operationally important. If full bodies are unnecessary, keep the
+answer concise and say why.

eval/agentic/workloads/package-dependencies.md

@@ -0,0 +1,6 @@
+# Workload: Dependency Graph Assessment
+
+You are reviewing `npm:express` for dependency footprint risk. Explain its
+runtime dependency shape, whether optional or non-runtime dependency groups are
+relevant to the adoption decision, and whether transitive dependency details
+change your recommendation.

eval/agentic/workloads/package-overview-vulnerabilities.md

@@ -0,0 +1,6 @@
+# Workload: Package Overview And Vulnerability Check
+
+You are evaluating whether `npm:express` is a reasonable dependency for a new
+Node.js service. Summarize the latest package metadata and whether known
+vulnerabilities should block adoption. Include version, license, repository
+health signals if available, active advisory status, and any caveats.

eval/agentic/workloads/unified-search-investigation.md

@@ -0,0 +1,6 @@
+# Workload: Cross-Surface Search Investigation
+
+You are comparing how Express exposes `Router` across package metadata, source,
+and public API references. Investigate `npm:express` and explain whether the
+search results were enough by themselves or whether you needed follow-up reads.
+Report any warnings, incomplete results, or follow-up status checks you saw.

package.json

@@ -35,6 +35,7 @@
     "inspector": "npx @modelcontextprotocol/inspector bun run dev mcp",
     "smoke:cli": "bun run scripts/cli-smoke.ts",
     "smoke:mcp": "bun run scripts/mcp-smoke.ts",
+    "agent:e2e": "bun run scripts/agent-eval.ts",
     "test": "bun test",
     "typecheck": "tsc",
     "format": "biome format --write .",