Skip to content

EVAL.md

Latest commit

 

History

History
110 lines (89 loc) · 6.51 KB

EVAL.md

File metadata and controls

110 lines (89 loc) · 6.51 KB
Raw

Retrieval-accuracy eval

Generated by npm run eval (eval/eval.mjs) on 2026-06-19. Re-run to refresh. Complements RESULTS.md (token efficiency). This file answers the harder question: when agentmap returns fewer tokens, are they the right tokens?

Why this exists

RESULTS.md proves agentmap puts far fewer tokens in front of the agent. It does not prove those tokens contain the correct answer — "fewer tokens" could mean "fewer correct answers". This eval measures retrieval correctness against ground truth derived live from real public repos, then shows accuracy and token cost together.

Method

Ground truth is derived at runtime from each cloned repo (not hand-authored, so it can't silently rot), using a different mechanism than agentmap's parser graph — so the comparison is a real cross-check, not circular. TS/JS fixtures cross-check the ts-morph graph; the PHP/Laravel fixture cross-checks the tree-sitter-php graph (see PHP section below):

  • Symbol definition — "where is symbol X defined?" Ground truth = the single file whose source declares X (regex over declaration sites). Only globally unique definitions are tested (no ambiguity). Compared: agentmap --find X (exact-name matches, in returned order) vs naive git grep -n X (every occurrence). Metric: top-1 / top-3 hit rate (is the definition file the 1st / among the first 3 results?).
  • Dependents / blast radius — "which files import module M?" Ground truth = files whose import statements resolve to M (TS: a relative-import resolver; PHP: PSR-4 use resolution). Compared: agentmap --relates M .dependents vs a naive git grep -l import-line baseline. Metric: precision / recall / F1 against the resolved set.

Scope alignment (so the comparison is fair both ways): test files (*.test.*, tests/, etc.) are excluded from ground truth and from both tools' outputs before scoring — otherwise agentmap's legitimate test-file importers would score as false positives. Type-only edges (import type / export type) are excluded from TS ground truth, because agentmap's ts-morph graph drops them by design — counting them would penalize recall for a documented behaviour rather than a defect. Each fixture is scoped to a sourceRoot. Token cost = chars/4 of the full default (human) output each tool puts in context (same heuristic as RESULTS.md, both sides).

Caveats — read these before quoting numbers. (1) The ground-truth resolvers are regex-based, not a TypeScript type-checker or PHP analyzer; they are the reference. (2) Definitions tested are uniquely-declared only — the easy, unambiguous cases. (3) agentmap --find lists barrel re-export sites alongside the real declaration, which is why top-1 trails top-3 — the definition is usually in the top 3, not always first. (4) Dependents recall reflects agentmap's value-import graph only (type-only edges are excluded from truth to match it). (5) Feature-level retrieval (--feature) is not scored. (6) Numbers move with upstream repos; resolved SHAs are recorded below.

PHP / Laravel ground truth

PHP uses a different module model than TS/JS — there are no relative-path imports. Symbols are referenced by fully-qualified class name (FQCN), and use statements import an FQCN into a file's namespace. So the PHP ground truth is FQCN-based, derived with a different mechanism than agentmap's tree-sitter-php graph (keeping the cross-check honest):

  • Symbol definition — regex over class / interface / trait / enum / function declaration sites (comments stripped first). Only globally-unique definitions in sourceRoot are tested. Compared: agentmap --find X vs git grep -w X.
  • Dependents — file B depends on file A when B has a use A\FQCN; statement that PSR-4-resolves to A. PSR-4 prefixes are read from composer.json (autoload + autoload-dev), longest-prefix-first. The grep baseline matches use …\ClassName; lines (the class name is the file basename — PSR-4 mandates one class per file named after it).

PHP caveats. (1) The PSR-4 resolver reads only composer.json psr-4 maps — psr-0, classmap, and files autoloading are not resolved (rare in modern Laravel). (2) The declaration regex catches top-level class/function declarations; dynamically-defined or eval'd classes are out of scope (and out of agentmap's scope too). (3) Dependents counts use imports only — runtime FQCN references (\Foo\Bar::class written inline without a use) are not counted as edges, matching agentmap's import-graph semantics. (4) Blade views (*.blade.php) are excluded from symbol-definition ground truth — they are templates, not declaration sites. (5) laravel/framework is large; the eval scopes ground truth to src/Illuminate and samples deterministically.

Results

Overall (pooled across fixtures)

Task n agentmap naive grep
Symbol definition — top-1 / top-3 hit 100 63% / 96% 41% / 81%
Dependents — recall / precision 62 98.7% / 100% 100% / 69.7%

Symbol-definition lookups cost a median ~2.4× fewer tokens than dumping git grep output, while landing the definition in the top 3 more often. For dependents the story is a precision win, not a token win: agentmap returns a clean importer list (high precision) where naive grep returns a noisy superset (high recall, low precision) — and --relates actually costs more tokens than grep -l because it returns the full blast radius (exports + imports + dependents + related), not just the file list.

Per fixture

Repo lang commit def n agentmap top1/top3 grep top1/top3 deps n agentmap recall/prec grep recall/prec
zod ts 912f0f51b0 25 64% / 84% 40% / 84% 20 98.3% / 100% 100% / 49.9%
zustand ts a1f685ca74 25 16% / 100% 24% / 80% 2 100% / 100% 100% / 65%
hono ts 27b7992f82 25 72% / 100% 24% / 72% 20 97.5% / 100% 100% / 66%
laravel-framework php a550e02594 25 100% / 100% 76% / 88% 20 100% / 100% 100% / 93.7%

Reproduce

npm run eval                         # all fixtures (TS/JS + Laravel)
node eval/eval.mjs --repo zod --sample 40
node eval/eval.mjs --repo laravel-framework  # PHP/Laravel fixture standalone
node eval/eval.mjs --refresh         # re-clone upstreams

Clones land in tmp/eval/ (gitignored). Network required; not part of CI.