diff --git a/docs/development/performance.md b/docs/development/performance.md index c04031a6..ea6ea940 100644 --- a/docs/development/performance.md +++ b/docs/development/performance.md @@ -829,7 +829,7 @@ The self-trial is also an accuracy fixture. CodeVetter's first inference blamed a bounded coverage-filename sort even though the workload held file count at one. That hypothesis was rejected from the captured workload identity before the observed offset helper was optimized. The active OpenSpec -[qualification](../../openspec/changes/add-runtime-performance-capsules/qualification.md) +[qualification](../../openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/qualification.md) records the complete evidence, the conservative shipping limitation, and the decision-explanation bug found and fixed during the loop. diff --git a/openspec/changes/add-runtime-performance-capsules/.openspec.yaml b/openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/.openspec.yaml similarity index 100% rename from openspec/changes/add-runtime-performance-capsules/.openspec.yaml rename to openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/.openspec.yaml diff --git a/openspec/changes/add-runtime-performance-capsules/design.md b/openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/design.md similarity index 100% rename from openspec/changes/add-runtime-performance-capsules/design.md rename to openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/design.md diff --git a/openspec/changes/add-runtime-performance-capsules/proposal.md b/openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/proposal.md similarity index 100% rename from openspec/changes/add-runtime-performance-capsules/proposal.md rename to openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/proposal.md diff --git a/openspec/changes/add-runtime-performance-capsules/qualification.md b/openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/qualification.md similarity index 100% rename from openspec/changes/add-runtime-performance-capsules/qualification.md rename to openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/qualification.md diff --git a/openspec/changes/add-runtime-performance-capsules/specs/runtime-performance-capsules/spec.md b/openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/specs/runtime-performance-capsules/spec.md similarity index 100% rename from openspec/changes/add-runtime-performance-capsules/specs/runtime-performance-capsules/spec.md rename to openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/specs/runtime-performance-capsules/spec.md diff --git a/openspec/changes/add-runtime-performance-capsules/tasks.md b/openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/tasks.md similarity index 100% rename from openspec/changes/add-runtime-performance-capsules/tasks.md rename to openspec/changes/archive/2026-08-15-add-runtime-performance-capsules/tasks.md diff --git a/openspec/changes/add-runtime-qualification-planner/.openspec.yaml b/openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/.openspec.yaml similarity index 100% rename from openspec/changes/add-runtime-qualification-planner/.openspec.yaml rename to openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/.openspec.yaml diff --git a/openspec/changes/add-runtime-qualification-planner/design.md b/openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/design.md similarity index 100% rename from openspec/changes/add-runtime-qualification-planner/design.md rename to openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/design.md diff --git a/openspec/changes/add-runtime-qualification-planner/proposal.md b/openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/proposal.md similarity index 100% rename from openspec/changes/add-runtime-qualification-planner/proposal.md rename to openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/proposal.md diff --git a/openspec/changes/add-runtime-qualification-planner/qualification.md b/openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/qualification.md similarity index 100% rename from openspec/changes/add-runtime-qualification-planner/qualification.md rename to openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/qualification.md diff --git a/openspec/changes/add-runtime-qualification-planner/specs/local-node-script-profiling/spec.md b/openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/specs/local-node-script-profiling/spec.md similarity index 100% rename from openspec/changes/add-runtime-qualification-planner/specs/local-node-script-profiling/spec.md rename to openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/specs/local-node-script-profiling/spec.md diff --git a/openspec/changes/add-runtime-qualification-planner/specs/performance-run-diagnostics/spec.md b/openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/specs/performance-run-diagnostics/spec.md similarity index 100% rename from openspec/changes/add-runtime-qualification-planner/specs/performance-run-diagnostics/spec.md rename to openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/specs/performance-run-diagnostics/spec.md diff --git a/openspec/changes/add-runtime-qualification-planner/specs/runtime-qualification-planning/spec.md b/openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/specs/runtime-qualification-planning/spec.md similarity index 100% rename from openspec/changes/add-runtime-qualification-planner/specs/runtime-qualification-planning/spec.md rename to openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/specs/runtime-qualification-planning/spec.md diff --git a/openspec/changes/add-runtime-qualification-planner/tasks.md b/openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/tasks.md similarity index 100% rename from openspec/changes/add-runtime-qualification-planner/tasks.md rename to openspec/changes/archive/2026-08-15-add-runtime-qualification-planner/tasks.md diff --git a/openspec/specs/local-node-script-profiling/spec.md b/openspec/specs/local-node-script-profiling/spec.md new file mode 100644 index 00000000..f574db58 --- /dev/null +++ b/openspec/specs/local-node-script-profiling/spec.md @@ -0,0 +1,23 @@ +# local-node-script-profiling Specification + +## Purpose +Define how CodeVetter profiles a repository-owned standalone JavaScript benchmark when the workload is an executable script rather than a test-runner test. +## Requirements +### Requirement: Standalone Node profiling is closed and exact +CodeVetter SHALL accept `node-script` only for one repository-contained `.js`, `.mjs`, or `.cjs` target. It MUST invoke the current Node executable with that target and no caller-supplied arguments, shell, package script, install step, or inherited application environment. + +#### Scenario: Local JavaScript benchmark is selected +- **WHEN** an agent explicitly profiles a repository-relative `.mjs` benchmark through `node-script` +- **THEN** CodeVetter executes only that file with the bounded sample, warmup, timeout, environment, and process-cleanup policy + +#### Scenario: Non-JavaScript target is supplied +- **WHEN** a `node-script` target has an unsupported extension or escapes the repository +- **THEN** CodeVetter rejects it before starting a process + +### Requirement: Node script evidence uses the existing performance contract +A successful standalone script profile SHALL emit the existing versioned performance capsule with `node-script` adapter identity, wall-time samples, bounded console benchmark metrics, V8 source profiles, limitations, and disposable artifact guarantees. CodeVetter MUST NOT describe the script as a test or infer correctness from exit success alone. + +#### Scenario: Script emits benchmark metrics +- **WHEN** a successful script emits bounded `[benchmark]` metrics and repository-owned V8 samples +- **THEN** the capsule reports median unprofiled metrics and source attribution separately +- **AND** labels the scope as a Node script rather than a test-runner test diff --git a/openspec/specs/performance-run-diagnostics/spec.md b/openspec/specs/performance-run-diagnostics/spec.md new file mode 100644 index 00000000..c2211f1f --- /dev/null +++ b/openspec/specs/performance-run-diagnostics/spec.md @@ -0,0 +1,27 @@ +# performance-run-diagnostics Specification + +## Purpose +Ensure incomplete performance runs contain enough bounded runtime evidence for an agent to distinguish a broken workload from a profiler or runner compatibility problem. +## Requirements +### Requirement: Failed passes retain bounded redacted evidence +Each unsuccessful warmup, measurement, metrics, profile, or coverage execution SHALL report its phase, index, terminal state, duration, workload-selection state, and bounded redacted operational error, stdout, and stderr. Successful executions SHALL omit failure output. + +#### Scenario: Profiler option is unsupported +- **WHEN** a profiling pass exits non-zero because the installed runner rejects an option +- **THEN** the performance capsule includes the bounded redacted runner error on that exact pass +- **AND** the diagnosis remains `no_confidence` + +#### Scenario: Failure output contains a credential-shaped value +- **WHEN** failed process output contains a token or repository path +- **THEN** retained failure evidence contains redaction markers and no raw sensitive value + +### Requirement: Bounded output does not erase confirmed selection +Every execution summary SHALL record whether bounded output directly confirms that the requested workload ran. For repeated successful Vitest passes, CodeVetter MAY accept one exact passed-selection confirmation for the fixed command when another successful JSON report was truncated, but MUST NOT accept a zero-match or failed execution as complete. + +#### Scenario: Exact Vitest JSON is truncated +- **WHEN** measurement JSON exceeds the output bound but a successful metrics or profile pass confirms the exact test identity +- **THEN** CodeVetter treats selection as confirmed while disclosing output truncation + +#### Scenario: No pass confirms the exact identity +- **WHEN** every bounded report lacks evidence that the selected test ran +- **THEN** the performance capsule remains `no_confidence` even if processes exited zero diff --git a/openspec/specs/runtime-performance-capsules/spec.md b/openspec/specs/runtime-performance-capsules/spec.md new file mode 100644 index 00000000..add0cd48 --- /dev/null +++ b/openspec/specs/runtime-performance-capsules/spec.md @@ -0,0 +1,226 @@ +# runtime-performance-capsules Specification + +## Purpose +Define bounded runtime performance evidence that helps coding agents detect regressions and locate repository-owned bottleneck candidates in exact Node, React, and Go workflows. +## Requirements +### Requirement: Profiling execution is exact, bounded, and opt-in +CodeVetter SHALL profile only a closed supported adapter with one +repository-relative target and optional exact workload name. Each request MUST +declare bounded warmup and sample counts, use separated process arguments, and +apply the existing repository-containment, timeout, minimal-environment, and +owned-process cleanup guarantees. + +#### Scenario: Exact Node workload is profiled +- **WHEN** a user selects a Node test or Vitest file and an exact test identity +- **THEN** CodeVetter runs only that scope for the bounded warmup, measurement, and profiling passes + +#### Scenario: Exact Go benchmark is profiled +- **WHEN** a user selects a Go benchmark file and exact benchmark identity +- **THEN** CodeVetter runs that package with an exact benchmark expression and bounded count without running unrelated tests + +#### Scenario: Profile target escapes the repository +- **WHEN** the target is absolute, traverses outside the repository, or resolves through an escaping symlink +- **THEN** CodeVetter rejects the request before starting a process + +#### Scenario: Repository contains untracked files +- **WHEN** the profiled worktree contains an untracked non-ignored file +- **THEN** the capsule records a dirty snapshot without fabricating changed-line intersections for that file + +### Requirement: Performance capsules separate measurements from findings +Each successful request SHALL emit one versioned Runtime Performance Capsule +containing subject identity, exact scope, terminal executions, wall-time +distribution, runtime-specific measurements, repository-owned hotspots, +deterministic findings, limitations, and capture coverage. Captured values MUST +remain separate from comparisons and unverified optimization hypotheses. + +#### Scenario: V8 profile contains repository source samples +- **WHEN** a Node or Vitest profiling pass produces valid V8 CPU profiles +- **THEN** the capsule reports bounded repository-relative hotspot functions, source locations, self time, and sample share as observed evidence + +#### Scenario: Go benchmark reports allocations +- **WHEN** a Go benchmark emits valid benchmark and allocation measurements +- **THEN** the capsule reports bounded `ns/op`, `B/op`, and `allocs/op` values with benchmark identity and provenance + +#### Scenario: Go profiles contain repository symbols +- **WHEN** the diagnostic Go pass emits CPU and allocation profiles that the installed toolchain can read +- **THEN** the capsule reports bounded repository-relative functions, source lines, flat and cumulative values, profile kind, and sample share without retaining the raw profiles + +#### Scenario: Runtime profile lacks application samples +- **WHEN** profiling captures only runner, dependency, or test-harness work +- **THEN** CodeVetter reports the coverage limitation and does not name an application bottleneck + +#### Scenario: Vitest startup dominates the measured scope +- **WHEN** reported assertion time is less than the bounded share of exact-scope process wall time +- **THEN** CodeVetter records a startup-dominated finding and does not attribute the process latency to product code + +### Requirement: Baseline comparisons are explicit and reproducible +CodeVetter SHALL compare a current capsule only with an explicitly supplied, +compatible baseline capsule. The comparison MUST record both subject identities, +metric, sample counts, absolute and percentage deltas, and threshold policy. +Incompatible or statistically insufficient evidence MUST remain +`no_confidence`. + +#### Scenario: Wall time materially regresses +- **WHEN** the current median exceeds the compatible baseline by both the recorded relative and absolute thresholds +- **THEN** the capsule deterministically reports a regression with the measured deltas and exits with the regression outcome + +#### Scenario: Current and baseline scopes differ +- **WHEN** adapter, target, or exact workload identity differs between current and baseline +- **THEN** CodeVetter refuses the comparison and reports `no_confidence` + +#### Scenario: No baseline is supplied +- **WHEN** profiling completes without a baseline capsule +- **THEN** CodeVetter reports measurements and hotspot candidates without claiming regression or improvement + +#### Scenario: Host timing is unstable +- **WHEN** current or baseline wall-time samples exceed the recorded variability threshold +- **THEN** CodeVetter reports host-load and sample-spread evidence and refuses to claim a regression + +### Requirement: Profiling artifacts are private, bounded, and disposable +CodeVetter MUST create runtime profiles only in an owned temporary directory, +redact repository prefixes and credential-shaped values before output, bound all +measurements and hotspots, disclose truncation, and remove the temporary profile +directory after parsing. It MUST NOT modify tracked target files or retain raw +profiles by default. + +#### Scenario: Profile output contains sensitive material +- **WHEN** runner output or a source label contains credential-shaped data +- **THEN** the capsule contains a redaction marker and records the redaction count + +#### Scenario: Profile contains excessive nodes +- **WHEN** runtime evidence exceeds collection bounds +- **THEN** the capsule retains only the ranked bounded subset and discloses omitted evidence + +### Requirement: Machine outcomes distinguish regression from uncertainty +The profiling CLI SHALL emit exactly one JSON document. It SHALL exit `0` for a +completed measurement with no demonstrated regression, `1` for a demonstrated +compatible-baseline regression, and `2` for invalid input, failed workloads, +timeouts, incompatible baselines, incomplete measurements, or cleanup failure. + +#### Scenario: Workload fails during profiling +- **WHEN** any required measured or profiling execution exits unsuccessfully +- **THEN** CodeVetter emits `no_confidence`, records the failed execution, and exits `2` + +### Requirement: Agents receive an evidence-linked performance diagnosis +CodeVetter SHALL expose one `diagnose-performance` operation that profiles an +exact supported workload and emits a versioned diagnosis containing the +originating capsule, ranked observations, explicit inferences, unverified +hypotheses, one next bounded action, and a same-scope verification recipe. The +diagnosis MUST be deterministic and MUST NOT invoke a model or modify source. + +#### Scenario: Go benchmark exposes allocation pressure +- **WHEN** a successful Go capsule contains allocation measurements and repository-owned allocation paths +- **THEN** the diagnosis ranks the measured `B/op` and `allocs/op`, identifies the strongest path as an inferred candidate, and requires the same benchmark to verify an optimization + +#### Scenario: Deterministic catalogue benchmark scales superlinearly +- **WHEN** a successful capsule contains at least two same-unit metrics whose names encode increasing input sizes +- **THEN** the diagnosis reports the endpoint ratios and scaling exponent as observed evidence and identifies the leading repository hotspot only as an optimization candidate + +#### Scenario: Runner startup dominates +- **WHEN** the capsule contains a startup-dominated finding +- **THEN** the diagnosis does not name application code as the primary issue and asks for a longer or batched representative workload + +#### Scenario: Profiling evidence is incomplete +- **WHEN** the originating capsule has `no_confidence` or lacks the observations required for a diagnosis +- **THEN** the diagnosis preserves the limitations and recommends a bounded evidence-improvement experiment rather than an optimization + +#### Scenario: Agent applies an optimization +- **WHEN** the diagnosis identifies an actionable candidate +- **THEN** its unverified hypothesis states what metric should move and its verification recipe selects the identical adapter, target, workload name, and sample policy + +### Requirement: Runtime candidates receive bounded source context +CodeVetter SHALL inspect source only for bounded repository-owned locations +already selected by runtime evidence. Each source observation MUST record its +file and line window, redacted excerpt, matched pattern and exact pattern lines, +and containment limitations. Source patterns MUST remain distinct from inferred +optimization claims. + +#### Scenario: Scaling hotspot intersects sort then slice +- **WHEN** a growing deterministic scale curve points to a JavaScript or TypeScript source window that fully sorts mapped candidates before taking a bounded slice and per-input cost materially increases +- **THEN** CodeVetter records the source pattern as observed evidence and proposes bounded top-k selection or deferred result materialization only as a falsifiable hypothesis + +#### Scenario: Runtime-selected source escapes containment +- **WHEN** a source location is absolute, traverses outside the repository, resolves through an escaping symlink, exceeds the file bound, or uses an unsupported source type +- **THEN** CodeVetter omits the excerpt, records the limitation, and does not weaken the runtime diagnosis + +#### Scenario: Hot TypeScript method retains only the first split segment +- **WHEN** a measured growing CPU path reaches a TypeScript method that splits a string with limit one and reads only element zero +- **THEN** CodeVetter confines inspection to that method and proposes direct delimiter search and slicing only as a falsifiable hypothesis + +#### Scenario: Growing path repeatedly scans materialized object keys +- **WHEN** a superlinear measured path materializes object keys and repeatedly performs linear membership checks over that array +- **THEN** CodeVetter records the exact key-materialization and membership lines and proposes indexed membership only as a falsifiable hypothesis + +### Requirement: Optimization verification uses identical domain evidence +CodeVetter SHALL expose `verify-optimization` for a repository-contained +baseline performance capsule or diagnosis and a newly captured identical scope. +It MUST verify workload success and compatibility before comparing bounded +domain metrics, and MUST return `confirmed`, `rejected`, `inconclusive`, or +`no_confidence` with exact deltas and policy. + +#### Scenario: Catalogue optimization improves high-end scaling +- **WHEN** baseline and current capsules contain the same encoded input sizes and units, the largest-input time improves materially, no smaller input materially regresses, and both workloads pass +- **THEN** verification reports `confirmed` with per-input deltas and exponent movement + +#### Scenario: Go allocation optimization reduces allocations +- **WHEN** identical Go benchmarks pass and B/op or allocs/op improves materially without an unacceptable ns/op regression +- **THEN** verification reports `confirmed` with all three metric deltas + +#### Scenario: Workload identities differ +- **WHEN** schema, adapter, target, exact workload name, scale inputs, units, or Go benchmark identity differ +- **THEN** verification reports `no_confidence` and makes no optimization claim + +#### Scenario: Candidate does not improve its target metric +- **WHEN** compatible before/after evidence is stable within the recorded policy +- **THEN** verification reports `inconclusive`; when the target metric materially regresses it reports `rejected` + +### Requirement: Paired verification controls temporal host drift +CodeVetter SHALL support an opt-in paired verification operation when baseline +and candidate are available as independently runnable contained repositories. +It MUST alternate execution order, require exact workload and metric identity, +record the bounded schedule, and refuse confirmation if either side fails or +does not provide repeated comparable measurements. + +#### Scenario: Two runnable revisions expose the same benchmark +- **WHEN** baseline and candidate repositories contain the same exact benchmark and emit matching domain metrics +- **THEN** CodeVetter alternates their run order, compares their repeated median measurements, and labels the result as paired evidence + +#### Scenario: One paired workload fails +- **WHEN** any required baseline or candidate execution fails, times out, or does not execute the exact selected workload +- **THEN** paired verification reports `no_confidence` and cannot confirm the optimization + +### Requirement: Performance coverage is qualified on owned and real targets +CodeVetter SHALL maintain hermetic fixtures for wall-time distributions, V8 +hotspots, Go benchmark parsing, redaction, comparison outcomes, and incomplete +profiles. Real-project performance claims MUST identify the target revision, +machine context, workload, sample policy, and observed limitations. + +#### Scenario: App Health qualification runs +- **WHEN** the profiler is exercised against selected App Health Node and Go benchmark scopes +- **THEN** results report measured coverage and gaps without changing App Health tracked files or contacting production services + +#### Scenario: Consumer browser qualification runs +- **WHEN** a supported local browser connector traces an exact Significant Hobbies loopback journey +- **THEN** the qualification records vitals, network and main-thread observations, connector identity, run bounds, and limitations without contacting hosted services + +#### Scenario: Consumer algorithm scale qualification runs +- **WHEN** Anime List recommendation logic is exercised at fixed deterministic catalogue sizes +- **THEN** the qualification records per-size timing, result identity, scaling behavior, and whether application work is material relative to runner overhead + +#### Scenario: Open-source qualification runs +- **WHEN** CodeVetter profiles a bounded public Node or Go project with an existing local test or benchmark surface +- **THEN** the qualification records the immutable upstream revision, local-only workload, diagnosis, verification outcome, retained changes, and cleanup status + +### Requirement: Self-improvement qualification + +CodeVetter SHALL be able to profile a deterministic workload around one of its +own runtime-evidence operations without replacing runtime evidence with source +review alone. + +#### Scenario: CodeVetter profiles itself +- **WHEN** an agent captures an exact scale workload around a repository-owned product operation +- **THEN** CodeVetter reports observed scale and source attribution before any optimization claim is accepted + +#### Scenario: A self-improvement candidate is applied +- **WHEN** an agent changes the runtime-selected product path +- **THEN** the identical correctness and scale workload is rerun and the verifier distinguishes measured improvement from inference and unverified follow-up diff --git a/openspec/specs/runtime-qualification-planning/spec.md b/openspec/specs/runtime-qualification-planning/spec.md new file mode 100644 index 00000000..25395682 --- /dev/null +++ b/openspec/specs/runtime-qualification-planning/spec.md @@ -0,0 +1,69 @@ +# runtime-qualification-planning Specification + +## Purpose +Define how CodeVetter identifies trustworthy, bounded local performance workloads before an agent spends resources profiling or claims that an application bottleneck was measured. +## Requirements +### Requirement: Qualification is read-only and bounded +CodeVetter SHALL inspect only bounded repository metadata and source text when qualifying a repository. Qualification MUST NOT execute project code, install dependencies, start services, contact a network, modify the target repository, or treat a discovered runtime as proof that a representative workload exists. + +#### Scenario: Repository has a test dependency +- **WHEN** a package manifest declares Vitest but no suitable exact performance workload is found +- **THEN** qualification reports the supported runtime lane separately from workload readiness +- **AND** does not run Vitest or describe the repository as profile-ready + +#### Scenario: Repository contains nested tooling +- **WHEN** the only supported manifest or configuration belongs to a nested documentation, fixture, or tooling package +- **THEN** qualification records that evidence with its package scope +- **AND** does not attribute that runtime to the whole product without a representative candidate + +### Requirement: Candidates have exact identity and explainable rank +Each discovered candidate SHALL identify a supported adapter, repository-relative target, exact workload name when required by the adapter, package scope, bounded evidence, safety flags, and an explainable score. CodeVetter MUST rank explicit benchmarks and performance-named workloads ahead of generic tests and MUST NOT invent exact names that were not found in source evidence. + +#### Scenario: Go benchmark is discovered +- **WHEN** a bounded Go test file declares `BenchmarkParseRows` +- **THEN** the candidate uses the Go benchmark adapter, the containing package target, and the exact benchmark name +- **AND** cites the declaration as direct evidence + +#### Scenario: Generic unit tests are the only candidates +- **WHEN** supported unit tests exist but none carries benchmark, performance, scale, load, or timing evidence +- **THEN** CodeVetter may list a bounded low-ranked candidate set +- **AND** requires agent selection rather than choosing one as representative + +### Requirement: Readiness remains conservative +Qualification SHALL return exactly one repository status from `ready`, `needs_selection`, `no_representative_workload`, `unsupported`, or `inaccessible`. A repository is `ready` only when one supported exact candidate crosses the recorded qualification threshold without unsafe external-operation signals. Ambiguity, absent exact identity, browser-only work, and likely network or production dependencies MUST lower readiness rather than be silently accepted. + +#### Scenario: Explicit local performance test is unambiguous +- **WHEN** one supported exact workload has strong benchmark evidence and no external-operation warning +- **THEN** qualification returns `ready` with that candidate as the recommended next profiling scope + +#### Scenario: Performance test appears network-dependent +- **WHEN** candidate source includes bounded evidence of remote URLs, database clients, browser automation, or environment-gated services +- **THEN** qualification includes safety flags and does not return `ready` + +#### Scenario: No supported runtime exists +- **WHEN** bounded evidence establishes no Node, Vitest, or Go performance adapter +- **THEN** qualification returns `unsupported` with limitations + +### Requirement: Portfolio qualification is explicit and sequential +CodeVetter SHALL accept a versioned manifest containing bounded repository identifiers and paths, qualify entries sequentially, and emit one versioned aggregate report with one result per declared repository. The report MUST preserve input order, bound repository and candidate counts, omit absolute repository paths, and continue after an inaccessible or unsupported entry. + +#### Scenario: Mixed portfolio is qualified +- **WHEN** a manifest contains ready, unsupported, and missing repositories +- **THEN** the aggregate report includes all declared identifiers in order with their individual statuses +- **AND** summarizes status counts without failing the whole portfolio + +#### Scenario: Portfolio exceeds its bound +- **WHEN** a manifest declares more repositories than the contract permits +- **THEN** CodeVetter rejects it before inspecting any repository + +### Requirement: Agents receive the next safe operation +For a ready repository, qualification SHALL return a closed profiling recipe compatible with the existing performance operation. For every other status it SHALL return a deterministic next action explaining what evidence or operator choice is missing. CLI and MCP surfaces MUST preserve the same versioned result and uncertainty. + +#### Scenario: Ready candidate is returned through MCP +- **WHEN** an agent calls the local qualification tool for a repository with an unambiguous exact workload +- **THEN** the result includes the adapter, target, exact name, bounded sample policy, evidence, and limitations needed for a subsequent explicit profile call + +#### Scenario: Workload is not representative +- **WHEN** the planner finds only startup-dominated or generic test scopes +- **THEN** the next action asks for a batched or representative workload +- **AND** no optimization target is inferred