Archive shipped performance specs: batch 1

openspec/specs/autonomous-optimization-campaigns/spec.md

@@ -0,0 +1,114 @@
+# autonomous-optimization-campaigns Specification
+
+## Purpose
+Defines a bounded, resumable optimization campaign in which an agent may
+iterate on source code while CodeVetter remains the deterministic authority for
+correctness, performance evidence, resource limits, and promotion decisions.
+## Requirements
+### Requirement: Campaign scope is immutable and reviewable
+The system SHALL require a versioned campaign manifest that identifies the
+repository revision, allowed mutable files, exact correctness scopes, exact
+performance scope, sample policies, resource limits, experiment budget, and
+stop conditions. The system MUST reject unknown fields, escaping paths,
+unsupported adapters, incomplete scopes, and manifest identity drift after a
+baseline exists.
+
+#### Scenario: Valid campaign is initialized
+- **WHEN** an agent supplies a contained manifest with supported exact scopes and finite budgets
+- **THEN** the system records a stable campaign identity before any candidate is evaluated
+
+#### Scenario: Manifest changes after baseline
+- **WHEN** an agent changes an evaluation target, policy, protected path, or budget after the baseline was recorded
+- **THEN** the system refuses to compare the candidate with the prior baseline
+
+### Requirement: Baseline precedes experimentation
+The system SHALL execute every declared correctness scope and the declared
+performance scope before accepting a candidate. A failed, incomplete, or
+unstable baseline MUST leave the campaign in `no_confidence` and MUST NOT
+authorize experimentation.
+
+#### Scenario: Complete baseline
+- **WHEN** all exact correctness scopes pass and bounded performance evidence completes
+- **THEN** the system records the baseline as the first immutable experiment and exposes one next candidate action
+
+#### Scenario: Baseline correctness failure
+- **WHEN** any declared correctness scope fails or does not execute exactly
+- **THEN** the system records the failure and does not establish an incumbent
+
+### Requirement: Correctness gates every performance decision
+The system SHALL execute all declared correctness scopes against every
+candidate before promotion. A correctness failure MUST produce `discard` or
+`crash` regardless of measured performance, and a performance result MUST NOT
+override an authoritative correctness result.
+
+#### Scenario: Faster incorrect candidate
+- **WHEN** a candidate materially improves the performance metric but fails one correctness scope
+- **THEN** the system records the performance observation but returns `discard` with correctness as the controlling reason
+
+#### Scenario: Correctness-preserving improvement
+- **WHEN** every correctness scope passes and compatible performance evidence materially improves without a protected regression
+- **THEN** the system may classify the candidate as `promising` or `keep` according to evidence strength
+
+### Requirement: Screening and promotion have separate authority
+The system SHALL distinguish bounded screening evidence from promotion-quality
+evidence. Screening MAY return `promising`, `discard`, `crash`, or
+`no_confidence`; only promotion evidence meeting the configured sample floor,
+paired-workload identity, stability, correctness, and secondary-resource policy
+MAY return `keep`.
+
+#### Scenario: Three-sample material improvement
+- **WHEN** a correct candidate materially improves under a three-sample screening policy
+- **THEN** the system returns `promising` and requests promotion-quality verification instead of advancing the incumbent
+
+#### Scenario: Stable paired promotion
+- **WHEN** independently runnable incumbent and candidate checkouts pass exact correctness and ten-sample paired verification without limitations
+- **THEN** the system returns `keep` and records the candidate as the new incumbent
+
+### Requirement: Every experiment is durably attributable
+The system SHALL append an experiment record containing campaign identity,
+sequence, timestamp, repository revision, diff identity, hypothesis, evidence
+identity, measurements, correctness outcomes, decision, reason, and limitations.
+Existing experiment records MUST NOT be rewritten when later candidates run.
+
+#### Scenario: Candidate is discarded
+- **WHEN** a candidate is slower, incorrect, unstable, or immaterial
+- **THEN** the system appends the complete result and preserves the incumbent and all earlier records
+
+#### Scenario: Campaign resumes
+- **WHEN** an agent reopens a valid campaign after the prior process exits
+- **THEN** the system reconstructs the incumbent, remaining budget, experiment count, and next permitted action from validated durable records
+
+### Requirement: Autonomy is bounded by explicit stop conditions
+The system SHALL stop requesting experiments when the manifest's experiment,
+elapsed-time, consecutive-no-improvement, or consecutive-crash budget is
+exhausted. `no_confidence` MUST NOT silently consume or extend an unspecified
+budget, and the system MUST explain the controlling stop condition.
+
+#### Scenario: Plateau reached
+- **WHEN** the configured number of consecutive non-improving candidates is recorded
+- **THEN** the campaign status becomes `stopped` with `plateau` as the reason
+
+#### Scenario: Budget remains
+- **WHEN** the latest experiment is terminal and at least one declared budget remains
+- **THEN** the campaign exposes a bounded next action without generating or applying a source patch
+
+### Requirement: Agent strategy remains outside evidence authority
+The system SHALL NOT generate hypotheses with a model, edit application source,
+install dependencies, invoke arbitrary shell commands, reset Git state, or
+weaken declared checks. It SHALL expose evidence and deterministic decisions so
+an external agent program can choose and apply the next bounded experiment.
+
+#### Scenario: Agent requests a campaign decision
+- **WHEN** an agent submits a candidate hypothesis after editing within the declared boundary
+- **THEN** CodeVetter evaluates the candidate without claiming authorship of the patch or inference beyond captured evidence
+
+### Requirement: Machine operations are closed and repository scoped
+The system SHALL expose start, baseline, evaluate, inspect, and status behavior
+through machine-readable CLI and repository-scoped MCP operations with closed
+schemas. Campaign artifacts MUST remain under an explicit repository-contained
+directory and MUST redact environment values, credentials, query values, and
+machine paths from portable evidence.
+
+#### Scenario: Unknown campaign argument
+- **WHEN** an MCP or CLI caller supplies an unknown field, escaping artifact path, or unsupported operation
+- **THEN** the system fails closed without running a workload or mutating campaign state

openspec/specs/durable-performance-supervision/spec.md

@@ -0,0 +1,59 @@
+# durable-performance-supervision Specification
+
+## Purpose
+Define a durable outer execution boundary that preserves local performance-run identity and terminal evidence when the profiling child crashes, times out, is signaled, or cannot emit a valid capsule.
+## Requirements
+### Requirement: Supervision accepts only a closed profiling request
+CodeVetter SHALL supervise only the existing `diagnose-performance` operation with one supported adapter, repository-contained target, optional workload name, bounded samples, warmups, timeout, and safe caller-supplied run ID. The supervisor MUST construct process arguments directly and MUST NOT accept a shell command, arbitrary arguments, inherited application environment, or an artifact path outside `.codevetter/performance-runs/<run-id>/`.
+
+#### Scenario: Exact workload is supervised
+- **WHEN** an agent supplies a valid run ID and exact supported performance scope
+- **THEN** CodeVetter starts one owned diagnosis child with the same closed scope and records that scope before launch
+
+#### Scenario: Run ID is unsafe or already exists
+- **WHEN** the run ID escapes its artifact root, has an unsupported form, or names an existing run directory
+- **THEN** CodeVetter rejects the request without launching a child or overwriting evidence
+
+### Requirement: Receipt exists before the child can fail
+Before launching the diagnosis child, CodeVetter MUST atomically persist a versioned receipt containing run identity, repository revision, dirty state, exact scope, bounded policy, supervisor identity, `initialized` state, and no inferred performance result. After launch it SHALL atomically refresh a `running` heartbeat without rewriting source files.
+
+#### Scenario: Child is killed immediately after launch
+- **WHEN** the child terminates before producing output
+- **THEN** the pre-existing receipt still identifies the attempted scope and records its terminal signal or failure
+
+#### Scenario: Receipt is inspected while running
+- **WHEN** an agent reads an active run
+- **THEN** it receives the latest atomic heartbeat and last recorded state without observing partial JSON
+
+### Requirement: Every child outcome is classified without fabrication
+The supervisor SHALL finalize exactly one terminal state from `succeeded`, `failed`, `timed_out`, `signaled`, `spawn_failed`, or `invalid_result`. A run is `succeeded` only when the child exits zero and emits exactly one valid performance diagnosis document. Non-zero exits, timeouts, signals, missing output, malformed JSON, or an incompatible schema MUST retain operational evidence and MUST NOT become performance findings.
+
+#### Scenario: Valid diagnosis completes
+- **WHEN** the child exits zero with one valid diagnosis document
+- **THEN** the receipt records `succeeded`, the result digest, and a bounded result reference
+
+#### Scenario: Child exits from a signal
+- **WHEN** the operating system reports a child signal before valid completion
+- **THEN** the receipt records `signaled`, the signal, last heartbeat, and no performance conclusion
+
+#### Scenario: Child prints malformed JSON
+- **WHEN** the child exits but stdout is not exactly one valid diagnosis document
+- **THEN** the receipt records `invalid_result` with bounded redacted output evidence
+
+### Requirement: Artifacts are bounded, redacted, and inspectable
+CodeVetter SHALL retain only the atomic receipt, an optional validated result document, and bounded redacted failure output under the owned run directory. It MUST record byte bounds, truncation, redaction count, and result digest, and MUST expose read-only inspection by run ID through CLI and MCP without accepting a repository path after MCP startup.
+
+#### Scenario: Failure output contains secrets
+- **WHEN** child output includes credential-shaped values or absolute repository paths
+- **THEN** the stored and returned evidence contains redaction markers and bounded text only
+
+#### Scenario: Agent inspects a completed run
+- **WHEN** an agent requests an existing run ID
+- **THEN** CodeVetter returns the validated receipt and result summary or operational failure evidence without rerunning the workload
+
+### Requirement: Supervisor cleanup is owned and bounded
+On timeout, cancellation, or supervisor shutdown, CodeVetter SHALL terminate only the owned child process tree, wait a bounded grace interval, finalize the receipt when the supervisor remains alive, and stop heartbeat activity. It MUST NOT kill unrelated processes, delete target artifacts, or claim cleanup succeeded without observing child termination.
+
+#### Scenario: Workload exceeds its supervisor deadline
+- **WHEN** the child remains alive beyond the derived bounded run deadline
+- **THEN** CodeVetter terminates the owned process tree, records `timed_out`, and returns the operational outcome

openspec/specs/evidence-backed-product-comparisons/spec.md

@@ -0,0 +1,29 @@
+# evidence-backed-product-comparisons Specification
+
+## Purpose
+Define public comparison pages that help people and retrieval systems distinguish
+CodeVetter's execution-backed verification from adjacent code-review products.
+## Requirements
+### Requirement: Direct comparison routes
+The site SHALL publish canonical comparison pages for CodeRabbit and Greptile
+using the existing public editorial surface.
+
+#### Scenario: Visitor opens a comparison
+- **WHEN** a visitor requests either approved comparison route
+- **THEN** the site returns a readable page with a unique title, description, and canonical URL
+
+### Requirement: Evidence-bounded claims
+Each comparison MUST cite dated first-party sources, identify the CodeVetter
+evidence used, and disclose that no common head-to-head benchmark has been run.
+
+#### Scenario: Reader evaluates a performance claim
+- **WHEN** a reader inspects catch-rate, precision, speed, memory, or cost language
+- **THEN** the page makes no superiority claim unsupported by a shared controlled run
+
+### Requirement: Human and agent discovery
+Each comparison SHALL be included in the site's sitemap and agent-readable
+Markdown projection and SHALL link to adjacent verification and benchmark pages.
+
+#### Scenario: A crawler discovers the page
+- **WHEN** a search or AI crawler follows the site's existing discovery surfaces
+- **THEN** it can retrieve the comparison and its supporting internal links