Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion PROJECT_STATUS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Project Status

Last updated: 2026-08-11
Last updated: 2026-08-15

## Why / What

Expand All @@ -19,6 +19,22 @@ behavior. **Core Mode** protects that roadmap. **Side Quest Mode** permits
explicitly requested non-core work, but it must stay bounded and must not
silently redefine or displace the core.

## Current investment decision

As of 2026-08-15, CodeVetter remains a maintained local product and empirical
verification/performance research asset, but broad feature expansion is paused.
The repository has strong deterministic infrastructure, caught-bug evidence,
and several measured optimization case studies; it does **not** have the planned
ten active external users, three team pilots, or complete authenticated
multi-provider trials. Those missing outcomes must not be inferred from owner
dogfood, synthetic corpora, or local qualification.

New work should be limited to core verification reliability, regressions,
security, maintenance, and evidence requested by real users. Agent Island stays
opt-in, authenticated provider comparisons remain claim-closed, and no hosted
profiling or production-load system should be added until external pull and a
separately approved safety design justify renewed investment.

## Dependencies

External:
Expand All @@ -38,6 +54,26 @@ Internal (fleet):

## Timeline

- **2026-08-15 — Local performance execution became zero-egress by default
(release candidate):** Added immutable dry-run plans and cost/egress receipts
for exact performance scopes. Admitted runs bind repository revision, dirty
state, and target identity; execute one process at a time with no retries,
external requests, services, or monetary cost, and fail closed when evidence is remote, paid,
production-like, unknown-cost, or stale. Node-family work gets a runtime
network guard; macOS also applies a child-process network sandbox with
loopback support. Go is admitted only where CodeVetter can establish an OS
zero-egress boundary. CLI/MCP dry-run, durable supervision, direct profiling,
and optimization campaigns share the policy. The focused runtime suite passes
150 tests, including hermetic loopback, blocked-remote, stale-identity, and
zero-process blocked-receipt cases. No hosted endpoint, paid service,
production configuration, or new dependency was used.
- **2026-08-15 — Continuation gate resolved conservatively:** Preserved the
released local product, corpus, deterministic graders, runtime lab, and
documentation while pausing broad surface expansion. Ten-user validation,
team pilots, authenticated provider comparisons, and real Codex/Claude Agent
Island qualification were not completed and remain unsupported product-value
claims rather than silently passing by proxy.

- **2026-08-11 — Code-health coverage became executable:** Added CI-blocking
unused-code, duplication-regression, runtime dependency-cycle, and production
vulnerability gates. Removed six unused public type exports, confirmed zero
Expand Down
49 changes: 49 additions & 0 deletions docs/development/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -1037,6 +1037,51 @@ workloads. Agents can also call the read-only MCP operation
`qualify_runtime_repository` before `capture_local_flow` or an explicit
performance profile.

### Zero-egress execution planning

Qualification ranks a workload; admission decides whether CodeVetter may run
it. Dry-run the exact profile before execution:

```bash
pnpm --silent runtime:plan-performance -- \
--repo /path/to/repository \
--adapter node-script \
--target benchmark/parser.mjs \
--samples 3 \
--warmups 1 \
--timeout-ms 30000 \
--json
```

The equivalent read-only MCP operation is `plan_local_performance`. Its stable
plan identity binds the repository revision and dirty state, target content,
adapter, exact workload, and bounds. An admitted plan permits one concurrent
owned process, zero retries, zero external requests, no external services, zero
monetary cost, and a finite total wall clock. The profiler rechecks that identity immediately
before each process; a changed revision, dirty state, or target fails closed.

Node-family runs preload a deny guard for remote DNS, sockets, HTTP(S), fetch,
and WebSocket access. On macOS, the child process tree also runs under a network
sandbox; loopback remains available for local HTTP flows. Go execution uses the
macOS sandbox and is blocked on hosts where CodeVetter cannot establish a
dependable zero-egress boundary. `GOPROXY=off` is not treated as a sandbox.

Remote endpoints, dynamic unknown destinations, hosted/paid service signals,
and workloads identified as load, soak, stress, or production profiles are
blocked before project code runs. Supplying an approval identity records the
caller input but does not enable hosted execution in this version. Unknown cost
is reported as unknown, never as zero. Admitted, blocked, completed, failed,
and policy-violation outcomes carry a versioned cost-and-egress receipt in the
performance capsule.

This boundary separates three activities:

- local profiling is the only executable autonomous mode;
- a public smoke is a separately initiated, bounded availability check and is
not a performance or load test;
- hosted, production, load, soak, and stress profiling remain unsupported and
require future infrastructure outside this local flywheel.

## 14. Preserving a profiling attempt under failure

Use the outer supervisor when a workload may crash, receive a signal, exceed
Expand Down Expand Up @@ -1077,6 +1122,10 @@ or alter source control. Child output is byte-bounded and redacted before it can
become failure evidence; successful JSON is redacted, schema-validated, and
hashed before preservation.

The supervisor derives the same execution plan before launching its child. A
blocked plan writes a terminal `blocked` receipt with zero child processes,
requests, retries, services, and executed cost; project code is not started.

Recovery is deliberately conservative. A killed profiling child, timeout,
ordinary exit, spawn failure, or malformed result gets a terminal receipt and
authorizes no performance conclusion. If the supervisor or entire machine dies,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-08-15
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
## Context

The runtime profiler launches exact repository-owned Node, Vitest, Playwright,
and Go targets with bounded output and wall time. Qualification detects some
network and database signals, but direct CLI and campaign execution do not
share an immutable admission contract and the child process is not currently a
zero-egress sandbox. See `proposal.md` for the product risk.

## Goals / Non-Goals

**Goals:**

- Make the default autonomous policy explicit, deterministic, and inspectable.
- Block unsafe work before the first child process and enforce the admitted
policy at runtime where CodeVetter has a dependable boundary.
- Reuse one policy across direct profiles, supervision, and campaigns.
- Keep blocked outcomes useful to agents through a compact receipt.

**Non-Goals:**

- Add hosted profiling, production load generation, billing integrations, or
approval management.
- Claim an OS-level sandbox for runtimes that CodeVetter cannot isolate.
- Infer that a workload is safe merely because credentials are absent.

## Decisions

### Use a closed preflight plan as the authority

A new dependency-free governance module derives a plan from exact repository,
adapter, target, name, and duration inputs plus a bounded source scan. The plan
uses stable JSON hashing and closed validation. Every executable performance
entry point calls it; a separate CLI dry-run exposes the same result.

Alternative: keep safety flags only in qualification. Rejected because callers
can bypass qualification and flags do not bind the later execution identity.

### Support only local zero-egress autonomous mode

The first schema has one executable mode: `local_zero_egress`. Remote, paid,
unknown-cost, load, soak, stress, and production evidence is terminally blocked
even if an approval string is supplied. This makes maximum external requests,
retries, and cost exactly zero rather than estimates.

Alternative: implement hosted approval and pricing now. Rejected because it
would expand the product into cloud execution and billing before local safety is
proven.

### Enforce Node-family egress with a preload and block unsupported runtimes

Node, node-script, and Vitest processes receive a repository-owned preload that
rejects non-loopback DNS, sockets, HTTP(S), fetch, and WebSocket calls. Playwright
is admitted only with an explicit loopback URL and uses the same remote guard.
On macOS, every admitted workload also runs under `sandbox-exec`; this permits
Go test and benchmark targets with networking denied. On other platforms, Go
performance execution remains manually callable outside the autonomous
campaign but is not admitted until a portable, testable sandbox exists.
`GOPROXY=off` alone is not considered a network sandbox.

Alternative: static source scanning alone. Rejected because dependencies and
computed endpoints can escape a lexical scan.

### Count enforcement events, not inferred traffic

The preload emits a bounded machine marker when it blocks a request. The parent
captures markers separately from redacted stdout/stderr and records attempted
external requests without retrying. An ordinary completed local run has zero
attempted external requests; a policy violation remains zero successful
external requests but records the blocked attempt and a failed admission
receipt.

## Risks / Trade-offs

- **Node APIs may add new network entry points** → Keep the preload small,
deny-by-default around built-in network modules, and cover supported APIs with
hermetic tests.
- **Playwright often starts a local dev server outside the test process** → The
autonomous path admits only already-declared loopback targets; server startup
orchestration stays outside this change.
- **Go becomes less automatic** → Return an explicit unsupported-enforcement
receipt rather than claiming zero egress. Manual profiling remains available.
- **Source scanning can over-block words used in fixtures** → Runtime
enforcement controls Node admission; lexical signals explain and block only
clearly remote/paid/production targets.

## Migration Plan

1. Add plan/receipt contracts and hermetic validation tests.
2. Add the Node preload and prove loopback/remote behavior without internet.
3. Route dry-run, direct profile, supervision, and campaign paths through the
shared admission check.
4. Document the stricter local boundary and run focused runtime tests.
5. Roll back by removing the new admission calls; no persisted database or
production configuration requires migration.
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
## Why

CodeVetter's autonomous performance loop can execute project-owned test and
benchmark code, so a seemingly local profile can still contact deployed
applications or paid services. The profiler needs a machine-readable,
fail-closed execution policy before it can safely iterate without an operator
watching every run.

## What Changes

- Add an immutable performance-execution plan and receipt with exact duration,
process concurrency, retry, external-request, external-service, and monetary
bounds.
- Default autonomous profiling to a local zero-egress policy: one owned process,
no retries, no remote requests, no paid services, and a bounded wall clock.
- Add a dry-run operation that reports whether the exact workload is admitted
before project code executes.
- Reject browser flows without an explicit loopback-only target, workloads with
remote or unknown-cost signals, and runtimes for which zero-egress enforcement
is unavailable.
- Enforce Node-family zero-egress at runtime and emit a blocked/no-confidence
receipt when the workload attempts remote network access.
- Keep hosted, load, soak, stress, and production profiling unsupported by the
autonomous loop; this change does not modify Cloudflare or other production
configuration.

## Capabilities

### New Capabilities

- `performance-execution-governance`: Defines immutable dry-run admission,
local zero-egress execution, bounded receipts, and fail-closed treatment of
hosted or unknown-cost workloads.

### Modified Capabilities

- `autonomous-optimization-campaigns`: Requires every campaign execution to be
admitted by the local performance-execution policy before correctness or
performance code runs.

## Impact

- Affects the repository-owned runtime profiler, campaign service, CLI/MCP
contracts, focused tests, and local performance documentation.
- Adds no production dependency, hosted service, credential, database change,
or production configuration.
- Existing explicitly invoked local profiles become stricter when their target
has remote-network or unknown-cost evidence.
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
## ADDED Requirements

### Requirement: Campaign execution requires current local admission
The system SHALL derive and validate a current performance-execution plan before
running campaign correctness or performance scopes. A blocked, stale, or
identity-mismatched plan MUST leave the campaign in `no_confidence` and MUST NOT
execute project code or consume an experiment attempt.

#### Scenario: Baseline is admitted locally
- **WHEN** every declared campaign scope has a current admitted zero-egress plan
- **THEN** the campaign may execute the bounded baseline and attach the admission receipts to its evidence

#### Scenario: Candidate contains a remote workload
- **WHEN** any correctness or performance scope has remote, paid, or unknown-cost evidence
- **THEN** the campaign records `no_confidence` with the blocked admission receipt before executing any declared scope
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
## Purpose

Defines the fail-closed execution policy that keeps autonomous performance
profiling local, bounded, zero-egress, and independently auditable before any
project-owned workload is allowed to run.

## ADDED Requirements

### Requirement: Every performance run has an immutable admission plan
The system SHALL derive a versioned plan before project code executes. The plan
MUST bind repository and target identity, adapter, execution mode, maximum wall
time, process concurrency, retries, external requests, monetary cost, declared
external services, and approval identity. Unknown fields or unbounded values
MUST fail closed.

#### Scenario: Local dry-run is admitted
- **WHEN** a caller dry-runs an exact supported local workload with no remote or unknown-cost evidence
- **THEN** the system returns an admitted zero-egress plan with one process, zero retries, zero external requests, zero external services, zero monetary cost, and a finite duration

#### Scenario: Plan identity changes
- **WHEN** repository, target, adapter, budget, or execution-policy input differs from an earlier plan
- **THEN** the system returns a different immutable plan identity and does not reuse the prior admission

### Requirement: Autonomous execution is local and zero-egress
The autonomous profiler SHALL execute only adapters for which it can enforce a
local zero-egress policy. It MUST block remote network access at runtime, allow
browser access only to loopback targets, run at most one owned workload process,
and perform no automatic retry.

#### Scenario: Node workload attempts remote access
- **WHEN** an admitted Node-family workload attempts DNS, socket, HTTP, HTTPS, fetch, or WebSocket access to a non-loopback destination
- **THEN** the system blocks the operation, terminates or fails the workload, and reports a zero-egress policy violation without retrying

#### Scenario: Runtime cannot enforce zero egress
- **WHEN** the selected adapter lacks an enforceable zero-egress boundary
- **THEN** the system blocks before project code executes and names the unsupported enforcement boundary

### Requirement: Hosted and unknown-cost profiling remains closed
The autonomous profiler MUST NOT run hosted, paid, unknown-cost, load, soak,
stress, or production profiling. An approval identity MUST NOT silently widen
this product boundary; the plan SHALL report the unsupported execution mode and
maximum possible spend as unknown rather than zero.

#### Scenario: Hosted target is requested without approval
- **WHEN** a workload includes a remote endpoint, paid service, production marker, or unknown pricing and no exact approval identity
- **THEN** the system blocks before the first request and records every detected service and missing approval input

#### Scenario: Hosted target is requested with approval
- **WHEN** a caller supplies an approval identity for a hosted or paid workload
- **THEN** the autonomous profiler still reports hosted execution as unsupported and does not contact the service

### Requirement: Every admission outcome emits a cost and egress receipt
The system SHALL emit a machine-readable receipt for admitted, blocked, failed,
and completed plans. The receipt MUST preserve planned and observed duration,
process concurrency, retry count, external request count, external services,
cost posture, enforcement method, terminal reason, and limitations without
including credentials or absolute private paths.

#### Scenario: Workload is blocked before execution
- **WHEN** the plan is rejected because cost, service, approval, or enforcement evidence is missing
- **THEN** the receipt records zero executed requests and processes plus the projected bounds and exact blockers

#### Scenario: Local workload completes
- **WHEN** an admitted zero-egress workload exits within its bounds
- **THEN** the receipt records the observed local process and duration totals and confirms zero external requests, retries, services, and monetary cost
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
## 1. Closed governance contracts

- [x] 1.1 Add versioned immutable performance-execution plan and receipt contracts with stable identities and closed validation
- [x] 1.2 Derive bounded local admission from exact repository, adapter, target, workload, and duration inputs without executing project code
- [x] 1.3 Fail closed on hosted, paid, production, load, soak, stress, unknown-cost, and unsupported-enforcement evidence

## 2. Runtime enforcement and product operations

- [x] 2.1 Add a Node-family preload that permits loopback and blocks remote DNS, sockets, HTTP(S), fetch, and WebSocket access with bounded policy markers
- [x] 2.2 Apply the same admission and zero-egress policy to direct performance profiling and durable supervision
- [x] 2.3 Add a machine-readable `plan-performance` dry-run CLI operation and equivalent MCP operation
- [x] 2.4 Gate campaign baseline, screen, and promotion execution on current admitted plans without consuming blocked attempts

## 3. Verification and closeout

- [x] 3.1 Add hermetic boundary tests for allowed local work, remote attempts, unsupported adapters, unknown cost, stale identity, and bounded receipts
- [x] 3.2 Document local profiling, public smoke, hosted/load boundaries, dry-run usage, and receipt interpretation
- [x] 3.3 Run focused runtime tests, lint, docs validation, strict OpenSpec validation, and diff checks
- [x] 3.4 Link the shipped change to GitHub issue #105, sync the specification, archive the completed change, and update current product truth
Loading
Loading