diff --git a/PROJECT_STATUS.md b/PROJECT_STATUS.md index 9ee6547c..bb912206 100644 --- a/PROJECT_STATUS.md +++ b/PROJECT_STATUS.md @@ -1,6 +1,6 @@ # Project Status -Last updated: 2026-08-11 +Last updated: 2026-08-15 ## Why / What @@ -19,6 +19,22 @@ behavior. **Core Mode** protects that roadmap. **Side Quest Mode** permits explicitly requested non-core work, but it must stay bounded and must not silently redefine or displace the core. +## Current investment decision + +As of 2026-08-15, CodeVetter remains a maintained local product and empirical +verification/performance research asset, but broad feature expansion is paused. +The repository has strong deterministic infrastructure, caught-bug evidence, +and several measured optimization case studies; it does **not** have the planned +ten active external users, three team pilots, or complete authenticated +multi-provider trials. Those missing outcomes must not be inferred from owner +dogfood, synthetic corpora, or local qualification. + +New work should be limited to core verification reliability, regressions, +security, maintenance, and evidence requested by real users. Agent Island stays +opt-in, authenticated provider comparisons remain claim-closed, and no hosted +profiling or production-load system should be added until external pull and a +separately approved safety design justify renewed investment. + ## Dependencies External: @@ -38,6 +54,26 @@ Internal (fleet): ## Timeline +- **2026-08-15 — Local performance execution became zero-egress by default + (release candidate):** Added immutable dry-run plans and cost/egress receipts + for exact performance scopes. Admitted runs bind repository revision, dirty + state, and target identity; execute one process at a time with no retries, + external requests, services, or monetary cost, and fail closed when evidence is remote, paid, + production-like, unknown-cost, or stale. Node-family work gets a runtime + network guard; macOS also applies a child-process network sandbox with + loopback support. Go is admitted only where CodeVetter can establish an OS + zero-egress boundary. CLI/MCP dry-run, durable supervision, direct profiling, + and optimization campaigns share the policy. The focused runtime suite passes + 150 tests, including hermetic loopback, blocked-remote, stale-identity, and + zero-process blocked-receipt cases. No hosted endpoint, paid service, + production configuration, or new dependency was used. +- **2026-08-15 — Continuation gate resolved conservatively:** Preserved the + released local product, corpus, deterministic graders, runtime lab, and + documentation while pausing broad surface expansion. Ten-user validation, + team pilots, authenticated provider comparisons, and real Codex/Claude Agent + Island qualification were not completed and remain unsupported product-value + claims rather than silently passing by proxy. + - **2026-08-11 — Code-health coverage became executable:** Added CI-blocking unused-code, duplication-regression, runtime dependency-cycle, and production vulnerability gates. Removed six unused public type exports, confirmed zero diff --git a/docs/development/performance.md b/docs/development/performance.md index 46258e5d..22876f67 100644 --- a/docs/development/performance.md +++ b/docs/development/performance.md @@ -1037,6 +1037,51 @@ workloads. Agents can also call the read-only MCP operation `qualify_runtime_repository` before `capture_local_flow` or an explicit performance profile. +### Zero-egress execution planning + +Qualification ranks a workload; admission decides whether CodeVetter may run +it. Dry-run the exact profile before execution: + +```bash +pnpm --silent runtime:plan-performance -- \ + --repo /path/to/repository \ + --adapter node-script \ + --target benchmark/parser.mjs \ + --samples 3 \ + --warmups 1 \ + --timeout-ms 30000 \ + --json +``` + +The equivalent read-only MCP operation is `plan_local_performance`. Its stable +plan identity binds the repository revision and dirty state, target content, +adapter, exact workload, and bounds. An admitted plan permits one concurrent +owned process, zero retries, zero external requests, no external services, zero +monetary cost, and a finite total wall clock. The profiler rechecks that identity immediately +before each process; a changed revision, dirty state, or target fails closed. + +Node-family runs preload a deny guard for remote DNS, sockets, HTTP(S), fetch, +and WebSocket access. On macOS, the child process tree also runs under a network +sandbox; loopback remains available for local HTTP flows. Go execution uses the +macOS sandbox and is blocked on hosts where CodeVetter cannot establish a +dependable zero-egress boundary. `GOPROXY=off` is not treated as a sandbox. + +Remote endpoints, dynamic unknown destinations, hosted/paid service signals, +and workloads identified as load, soak, stress, or production profiles are +blocked before project code runs. Supplying an approval identity records the +caller input but does not enable hosted execution in this version. Unknown cost +is reported as unknown, never as zero. Admitted, blocked, completed, failed, +and policy-violation outcomes carry a versioned cost-and-egress receipt in the +performance capsule. + +This boundary separates three activities: + +- local profiling is the only executable autonomous mode; +- a public smoke is a separately initiated, bounded availability check and is + not a performance or load test; +- hosted, production, load, soak, and stress profiling remain unsupported and + require future infrastructure outside this local flywheel. + ## 14. Preserving a profiling attempt under failure Use the outer supervisor when a workload may crash, receive a signal, exceed @@ -1077,6 +1122,10 @@ or alter source control. Child output is byte-bounded and redacted before it can become failure evidence; successful JSON is redacted, schema-validated, and hashed before preservation. +The supervisor derives the same execution plan before launching its child. A +blocked plan writes a terminal `blocked` receipt with zero child processes, +requests, retries, services, and executed cost; project code is not started. + Recovery is deliberately conservative. A killed profiling child, timeout, ordinary exit, spawn failure, or malformed result gets a terminal receipt and authorizes no performance conclusion. If the supervisor or entire machine dies, diff --git a/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/.openspec.yaml b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/.openspec.yaml new file mode 100644 index 00000000..0c73c8f5 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-15 diff --git a/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/design.md b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/design.md new file mode 100644 index 00000000..c20c110e --- /dev/null +++ b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/design.md @@ -0,0 +1,94 @@ +## Context + +The runtime profiler launches exact repository-owned Node, Vitest, Playwright, +and Go targets with bounded output and wall time. Qualification detects some +network and database signals, but direct CLI and campaign execution do not +share an immutable admission contract and the child process is not currently a +zero-egress sandbox. See `proposal.md` for the product risk. + +## Goals / Non-Goals + +**Goals:** + +- Make the default autonomous policy explicit, deterministic, and inspectable. +- Block unsafe work before the first child process and enforce the admitted + policy at runtime where CodeVetter has a dependable boundary. +- Reuse one policy across direct profiles, supervision, and campaigns. +- Keep blocked outcomes useful to agents through a compact receipt. + +**Non-Goals:** + +- Add hosted profiling, production load generation, billing integrations, or + approval management. +- Claim an OS-level sandbox for runtimes that CodeVetter cannot isolate. +- Infer that a workload is safe merely because credentials are absent. + +## Decisions + +### Use a closed preflight plan as the authority + +A new dependency-free governance module derives a plan from exact repository, +adapter, target, name, and duration inputs plus a bounded source scan. The plan +uses stable JSON hashing and closed validation. Every executable performance +entry point calls it; a separate CLI dry-run exposes the same result. + +Alternative: keep safety flags only in qualification. Rejected because callers +can bypass qualification and flags do not bind the later execution identity. + +### Support only local zero-egress autonomous mode + +The first schema has one executable mode: `local_zero_egress`. Remote, paid, +unknown-cost, load, soak, stress, and production evidence is terminally blocked +even if an approval string is supplied. This makes maximum external requests, +retries, and cost exactly zero rather than estimates. + +Alternative: implement hosted approval and pricing now. Rejected because it +would expand the product into cloud execution and billing before local safety is +proven. + +### Enforce Node-family egress with a preload and block unsupported runtimes + +Node, node-script, and Vitest processes receive a repository-owned preload that +rejects non-loopback DNS, sockets, HTTP(S), fetch, and WebSocket calls. Playwright +is admitted only with an explicit loopback URL and uses the same remote guard. +On macOS, every admitted workload also runs under `sandbox-exec`; this permits +Go test and benchmark targets with networking denied. On other platforms, Go +performance execution remains manually callable outside the autonomous +campaign but is not admitted until a portable, testable sandbox exists. +`GOPROXY=off` alone is not considered a network sandbox. + +Alternative: static source scanning alone. Rejected because dependencies and +computed endpoints can escape a lexical scan. + +### Count enforcement events, not inferred traffic + +The preload emits a bounded machine marker when it blocks a request. The parent +captures markers separately from redacted stdout/stderr and records attempted +external requests without retrying. An ordinary completed local run has zero +attempted external requests; a policy violation remains zero successful +external requests but records the blocked attempt and a failed admission +receipt. + +## Risks / Trade-offs + +- **Node APIs may add new network entry points** → Keep the preload small, + deny-by-default around built-in network modules, and cover supported APIs with + hermetic tests. +- **Playwright often starts a local dev server outside the test process** → The + autonomous path admits only already-declared loopback targets; server startup + orchestration stays outside this change. +- **Go becomes less automatic** → Return an explicit unsupported-enforcement + receipt rather than claiming zero egress. Manual profiling remains available. +- **Source scanning can over-block words used in fixtures** → Runtime + enforcement controls Node admission; lexical signals explain and block only + clearly remote/paid/production targets. + +## Migration Plan + +1. Add plan/receipt contracts and hermetic validation tests. +2. Add the Node preload and prove loopback/remote behavior without internet. +3. Route dry-run, direct profile, supervision, and campaign paths through the + shared admission check. +4. Document the stricter local boundary and run focused runtime tests. +5. Roll back by removing the new admission calls; no persisted database or + production configuration requires migration. diff --git a/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/proposal.md b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/proposal.md new file mode 100644 index 00000000..3871a0e3 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/proposal.md @@ -0,0 +1,48 @@ +## Why + +CodeVetter's autonomous performance loop can execute project-owned test and +benchmark code, so a seemingly local profile can still contact deployed +applications or paid services. The profiler needs a machine-readable, +fail-closed execution policy before it can safely iterate without an operator +watching every run. + +## What Changes + +- Add an immutable performance-execution plan and receipt with exact duration, + process concurrency, retry, external-request, external-service, and monetary + bounds. +- Default autonomous profiling to a local zero-egress policy: one owned process, + no retries, no remote requests, no paid services, and a bounded wall clock. +- Add a dry-run operation that reports whether the exact workload is admitted + before project code executes. +- Reject browser flows without an explicit loopback-only target, workloads with + remote or unknown-cost signals, and runtimes for which zero-egress enforcement + is unavailable. +- Enforce Node-family zero-egress at runtime and emit a blocked/no-confidence + receipt when the workload attempts remote network access. +- Keep hosted, load, soak, stress, and production profiling unsupported by the + autonomous loop; this change does not modify Cloudflare or other production + configuration. + +## Capabilities + +### New Capabilities + +- `performance-execution-governance`: Defines immutable dry-run admission, + local zero-egress execution, bounded receipts, and fail-closed treatment of + hosted or unknown-cost workloads. + +### Modified Capabilities + +- `autonomous-optimization-campaigns`: Requires every campaign execution to be + admitted by the local performance-execution policy before correctness or + performance code runs. + +## Impact + +- Affects the repository-owned runtime profiler, campaign service, CLI/MCP + contracts, focused tests, and local performance documentation. +- Adds no production dependency, hosted service, credential, database change, + or production configuration. +- Existing explicitly invoked local profiles become stricter when their target + has remote-network or unknown-cost evidence. diff --git a/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/specs/autonomous-optimization-campaigns/spec.md b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/specs/autonomous-optimization-campaigns/spec.md new file mode 100644 index 00000000..e26b92e7 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/specs/autonomous-optimization-campaigns/spec.md @@ -0,0 +1,15 @@ +## ADDED Requirements + +### Requirement: Campaign execution requires current local admission +The system SHALL derive and validate a current performance-execution plan before +running campaign correctness or performance scopes. A blocked, stale, or +identity-mismatched plan MUST leave the campaign in `no_confidence` and MUST NOT +execute project code or consume an experiment attempt. + +#### Scenario: Baseline is admitted locally +- **WHEN** every declared campaign scope has a current admitted zero-egress plan +- **THEN** the campaign may execute the bounded baseline and attach the admission receipts to its evidence + +#### Scenario: Candidate contains a remote workload +- **WHEN** any correctness or performance scope has remote, paid, or unknown-cost evidence +- **THEN** the campaign records `no_confidence` with the blocked admission receipt before executing any declared scope diff --git a/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/specs/performance-execution-governance/spec.md b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/specs/performance-execution-governance/spec.md new file mode 100644 index 00000000..54cc4ddc --- /dev/null +++ b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/specs/performance-execution-governance/spec.md @@ -0,0 +1,65 @@ +## Purpose + +Defines the fail-closed execution policy that keeps autonomous performance +profiling local, bounded, zero-egress, and independently auditable before any +project-owned workload is allowed to run. + +## ADDED Requirements + +### Requirement: Every performance run has an immutable admission plan +The system SHALL derive a versioned plan before project code executes. The plan +MUST bind repository and target identity, adapter, execution mode, maximum wall +time, process concurrency, retries, external requests, monetary cost, declared +external services, and approval identity. Unknown fields or unbounded values +MUST fail closed. + +#### Scenario: Local dry-run is admitted +- **WHEN** a caller dry-runs an exact supported local workload with no remote or unknown-cost evidence +- **THEN** the system returns an admitted zero-egress plan with one process, zero retries, zero external requests, zero external services, zero monetary cost, and a finite duration + +#### Scenario: Plan identity changes +- **WHEN** repository, target, adapter, budget, or execution-policy input differs from an earlier plan +- **THEN** the system returns a different immutable plan identity and does not reuse the prior admission + +### Requirement: Autonomous execution is local and zero-egress +The autonomous profiler SHALL execute only adapters for which it can enforce a +local zero-egress policy. It MUST block remote network access at runtime, allow +browser access only to loopback targets, run at most one owned workload process, +and perform no automatic retry. + +#### Scenario: Node workload attempts remote access +- **WHEN** an admitted Node-family workload attempts DNS, socket, HTTP, HTTPS, fetch, or WebSocket access to a non-loopback destination +- **THEN** the system blocks the operation, terminates or fails the workload, and reports a zero-egress policy violation without retrying + +#### Scenario: Runtime cannot enforce zero egress +- **WHEN** the selected adapter lacks an enforceable zero-egress boundary +- **THEN** the system blocks before project code executes and names the unsupported enforcement boundary + +### Requirement: Hosted and unknown-cost profiling remains closed +The autonomous profiler MUST NOT run hosted, paid, unknown-cost, load, soak, +stress, or production profiling. An approval identity MUST NOT silently widen +this product boundary; the plan SHALL report the unsupported execution mode and +maximum possible spend as unknown rather than zero. + +#### Scenario: Hosted target is requested without approval +- **WHEN** a workload includes a remote endpoint, paid service, production marker, or unknown pricing and no exact approval identity +- **THEN** the system blocks before the first request and records every detected service and missing approval input + +#### Scenario: Hosted target is requested with approval +- **WHEN** a caller supplies an approval identity for a hosted or paid workload +- **THEN** the autonomous profiler still reports hosted execution as unsupported and does not contact the service + +### Requirement: Every admission outcome emits a cost and egress receipt +The system SHALL emit a machine-readable receipt for admitted, blocked, failed, +and completed plans. The receipt MUST preserve planned and observed duration, +process concurrency, retry count, external request count, external services, +cost posture, enforcement method, terminal reason, and limitations without +including credentials or absolute private paths. + +#### Scenario: Workload is blocked before execution +- **WHEN** the plan is rejected because cost, service, approval, or enforcement evidence is missing +- **THEN** the receipt records zero executed requests and processes plus the projected bounds and exact blockers + +#### Scenario: Local workload completes +- **WHEN** an admitted zero-egress workload exits within its bounds +- **THEN** the receipt records the observed local process and duration totals and confirms zero external requests, retries, services, and monetary cost diff --git a/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/tasks.md b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/tasks.md new file mode 100644 index 00000000..a8c605a2 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-guard-performance-cloud-spend/tasks.md @@ -0,0 +1,19 @@ +## 1. Closed governance contracts + +- [x] 1.1 Add versioned immutable performance-execution plan and receipt contracts with stable identities and closed validation +- [x] 1.2 Derive bounded local admission from exact repository, adapter, target, workload, and duration inputs without executing project code +- [x] 1.3 Fail closed on hosted, paid, production, load, soak, stress, unknown-cost, and unsupported-enforcement evidence + +## 2. Runtime enforcement and product operations + +- [x] 2.1 Add a Node-family preload that permits loopback and blocks remote DNS, sockets, HTTP(S), fetch, and WebSocket access with bounded policy markers +- [x] 2.2 Apply the same admission and zero-egress policy to direct performance profiling and durable supervision +- [x] 2.3 Add a machine-readable `plan-performance` dry-run CLI operation and equivalent MCP operation +- [x] 2.4 Gate campaign baseline, screen, and promotion execution on current admitted plans without consuming blocked attempts + +## 3. Verification and closeout + +- [x] 3.1 Add hermetic boundary tests for allowed local work, remote attempts, unsupported adapters, unknown cost, stale identity, and bounded receipts +- [x] 3.2 Document local profiling, public smoke, hosted/load boundaries, dry-run usage, and receipt interpretation +- [x] 3.3 Run focused runtime tests, lint, docs validation, strict OpenSpec validation, and diff checks +- [x] 3.4 Link the shipped change to GitHub issue #105, sync the specification, archive the completed change, and update current product truth diff --git a/openspec/specs/autonomous-optimization-campaigns/spec.md b/openspec/specs/autonomous-optimization-campaigns/spec.md index 77e08506..7f4d8379 100644 --- a/openspec/specs/autonomous-optimization-campaigns/spec.md +++ b/openspec/specs/autonomous-optimization-campaigns/spec.md @@ -112,3 +112,16 @@ machine paths from portable evidence. #### Scenario: Unknown campaign argument - **WHEN** an MCP or CLI caller supplies an unknown field, escaping artifact path, or unsupported operation - **THEN** the system fails closed without running a workload or mutating campaign state +### Requirement: Campaign execution requires current local admission +The system SHALL derive and validate a current performance-execution plan before +running campaign correctness or performance scopes. A blocked, stale, or +identity-mismatched plan MUST leave the campaign in `no_confidence` and MUST NOT +execute project code or consume an experiment attempt. + +#### Scenario: Baseline is admitted locally +- **WHEN** every declared campaign scope has a current admitted zero-egress plan +- **THEN** the campaign may execute the bounded baseline and attach the admission receipts to its evidence + +#### Scenario: Candidate contains a remote workload +- **WHEN** any correctness or performance scope has remote, paid, or unknown-cost evidence +- **THEN** the campaign records `no_confidence` with the blocked admission receipt before executing any declared scope diff --git a/openspec/specs/performance-execution-governance/spec.md b/openspec/specs/performance-execution-governance/spec.md new file mode 100644 index 00000000..66e59058 --- /dev/null +++ b/openspec/specs/performance-execution-governance/spec.md @@ -0,0 +1,64 @@ +# performance-execution-governance Specification + +## Purpose +Defines the fail-closed execution policy that keeps autonomous performance +profiling local, bounded, zero-egress, and independently auditable before any +project-owned workload is allowed to run. +## Requirements +### Requirement: Every performance run has an immutable admission plan +The system SHALL derive a versioned plan before project code executes. The plan +MUST bind repository and target identity, adapter, execution mode, maximum wall +time, process concurrency, retries, external requests, monetary cost, declared +external services, and approval identity. Unknown fields or unbounded values +MUST fail closed. + +#### Scenario: Local dry-run is admitted +- **WHEN** a caller dry-runs an exact supported local workload with no remote or unknown-cost evidence +- **THEN** the system returns an admitted zero-egress plan with one process, zero retries, zero external requests, zero external services, zero monetary cost, and a finite duration + +#### Scenario: Plan identity changes +- **WHEN** repository, target, adapter, budget, or execution-policy input differs from an earlier plan +- **THEN** the system returns a different immutable plan identity and does not reuse the prior admission + +### Requirement: Autonomous execution is local and zero-egress +The autonomous profiler SHALL execute only adapters for which it can enforce a +local zero-egress policy. It MUST block remote network access at runtime, allow +browser access only to loopback targets, run at most one owned workload process, +and perform no automatic retry. + +#### Scenario: Node workload attempts remote access +- **WHEN** an admitted Node-family workload attempts DNS, socket, HTTP, HTTPS, fetch, or WebSocket access to a non-loopback destination +- **THEN** the system blocks the operation, terminates or fails the workload, and reports a zero-egress policy violation without retrying + +#### Scenario: Runtime cannot enforce zero egress +- **WHEN** the selected adapter lacks an enforceable zero-egress boundary +- **THEN** the system blocks before project code executes and names the unsupported enforcement boundary + +### Requirement: Hosted and unknown-cost profiling remains closed +The autonomous profiler MUST NOT run hosted, paid, unknown-cost, load, soak, +stress, or production profiling. An approval identity MUST NOT silently widen +this product boundary; the plan SHALL report the unsupported execution mode and +maximum possible spend as unknown rather than zero. + +#### Scenario: Hosted target is requested without approval +- **WHEN** a workload includes a remote endpoint, paid service, production marker, or unknown pricing and no exact approval identity +- **THEN** the system blocks before the first request and records every detected service and missing approval input + +#### Scenario: Hosted target is requested with approval +- **WHEN** a caller supplies an approval identity for a hosted or paid workload +- **THEN** the autonomous profiler still reports hosted execution as unsupported and does not contact the service + +### Requirement: Every admission outcome emits a cost and egress receipt +The system SHALL emit a machine-readable receipt for admitted, blocked, failed, +and completed plans. The receipt MUST preserve planned and observed duration, +process concurrency, retry count, external request count, external services, +cost posture, enforcement method, terminal reason, and limitations without +including credentials or absolute private paths. + +#### Scenario: Workload is blocked before execution +- **WHEN** the plan is rejected because cost, service, approval, or enforcement evidence is missing +- **THEN** the receipt records zero executed requests and processes plus the projected bounds and exact blockers + +#### Scenario: Local workload completes +- **WHEN** an admitted zero-egress workload exits within its bounds +- **THEN** the receipt records the observed local process and duration totals and confirms zero external requests, retries, services, and monetary cost diff --git a/package.json b/package.json index 053dc345..697ce84f 100644 --- a/package.json +++ b/package.json @@ -25,6 +25,7 @@ "runtime:detect": "node scripts/runtime-failure-capsule/cli.mjs detect", "runtime:qualify": "node scripts/runtime-failure-capsule/cli.mjs qualify", "runtime:qualify-portfolio": "node scripts/runtime-failure-capsule/cli.mjs qualify-portfolio", + "runtime:plan-performance": "node scripts/runtime-failure-capsule/cli.mjs plan-performance", "runtime:plan-flow-campaign": "node scripts/runtime-failure-capsule/cli.mjs plan-flow-campaign", "runtime:supervise-performance": "node scripts/runtime-failure-capsule/cli.mjs supervise-performance", "runtime:inspect-performance-run": "node scripts/runtime-failure-capsule/cli.mjs inspect-performance-run", diff --git a/scripts/runtime-failure-capsule/campaign.mjs b/scripts/runtime-failure-capsule/campaign.mjs index e7fa0324..01f45bc5 100644 --- a/scripts/runtime-failure-capsule/campaign.mjs +++ b/scripts/runtime-failure-capsule/campaign.mjs @@ -15,6 +15,10 @@ import { } from './campaign-contracts.mjs'; import { assessChangeCost, inspectChangeCost } from './change-cost.mjs'; import { parseVitestSelection } from './capsule.mjs'; +import { + createPerformanceExecutionReceipt, + planPerformanceExecution, +} from './execution-governance.mjs'; import { verifyOptimizationCapsules } from './optimization-verification.mjs'; import { verifyPairedRepositories } from './paired-verification.mjs'; import { profileRepository } from './performance.mjs'; @@ -30,6 +34,8 @@ const ENGINE_FILES = [ 'capsule.mjs', 'change-cost.mjs', 'contracts.mjs', + 'execution-governance.mjs', + 'node-egress-preload.mjs', 'optimization-verification.mjs', 'paired-verification.mjs', 'performance-diagnosis.mjs', @@ -109,11 +115,22 @@ async function baselineCampaign(root, input, dependencies) { if (priorStatus.status === 'stopped') throw new Error(`campaign stopped: ${priorStatus.stop_reason}`); + const admission = await admitCampaignScopes(root, campaign.manifest, 'screening'); const repository = await inspectRepositoryState(root, campaign.manifest); assertAllowedChanges(campaign.manifest, repository.changed_files); - const correctness = await runCorrectness(root, campaign.manifest.correctness, dependencies); + const correctness = await runCorrectness( + root, + campaign.manifest.correctness, + dependencies, + 'candidate', + admission.correctness + ); let performance = null; - let evidence = { correctness: correctness.results, performance_capsule: null }; + let evidence = { + correctness: correctness.results, + correctness_execution_governance: correctness.executionGovernance, + performance_capsule: null, + }; let decision; const limitations = [...correctness.limitations]; @@ -129,7 +146,11 @@ async function baselineCampaign(root, input, dependencies) { samples: scope.screening.samples, warmups: scope.screening.warmups, }); - evidence = { correctness: correctness.results, performance_capsule: capsule }; + evidence = { + correctness: correctness.results, + correctness_execution_governance: correctness.executionGovernance, + performance_capsule: capsule, + }; if (capsule.verdict.status === 'no_confidence') { limitations.push(...capsule.limitations); decision = { @@ -191,6 +212,7 @@ async function screenCampaign(root, input, dependencies) { if (status.next_action?.kind === 'promote_candidate') { throw new Error('the latest promising candidate must be promoted or discarded first'); } + const admission = await admitCampaignScopes(root, campaign.manifest, 'screening'); const hypothesis = sanitizeHypothesis(input?.hypothesis, root); const repository = await inspectRepositoryState(root, campaign.manifest); assertAllowedChanges(campaign.manifest, repository.changed_files); @@ -204,11 +226,18 @@ async function screenCampaign(root, input, dependencies) { const attempt = status.experiments + 1; const correctness = changeCost.violations.length === 0 - ? await runCorrectness(root, campaign.manifest.correctness, dependencies) - : { status: 'not_run', results: [], limitations: [] }; + ? await runCorrectness( + root, + campaign.manifest.correctness, + dependencies, + 'candidate', + admission.correctness + ) + : { status: 'not_run', results: [], limitations: [], executionGovernance: [] }; const limitations = [...correctness.limitations]; let evidence = { correctness: correctness.results, + correctness_execution_governance: correctness.executionGovernance, performance_capsule: null, verification: null, change_cost: changeCost, @@ -239,6 +268,7 @@ async function screenCampaign(root, input, dependencies) { const verification = dependencies.verifyOptimizationCapsules(baseline, current); evidence = { correctness: correctness.results, + correctness_execution_governance: correctness.executionGovernance, performance_capsule: current, verification, change_cost: changeCost, @@ -294,6 +324,10 @@ async function promoteCampaign(root, input, dependencies) { } const incumbentRoot = await realpath(resolve(input.incumbent_repository)); if (incumbentRoot === root) throw new Error('incumbent and candidate repositories must differ'); + const [candidateAdmission, incumbentAdmission] = await Promise.all([ + admitCampaignScopes(root, campaign.manifest, 'promotion'), + admitCampaignScopes(incumbentRoot, campaign.manifest, 'promotion'), + ]); const [candidateRepository, incumbentRepository] = await Promise.all([ inspectRepositoryState(root, campaign.manifest), inspectRepositoryState(incumbentRoot, campaign.manifest), @@ -308,12 +342,31 @@ async function promoteCampaign(root, input, dependencies) { } const [incumbentCorrectness, candidateCorrectness] = await Promise.all([ - runCorrectness(incumbentRoot, campaign.manifest.correctness, dependencies, 'incumbent'), - runCorrectness(root, campaign.manifest.correctness, dependencies, 'candidate'), + runCorrectness( + incumbentRoot, + campaign.manifest.correctness, + dependencies, + 'incumbent', + incumbentAdmission.correctness + ), + runCorrectness( + root, + campaign.manifest.correctness, + dependencies, + 'candidate', + candidateAdmission.correctness + ), ]); const correctness = [...incumbentCorrectness.results, ...candidateCorrectness.results]; const limitations = [...incumbentCorrectness.limitations, ...candidateCorrectness.limitations]; - let evidence = { correctness, verification: null }; + let evidence = { + correctness, + correctness_execution_governance: [ + ...incumbentCorrectness.executionGovernance, + ...candidateCorrectness.executionGovernance, + ], + verification: null, + }; let decision; if (incumbentCorrectness.status !== 'passed') { @@ -336,7 +389,14 @@ async function promoteCampaign(root, input, dependencies) { samples: scope.promotion.samples, warmups: scope.promotion.warmups, }); - evidence = { correctness, verification }; + evidence = { + correctness, + correctness_execution_governance: [ + ...incumbentCorrectness.executionGovernance, + ...candidateCorrectness.executionGovernance, + ], + verification, + }; limitations.push(...verification.limitations); decision = promotionDecision(verification); } catch (error) { @@ -477,19 +537,30 @@ export function deriveCampaignStatus(manifest, records, now = new Date()) { }; } -async function runCorrectness(root, scopes, dependencies, role = 'candidate') { +async function runCorrectness(root, scopes, dependencies, role = 'candidate', executionPlans = []) { const results = []; const limitations = []; - for (const scope of scopes) { + const executionGovernance = []; + for (const [index, scope] of scopes.entries()) { const execution = await dependencies.runClosedAdapter({ repositoryRoot: root, adapter: scope.adapter, target: scope.target, name: scope.name, timeoutMs: scope.timeout_ms, + executionPlan: executionPlans[index] ?? null, }); const normalized = normalizeCorrectnessExecution(scope, execution, role); results.push(normalized); + if (executionPlans[index]) { + executionGovernance.push({ + role, + plan: executionPlans[index], + receipt: createPerformanceExecutionReceipt(executionPlans[index], [ + { phase: 'correctness', index, execution }, + ]), + }); + } if (normalized.limitation) limitations.push(normalized.limitation); } const statuses = new Set(results.map((result) => result.status)); @@ -500,7 +571,45 @@ async function runCorrectness(root, scopes, dependencies, role = 'candidate') { : statuses.has('no_confidence') ? 'no_confidence' : 'passed'; - return { status, results, limitations }; + return { status, results, limitations, executionGovernance }; +} + +async function admitCampaignScopes(root, manifest, phase) { + const correctness = await Promise.all( + manifest.correctness.map((scope) => + planPerformanceExecution({ + repositoryRoot: root, + adapter: scope.adapter, + target: scope.target, + name: scope.name, + timeoutMs: scope.timeout_ms, + processCount: 1, + }) + ) + ); + const samplePolicy = manifest.performance[phase]; + const profileRuns = manifest.performance.adapter === 'go-bench' ? 1 : 2; + const metricRuns = ['node-test', 'node-script', 'vitest'].includes(manifest.performance.adapter) + ? samplePolicy.samples + : 0; + const performance = await planPerformanceExecution({ + repositoryRoot: root, + adapter: manifest.performance.adapter, + target: manifest.performance.target, + name: manifest.performance.name, + timeoutMs: manifest.performance.timeout_ms, + processCount: samplePolicy.samples + samplePolicy.warmups + metricRuns + profileRuns, + }); + const blocked = [...correctness, performance].filter( + (plan) => plan.decision.status === 'blocked' + ); + if (blocked.length > 0) { + const reasons = blocked.flatMap((plan) => plan.decision.blockers); + throw new Error( + `campaign execution blocked before project code: ${[...new Set(reasons)].join(' ')}` + ); + } + return { correctness, performance }; } export function normalizeCorrectnessExecution(scope, execution, role = 'candidate') { diff --git a/scripts/runtime-failure-capsule/cli.mjs b/scripts/runtime-failure-capsule/cli.mjs index b0f0dce0..74670e4f 100644 --- a/scripts/runtime-failure-capsule/cli.mjs +++ b/scripts/runtime-failure-capsule/cli.mjs @@ -17,14 +17,108 @@ import { detectRuntimeLanes } from './detect.mjs'; import { captureFlowRepository } from './flow.mjs'; import { planFlowOptimizationCampaign } from './flow-campaign-planner.mjs'; import { diagnosePerformanceRepository } from './performance-diagnosis.mjs'; +import { planPerformanceExecution } from './execution-governance.mjs'; import { verifyOptimizationCapsules } from './optimization-verification.mjs'; import { verifyPairedRepositories } from './paired-verification.mjs'; -import { loadPerformanceCapsule, profileRepository } from './performance.mjs'; +import { + loadPerformanceCapsule, + plannedProfileProcessCount, + profileRepository, +} from './performance.mjs'; import { qualifyPortfolioManifest, qualifyRepository } from './qualification.mjs'; import { redactText } from './redact.mjs'; import { runClosedAdapter } from './runner.mjs'; import { inspectSupervisedRun, supervisePerformanceRun } from './supervision.mjs'; +const OPERATIONS = [ + 'detect', + 'run', + 'import', + 'profile', + 'diagnose-performance', + 'verify-optimization', + 'verify-paired-optimization', + 'capture-flow', + 'qualify', + 'qualify-portfolio', + 'plan-performance', + 'supervise-performance', + 'inspect-performance-run', + 'plan-flow-campaign', +]; +const PLANNING_OPERATIONS = new Set(['plan-performance', 'plan-flow-campaign']); +const PROFILE_OPTIONS = [ + 'repo', + 'adapter', + 'target', + 'name', + 'timeout-ms', + 'samples', + 'warmups', + 'baseline', + 'regression-percent', + 'regression-ms', + 'vite-build-dir', + 'vite-entry', + 'json', +]; +const ALLOWED_OPTIONS = { + detect: ['repo', 'json'], + qualify: ['repo', 'json'], + 'qualify-portfolio': ['manifest', 'json'], + 'plan-performance': [ + 'repo', + 'adapter', + 'target', + 'name', + 'timeout-ms', + 'samples', + 'warmups', + 'approval-id', + 'json', + ], + 'plan-flow-campaign': [ + 'repo', + 'priority-manifest', + 'max-flows', + 'timeout-ms', + 'samples', + 'warmups', + 'json', + ], + 'inspect-performance-run': ['repo', 'run-id', 'json'], + 'supervise-performance': [ + 'repo', + 'run-id', + 'adapter', + 'target', + 'name', + 'timeout-ms', + 'samples', + 'warmups', + 'json', + ], + run: ['repo', 'adapter', 'target', 'name', 'diff', 'timeout-ms', 'json'], + import: ['repo', 'kind', 'receipt', 'diff', 'json'], + 'verify-paired-optimization': [ + 'repo', + 'baseline-repo', + 'adapter', + 'target', + 'name', + 'timeout-ms', + 'samples', + 'warmups', + 'vite-build-dir', + 'vite-entry', + 'json', + ], + profile: PROFILE_OPTIONS, + 'diagnose-performance': PROFILE_OPTIONS, + 'verify-optimization': PROFILE_OPTIONS, + 'capture-flow': PROFILE_OPTIONS, +}; + export async function main(argv = process.argv.slice(2)) { let repositoryRoot = resolve(process.cwd()); try { @@ -43,32 +137,8 @@ export async function main(argv = process.argv.slice(2)) { writeJson(await qualifyPortfolioManifest(required(options, 'manifest'))); return 0; } - if (operation === 'plan-flow-campaign') { - const result = await planFlowOptimizationCampaign({ - repositoryRoot, - priorityManifestPath: options['priority-manifest'], - maxFlows: boundedCount(options['max-flows'], { - name: 'max flows', - defaultValue: 3, - minimum: 1, - maximum: LIMITS.campaignFlows, - }), - samples: boundedCount(options.samples, { - name: 'samples', - defaultValue: LIMITS.defaultSamples, - minimum: LIMITS.minimumSamples, - maximum: LIMITS.maximumSamples, - }), - warmups: boundedCount(options.warmups, { - name: 'warmups', - defaultValue: LIMITS.defaultWarmups, - maximum: LIMITS.maximumWarmups, - }), - timeoutMs: boundedTimeout(options['timeout-ms']), - }); - writeJson(result); - return result.verdict.status === 'no_confidence' ? 2 : 0; - } + if (PLANNING_OPERATIONS.has(operation)) + return runPlanningOperation(operation, repositoryRoot, options); if (operation === 'supervise-performance') { const receipt = await supervisePerformanceRun({ repositoryRoot, @@ -244,25 +314,9 @@ export async function main(argv = process.argv.slice(2)) { function parseArguments(argv) { const [operation, ...rest] = argv; - if ( - ![ - 'detect', - 'run', - 'import', - 'profile', - 'diagnose-performance', - 'verify-optimization', - 'verify-paired-optimization', - 'capture-flow', - 'qualify', - 'qualify-portfolio', - 'supervise-performance', - 'inspect-performance-run', - 'plan-flow-campaign', - ].includes(operation) - ) { + if (!OPERATIONS.includes(operation)) { throw new Error( - 'usage: cli.mjs [--repo PATH] [operation options] [--json]' + 'usage: cli.mjs [--repo PATH] [operation options] [--json]' ); } const normalizedRest = rest[0] === '--' ? rest.slice(1) : rest; @@ -289,66 +343,63 @@ function parseArguments(argv) { options[key] = value; index += 1; } - const allowed = new Set( - operation === 'detect' || operation === 'qualify' - ? ['repo', 'json'] - : operation === 'qualify-portfolio' - ? ['manifest', 'json'] - : operation === 'plan-flow-campaign' - ? ['repo', 'priority-manifest', 'max-flows', 'timeout-ms', 'samples', 'warmups', 'json'] - : operation === 'inspect-performance-run' - ? ['repo', 'run-id', 'json'] - : operation === 'supervise-performance' - ? [ - 'repo', - 'run-id', - 'adapter', - 'target', - 'name', - 'timeout-ms', - 'samples', - 'warmups', - 'json', - ] - : operation === 'run' - ? ['repo', 'adapter', 'target', 'name', 'diff', 'timeout-ms', 'json'] - : operation === 'import' - ? ['repo', 'kind', 'receipt', 'diff', 'json'] - : operation === 'verify-paired-optimization' - ? [ - 'repo', - 'baseline-repo', - 'adapter', - 'target', - 'name', - 'timeout-ms', - 'samples', - 'warmups', - 'vite-build-dir', - 'vite-entry', - 'json', - ] - : [ - 'repo', - 'adapter', - 'target', - 'name', - 'timeout-ms', - 'samples', - 'warmups', - 'baseline', - 'regression-percent', - 'regression-ms', - 'vite-build-dir', - 'vite-entry', - 'json', - ] - ); + const allowed = new Set(ALLOWED_OPTIONS[operation]); const unknown = Object.keys(options).find((key) => !allowed.has(key)); if (unknown) throw new Error(`unknown option for ${operation}: --${unknown}`); return { operation, options }; } +async function runPlanningOperation(operation, repositoryRoot, options) { + if (operation === 'plan-performance') { + const adapter = assertProfileAdapter(required(options, 'adapter')); + const samples = profileSamples(options.samples); + const warmups = profileWarmups(options.warmups); + const plan = await planPerformanceExecution({ + repositoryRoot, + adapter, + target: required(options, 'target'), + name: options.name, + timeoutMs: boundedTimeout(options['timeout-ms']), + processCount: plannedProfileProcessCount({ adapter, samples, warmups }), + approvalIdentity: options['approval-id'], + }); + writeJson(plan); + return plan.decision.status === 'admitted' ? 0 : 2; + } + const result = await planFlowOptimizationCampaign({ + repositoryRoot, + priorityManifestPath: options['priority-manifest'], + maxFlows: boundedCount(options['max-flows'], { + name: 'max flows', + defaultValue: 3, + minimum: 1, + maximum: LIMITS.campaignFlows, + }), + samples: profileSamples(options.samples), + warmups: profileWarmups(options.warmups), + timeoutMs: boundedTimeout(options['timeout-ms']), + }); + writeJson(result); + return result.verdict.status === 'no_confidence' ? 2 : 0; +} + +function profileSamples(value) { + return boundedCount(value, { + name: 'samples', + defaultValue: LIMITS.defaultSamples, + minimum: LIMITS.minimumSamples, + maximum: LIMITS.maximumSamples, + }); +} + +function profileWarmups(value) { + return boundedCount(value, { + name: 'warmups', + defaultValue: LIMITS.defaultWarmups, + maximum: LIMITS.maximumWarmups, + }); +} + function required(options, key) { const value = options[key]; if (typeof value !== 'string' || value.length === 0) throw new Error(`missing --${key}`); diff --git a/scripts/runtime-failure-capsule/contracts.mjs b/scripts/runtime-failure-capsule/contracts.mjs index cc8cb141..ca5c0f5d 100644 --- a/scripts/runtime-failure-capsule/contracts.mjs +++ b/scripts/runtime-failure-capsule/contracts.mjs @@ -177,6 +177,21 @@ export function validatePerformanceCapsule(capsule) { if (!Array.isArray(capsule?.findings)) errors.push('findings must be an array'); if (!Array.isArray(capsule?.unverified)) errors.push('unverified must be an array'); if (!Array.isArray(capsule?.limitations)) errors.push('limitations must be an array'); + if (capsule?.execution_governance !== null && capsule?.execution_governance !== undefined) { + if (capsule.execution_governance?.plan?.schema_version !== 'performance-execution-plan/v1') { + errors.push('invalid execution_governance.plan'); + } + if ( + capsule.execution_governance?.receipt?.schema_version !== 'performance-execution-receipt/v1' + ) { + errors.push('invalid execution_governance.receipt'); + } + if ( + capsule.execution_governance?.plan?.plan_id !== capsule.execution_governance?.receipt?.plan_id + ) { + errors.push('execution_governance identity mismatch'); + } + } return errors; } diff --git a/scripts/runtime-failure-capsule/execution-governance.mjs b/scripts/runtime-failure-capsule/execution-governance.mjs new file mode 100644 index 00000000..479412ff --- /dev/null +++ b/scripts/runtime-failure-capsule/execution-governance.mjs @@ -0,0 +1,524 @@ +import { createHash } from 'node:crypto'; +import { readFile, realpath, stat } from 'node:fs/promises'; +import { isAbsolute, resolve } from 'node:path'; + +import { LIMITS, PROFILE_ADAPTERS, repositoryRelative } from './contracts.mjs'; +import { inspectGitDiff } from './git-diff.mjs'; + +export const PERFORMANCE_EXECUTION_PLAN_SCHEMA_VERSION = 'performance-execution-plan/v1'; +export const PERFORMANCE_EXECUTION_RECEIPT_SCHEMA_VERSION = 'performance-execution-receipt/v1'; + +const MAX_SOURCE_BYTES = 512 * 1024; +const GOVERNED_ADAPTERS = Object.freeze([...PROFILE_ADAPTERS, 'go-test']); +const NODE_ADAPTERS = new Set(['node-test', 'node-script', 'vitest']); +const LOOPBACK_URL = + /^(?:https?|wss?):\/\/(?:localhost|127(?:\.\d{1,3}){3}|\[::1\])(?::(?:\d+|\$\{[^}]+\}))?(?:[/?#]|$)/i; +const URL_LITERAL = /(?:https?|wss?):\/\/[^\s'"`<>)]+/gi; +const DYNAMIC_NETWORK_CALL = + /\b(?:fetch|axios(?:\.[A-Za-z]+)?|got|ky|https?\.request|new\s+WebSocket)\s*\(\s*(?!['"`]https?:\/\/(?:localhost|127\.|\[::1\]))/i; +const REMOTE_SERVICE = + /\b(?:postgres(?:ql)?|mysql|mongodb|redis|supabase|firebase|dynamodb|cloudflare|workers\s*ai|openai|anthropic|openrouter|stripe)\b/i; +const CLOUDFLARE_SERVICE = + /\b(?:DurableObject|browser\s+rendering|Workers\s+AI|env\.(?:D1|R2|KV)|(?:D1|R2|KV)Database)\b/i; +const SUBPROCESS_NETWORK_ESCAPE = + /(?:from\s+['"]node:child_process['"]|require\s*\(\s*['"](?:node:)?child_process['"]\s*\)|\b(?:spawn|exec|execFile)\s*\()/; +const FORBIDDEN_WORKLOAD = + /\b(?:load[-_ ]?test|soak|stress[-_ ]?test|production[-_ ]?(?:profile|load|test))\b/i; + +export async function planPerformanceExecution({ + repositoryRoot, + adapter, + target, + name = null, + timeoutMs, + processCount = 1, + approvalIdentity = null, +}) { + const root = await realpath(resolve(repositoryRoot)); + if (!GOVERNED_ADAPTERS.includes(adapter)) throw new Error(`unsupported adapter: ${adapter}`); + const safeTarget = await inspectTarget(root, target); + const git = await inspectGitDiff(root); + const signals = inspectSafetySignals(safeTarget.source, { adapter, target, name }); + const enforcement = enforcementFor(adapter); + const blockers = [...signals.blockers]; + if (enforcement.kind === 'unavailable') blockers.push(enforcement.reason); + if (adapter === 'playwright' && signals.loopback_urls.length === 0) { + blockers.push('Playwright zero-egress admission requires an explicit loopback URL.'); + } + const maxDurationMs = boundedTotalDuration(timeoutMs, processCount); + const decision = blockers.length === 0 ? 'admitted' : 'blocked'; + const maximumCostMicrousd = decision === 'admitted' ? 0 : signals.unknown_cost ? null : 0; + const payload = { + schema_version: PERFORMANCE_EXECUTION_PLAN_SCHEMA_VERSION, + subject: { + repository_revision: git.repository_revision, + diff_identity: git.diff_identity, + dirty: git.dirty, + target_sha256: sha256(safeTarget.source), + }, + scope: { adapter, target: safeTarget.relative, name }, + mode: 'local_zero_egress', + limits: { + max_wall_time_ms: maxDurationMs, + max_processes: processCount, + max_concurrency: 1, + max_retries: 0, + max_external_requests: 0, + max_cost_microusd: maximumCostMicrousd, + }, + external_services: signals.external_services, + approval_identity: normalizeApprovalIdentity(approvalIdentity), + enforcement, + decision: { + status: decision, + reason: + decision === 'admitted' + ? 'The exact workload is admitted for bounded local zero-egress execution.' + : 'The workload is blocked before project code executes.', + blockers: [...new Set(blockers)].sort(), + }, + limitations: signals.limitations, + }; + const plan = { ...payload, plan_id: sha256(stableStringify(payload)) }; + assertPerformanceExecutionPlan(plan); + return plan; +} + +export function createPerformanceExecutionReceipt(plan, executions = []) { + assertPerformanceExecutionPlan(plan); + const markers = executions.flatMap((entry) => + blockedEgressMarkers(entry.execution?.stderr ?? '') + ); + const outcome = summarizeReceiptOutcome(plan, executions, markers); + const receipt = { + schema_version: PERFORMANCE_EXECUTION_RECEIPT_SCHEMA_VERSION, + plan_id: plan.plan_id, + decision: plan.decision.status, + status: outcome.status, + planned: plan.limits, + observed: summarizeReceiptObservations(plan, executions, markers), + enforcement: plan.enforcement, + terminal_reason: outcome.reason, + limitations: [...plan.limitations], + }; + assertPerformanceExecutionReceipt(receipt); + return receipt; +} + +function summarizeReceiptOutcome(plan, executions, markers) { + if (plan.decision.status === 'blocked') { + return { status: 'blocked', reason: plan.decision.blockers.join(' ') }; + } + if (markers.length > 0) { + return { + status: 'policy_violation', + reason: 'The zero-egress boundary blocked a remote network attempt.', + }; + } + if (executions.length === 0) { + return { status: 'admitted', reason: 'The workload is admitted but has not executed.' }; + } + const completed = executions.every( + (entry) => entry.execution?.status === 'exited' && entry.execution?.exitCode === 0 + ); + return completed + ? { + status: 'completed', + reason: 'The admitted local workload completed within the execution policy.', + } + : { + status: 'failed', + reason: 'The admitted local workload did not complete successfully.', + }; +} + +function summarizeReceiptObservations(plan, executions, markers) { + return { + wall_time_ms: executions.reduce( + (total, entry) => total + Math.max(0, entry.execution?.durationMs ?? 0), + 0 + ), + processes: executions.length, + max_concurrency: executions.length > 0 ? 1 : 0, + retries: 0, + successful_external_requests: 0, + blocked_external_attempts: markers.length, + external_services: [], + cost_microusd: plan.decision.status === 'admitted' ? 0 : null, + }; +} + +export async function assertPerformanceExecutionPlanCurrent({ + plan, + repositoryRoot, + adapter, + target, + name = null, +}) { + assertPerformanceExecutionPlan(plan); + const root = await realpath(resolve(repositoryRoot)); + const safeTarget = await inspectTarget(root, target); + const git = await inspectGitDiff(root); + if ( + plan.scope.adapter !== adapter || + plan.scope.target !== safeTarget.relative || + plan.scope.name !== name || + plan.subject.repository_revision !== git.repository_revision || + plan.subject.diff_identity !== git.diff_identity || + plan.subject.dirty !== git.dirty || + plan.subject.target_sha256 !== sha256(safeTarget.source) + ) { + throw new Error('performance execution plan identity is stale'); + } + return plan; +} + +export function assertPerformanceExecutionPlan(value) { + const errors = validatePerformanceExecutionPlan(value); + if (errors.length > 0) + throw new Error(`invalid performance execution plan: ${errors.join('; ')}`); + return value; +} + +export function validatePerformanceExecutionPlan(value) { + const errors = []; + if (!plainObject(value)) return ['plan must be an object']; + closed( + value, + [ + 'schema_version', + 'plan_id', + 'subject', + 'scope', + 'mode', + 'limits', + 'external_services', + 'approval_identity', + 'enforcement', + 'decision', + 'limitations', + ], + 'plan', + errors + ); + if (value.schema_version !== PERFORMANCE_EXECUTION_PLAN_SCHEMA_VERSION) + errors.push('invalid schema_version'); + if (!/^[0-9a-f]{64}$/.test(value.plan_id ?? '')) errors.push('plan_id is invalid'); + validatePlanSubject(value.subject, errors); + validatePlanScope(value.scope, errors); + if (value.mode !== 'local_zero_egress') errors.push('mode is invalid'); + validateLimits(value.limits, errors); + if (!Array.isArray(value.external_services)) errors.push('external_services must be an array'); + if ( + value.approval_identity !== null && + !/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(value.approval_identity) + ) + errors.push('approval_identity is invalid'); + validateEnforcement(value.enforcement, errors); + validatePlanDecision(value.decision, errors); + if (!stringArray(value.external_services)) errors.push('external_services must be an array'); + if (!stringArray(value.limitations)) errors.push('limitations must be an array'); + const { plan_id: _planId, ...payload } = value; + if (value.plan_id && value.plan_id !== sha256(stableStringify(payload))) + errors.push('plan_id does not match content'); + return errors; +} + +function validatePlanSubject(value, errors) { + closed( + value, + ['repository_revision', 'diff_identity', 'dirty', 'target_sha256'], + 'subject', + errors + ); + if (!plainObject(value)) return errors.push('subject is invalid'); + const valid = + /^[0-9a-f]{40,64}$/.test(value.repository_revision ?? '') && + typeof value.diff_identity === 'string' && + value.diff_identity.length > 0 && + value.diff_identity.length <= 200 && + typeof value.dirty === 'boolean' && + /^[0-9a-f]{64}$/.test(value.target_sha256 ?? ''); + if (!valid) errors.push('subject is invalid'); +} + +function validatePlanScope(value, errors) { + closed(value, ['adapter', 'target', 'name'], 'scope', errors); + if (!plainObject(value)) return errors.push('scope is invalid'); + const valid = + GOVERNED_ADAPTERS.includes(value.adapter) && + typeof value.target === 'string' && + value.target.length > 0 && + (value.name === null || typeof value.name === 'string'); + if (!valid) errors.push('scope is invalid'); +} + +function validateEnforcement(value, errors) { + closed(value, ['kind', 'network_scope', 'reason'], 'enforcement', errors); + if (!plainObject(value)) return errors.push('enforcement is invalid'); + const validKinds = ['node_preload', 'macos_sandbox_node_preload', 'macos_sandbox', 'unavailable']; + if (!validKinds.includes(value.kind)) errors.push('enforcement is invalid'); + if (!['none', 'loopback_only', 'unknown'].includes(value.network_scope)) + errors.push('enforcement.network_scope is invalid'); +} + +function validatePlanDecision(value, errors) { + closed(value, ['status', 'reason', 'blockers'], 'decision', errors); + if (!plainObject(value)) return errors.push('decision is invalid'); + const valid = + ['admitted', 'blocked'].includes(value.status) && + typeof value.reason === 'string' && + value.reason.length > 0 && + stringArray(value.blockers); + if (!valid) errors.push('decision is invalid'); +} + +export function assertPerformanceExecutionReceipt(value) { + const errors = validatePerformanceExecutionReceipt(value); + if (errors.length > 0) + throw new Error(`invalid performance execution receipt: ${errors.join('; ')}`); + return value; +} + +export function validatePerformanceExecutionReceipt(value) { + const errors = []; + if (!plainObject(value)) return ['receipt must be an object']; + closed( + value, + [ + 'schema_version', + 'plan_id', + 'decision', + 'status', + 'planned', + 'observed', + 'enforcement', + 'terminal_reason', + 'limitations', + ], + 'receipt', + errors + ); + if (value.schema_version !== PERFORMANCE_EXECUTION_RECEIPT_SCHEMA_VERSION) + errors.push('invalid schema_version'); + if (!/^[0-9a-f]{64}$/.test(value.plan_id ?? '')) errors.push('plan_id is invalid'); + if (!['admitted', 'blocked'].includes(value.decision)) errors.push('decision is invalid'); + if (!['admitted', 'blocked', 'completed', 'failed', 'policy_violation'].includes(value.status)) + errors.push('status is invalid'); + validateLimits(value.planned, errors); + validateReceiptObservations(value.observed, errors); + validateEnforcement(value.enforcement, errors); + if (typeof value.terminal_reason !== 'string' || value.terminal_reason.length === 0) + errors.push('terminal_reason is invalid'); + if (!stringArray(value.limitations)) errors.push('limitations must be an array'); + return errors; +} + +function validateReceiptObservations(value, errors) { + if (!plainObject(value)) return errors.push('observed is invalid'); + closed( + value, + [ + 'wall_time_ms', + 'processes', + 'max_concurrency', + 'retries', + 'successful_external_requests', + 'blocked_external_attempts', + 'external_services', + 'cost_microusd', + ], + 'observed', + errors + ); + for (const field of [ + 'wall_time_ms', + 'processes', + 'max_concurrency', + 'retries', + 'successful_external_requests', + 'blocked_external_attempts', + ]) { + if (!Number.isInteger(value[field]) || value[field] < 0) + errors.push(`observed.${field} is invalid`); + } + if (!stringArray(value.external_services)) errors.push('observed.external_services is invalid'); + if (value.cost_microusd === null) return; + if (!Number.isInteger(value.cost_microusd) || value.cost_microusd < 0) + errors.push('observed.cost_microusd is invalid'); +} + +export function blockedEgressMarkers(stderr) { + return String(stderr) + .split(/\r?\n/) + .filter((line) => line.startsWith('CODEVETTER_EGRESS_BLOCKED ')) + .slice(0, 32) + .map((line) => { + try { + return JSON.parse(line.slice('CODEVETTER_EGRESS_BLOCKED '.length)); + } catch { + return { kind: 'unknown', destination: '' }; + } + }); +} + +function inspectSafetySignals(source, { adapter, target, name }) { + const blockers = []; + const externalServices = new Set(); + const urls = [...source.matchAll(URL_LITERAL)].map((match) => match[0]); + const remoteUrls = urls.filter((url) => !LOOPBACK_URL.test(url)); + const loopbackUrls = urls.filter((url) => LOOPBACK_URL.test(url)); + for (const url of remoteUrls) { + try { + externalServices.add(new URL(url).hostname.toLowerCase()); + } catch { + externalServices.add('unknown-remote-endpoint'); + } + } + if (remoteUrls.length > 0) blockers.push('The workload contains a non-loopback endpoint.'); + if (DYNAMIC_NETWORK_CALL.test(source)) { + blockers.push( + 'The workload contains a network call whose destination is not a literal loopback URL.' + ); + externalServices.add('unknown-dynamic-endpoint'); + } + if (REMOTE_SERVICE.test(source) || CLOUDFLARE_SERVICE.test(source)) { + blockers.push( + 'The workload contains a hosted or paid service signal with unknown execution cost.' + ); + externalServices.add('unknown-hosted-service'); + } + if (SUBPROCESS_NETWORK_ESCAPE.test(source)) { + blockers.push( + 'The workload can launch a subprocess outside the portable Node zero-egress guard.' + ); + } + if (FORBIDDEN_WORKLOAD.test(`${target} ${name ?? ''} ${source}`)) { + blockers.push('Autonomous load, soak, stress, and production profiling is unsupported.'); + } + return { + blockers, + external_services: [...externalServices].sort(), + loopback_urls: loopbackUrls, + unknown_cost: + remoteUrls.length > 0 || + DYNAMIC_NETWORK_CALL.test(source) || + REMOTE_SERVICE.test(source) || + CLOUDFLARE_SERVICE.test(source), + limitations: [ + adapter === 'playwright' + ? 'Browser execution is admitted only for explicit loopback targets.' + : 'Admission applies only to the exact repository-owned workload identity.', + ], + }; +} + +function enforcementFor(adapter) { + if (process.platform === 'darwin') { + return NODE_ADAPTERS.has(adapter) || adapter === 'playwright' + ? { kind: 'macos_sandbox_node_preload', network_scope: 'loopback_only' } + : { kind: 'macos_sandbox', network_scope: 'none' }; + } + if (NODE_ADAPTERS.has(adapter)) return { kind: 'node_preload', network_scope: 'loopback_only' }; + return { + kind: 'unavailable', + network_scope: 'unknown', + reason: `Portable zero-egress enforcement is unavailable for ${adapter} on ${process.platform}.`, + }; +} + +async function inspectTarget(root, target) { + if (typeof target !== 'string' || target.length === 0 || isAbsolute(target)) + throw new Error('target must be a repository-relative file'); + const absolute = resolve(root, target); + const relative = repositoryRelative(root, absolute); + if (relative === null) throw new Error('target escapes repository'); + const resolved = await realpath(absolute); + if (repositoryRelative(root, resolved) === null) + throw new Error('target symlink escapes repository'); + const metadata = await stat(resolved); + if (!metadata.isFile() || metadata.size > MAX_SOURCE_BYTES) + throw new Error('target must be a bounded regular file'); + return { relative, source: await readFile(resolved, 'utf8') }; +} + +function validateLimits(value, errors) { + if (!plainObject(value)) { + errors.push('limits are invalid'); + return; + } + closed( + value, + [ + 'max_wall_time_ms', + 'max_processes', + 'max_concurrency', + 'max_retries', + 'max_external_requests', + 'max_cost_microusd', + ], + 'limits', + errors + ); + const exact = { max_concurrency: 1, max_retries: 0, max_external_requests: 0 }; + for (const [field, expected] of Object.entries(exact)) + if (value[field] !== expected) errors.push(`limits.${field} must equal ${expected}`); + if ( + !Number.isInteger(value.max_wall_time_ms) || + value.max_wall_time_ms < 100 || + value.max_wall_time_ms > LIMITS.maximumTimeoutMs * 64 + ) + errors.push('limits.max_wall_time_ms is invalid'); + if (!Number.isInteger(value.max_processes) || value.max_processes < 1 || value.max_processes > 64) + errors.push('limits.max_processes is invalid'); + if (value.max_cost_microusd !== null && value.max_cost_microusd !== 0) + errors.push('limits.max_cost_microusd is invalid'); +} + +function boundedTotalDuration(timeoutMs, processCount) { + if (!Number.isInteger(timeoutMs) || timeoutMs < 100 || timeoutMs > LIMITS.maximumTimeoutMs) + throw new Error('timeout is outside the performance execution bound'); + if (!Number.isInteger(processCount) || processCount < 1 || processCount > 64) + throw new Error('process count is outside the performance execution bound'); + return timeoutMs * processCount; +} + +function normalizeApprovalIdentity(value) { + if (value === null || value === undefined || value === '') return null; + if (typeof value !== 'string' || !/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(value)) + throw new Error('approval identity is invalid'); + return value; +} + +function closed(value, allowed, label, errors) { + if (!plainObject(value)) return; + const keys = new Set(allowed); + for (const key of Object.keys(value)) + if (!keys.has(key)) errors.push(`${label} contains unknown field: ${key}`); +} + +function plainObject(value) { + return Boolean(value && typeof value === 'object' && !Array.isArray(value)); +} + +function stringArray(value) { + return Array.isArray(value) && value.every((entry) => typeof entry === 'string'); +} + +function stableStringify(value) { + return JSON.stringify(sortValue(value)); +} + +function sortValue(value) { + if (Array.isArray(value)) return value.map(sortValue); + if (!plainObject(value)) return value; + return Object.fromEntries( + Object.keys(value) + .sort() + .map((key) => [key, sortValue(value[key])]) + ); +} + +function sha256(value) { + return createHash('sha256').update(value).digest('hex'); +} diff --git a/scripts/runtime-failure-capsule/execution-governance.test.mjs b/scripts/runtime-failure-capsule/execution-governance.test.mjs new file mode 100644 index 00000000..7e0ef16b --- /dev/null +++ b/scripts/runtime-failure-capsule/execution-governance.test.mjs @@ -0,0 +1,251 @@ +import assert from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import { mkdtemp, mkdir, rm, writeFile } from 'node:fs/promises'; +import http from 'node:http'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { promisify } from 'node:util'; +import test from 'node:test'; + +import { + createPerformanceExecutionReceipt, + planPerformanceExecution, + validatePerformanceExecutionPlan, +} from './execution-governance.mjs'; +import { runClosedAdapter } from './runner.mjs'; +import { profileRepository } from './performance.mjs'; +import { supervisePerformanceRun } from './supervision.mjs'; + +const run = promisify(execFile); + +test('local plan is immutable, bounded, and stable without running project code', async (context) => { + const root = await fixture(context, { + 'bench.mjs': "process.stdout.write('not executed');\n", + }); + const input = { + repositoryRoot: root, + adapter: 'node-script', + target: 'bench.mjs', + timeoutMs: 1_000, + processCount: 4, + }; + const first = await planPerformanceExecution(input); + const second = await planPerformanceExecution(input); + + assert.equal(first.decision.status, 'admitted'); + assert.equal(first.plan_id, second.plan_id); + assert.deepEqual(first.limits, { + max_wall_time_ms: 4_000, + max_processes: 4, + max_concurrency: 1, + max_retries: 0, + max_external_requests: 0, + max_cost_microusd: 0, + }); + assert.deepEqual(first.external_services, []); + assert.ok( + validatePerformanceExecutionPlan({ ...first, unexpected: true }).includes( + 'plan contains unknown field: unexpected' + ) + ); + assert.ok( + validatePerformanceExecutionPlan({ + ...first, + limits: { ...first.limits, unexpected: true }, + }).includes('limits contains unknown field: unexpected') + ); + assert.ok( + validatePerformanceExecutionPlan({ + ...first, + decision: { ...first.decision, blockers: [42] }, + }).includes('decision is invalid') + ); +}); + +test('remote and production-like plans fail closed even with an approval identity', async (context) => { + const root = await fixture(context, { + 'stress-production.mjs': "await fetch('https://api.openai.com/v1/models');\n", + }); + const plan = await planPerformanceExecution({ + repositoryRoot: root, + adapter: 'node-script', + target: 'stress-production.mjs', + name: 'production stress test', + timeoutMs: 1_000, + processCount: 1, + approvalIdentity: 'owner-approved-123', + }); + const receipt = createPerformanceExecutionReceipt(plan); + + assert.equal(plan.decision.status, 'blocked'); + assert.equal(plan.approval_identity, 'owner-approved-123'); + assert.ok(plan.external_services.includes('api.openai.com')); + assert.equal(plan.limits.max_cost_microusd, null); + assert.equal(receipt.status, 'blocked'); + assert.equal(receipt.observed.processes, 0); + assert.equal(receipt.observed.successful_external_requests, 0); + assert.equal(receipt.observed.cost_microusd, null); +}); + +test('runtime guard permits loopback and blocks imported remote access without internet', async (context) => { + const server = http.createServer((_request, response) => response.end('ok')); + await new Promise((resolvePromise) => server.listen(0, '127.0.0.1', resolvePromise)); + context.after(() => new Promise((resolvePromise) => server.close(resolvePromise))); + const address = server.address(); + const root = await fixture(context, { + 'local.mjs': "import './local-dependency.mjs';\n", + 'local-dependency.mjs': `const response = await fetch('http://127.0.0.1:${address.port}/'); if ((await response.text()) !== 'ok') process.exitCode = 1;\n`, + 'remote.mjs': "import './remote-dependency.mjs';\n", + 'remote-dependency.mjs': "await fetch('http://192.0.2.1/');\n", + }); + const localPlan = await planPerformanceExecution({ + repositoryRoot: root, + adapter: 'node-script', + target: 'local.mjs', + timeoutMs: 5_000, + processCount: 1, + }); + const local = await runClosedAdapter({ + repositoryRoot: root, + adapter: 'node-script', + target: 'local.mjs', + timeoutMs: 5_000, + executionPlan: localPlan, + }); + assert.equal(local.status, 'exited'); + assert.equal(local.exitCode, 0, local.stderr); + + const remotePlan = await planPerformanceExecution({ + repositoryRoot: root, + adapter: 'node-script', + target: 'remote.mjs', + timeoutMs: 5_000, + processCount: 1, + }); + const remote = await runClosedAdapter({ + repositoryRoot: root, + adapter: 'node-script', + target: 'remote.mjs', + timeoutMs: 5_000, + executionPlan: remotePlan, + }); + const receipt = createPerformanceExecutionReceipt(remotePlan, [ + { phase: 'measurement', index: 0, execution: remote }, + ]); + assert.notEqual(remote.exitCode, 0); + assert.match(remote.stderr, /CODEVETTER_EGRESS_BLOCKED/); + assert.equal(receipt.status, 'policy_violation'); + assert.equal(receipt.observed.blocked_external_attempts, 1); + assert.equal(receipt.observed.successful_external_requests, 0); + assert.equal(receipt.observed.retries, 0); +}); + +test('runner rejects a stale target identity before execution', async (context) => { + const root = await fixture(context, { 'bench.mjs': "process.stdout.write('old');\n" }); + const plan = await planPerformanceExecution({ + repositoryRoot: root, + adapter: 'node-script', + target: 'bench.mjs', + timeoutMs: 1_000, + processCount: 1, + }); + await writeFile(join(root, 'bench.mjs'), "process.stdout.write('new');\n"); + await assert.rejects( + runClosedAdapter({ + repositoryRoot: root, + adapter: 'node-script', + target: 'bench.mjs', + timeoutMs: 1_000, + executionPlan: plan, + }), + /plan identity is stale/ + ); +}); + +test('profile and dry-run CLI emit machine-readable blocked evidence before project code', async (context) => { + const root = await fixture(context, { + 'remote.mjs': + "await fetch('https://paid.example.test/work');\nprocess.stdout.write('must-not-run');\n", + }); + const capsule = await profileRepository({ + repositoryRoot: root, + adapter: 'node-script', + target: 'remote.mjs', + timeoutMs: 1_000, + samples: 2, + warmups: 0, + }); + assert.equal(capsule.verdict.status, 'no_confidence'); + assert.equal(capsule.observed.executions.length, 0); + assert.equal(capsule.execution_governance.receipt.status, 'blocked'); + assert.equal(capsule.execution_governance.receipt.observed.processes, 0); + + const supervised = await supervisePerformanceRun({ + repositoryRoot: root, + runId: 'blocked-remote-run', + adapter: 'node-script', + target: 'remote.mjs', + timeoutMs: 1_000, + samples: 2, + warmups: 0, + }); + assert.equal(supervised.state, 'blocked'); + assert.equal(supervised.child.pid, null); + assert.equal(supervised.execution_governance.receipt.observed.processes, 0); + + await assert.rejects( + run( + process.execPath, + [ + join(import.meta.dirname, 'cli.mjs'), + 'plan-performance', + '--repo', + root, + '--adapter', + 'node-script', + '--target', + 'remote.mjs', + '--samples', + '2', + '--warmups', + '0', + '--timeout-ms', + '1000', + '--json', + ], + { cwd: root } + ), + (error) => { + const plan = JSON.parse(error.stdout); + assert.equal(error.code, 2); + assert.equal(plan.decision.status, 'blocked'); + assert.equal(plan.limits.max_cost_microusd, null); + return true; + } + ); +}); + +async function fixture(context, files) { + const root = await mkdtemp(join(tmpdir(), 'codevetter-execution-governance-')); + context.after(() => rm(root, { recursive: true, force: true })); + for (const [path, contents] of Object.entries(files)) { + await mkdir(join(root, path, '..'), { recursive: true }); + await writeFile(join(root, path), contents); + } + await run('git', ['init', '-q'], { cwd: root }); + await run('git', ['add', '.'], { cwd: root }); + await run( + 'git', + [ + '-c', + 'user.name=CodeVetter', + '-c', + 'user.email=codevetter@example.invalid', + 'commit', + '-qm', + 'fixture', + ], + { cwd: root } + ); + return root; +} diff --git a/scripts/runtime-failure-capsule/local-flow-runtime.test.mjs b/scripts/runtime-failure-capsule/local-flow-runtime.test.mjs index 391304db..ae647473 100644 --- a/scripts/runtime-failure-capsule/local-flow-runtime.test.mjs +++ b/scripts/runtime-failure-capsule/local-flow-runtime.test.mjs @@ -179,6 +179,7 @@ test('runtime MCP exposes product capabilities and fails closed on unknown captu tools.map((tool) => tool.name), [ 'qualify_runtime_repository', + 'plan_local_performance', 'profile_local_performance', 'verify_paired_performance', 'inspect_performance_run', @@ -199,14 +200,15 @@ test('runtime MCP exposes product capabilities and fails closed on unknown captu ] ); assert.equal(tools[0].annotations.readOnlyHint, true); - assert.equal(tools[1].annotations.readOnlyHint, false); + assert.equal(tools[1].annotations.readOnlyHint, true); assert.equal(tools[2].annotations.readOnlyHint, false); - assert.equal(tools[3].annotations.readOnlyHint, true); - assert.equal(tools[4].annotations.readOnlyHint, false); + assert.equal(tools[3].annotations.readOnlyHint, false); + assert.equal(tools[4].annotations.readOnlyHint, true); + assert.equal(tools[5].annotations.readOnlyHint, false); const handle = await createRuntimeMcpHandler(root); const listed = await handle({ jsonrpc: '2.0', id: 1, method: 'tools/list' }); - assert.equal(listed.result.tools.length, 18); + assert.equal(listed.result.tools.length, 19); const qualification = await handle({ jsonrpc: '2.0', id: 2, @@ -333,7 +335,7 @@ test('runtime MCP process speaks line-delimited JSON-RPC without network setup', { jsonrpc: '2.0', id: 2, method: 'tools/list', params: {} }, ]); assert.equal(responses[0].result.serverInfo.name, 'codevetter-local-runtime'); - assert.equal(responses[1].result.tools.length, 18); + assert.equal(responses[1].result.tools.length, 19); }); test('validates the required recursive flow contract', () => { diff --git a/scripts/runtime-failure-capsule/mcp.mjs b/scripts/runtime-failure-capsule/mcp.mjs index 4313394e..1f4ae0e8 100644 --- a/scripts/runtime-failure-capsule/mcp.mjs +++ b/scripts/runtime-failure-capsule/mcp.mjs @@ -4,6 +4,7 @@ import { fileURLToPath } from 'node:url'; import { createOptimizationCampaignService } from './campaign.mjs'; import { createOptimizationContributionService } from './contribution.mjs'; +import { planPerformanceExecution } from './execution-governance.mjs'; import { LIMITS, PROFILE_ADAPTERS, @@ -16,12 +17,16 @@ import { planFlowOptimizationCampaign } from './flow-campaign-planner.mjs'; import { qualifyRepository } from './qualification.mjs'; import { redactText } from './redact.mjs'; import { inspectSupervisedRun } from './supervision.mjs'; -import { profileRepository } from './performance.mjs'; +import { plannedProfileProcessCount, profileRepository } from './performance.mjs'; import { verifyPairedRepositories } from './paired-verification.mjs'; const PROTOCOL_VERSION = '2025-03-26'; const SERVER_INFO = { name: 'codevetter-local-runtime', version: '0.1.0' }; -const PERFORMANCE_TOOLS = new Set(['profile_local_performance', 'verify_paired_performance']); +const PERFORMANCE_TOOLS = new Set([ + 'plan_local_performance', + 'profile_local_performance', + 'verify_paired_performance', +]); export async function createRuntimeMcpHandler(repositoryRoot, options = {}) { const flowService = options.flowService ?? (await createLocalFlowService(repositoryRoot)); @@ -188,6 +193,9 @@ async function callTool( } async function callPerformanceTool({ name, args, repositoryRoot, incumbentRepositoryRoot }) { + if (name === 'plan_local_performance') { + return callPerformancePlan(args, repositoryRoot); + } closedArguments( args, ['adapter', 'target'], @@ -216,6 +224,28 @@ async function callPerformanceTool({ name, args, repositoryRoot, incumbentReposi return profileRepository(input); } +function callPerformancePlan(args, repositoryRoot) { + closedArguments( + args, + ['adapter', 'target'], + ['name', 'approval_identity'], + [], + ['samples', 'warmups', 'timeout_ms'] + ); + const adapter = assertProfileAdapter(args.adapter); + const samples = boundedSamples(args.samples); + const warmups = boundedWarmups(args.warmups); + return planPerformanceExecution({ + repositoryRoot, + adapter, + target: args.target, + name: args.name, + timeoutMs: boundedTimeout(args.timeout_ms), + processCount: plannedProfileProcessCount({ adapter, samples, warmups }), + approvalIdentity: args.approval_identity, + }); +} + function closedArguments( value, required, @@ -272,6 +302,22 @@ export function toolDefinitions() { properties: {}, }, }, + { + name: 'plan_local_performance', + description: + 'Dry-run one exact performance scope and report immutable zero-egress, duration, retry, request, service, and cost bounds without executing project code.', + annotations: readAnnotations, + inputSchema: { + ...performanceInputSchema(), + properties: { + ...performanceInputSchema().properties, + approval_identity: { + type: 'string', + pattern: '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$', + }, + }, + }, + }, { name: 'profile_local_performance', description: diff --git a/scripts/runtime-failure-capsule/node-egress-preload.mjs b/scripts/runtime-failure-capsule/node-egress-preload.mjs new file mode 100644 index 00000000..48fd6e1e --- /dev/null +++ b/scripts/runtime-failure-capsule/node-egress-preload.mjs @@ -0,0 +1,86 @@ +import dns from 'node:dns'; +import http from 'node:http'; +import https from 'node:https'; +import net from 'node:net'; +import tls from 'node:tls'; + +const BLOCK_CODE = 'CODEVETTER_REMOTE_EGRESS_BLOCKED'; + +function isLoopback(hostname) { + const value = String(hostname ?? '') + .replace(/^\[|\]$/g, '') + .toLowerCase(); + return value === 'localhost' || value === '::1' || /^127(?:\.\d{1,3}){3}$/.test(value); +} + +function destination(args) { + const first = args[0]; + if (first instanceof URL) return first.hostname; + if (typeof first === 'string') { + try { + return new URL(first).hostname; + } catch { + return first; + } + } + if (first && typeof first === 'object') return first.hostname ?? first.host ?? 'localhost'; + return args[1]?.hostname ?? args[1]?.host ?? 'localhost'; +} + +function deny(kind, hostname) { + const safeDestination = String(hostname ?? '').slice(0, 255); + process.stderr.write( + `CODEVETTER_EGRESS_BLOCKED ${JSON.stringify({ kind, destination: safeDestination })}\n` + ); + const error = new Error(`CodeVetter blocked remote ${kind} access to ${safeDestination}`); + error.code = BLOCK_CODE; + throw error; +} + +function guard(object, method, kind, pick = destination) { + const original = object?.[method]; + if (typeof original !== 'function') return; + object[method] = function guarded(...args) { + const hostname = pick(args); + if (!isLoopback(hostname)) return deny(kind, hostname); + return original.apply(this, args); + }; +} + +for (const method of ['connect', 'createConnection']) guard(net, method, 'socket'); +guard(tls, 'connect', 'tls'); +for (const method of ['request', 'get']) { + guard(http, method, 'http'); + guard(https, method, 'https'); +} +for (const method of ['lookup', 'resolve', 'resolve4', 'resolve6']) { + guard(dns, method, 'dns', (args) => args[0]); +} + +if (typeof globalThis.fetch === 'function') { + const originalFetch = globalThis.fetch; + globalThis.fetch = function guardedFetch(input, init) { + const hostname = destination([input]); + if (!isLoopback(hostname)) return Promise.reject(policyError('fetch', hostname)); + return originalFetch(input, init); + }; +} + +if (typeof globalThis.WebSocket === 'function') { + const OriginalWebSocket = globalThis.WebSocket; + globalThis.WebSocket = class GuardedWebSocket extends OriginalWebSocket { + constructor(url, protocols) { + const hostname = destination([url]); + if (!isLoopback(hostname)) deny('websocket', hostname); + super(url, protocols); + } + }; +} + +function policyError(kind, hostname) { + try { + deny(kind, hostname); + } catch (error) { + return error; + } +} diff --git a/scripts/runtime-failure-capsule/paired-verification.mjs b/scripts/runtime-failure-capsule/paired-verification.mjs index 1485adf9..73e4886d 100644 --- a/scripts/runtime-failure-capsule/paired-verification.mjs +++ b/scripts/runtime-failure-capsule/paired-verification.mjs @@ -192,8 +192,12 @@ function pairedCapsule({ }, viteArtifact, }); + const usesWorkloadMetric = + capsule.observed.console_metrics.length > 0 || capsule.observed.go_benchmarks.length > 0; capsule.limitations = capsule.limitations.filter( - (limitation) => !DIAGNOSTIC_ONLY_LIMITATIONS.has(limitation) + (limitation) => + !DIAGNOSTIC_ONLY_LIMITATIONS.has(limitation) && + !(usesWorkloadMetric && limitation.startsWith('Wall-time samples varied by')) ); capsule.capture.profile_kind = 'paired_timing_only'; return capsule; diff --git a/scripts/runtime-failure-capsule/paired-verification.test.mjs b/scripts/runtime-failure-capsule/paired-verification.test.mjs index 242dc5f3..ff3e793a 100644 --- a/scripts/runtime-failure-capsule/paired-verification.test.mjs +++ b/scripts/runtime-failure-capsule/paired-verification.test.mjs @@ -36,7 +36,7 @@ test('alternates two runnable repositories and confirms matching scale evidence' ['baseline', 'current', 'current', 'baseline', 'baseline', 'current'] ); assert.equal(report.baseline_capsule.observed.console_metrics[0].metrics[0].sample_count, 3); - assert.equal(report.limitations.length, 0); + assert.equal(report.limitations.length, 0, JSON.stringify(report.limitations)); }); test('fails closed when one paired workload does not complete', async (context) => { diff --git a/scripts/runtime-failure-capsule/performance.mjs b/scripts/runtime-failure-capsule/performance.mjs index 9b3f68a8..77b77501 100644 --- a/scripts/runtime-failure-capsule/performance.mjs +++ b/scripts/runtime-failure-capsule/performance.mjs @@ -12,6 +12,10 @@ import { validatePerformanceCapsule, } from './contracts.mjs'; import { inspectGitDiff, rankRelevantChanges } from './git-diff.mjs'; +import { + createPerformanceExecutionReceipt, + planPerformanceExecution, +} from './execution-governance.mjs'; import { collectV8FunctionCoverage, emptyFunctionCoverage } from './function-coverage.mjs'; import { redactText } from './redact.mjs'; import { inspectGoProfile, runClosedAdapter } from './runner.mjs'; @@ -48,6 +52,28 @@ export async function profileRepository({ validateProfileScope(adapter, name); const lexicalRoot = resolve(repositoryRoot); const root = await realpath(lexicalRoot); + const processCount = plannedProfileProcessCount({ adapter, samples, warmups, captureFlow }); + const executionPlan = await planPerformanceExecution({ + repositoryRoot: root, + adapter, + target, + name, + timeoutMs, + processCount, + }); + if (executionPlan.decision.status === 'blocked') { + return createBlockedPerformanceCapsule({ + root, + lexicalRoot, + git: await inspectGitDiff(root), + adapter, + target, + name, + samples, + warmups, + executionPlan, + }); + } const temporaryDirectory = await mkdtemp(join(tmpdir(), 'codevetter-profile-')); const executions = []; let profileEvidence = emptyProfileEvidence(adapter); @@ -67,6 +93,7 @@ export async function profileRepository({ target, name, timeoutMs, + executionPlan, }), }); } @@ -80,6 +107,7 @@ export async function profileRepository({ target, name, timeoutMs, + executionPlan, }), }); } @@ -95,6 +123,7 @@ export async function profileRepository({ name, timeoutMs, vitestReporter: adapter === 'vitest' ? 'verbose' : undefined, + executionPlan, }), }); } @@ -112,6 +141,7 @@ export async function profileRepository({ name, timeoutMs, flowDirectory, + executionPlan, }), }); flowEvidence = await collectNodeFlowEvents(flowDirectory); @@ -129,6 +159,7 @@ export async function profileRepository({ timeoutMs, coverageDirectory, vitestReporter: adapter === 'vitest' ? 'dot' : undefined, + executionPlan, }), }); functionCoverage = await collectV8FunctionCoverage(coverageDirectory, root); @@ -148,6 +179,7 @@ export async function profileRepository({ name, timeoutMs, profileDirectory, + executionPlan, }), }); profileRuns.push( @@ -187,6 +219,8 @@ export async function profileRepository({ regressionPercent, regressionMs, viteArtifact, + executionPlan, + executionReceipt: createPerformanceExecutionReceipt(executionPlan, executions), }); } @@ -209,6 +243,8 @@ export function createPerformanceCapsule({ regressionPercent = 20, regressionMs = 25, viteArtifact = null, + executionPlan = null, + executionReceipt = null, }) { let redactionCount = 0; let outputTruncated = false; @@ -303,6 +339,7 @@ export function createPerformanceCapsule({ if (cleanupFailed) limitations.push('Owned temporary profiling artifacts could not be completely removed.'); if (outputTruncated) limitations.push('Runner output was truncated before normalization.'); + limitations.push(...executionGovernanceLimitations(executionReceipt)); limitations.push(...playwrightLimitations(adapter, playwrightTest)); if (adapter === 'go-bench' && goBenchmarks.length === 0) { limitations.push('No matching Go benchmark measurement was captured.'); @@ -547,6 +584,7 @@ export function createPerformanceCapsule({ coverage_bytes: functionCoverage.coverage_bytes, coverage_functions: functionCoverage.functions.length, }, + execution_governance: executionGovernance(executionPlan, executionReceipt), verdict: { status: verdict, reason: @@ -562,6 +600,57 @@ export function createPerformanceCapsule({ return capsule; } +function executionGovernance(plan, receipt) { + if (!plan || !receipt) return null; + return { plan, receipt }; +} + +function executionGovernanceLimitations(receipt) { + return receipt?.status === 'policy_violation' + ? ['The zero-egress boundary blocked a remote network attempt.'] + : []; +} + +function createBlockedPerformanceCapsule({ + root, + lexicalRoot, + git, + adapter, + target, + name, + samples, + warmups, + executionPlan, +}) { + const executionReceipt = createPerformanceExecutionReceipt(executionPlan); + return createPerformanceCapsule({ + root, + lexicalRoot, + git, + adapter, + target, + name, + samples, + warmups, + executions: [], + profileEvidence: emptyProfileEvidence(adapter), + profileRuns: [], + flowEvidence: emptyFlowEvidence(), + functionCoverage: emptyFunctionCoverage(), + cleanupFailed: false, + baseline: null, + viteArtifact: null, + executionPlan, + executionReceipt, + }); +} + +export function plannedProfileProcessCount({ adapter, samples, warmups, captureFlow = false }) { + const metrics = ['node-test', 'node-script', 'vitest'].includes(adapter) ? samples : 0; + const flows = captureFlow && ['node-test', 'vitest'].includes(adapter) ? 2 : 0; + return warmups + samples + metrics + flows + profileRunsFor(adapter); +} + function summarizeExecution(entry, adapter, name) { const summary = { phase: entry.phase, diff --git a/scripts/runtime-failure-capsule/runner.mjs b/scripts/runtime-failure-capsule/runner.mjs index e0b8ea4a..99ec132a 100644 --- a/scripts/runtime-failure-capsule/runner.mjs +++ b/scripts/runtime-failure-capsule/runner.mjs @@ -4,6 +4,7 @@ import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; import { LIMITS, boundedTimeout, repositoryRelative } from './contracts.mjs'; +import { assertPerformanceExecutionPlanCurrent } from './execution-governance.mjs'; export async function runClosedAdapter({ repositoryRoot, @@ -16,6 +17,7 @@ export async function runClosedAdapter({ coverageDirectory, benchmarkCount = 1, vitestReporter, + executionPlan = null, }) { const root = await realpath(resolve(repositoryRoot)); const scope = await resolveTarget(root, target); @@ -53,16 +55,35 @@ export async function runClosedAdapter({ environmentValues: [], }; } + if ( + executionPlan?.decision?.status !== undefined && + executionPlan.decision.status !== 'admitted' + ) { + throw new Error('performance execution plan is not admitted'); + } + if (executionPlan) { + await assertPerformanceExecutionPlanCurrent({ + plan: executionPlan, + repositoryRoot: root, + adapter, + target: scope.relative, + name: name ?? null, + }); + } + const zeroEgress = executionPlan !== null; const environment = minimalEnvironment({ profileDirectory: profileDirectory && ['node-test', 'node-script'].includes(adapter) ? profileDirectory : null, flowDirectory: flowDirectory && ['node-test', 'vitest'].includes(adapter) ? flowDirectory : null, coverageDirectory: coverageDirectory && adapter === 'node-test' ? coverageDirectory : null, + zeroEgress: + zeroEgress && ['node-test', 'node-script', 'vitest', 'playwright'].includes(adapter), }); + const ownedCommand = governedCommand(command, executionPlan); const execution = await runOwnedProcess({ - program: command.program, - args: command.args, + program: ownedCommand.program, + args: ownedCommand.args, cwd: command.cwd, environment, timeoutMs: timeout, @@ -328,6 +349,7 @@ function minimalEnvironment({ profileDirectory = null, flowDirectory = null, coverageDirectory = null, + zeroEgress = false, } = {}) { const allowed = ['PATH', 'HOME', 'TMPDIR', 'TMP', 'TEMP', 'SYSTEMROOT', 'COMSPEC', 'PATHEXT']; const environment = { CI: '1', FORCE_COLOR: '0', NO_COLOR: '1' }; @@ -342,10 +364,29 @@ function minimalEnvironment({ environment.CODEVETTER_FLOW_DIRECTORY = flowDirectory; } if (coverageDirectory) environment.NODE_V8_COVERAGE = coverageDirectory; + if (zeroEgress) { + const preload = fileURLToPath(new URL('./node-egress-preload.mjs', import.meta.url)); + nodeOptions.push(`--import=${pathToFileURL(preload).href}`); + environment.CODEVETTER_NETWORK_POLICY = 'loopback-only'; + } if (nodeOptions.length > 0) environment.NODE_OPTIONS = nodeOptions.join(' '); return environment; } +function governedCommand(command, executionPlan) { + if (!executionPlan || !executionPlan.enforcement?.kind?.startsWith('macos_sandbox')) { + return { program: command.program, args: command.args }; + } + const allowLoopback = executionPlan.enforcement.network_scope === 'loopback_only'; + const policy = allowLoopback + ? '(version 1)(allow default)(deny network*)(allow network-inbound (local ip "*:*"))(allow network-outbound (remote ip "localhost:*"))' + : '(version 1)(allow default)(deny network*)'; + return { + program: '/usr/bin/sandbox-exec', + args: ['-p', policy, command.program, ...command.args], + }; +} + function runOwnedProcess({ program, args, cwd, environment, timeoutMs }) { return new Promise((resolvePromise) => { const startedAt = Date.now(); diff --git a/scripts/runtime-failure-capsule/supervision-contracts.mjs b/scripts/runtime-failure-capsule/supervision-contracts.mjs index e797b0bc..6a2e66af 100644 --- a/scripts/runtime-failure-capsule/supervision-contracts.mjs +++ b/scripts/runtime-failure-capsule/supervision-contracts.mjs @@ -13,6 +13,7 @@ export const SUPERVISION_LIMITS = Object.freeze({ export const SUPERVISED_RUN_STATES = Object.freeze([ 'initialized', + 'blocked', 'running', 'succeeded', 'failed', @@ -51,6 +52,7 @@ export function validateSupervisedRunReceipt(value) { 'state', 'subject', 'scope', + 'execution_governance', 'policy', 'supervisor', 'lifecycle', @@ -72,6 +74,7 @@ export function validateSupervisedRunReceipt(value) { if (!SUPERVISED_RUN_STATES.includes(value.state)) errors.push('invalid state'); validateSubject(value.subject, errors); validateScope(value.scope, errors); + validateExecutionGovernance(value.execution_governance, errors); validatePolicy(value.policy, errors); validateSupervisor(value.supervisor, errors); validateLifecycle(value.lifecycle, errors); @@ -90,6 +93,20 @@ export function validateSupervisedRunReceipt(value) { return errors; } +function validateExecutionGovernance(value, errors) { + if (value === null || value === undefined) return; + if (!objectWithKeys(value, ['plan', 'receipt'], 'execution_governance', errors)) return; + if (value.plan?.schema_version !== 'performance-execution-plan/v1') { + errors.push('execution_governance.plan is invalid'); + } + if (value.receipt?.schema_version !== 'performance-execution-receipt/v1') { + errors.push('execution_governance.receipt is invalid'); + } + if (value.plan?.plan_id !== value.receipt?.plan_id) { + errors.push('execution_governance identity mismatch'); + } +} + function validateSubject(value, errors) { if (!objectWithKeys(value, ['repository_revision', 'dirty'], 'subject', errors)) return; if (typeof value.repository_revision !== 'string' || value.repository_revision.length === 0) { diff --git a/scripts/runtime-failure-capsule/supervision.mjs b/scripts/runtime-failure-capsule/supervision.mjs index 4908695d..041247b9 100644 --- a/scripts/runtime-failure-capsule/supervision.mjs +++ b/scripts/runtime-failure-capsule/supervision.mjs @@ -13,6 +13,11 @@ import { validatePerformanceDiagnosis, } from './contracts.mjs'; import { inspectGitDiff } from './git-diff.mjs'; +import { + createPerformanceExecutionReceipt, + planPerformanceExecution, +} from './execution-governance.mjs'; +import { plannedProfileProcessCount } from './performance.mjs'; import { redactJsonValue, redactText } from './redact.mjs'; import { SUPERVISED_RUN_SCHEMA_VERSION, @@ -58,6 +63,18 @@ export async function supervisePerformanceRun({ } await assertTarget(root, target); const git = await inspectGitDiff(root); + const executionPlan = await planPerformanceExecution({ + repositoryRoot: root, + adapter: safeAdapter, + target, + name, + timeoutMs: safeTimeout, + processCount: plannedProfileProcessCount({ + adapter: safeAdapter, + samples: safeSamples, + warmups: safeWarmups, + }), + }); const relativeDirectory = `${RUNS_DIRECTORY}/${safeRunId}`; const lexicalRunsDirectory = resolve(root, RUNS_DIRECTORY); await mkdir(lexicalRunsDirectory, { recursive: true }); @@ -75,6 +92,10 @@ export async function supervisePerformanceRun({ state: 'initialized', subject: { repository_revision: git.repository_revision, dirty: git.dirty }, scope: { adapter: safeAdapter, target, name: name ?? null }, + execution_governance: { + plan: executionPlan, + receipt: createPerformanceExecutionReceipt(executionPlan), + }, policy: { samples: safeSamples, warmups: safeWarmups, @@ -102,6 +123,26 @@ export async function supervisePerformanceRun({ }, limitations: ['A supervised run proves only the recorded local performance scope.'], }; + if (executionPlan.decision.status === 'blocked') { + const completedAt = new Date().toISOString(); + receipt = { + ...receipt, + state: 'blocked', + lifecycle: { ...receipt.lifecycle, completed_at: completedAt }, + failure: { + kind: 'blocked', + operational_error: executionPlan.decision.blockers.join(' '), + stdout: null, + stderr: null, + }, + limitations: [ + ...receipt.limitations, + 'No project code executed because the performance execution plan was blocked.', + ], + }; + await writeReceipt(receiptPath, receipt); + return assertSupervisedRunReceipt(receipt); + } await writeReceipt(receiptPath, receipt); const command =