diff --git a/.osc/plans/done/173-codex-token-efficiency-proof-amendment-1.md b/.osc/plans/done/173-codex-token-efficiency-proof-amendment-1.md new file mode 100644 index 00000000..a8b7f326 --- /dev/null +++ b/.osc/plans/done/173-codex-token-efficiency-proof-amendment-1.md @@ -0,0 +1,23 @@ +# Amendment 1: 173-codex-token-efficiency-proof + +## Parent + +173-codex-token-efficiency-proof + +## Date + +2026-06-18 + +## Learning + +The Codex 2x cold-resume fixture is useful but too easy to over-read: it currently proves one bounded token-efficiency decision, while the next proof-battery work needs to make adjacent non-claims impossible to accidentally promote. Human-reviewer replication, controlled ablations, and cold-resume packet quality should be first-class, source-labeled fixture metadata even when their status is `not_demonstrated` or `mixed_not_proven`. + +## New direction + +Harden the checked-in proof surface by adding a fail-closed evidence-battery layer to proof manifests. The current Codex fixture may still pass only for its stated source-labeled 2x cold-resume claim, but its report must also disclose which evidence items are required for that pass and which adjacent claims are explicitly outside the pass gate: human-reviewer replication, controlled ablations, and broader fixture coverage. Add a cold-resume packet contract so the compact packet's required fields are mechanically named and source-linked instead of implied by prose. + +## Impact on acceptance criteria + +- AC1 and AC3 gain a stricter source-labeling requirement: the Codex fixture must include evidence-battery entries and a cold-resume packet contract that point only to committed, public-safe local fixture files. +- AC4 gains explicit non-claim disclosures: human-reviewer replication and controlled ablations must be represented as `not_demonstrated` / `mixed_not_proven` evidence entries unless and until real receipts exist. +- AC5 expands from targeted proof tests to include fail-closed evidence-battery validation, package-payload coverage for the new fixture sources, and docs that keep the Codex 2x claim bounded. diff --git a/.osc/plans/active/173-codex-token-efficiency-proof.md b/.osc/plans/done/173-codex-token-efficiency-proof.md similarity index 63% rename from .osc/plans/active/173-codex-token-efficiency-proof.md rename to .osc/plans/done/173-codex-token-efficiency-proof.md index 50ae163f..e4456bab 100644 --- a/.osc/plans/active/173-codex-token-efficiency-proof.md +++ b/.osc/plans/done/173-codex-token-efficiency-proof.md @@ -2,7 +2,7 @@ ## Status -active +done ## Context @@ -40,11 +40,11 @@ and uses at least 2x fewer Codex-reported total tokens than the naked Codex lane ## Acceptance criteria -- [ ] `osc prove compare examples/proof/codex-token-efficient-resume/manifest.json --format markdown` reports PASS with a token metric ratio of at least 2x. -- [ ] If a proof manifest sets a minimum ratio that is not met, `compareProofManifest` marks the bounded proof as failed/inconclusive. -- [ ] The new fixture's live Codex receipts are source-labeled, committed, and include enough usage data to recompute the medians. -- [ ] Public docs say exactly what the 2x fixture proves and what it does not prove. -- [ ] Targeted tests and the full project test suite pass. +- [x] `osc prove compare examples/proof/codex-token-efficient-resume/manifest.json --format markdown` reports PASS with a token metric ratio of at least 2x. Evidence: `examples/proof/codex-token-efficient-resume/manifest.json`, `examples/proof/codex-token-efficient-resume/receipts/aggregate.json`, `.osc/releases/2026-06-18-173-codex-token-efficiency-proof.md`. +- [x] If a proof manifest sets a minimum ratio that is not met, `compareProofManifest` marks the bounded proof as failed/inconclusive. Evidence: `src/compare.ts`, `tests/proof.test.ts`, `.osc/releases/2026-06-18-173-codex-token-efficiency-proof.md`. +- [x] The new fixture's live Codex receipts are source-labeled, committed, and include enough usage data to recompute the medians. Evidence: `examples/proof/codex-token-efficient-resume/receipts/aggregate.json`, `examples/proof/codex-token-efficient-resume/receipts/control-r1.json`, `examples/proof/codex-token-efficient-resume/receipts/scaffolded-r1.json`. +- [x] Public docs say exactly what the 2x fixture proves and what it does not prove. Evidence: `docs/PROOF_HARNESS.md`, `docs/SCHEMA_REGISTRY.md`, `examples/proof/codex-token-efficient-resume/evidence/human-reviewer-replication-boundary.md`, `examples/proof/codex-token-efficient-resume/evidence/controlled-ablations-boundary.md`. +- [x] Targeted tests and the full project test suite pass. Evidence: `tests/proof.test.ts`, `tests/package-payload.test.ts`, `.osc/releases/2026-06-18-173-codex-token-efficiency-proof.md`. ## Verification steps diff --git a/.osc/releases/2026-06-18-173-codex-token-efficiency-proof.md b/.osc/releases/2026-06-18-173-codex-token-efficiency-proof.md new file mode 100644 index 00000000..fe8b9da5 --- /dev/null +++ b/.osc/releases/2026-06-18-173-codex-token-efficiency-proof.md @@ -0,0 +1,40 @@ +# Release / Evidence Note: 173-codex-token-efficiency-proof + +## Summary + +Hardened the Codex 2x cold-resume proof fixture with fail-closed evidence-battery rows, explicit non-claim disclosures for human-reviewer replication and controlled ablations, and a cold-resume packet contract recorded in the aggregate receipt. The bounded Codex 2x claim remains source-labeled to committed receipts and still does not claim broad workload dominance, human replication, release, merge, publish, deployment, or compliance approval. + +## Traceability + +- Roadmap / issue / task: owner prompt to strengthen bounded proof battery; implemented through plan 173 amendment 1. +- Plan: `.osc/plans/done/173-codex-token-efficiency-proof.md`. +- Amendment: `.osc/plans/done/173-codex-token-efficiency-proof-amendment-1.md`. +- Run ID / run packet: `N/A` — no runtime was spawned; this is a fixture/manifest/docs/test hardening slice over committed receipts. +- Branch / PR: branch `proof-battery-hardening`; PR pending owner review. + +## Verification + +- `./verify.sh --quick --quiet && npm run osc -- resume --max-chars 12000` — exit 0; resumed active plan 173. +- `node examples/proof/codex-token-efficient-resume/score-fixture.mjs --reuse-committed-receipts` — exit 0; regenerated `receipts/aggregate.json` and `manifest.json`; reported ratio `4.330033` with control tokens `137327` and scaffolded tokens `31715`. +- `npm run osc -- prove check examples/proof/codex-token-efficient-resume/manifest.json` — PASS, `0 warning(s)`. +- `npm run osc -- prove compare examples/proof/codex-token-efficient-resume/manifest.json --format markdown` — PASS; evidence battery pass; required rows `codex-2x-cold-resume-replicates` and `cold-resume-packet-contract` demonstrated; human-reviewer replication `not_demonstrated`; controlled ablations `mixed_not_proven`. +- `npm test -- --run tests/proof.test.ts tests/package-payload.test.ts tests/public-positioning.test.ts tests/blueprint-mega.test.ts` — 4 files passed, 34 tests passed. +- `npm run build` — exit 0 (`build:core` and `build:runtime-omx`). +- `npm test` — 48 files passed, 541 tests passed. +- `./verify.sh --strict` — 9 pass, 0 fail, 1 warn (`Plan immutability check skipped` in this git-worktree checkout). +- `npm run osc -- doctor --check secret-scan` — PASS, no obvious token/webhook strings found. +- `git diff --check` — exit 0. +- Added-line public safety scan for names, private paths, Discord IDs, tokens/webhooks/secrets/API keys — no private identity/path/secret hit; benign matches were proof-token wording and fixture paths. + +## Outcome + +approval: + status: approved + rationale: Local plan closeout is approved for this branch because the source-labeled fixture hardening, docs, tests, build, proof compare/check, secret scan, and diff checks passed; this is not merge, publish, release, deployment, or compliance approval. + +The proof harness now renders an evidence-battery section in `osc prove compare`, validates every evidence-battery source ref with the same local/public-safe fail-closed rules as metric refs, and blocks `boundedProof` when a required item is not `demonstrated` or `reproduced`. The Codex cold-resume fixture records a packet-field contract and explicitly keeps human-reviewer replication and controlled ablation claims out of the PASS gate until real receipts exist. + +## Follow-up + +- Open PR / owner review / merge remain pending; no merge, publish, npm release, GitHub Release, deployment, or compliance approval happened. +- Real human-reviewer replication and controlled ablation runs remain future source-labeled receipts, not current claims. diff --git a/MISSION.md b/MISSION.md index a37b5747..4f622f9e 100644 --- a/MISSION.md +++ b/MISSION.md @@ -53,6 +53,8 @@ One-line dated entries for every scope pivot. Format: `YYYY-MM-DD: - 2026-06-18: closed v1-public-positioning-polish — polished v1 public positioning and release-readiness boundaries +- 2026-06-18: closed 173-codex-token-efficiency-proof — closed proof-battery hardening with fail-closed evidence battery and source-labeled Codex 2x fixture boundaries +- 2026-06-18: Owner expanded proof-battery hardening: add source-labeled evidence-battery metadata for more fixtures, human-reviewer replication status, controlled ablation status, and cold-resume packet contract without broadening the Codex 2x claim. — see .osc/plans/done/173-codex-token-efficiency-proof-amendment-1.md - 2026-06-15: closed 172-public-readiness-package-sync — Published open-scaffold@0.32.1 to npm latest and created GitHub Release v0.32.1 as Latest. - 2026-06-15: closed public-readiness-hardening — hardened public readiness messaging - 2026-06-13: closed 171-capture-package-sync — Published open-scaffold@0.32.0 to npm latest and created GitHub Release v0.32.0 as Latest. diff --git a/docs/PROOF_HARNESS.md b/docs/PROOF_HARNESS.md index 2695904e..d2ebed35 100644 --- a/docs/PROOF_HARNESS.md +++ b/docs/PROOF_HARNESS.md @@ -118,6 +118,25 @@ now target what a human reader needs from the answer: readability, comprehension clarity, enough decision detail, unambiguous resume routing, and explicit authority boundaries. It is not a blind human-reader study. +The fixture now carries a fail-closed evidence battery in `manifest.json`: + +| Evidence item | Status | Required for PASS | Boundary | +|---|---|---:|---| +| `codex-2x-cold-resume-replicates` | `demonstrated` | yes | three read-only Codex replicates per arm for one paused-work decision | +| `cold-resume-packet-contract` | `demonstrated` | yes | packet-field contract for this fixture only, not a universal resume schema | +| `human-reviewer-replication` | `not_demonstrated` | no | no blind human-reader replication is claimed for this fixture | +| `controlled-ablations` | `mixed_not_proven` | no | no minimal-checklist, packet-only, or alternate-packet ablation is claimed here | + +`osc prove compare` blocks the bounded PASS if any required evidence-battery item +is not `demonstrated` or `reproduced`. Disclosure-only rows keep adjacent claims +visible but cannot support the pass gate. A manifest with no `evidence_battery` +rows reports `evidenceBatteryStatus: not_evaluated` (and `evidenceBatteryPass: +null`) rather than `pass`, so the report never claims "passes required +evidence-battery items" for a fixture that supplied no battery; it still does +not block `boundedProof` on its own. The human-reviewer and ablation rows point at committed boundary files under +`examples/proof/codex-token-efficient-resume/evidence/`; they are not substitutes +for real receipts. + ## Legacy evolution-controller fixture Before the Codex 2x cold-resume fixture, the first checked-in `osc prove` @@ -191,7 +210,34 @@ A manifest declares the two arms and a set of metrics: } ``` -Required metric categories are `quality`, `tokens`, `speed`, and `evolution`. `osc prove check` fails when a category is missing or a source ref is missing/private. +Required metric categories are `quality`, `tokens`, `speed`, and `evolution`. `osc prove check` fails when a category is missing or a source ref is missing/private. Optional `evidence_battery` rows make fixture/replication/ablation claims explicit: + +```json +{ + "evidence_battery": [ + { + "id": "human-reviewer-replication", + "kind": "human_reviewer_replication", + "status": "not_demonstrated", + "required_for_pass": false, + "claim": "No blind human-reader replication is claimed for this fixture.", + "boundary": "Deterministic reader-usability scoring only until real human receipts exist.", + "source_refs": ["evidence/human-reviewer-replication-boundary.md"] + } + ] +} +``` + +Evidence-battery `source_refs` follow the same fail-closed rules as metric sources: committed fixture-local files only, no URLs, no absolute paths, no parent escapes, and no private/runtime directories. + +A manifest-level `required_evidence` array (of evidence-battery IDs) protects against accidental omission: every listed ID must be *present* in `evidence_battery` and `demonstrated`/`reproduced`, or the gate blocks. This composes with per-row `required_for_pass` (which catches a present row being downgraded to `not_demonstrated`) so deletion and status-downgrade are both fail-closed. + +```json +{ + "required_evidence": ["codex-2x-cold-resume-replicates", "cold-resume-packet-contract"], + "evidence_battery": [ "..." ] +} +``` ## Boundary @@ -228,6 +274,9 @@ Core bench suites write `.osc/bench//aggregate.json` and `.osc/bench/< | Claim | Status | Notes | |---|---|---| | Codex cold-resume 2x token-efficiency fixture passes `osc prove compare` | demonstrated | `examples/proof/codex-token-efficient-resume/`; median reported total tokens 137,327 vs 31,715, ratio 4.330033x, quality tied 6/6 on a deterministic human-facing reader-usability rubric | +| Codex cold-resume evidence-battery required rows | demonstrated | required rows are `codex-2x-cold-resume-replicates` and `cold-resume-packet-contract`; either missing/not-demonstrated row makes `boundedProof=false` | +| Codex fixture human-reviewer replication | not_demonstrated | explicitly disclosure-only; see `examples/proof/codex-token-efficient-resume/evidence/human-reviewer-replication-boundary.md` | +| Codex fixture controlled ablations | mixed_not_proven | explicitly disclosure-only; no minimal-checklist/packet-only/alternate-packet ablation is claimed by the current fixture | | Legacy evolution-controller fixture passes `osc prove check` | demonstrated | `examples/proof/scaffold-vs-naked-codex/`; retained as a legacy example fixture, not the current headline result | | Legacy decision quality preserved (5/5 both arms) | demonstrated | committed receipts for `examples/proof/scaffold-vs-naked-codex/` | | Legacy prompt payload reduction (~11.6×) | demonstrated | committed receipts for `examples/proof/scaffold-vs-naked-codex/` | diff --git a/docs/SCHEMA_REGISTRY.md b/docs/SCHEMA_REGISTRY.md index 5d915744..7ee87659 100644 --- a/docs/SCHEMA_REGISTRY.md +++ b/docs/SCHEMA_REGISTRY.md @@ -28,6 +28,8 @@ Use `osc schemas list --json` for the machine-readable list. Representative sche - `open-scaffold.audit-envelope.v1` — digest-only audit envelope for curated local artifacts. - `open-scaffold.evaluation.v1` — acceptance-criteria evaluation envelope scaffold / external scorer reference. - `open-scaffold.evolution-judgment-checkpoint.v1` — retry authorization gate built from evolution analysis plus optional independent judge input. +- `open-scaffold.proof-comparison.v1` / `open-scaffold.proof-comparison-result.v1` — source-labeled proof manifests and computed comparison verdicts, including evidence-battery rows for fixture/replication/ablation status. +- `open-scaffold.proof-aggregate.v1` — per-arm proof medians, derived deltas, quality-rubric metadata, and optional cold-resume packet contract. - `osc.ambient-work-record.v1` — post-hoc work record extracted from observed run facts so workers do not hand-write bookkeeping in-loop. - `osc.feedback.v1` and `osc.feedback-analysis.v1` — feedback records and repair hypotheses. - `osc.accepted-improvement.v1` — accepted lesson files for relevant future-run inheritance. diff --git a/examples/proof/codex-token-efficient-resume/evidence/controlled-ablations-boundary.md b/examples/proof/codex-token-efficient-resume/evidence/controlled-ablations-boundary.md new file mode 100644 index 00000000..8b527fcb --- /dev/null +++ b/examples/proof/codex-token-efficient-resume/evidence/controlled-ablations-boundary.md @@ -0,0 +1,22 @@ +# Controlled ablation boundary + +Status: `mixed_not_proven` / disclosure-only for this checked-in Codex 2x cold-resume fixture. + +What is demonstrated here: raw paused-session artifacts versus one Open Scaffold compact resume capsule for one paused-work decision, with three Codex replicates per arm and source-labeled receipts. + +What is not demonstrated here: + +- a minimal-checklist control; +- a packet-only ablation that removes Open Scaffold framing while keeping the same compact facts; +- alternate packet shapes or budgets; +- a broad causal claim that Open Scaffold, rather than compact source facts for this task, caused all token savings. + +Minimum ablation packet before any ablation claim may pass: + +1. preregistered ablation arms and kill rules; +2. equal model/runtime settings across arms; +3. per-run usage receipts from trusted adapter metadata, not model text; +4. answer artifacts and a scoring protocol that preserves quality ties as ties; +5. aggregate report with exact reproduction enum: `reproduced`, `partially_reproduced`, or `not_reproduced`. + +Until those files exist for this fixture, ablation evidence cannot support the bounded PASS and must remain disclosure-only. diff --git a/examples/proof/codex-token-efficient-resume/evidence/human-reviewer-replication-boundary.md b/examples/proof/codex-token-efficient-resume/evidence/human-reviewer-replication-boundary.md new file mode 100644 index 00000000..a3e4075c --- /dev/null +++ b/examples/proof/codex-token-efficient-resume/evidence/human-reviewer-replication-boundary.md @@ -0,0 +1,17 @@ +# Human-reviewer replication boundary + +Status: `not_demonstrated` for this checked-in Codex 2x cold-resume fixture. + +What is demonstrated here: committed Codex receipts plus deterministic, human-facing reader-usability scoring over the answers in `../answers/`, summarized in `../receipts/aggregate.json`. + +What is not demonstrated here: blind human reviewers independently reconstructing the paused-work decision from with-record and without-record packets. + +Minimum source-labeled replication packet before this status may change: + +1. preregistered human-review protocol and answer key committed before review; +2. blinded packet pair(s), reviewer instructions, and reviewer identity anonymization policy; +3. raw reviewer responses or sanitized per-question receipts; +4. grading script/output and aggregate accuracy/confabulation/cost or time summary; +5. explicit boundary saying the result is human-reviewer replication for the named fixture only, not universal proof. + +Until those files exist, any human-reviewer claim must remain fail-closed as `not_demonstrated`. diff --git a/examples/proof/codex-token-efficient-resume/manifest.json b/examples/proof/codex-token-efficient-resume/manifest.json index 7d0400a3..f5995a06 100644 --- a/examples/proof/codex-token-efficient-resume/manifest.json +++ b/examples/proof/codex-token-efficient-resume/manifest.json @@ -106,6 +106,66 @@ "notes": "This is the mechanism difference: raw artifacts vs a compiled Open Scaffold resume capsule. It is not a claim that the model became smarter." } ], + "evidence_battery": [ + { + "id": "codex-2x-cold-resume-replicates", + "kind": "cold_resume_fixture", + "status": "demonstrated", + "required_for_pass": true, + "claim": "3 read-only Codex replicates per arm preserved decision quality at 6/6 while reducing median Codex-reported total tokens by 4.330033x.", + "boundary": "One paused-work cold-resume decision on codex-cli exec / gpt-5.5; not a universal workload, model, or production-readiness claim.", + "source_refs": [ + "receipts/aggregate.json", + "receipts/control-r1.json", + "receipts/control-r2.json", + "receipts/control-r3.json", + "receipts/scaffolded-r1.json", + "receipts/scaffolded-r2.json", + "receipts/scaffolded-r3.json" + ] + }, + { + "id": "cold-resume-packet-contract", + "kind": "cold_resume_packet", + "status": "demonstrated", + "required_for_pass": true, + "claim": "The scaffolded arm uses a compact packet with explicit plan, objective, action, resume pointer, acceptance summary, reasons, next fields, evidence refs, and authority boundary.", + "boundary": "Packet-field contract for this fixture only; it does not define a universal resume schema or claim human-reviewer replication.", + "source_refs": [ + "prompts/scaffolded-resume-capsule-prompt.md", + "generate-fixture.mjs", + "receipts/aggregate.json" + ] + }, + { + "id": "human-reviewer-replication", + "kind": "human_reviewer_replication", + "status": "not_demonstrated", + "required_for_pass": false, + "claim": "No blind human-reviewer replication is claimed for this Codex 2x fixture.", + "boundary": "The shipped quality score is deterministic and human-facing; a real human-reader replication needs separate preregistration, answer keys, blinded packets, and receipts.", + "source_refs": [ + "evidence/human-reviewer-replication-boundary.md", + "receipts/aggregate.json" + ] + }, + { + "id": "controlled-ablations", + "kind": "controlled_ablation", + "status": "mixed_not_proven", + "required_for_pass": false, + "claim": "No minimal-checklist, packet-only, or alternate-packet ablation is claimed by this fixture.", + "boundary": "Ablations are disclosure-only here and cannot support broad dominance; future ablation claims must ship their own committed receipts.", + "source_refs": [ + "evidence/controlled-ablations-boundary.md", + "receipts/aggregate.json" + ] + } + ], + "required_evidence": [ + "codex-2x-cold-resume-replicates", + "cold-resume-packet-contract" + ], "caveats": [ "Bounded fixture proof only: one paused-work cold-resume decision, not all AI work or all repositories.", "Codex CLI did not emit total_tokens directly in this environment; receipts record input/output splits and derive total as input_tokens + output_tokens.", diff --git a/examples/proof/codex-token-efficient-resume/receipts/aggregate.json b/examples/proof/codex-token-efficient-resume/receipts/aggregate.json index ea360a02..865b4afa 100644 --- a/examples/proof/codex-token-efficient-resume/receipts/aggregate.json +++ b/examples/proof/codex-token-efficient-resume/receipts/aggregate.json @@ -37,6 +37,26 @@ } ] }, + "cold_resume_packet_contract": { + "id": "cold-resume-packet-v2-readable-closeout", + "required_fields": [ + "plan", + "objective", + "action", + "resume_current_frontier_evaluation", + "acceptance_summary", + "reasons", + "required_next_fields", + "evidence_refs", + "authority_boundary" + ], + "source_refs": [ + "prompts/scaffolded-resume-capsule-prompt.md", + "generate-fixture.mjs" + ], + "fail_closed_if_missing": true, + "boundary": "Packet-field contract for this fixture only; it does not claim a stable public schema or universal resume format." + }, "arms": { "control": { "prompt_bytes": 419233, diff --git a/examples/proof/codex-token-efficient-resume/score-fixture.mjs b/examples/proof/codex-token-efficient-resume/score-fixture.mjs index ac4adacf..acbb7fb8 100644 --- a/examples/proof/codex-token-efficient-resume/score-fixture.mjs +++ b/examples/proof/codex-token-efficient-resume/score-fixture.mjs @@ -30,6 +30,26 @@ const qualityRubric = { { id: 'reader_boundary_is_plain', label: 'The boundary plainly says this is decision support, not approval/release/deployment/compliance.' }, ], }; +const coldResumePacketContract = { + id: 'cold-resume-packet-v2-readable-closeout', + required_fields: [ + 'plan', + 'objective', + 'action', + 'resume_current_frontier_evaluation', + 'acceptance_summary', + 'reasons', + 'required_next_fields', + 'evidence_refs', + 'authority_boundary', + ], + source_refs: [ + 'prompts/scaffolded-resume-capsule-prompt.md', + 'generate-fixture.mjs', + ], + fail_closed_if_missing: true, + boundary: 'Packet-field contract for this fixture only; it does not claim a stable public schema or universal resume format.', +}; function readJson(path) { return JSON.parse(readFileSync(path, 'utf8').replace(/^\uFEFF/, '')); @@ -53,6 +73,27 @@ function round(value) { return Number(value.toFixed(6)); } +function assertColdResumePacketContract() { + const packet = readFileSync(join(fixtureRoot, 'prompts', 'scaffolded-resume-capsule-prompt.md'), 'utf8').toLowerCase(); + const checks = { + plan: 'plan:', + objective: 'objective:', + action: 'action:', + resume_current_frontier_evaluation: 'resume: current=', + acceptance_summary: 'acceptance: 5/5 pass', + reasons: 'reasons:', + required_next_fields: 'required next fields:', + evidence_refs: 'evidence refs:', + authority_boundary: 'boundary:', + }; + const missing = Object.entries(checks) + .filter(([, needle]) => !packet.includes(needle)) + .map(([field]) => field); + if (missing.length > 0) { + throw new Error(`cold resume packet contract missing fields: ${missing.join(', ')}`); + } +} + function usageFromEvents(events) { const turn = [...events].reverse().find((event) => event.type === 'turn.completed' && event.usage); if (!turn) throw new Error('missing turn.completed usage event'); @@ -116,6 +157,8 @@ function scoreAnswer(answer) { return { rubric: qualityRubric.id, human_facing: true, score: checks.filter((check) => check.pass).length, total: checks.length, checks }; } +assertColdResumePacketContract(); + const receipts = []; for (const arm of arms) { for (const replicate of replicates) { @@ -184,6 +227,7 @@ const aggregate = { scorer: 'score-fixture.mjs', }, quality_rubric: qualityRubric, + cold_resume_packet_contract: coldResumePacketContract, arms: { control, scaffolded }, deltas: { prompt_payload_reduction_ratio: round(control.prompt_bytes / scaffolded.prompt_bytes), @@ -280,6 +324,48 @@ const manifest = { notes: 'This is the mechanism difference: raw artifacts vs a compiled Open Scaffold resume capsule. It is not a claim that the model became smarter.', }, ], + evidence_battery: [ + { + id: 'codex-2x-cold-resume-replicates', + kind: 'cold_resume_fixture', + status: 'demonstrated', + required_for_pass: true, + claim: `${replicates.length} read-only Codex replicates per arm preserved decision quality at ${scaffolded.median_quality_score}/${scaffolded.quality_score_total} while reducing median Codex-reported total tokens by ${aggregate.deltas.codex_total_token_reduction_ratio_median}x.`, + boundary: 'One paused-work cold-resume decision on codex-cli exec / gpt-5.5; not a universal workload, model, or production-readiness claim.', + source_refs: sourceRefs, + }, + { + id: 'cold-resume-packet-contract', + kind: 'cold_resume_packet', + status: 'demonstrated', + required_for_pass: true, + claim: 'The scaffolded arm uses a compact packet with explicit plan, objective, action, resume pointer, acceptance summary, reasons, next fields, evidence refs, and authority boundary.', + boundary: 'Packet-field contract for this fixture only; it does not define a universal resume schema or claim human-reviewer replication.', + source_refs: ['prompts/scaffolded-resume-capsule-prompt.md', 'generate-fixture.mjs', 'receipts/aggregate.json'], + }, + { + id: 'human-reviewer-replication', + kind: 'human_reviewer_replication', + status: 'not_demonstrated', + required_for_pass: false, + claim: 'No blind human-reviewer replication is claimed for this Codex 2x fixture.', + boundary: 'The shipped quality score is deterministic and human-facing; a real human-reader replication needs separate preregistration, answer keys, blinded packets, and receipts.', + source_refs: ['evidence/human-reviewer-replication-boundary.md', 'receipts/aggregate.json'], + }, + { + id: 'controlled-ablations', + kind: 'controlled_ablation', + status: 'mixed_not_proven', + required_for_pass: false, + claim: 'No minimal-checklist, packet-only, or alternate-packet ablation is claimed by this fixture.', + boundary: 'Ablations are disclosure-only here and cannot support broad dominance; future ablation claims must ship their own committed receipts.', + source_refs: ['evidence/controlled-ablations-boundary.md', 'receipts/aggregate.json'], + }, + ], + required_evidence: [ + 'codex-2x-cold-resume-replicates', + 'cold-resume-packet-contract', + ], caveats: [ 'Bounded fixture proof only: one paused-work cold-resume decision, not all AI work or all repositories.', 'Codex CLI did not emit total_tokens directly in this environment; receipts record input/output splits and derive total as input_tokens + output_tokens.', diff --git a/src/compare.ts b/src/compare.ts index 75727d71..5e846c55 100644 --- a/src/compare.ts +++ b/src/compare.ts @@ -354,9 +354,15 @@ export function renderAttemptComparisonJson(comparison: BareAttemptComparison): export const PROOF_COMPARISON_SCHEMA = 'open-scaffold.proof-comparison.v1', PROOF_COMPARISON_RESULT_SCHEMA = 'open-scaffold.proof-comparison-result.v1'; export type ProofRenderFormat = 'terminal' | 'markdown' | 'json'; type ProofCategory = 'quality' | 'tokens' | 'speed' | 'evolution'; type ProofDirection = 'higher' | 'lower'; type ProofWinner = 'scaffolded' | 'control' | 'tie'; type ProofRatio = number | 'unbounded' | null; +type ProofEvidenceKind = 'cold_resume_fixture' | 'cold_resume_packet' | 'human_reviewer_replication' | 'controlled_ablation' | 'legacy_fixture' | 'reviewability_fixture' | 'other'; +type ProofEvidenceStatus = 'demonstrated' | 'reproduced' | 'partially_reproduced' | 'not_reproduced' | 'not_demonstrated' | 'mixed_not_proven' | 'blocked'; type ProofIssue = { level: 'fail' | 'warn'; code: string; message: string; path?: string }; type ProofValidationResult = { failures: ProofIssue[]; warnings: ProofIssue[] }; type ProofMetric = { id: string; label: string; category: ProofCategory; unit: string; direction: ProofDirection; control: number; scaffolded: number; source_refs: string[]; notes?: string | null; minimum_ratio?: number | null }; +type ProofEvidenceItem = { id: string; kind: ProofEvidenceKind; status: ProofEvidenceStatus; requiredForPass: boolean; claim: string; boundary: string; sourceRefs: string[]; notes?: string | null }; const PROOF_CATEGORIES: ProofCategory[] = ['quality', 'tokens', 'speed', 'evolution']; +const PROOF_EVIDENCE_KINDS: ProofEvidenceKind[] = ['cold_resume_fixture', 'cold_resume_packet', 'human_reviewer_replication', 'controlled_ablation', 'legacy_fixture', 'reviewability_fixture', 'other']; +const PROOF_EVIDENCE_STATUSES: ProofEvidenceStatus[] = ['demonstrated', 'reproduced', 'partially_reproduced', 'not_reproduced', 'not_demonstrated', 'mixed_not_proven', 'blocked']; +const PROOF_EVIDENCE_PASS_STATUSES = new Set(['demonstrated', 'reproduced']); const proofStrings = (value: unknown) => Array.isArray(value) ? value.filter((item): item is string => typeof item === 'string' && item.trim().length > 0) : []; const proofIssue = (level: 'fail' | 'warn', code: string, message: string, path?: string): ProofIssue => ({ level, code, message, path }); const proofArm = (value: unknown, fallback: string) => { const raw = isRecord(value) ? value : {}; return { id: asString(raw.id)?.trim() || fallback, label: asString(raw.label)?.trim() || fallback, runtime: asString(raw.runtime) }; }; @@ -373,6 +379,30 @@ function proofMetric(value: unknown, index: number, manifestDir: string, failure if (category === 'speed' && direction !== 'lower') warnings.push(proofIssue('warn', 'unexpected-speed-direction', 'Speed metrics normally use direction: lower.', `${path}.direction`)); return id && label && category && PROOF_CATEGORIES.includes(category) && unit && direction && ['higher', 'lower'].includes(direction) && control !== null && scaffolded !== null ? { id, label, category, unit, direction, control, scaffolded, source_refs: refs, notes: asString(value.notes), minimum_ratio: minimumRatio } : null; } +function proofEvidenceItem(value: unknown, index: number, manifestDir: string, failures: ProofIssue[]): ProofEvidenceItem | null { + const path = `evidence_battery[${index}]`; + if (!isRecord(value)) { failures.push(proofIssue('fail', 'invalid-evidence-item', 'Evidence-battery item must be an object.', path)); return null; } + const id = asString(value.id)?.trim() || null, kind = asString(value.kind) as ProofEvidenceKind | null, status = asString(value.status) as ProofEvidenceStatus | null, claim = asString(value.claim)?.trim() || null, boundary = asString(value.boundary)?.trim() || null, refs = proofStrings(value.source_refs); + const requiredForPass = typeof value.required_for_pass === 'boolean' ? value.required_for_pass : false; + for (const [bad, code, message, suffix] of [[!id, 'missing-evidence-id', 'Evidence-battery item must include id.', 'id'], [!kind || !PROOF_EVIDENCE_KINDS.includes(kind), 'invalid-evidence-kind', `Evidence-battery kind must be one of: ${PROOF_EVIDENCE_KINDS.join(', ')}.`, 'kind'], [!status || !PROOF_EVIDENCE_STATUSES.includes(status), 'invalid-evidence-status', `Evidence-battery status must be one of: ${PROOF_EVIDENCE_STATUSES.join(', ')}.`, 'status'], [!claim, 'missing-evidence-claim', 'Evidence-battery item must include a bounded claim.', 'claim'], [!boundary, 'missing-evidence-boundary', 'Evidence-battery item must include a boundary statement.', 'boundary'], [refs.length === 0, 'missing-source-ref', 'Evidence-battery item must include at least one source_refs entry.', 'source_refs']] as const) if (bad) failures.push(proofIssue('fail', code, message, `${path}.${suffix}`)); + if (Object.hasOwn(value, 'required_for_pass') && typeof value.required_for_pass !== 'boolean') failures.push(proofIssue('fail', 'invalid-evidence-required-for-pass', 'required_for_pass must be boolean when present.', `${path}.required_for_pass`)); + for (const ref of refs) proofRefIsPrivate(ref) ? failures.push(proofIssue('fail', 'private-source-ref', `Source ref must be repo/public-safe and relative, not private: ${ref}`, `${path}.source_refs`)) : !proofRefExists(manifestDir, ref) && failures.push(proofIssue('fail', 'missing-source-ref', `Source ref does not exist: ${ref}`, `${path}.source_refs`)); + return id && kind && PROOF_EVIDENCE_KINDS.includes(kind) && status && PROOF_EVIDENCE_STATUSES.includes(status) && claim && boundary && refs.length > 0 ? { id, kind, status, requiredForPass, claim, boundary, sourceRefs: refs, notes: asString(value.notes) } : null; +} +function proofEvidenceBattery(raw: Record, manifestDir: string, failures: ProofIssue[]): ProofEvidenceItem[] { + if (!Object.hasOwn(raw, 'evidence_battery')) return []; + if (!Array.isArray(raw.evidence_battery)) { failures.push(proofIssue('fail', 'invalid-evidence-battery', 'evidence_battery must be an array when present.', 'evidence_battery')); return []; } + return raw.evidence_battery.map((item, index) => proofEvidenceItem(item, index, manifestDir, failures)).filter((item): item is ProofEvidenceItem => Boolean(item)); +} +function proofRequiredEvidence(raw: Record, failures: ProofIssue[]): string[] { + if (!Object.hasOwn(raw, 'required_evidence')) return []; + if (!Array.isArray(raw.required_evidence)) { failures.push(proofIssue('fail', 'invalid-required-evidence', 'required_evidence must be an array of evidence IDs when present.', 'required_evidence')); return []; } + const ids = raw.required_evidence.filter((id): id is string => typeof id === 'string' && id.trim().length > 0).map((id) => id.trim()); + if (ids.length !== raw.required_evidence.length) failures.push(proofIssue('fail', 'invalid-required-evidence-id', 'Each required_evidence entry must be a non-empty string.', 'required_evidence')); + const dupes = ids.filter((id, index) => ids.indexOf(id) !== index); + if (dupes.length > 0) failures.push(proofIssue('fail', 'duplicate-required-evidence-id', `required_evidence must not contain duplicate IDs: ${[...new Set(dupes)].join(', ')}`, 'required_evidence')); + return ids; +} function validateProofRaw(raw: unknown, manifestDir: string): ProofValidationResult { const failures: ProofIssue[] = [], warnings: ProofIssue[] = []; if (!isRecord(raw)) return { failures: [proofIssue('fail', 'invalid-manifest', 'Proof comparison manifest must be a JSON object.')], warnings }; @@ -383,6 +413,8 @@ function validateProofRaw(raw: unknown, manifestDir: string): ProofValidationRes else for (const label of ['control', 'scaffolded']) { const arm = arms[label]; if (!isRecord(arm)) failures.push(proofIssue('fail', 'invalid-arm', `${label} arm must be an object.`, `arms.${label}`)); else for (const field of ['id', 'label']) if (!asString(arm[field])?.trim()) failures.push(proofIssue('fail', `missing-arm-${field}`, `${label} arm must include ${field}.`, `arms.${label}.${field}`)); } if (!Array.isArray(raw.metrics) || raw.metrics.length === 0) failures.push(proofIssue('fail', 'missing-metrics', 'Manifest must include at least one metric.', 'metrics')); else { const categories = new Set(); raw.metrics.forEach((metric, index) => { const parsed = proofMetric(metric, index, manifestDir, failures, warnings); if (parsed) categories.add(parsed.category); }); for (const category of PROOF_CATEGORIES) if (!categories.has(category)) failures.push(proofIssue('fail', 'missing-required-category', `Manifest must include a ${category} metric.`, 'metrics')); } + const evidenceItems = proofEvidenceBattery(raw, manifestDir, failures), evidenceIds = new Set(evidenceItems.map((item) => item.id)); + for (const id of proofRequiredEvidence(raw, failures)) if (!evidenceIds.has(id)) failures.push(proofIssue('fail', 'missing-required-evidence-row', `required_evidence ID is not present in evidence_battery: ${id}`, 'required_evidence')); return { failures, warnings }; } function resolveProofManifestPath(pathArg: string): string { const callerRelative = resolve(pathArg), normalized = normalizeRelativePath(pathArg); if (existsSync(callerRelative) || isAbsolute(pathArg) || normalized === '..' || normalized.startsWith('../')) return callerRelative; const packagedPath = join(packageRoot, ...normalized.split('/')); return (normalized === 'examples/proof' || normalized.startsWith(packagedProofExamplePrefix)) && existsSync(packagedPath) ? packagedPath : callerRelative; } @@ -398,13 +430,24 @@ function proofCategoryStatus(metrics: Array, validation = validateProofRaw(raw, manifestDir); if (validation.failures.length > 0) throw new Error(`Invalid proof comparison manifest: ${validation.failures.map((failure) => failure.code).join(', ')}`); const metrics = ((raw.metrics as unknown[]) ?? []).map((metric, index) => proofMetric(metric, index, manifestDir, [], [])).filter((metric): metric is ProofMetric => Boolean(metric)).map((metric) => { const winner = proofWinner(metric), thresholdRatio = proofScaffoldedRatio(metric), improvementRatio = proofReportedRatio(thresholdRatio), minimumRatio = metric.minimum_ratio ?? null, minimumRatioPassed = proofRatioPasses(thresholdRatio, minimumRatio, winner); return { ...metric, winner, delta: Number((metric.scaffolded - metric.control).toFixed(6)), improvementRatio, minimumRatio, minimumRatioActual: thresholdRatio, minimumRatioPassed, sourceRefs: metric.source_refs }; }); + const evidenceBattery = proofEvidenceBattery(raw, manifestDir, []), requiredEvidence = proofStrings(raw.required_evidence).map((id) => id.trim()), requiredEvidenceSet = new Set(requiredEvidence), evidenceBatteryBlocking = evidenceBattery.filter((item) => (item.requiredForPass || requiredEvidenceSet.has(item.id)) && !PROOF_EVIDENCE_PASS_STATUSES.has(item.status)).map((item) => item.id), evidenceBatteryApplicable = evidenceBattery.length > 0 || requiredEvidence.length > 0, evidenceBatteryPresent = evidenceBattery.length > 0, evidenceBatteryPass: boolean | null = !evidenceBatteryApplicable ? null : evidenceBatteryBlocking.length === 0, evidenceBatteryStatus: 'pass' | 'fail' | 'not_evaluated' = !evidenceBatteryApplicable ? 'not_evaluated' : evidenceBatteryBlocking.length === 0 ? 'pass' : 'fail'; const categories = Object.fromEntries(PROOF_CATEGORIES.map((category) => [category, proofCategoryStatus(metrics, category)])) as Record, scaffoldedWins = metrics.filter((metric) => metric.winner === 'scaffolded').length, controlWins = metrics.filter((metric) => metric.winner === 'control').length, ties = metrics.filter((metric) => metric.winner === 'tie').length; - const thresholdViolations = metrics.filter((metric) => metric.minimumRatioPassed === false).map((metric) => ({ metricId: metric.id, required: metric.minimumRatio, actual: metric.minimumRatioActual })), thresholdsPass = thresholdViolations.length === 0, boundedProof = thresholdsPass && categories.quality !== 'missing' && categories.quality !== 'regressed' && categories.evolution === 'improved' && categories.tokens !== 'regressed' && categories.speed !== 'regressed' && controlWins === 0; - return { schema: PROOF_COMPARISON_RESULT_SCHEMA, comparisonId: asString(raw.comparison_id) ?? 'proof-comparison', title: asString(raw.title) ?? 'Proof comparison', question: asString(raw.question) ?? 'Does the scaffolded arm improve the bounded comparison?', arms: { control: proofArm(isRecord(raw.arms) ? raw.arms.control : null, 'control'), scaffolded: proofArm(isRecord(raw.arms) ? raw.arms.scaffolded : null, 'scaffolded') }, metrics, summary: { scaffoldedWins, controlWins, ties, categories, thresholdsPass, thresholdViolations, boundedProof, verdict: boundedProof ? 'PASS: bounded source-labeled comparison preserves decision quality, satisfies required improvement ratios, and improves evolution without token/speed regression.' : 'FAIL/INCONCLUSIVE: this bounded comparison does not show the scaffolded arm is better at the required threshold; inspect metric regressions, missing categories, or minimum-ratio failures.' }, caveats: proofStrings(raw.caveats).length > 0 ? proofStrings(raw.caveats) : ['Bounded comparison only; not a universal benchmark or model-ranking claim.'], validation }; + const thresholdViolations = metrics.filter((metric) => metric.minimumRatioPassed === false).map((metric) => ({ metricId: metric.id, required: metric.minimumRatio, actual: metric.minimumRatioActual })), thresholdsPass = thresholdViolations.length === 0, metricProof = thresholdsPass && categories.quality !== 'missing' && categories.quality !== 'regressed' && categories.evolution === 'improved' && categories.tokens !== 'regressed' && categories.speed !== 'regressed' && controlWins === 0, boundedProof = metricProof && evidenceBatteryPass !== false; + return { schema: PROOF_COMPARISON_RESULT_SCHEMA, comparisonId: asString(raw.comparison_id) ?? 'proof-comparison', title: asString(raw.title) ?? 'Proof comparison', question: asString(raw.question) ?? 'Does the scaffolded arm improve the bounded comparison?', arms: { control: proofArm(isRecord(raw.arms) ? raw.arms.control : null, 'control'), scaffolded: proofArm(isRecord(raw.arms) ? raw.arms.scaffolded : null, 'scaffolded') }, metrics, evidenceBattery, summary: { scaffoldedWins, controlWins, ties, categories, thresholdsPass, thresholdViolations, evidenceBatteryPresent, evidenceBatteryRequired: requiredEvidence, evidenceBatteryPass, evidenceBatteryStatus, evidenceBatteryBlocking, boundedProof, verdict: boundedProof ? (evidenceBatteryStatus === 'pass' ? 'PASS: bounded source-labeled comparison preserves decision quality, satisfies required improvement ratios, passes required evidence-battery items, and improves evolution without token/speed regression.' : 'PASS: bounded source-labeled comparison preserves decision quality, satisfies required improvement ratios, and improves evolution without token/speed regression. (No evidence_battery rows supplied; the evidence-battery gate was not evaluated.)') : evidenceBatteryStatus === 'fail' ? 'FAIL/INCONCLUSIVE: a required evidence-battery item is not demonstrated/reproduced; this claim stays fail-closed until source-labeled evidence exists.' : 'FAIL/INCONCLUSIVE: this bounded comparison does not show the scaffolded arm is better at the required threshold; inspect metric regressions, missing categories, or minimum-ratio failures.' }, caveats: proofStrings(raw.caveats).length > 0 ? proofStrings(raw.caveats) : ['Bounded comparison only; not a universal benchmark or model-ranking claim.'], validation }; } const proofFmt = (value: ProofRatio) => value === null ? 'n/a' : value === 'unbounded' ? '∞' : Number.isInteger(value) ? String(value) : String(Number(value.toFixed(6))); export function renderProofComparison(result: ReturnType, format: ProofRenderFormat = 'terminal'): string { if (format === 'json') return `${JSON.stringify(result, null, 2)}\n`; - const lines = [format === 'markdown' ? `# ${result.title}` : result.title, '', `Question: ${result.question}`, `Control arm: ${result.arms.control.label} (${result.arms.control.id})`, `Scaffolded arm: ${result.arms.scaffolded.label} (${result.arms.scaffolded.id})`, '', `Bounded proof verdict: ${result.summary.boundedProof ? 'PASS' : 'FAIL/INCONCLUSIVE'}`, result.summary.verdict, `Category status: ${PROOF_CATEGORIES.map((category) => `${category}=${result.summary.categories[category]}`).join(', ')}`, `Minimum ratios: ${result.summary.thresholdsPass ? 'pass' : 'fail'}`, `Wins: scaffolded=${result.summary.scaffoldedWins}, control=${result.summary.controlWins}, ties=${result.summary.ties}`, '', '| Metric | Category | Control | Scaffolded | Winner | Ratio | Minimum | Sources |', '|---|---|---:|---:|---|---:|---:|---|', ...result.metrics.map((metric) => `| ${metric.id} | ${metric.category} | ${proofFmt(metric.control)} ${metric.unit} | ${proofFmt(metric.scaffolded)} ${metric.unit} | ${metric.winner} | ${metric.improvementRatio === null ? 'n/a' : `${proofFmt(metric.improvementRatio)}x`} | ${metric.minimumRatio === null ? 'n/a' : `${proofFmt(metric.minimumRatio)}x ${metric.minimumRatioPassed ? 'pass' : 'fail'}`} | ${metric.sourceRefs.join('
')} |`), '', 'Caveats', ...result.caveats.map((caveat) => `- ${caveat}`), '- This report compares the supplied receipts only. It does not prove all tasks, all models, or all operators will improve; it does not make models smarter, rank models, certify correctness, or authorize release/merge/deploy actions.']; + const batteryStatusLabel = result.summary.evidenceBatteryStatus === 'not_evaluated' ? 'not evaluated' : result.summary.evidenceBatteryStatus; + const requiredSet = new Set(result.summary.evidenceBatteryRequired ?? []); + const batteryLines = result.evidenceBattery.length > 0 ? [ + '', + format === 'markdown' ? '## Proof battery' : 'Proof battery', + `Evidence-battery gate: ${batteryStatusLabel}${result.summary.evidenceBatteryBlocking.length > 0 ? ` (blocking: ${result.summary.evidenceBatteryBlocking.join(', ')})` : ''}`, + '| Evidence item | Kind | Status | Required | Claim | Boundary | Sources |', + '|---|---|---|---|---|---|---|', + ...result.evidenceBattery.map((item) => `| ${item.id} | ${item.kind} | ${item.status} | ${item.requiredForPass || requiredSet.has(item.id) ? 'required' : 'disclosure-only'} | ${escapeCell(item.claim)} | ${escapeCell(item.boundary)} | ${item.sourceRefs.join('
')} |`), + ] : ['', format === 'markdown' ? '## Proof battery' : 'Proof battery', `Evidence-battery gate: ${batteryStatusLabel} (no evidence_battery rows supplied; the gate was not evaluated for this manifest)`]; + const lines = [format === 'markdown' ? `# ${result.title}` : result.title, '', `Question: ${result.question}`, `Control arm: ${result.arms.control.label} (${result.arms.control.id})`, `Scaffolded arm: ${result.arms.scaffolded.label} (${result.arms.scaffolded.id})`, '', `Bounded proof verdict: ${result.summary.boundedProof ? 'PASS' : 'FAIL/INCONCLUSIVE'}`, result.summary.verdict, `Category status: ${PROOF_CATEGORIES.map((category) => `${category}=${result.summary.categories[category]}`).join(', ')}`, `Minimum ratios: ${result.summary.thresholdsPass ? 'pass' : 'fail'}`, `Evidence battery: ${batteryStatusLabel}`, `Wins: scaffolded=${result.summary.scaffoldedWins}, control=${result.summary.controlWins}, ties=${result.summary.ties}`, '', '| Metric | Category | Control | Scaffolded | Winner | Ratio | Minimum | Sources |', '|---|---|---:|---:|---|---:|---:|---|', ...result.metrics.map((metric) => `| ${metric.id} | ${metric.category} | ${proofFmt(metric.control)} ${metric.unit} | ${proofFmt(metric.scaffolded)} ${metric.unit} | ${metric.winner} | ${metric.improvementRatio === null ? 'n/a' : `${proofFmt(metric.improvementRatio)}x`} | ${metric.minimumRatio === null ? 'n/a' : `${proofFmt(metric.minimumRatio)}x ${metric.minimumRatioPassed ? 'pass' : 'fail'}`} | ${metric.sourceRefs.join('
')} |`), ...batteryLines, '', 'Caveats', ...result.caveats.map((caveat) => `- ${caveat}`), '- This report compares the supplied receipts only. It does not prove all tasks, all models, or all operators will improve; it does not make models smarter, rank models, certify correctness, or authorize release/merge/deploy actions.']; return `${lines.join('\n')}\n`; } diff --git a/src/schema-registry.ts b/src/schema-registry.ts index 5105e44d..d39cfea6 100644 --- a/src/schema-registry.ts +++ b/src/schema-registry.ts @@ -17,11 +17,11 @@ export const SCHEMA_REGISTRY: SchemaRegistryEntry[] = [ { id: 'open-scaffold.attempt-comparison.v1', owner: 'src/compare.ts', maturity: 'stable', emittedBy: ['osc compare --json'], shape: 'Read-only comparison of two attempt directories with file diffs and reviewable decision support.' }, { id: 'open-scaffold.evidence.v1', owner: 'docs/SLICE_CLOSE_PROTOCOL.md', maturity: 'stable', emittedBy: ['evidence receipts and postflight notes'], shape: 'Evidence receipt with approval status, rationale, artifacts, verification, follow-up, and boundary statements.' }, { id: 'open-scaffold.audit-envelope.v1', owner: 'src/audit.ts', maturity: 'lab', emittedBy: ['osc audit init'], shape: 'Digest-only artifact audit envelope for local structural review.' }, - { id: 'open-scaffold.proof-comparison.v1', owner: 'src/compare.ts', maturity: 'lab', emittedBy: ['proof manifests consumed by osc prove compare/check'], shape: 'Source-labeled scaffolded-vs-control comparison manifest with quality, token, speed, and evolution metrics plus optional per-metric minimum improvement ratios.' }, - { id: 'open-scaffold.proof-comparison-result.v1', owner: 'src/compare.ts', maturity: 'lab', emittedBy: ['osc prove compare --format json'], shape: 'Computed receipt-comparison verdict with per-metric winners, category status, minimum-ratio threshold status, caveats, and validation summary.' }, + { id: 'open-scaffold.proof-comparison.v1', owner: 'src/compare.ts', maturity: 'lab', emittedBy: ['proof manifests consumed by osc prove compare/check'], shape: 'Source-labeled scaffolded-vs-control comparison manifest with quality, token, speed, and evolution metrics, optional per-metric minimum improvement ratios, and optional evidence_battery rows for fixture/replication/ablation status.' }, + { id: 'open-scaffold.proof-comparison-result.v1', owner: 'src/compare.ts', maturity: 'lab', emittedBy: ['osc prove compare --format json'], shape: 'Computed receipt-comparison verdict with per-metric winners, category status, minimum-ratio threshold status, evidence-battery present/pass/status/blocking (not_evaluated when no battery rows), caveats, and validation summary.' }, { id: 'open-scaffold.proof-receipt.v1', owner: 'docs/PROOF_HARNESS.md', maturity: 'lab', emittedBy: ['examples/proof/scaffold-vs-naked-codex/receipts/*.json'], shape: 'Sanitized per-run receipt for a bounded proof fixture; raw private logs stay out of the public repo.' }, { id: 'open-scaffold.codex-token-efficiency-receipt.v1', owner: 'examples/proof/codex-token-efficient-resume/score-fixture.mjs', maturity: 'lab', emittedBy: ['examples/proof/codex-token-efficient-resume/score-fixture.mjs', 'examples/proof/codex-token-efficient-resume/receipts/{control,scaffolded}-r*.json'], shape: 'Specialized per-run Codex token-efficiency proof receipt with prompt and answer refs, Codex usage and wall time provenance, deterministic quality rubric scores, and explicit bounded-proof safety flags.' }, - { id: 'open-scaffold.proof-aggregate.v1', owner: 'docs/PROOF_HARNESS.md', maturity: 'lab', emittedBy: ['examples/proof/*/receipts/aggregate.json'], shape: 'Aggregate receipt with per-arm medians and derived deltas for a bounded proof fixture.' }, + { id: 'open-scaffold.proof-aggregate.v1', owner: 'docs/PROOF_HARNESS.md', maturity: 'lab', emittedBy: ['examples/proof/*/receipts/aggregate.json'], shape: 'Aggregate receipt with per-arm medians, derived deltas, quality rubric metadata, and optional cold-resume packet contract for a bounded proof fixture.' }, { id: 'open-scaffold.evaluation.v1', owner: 'src/cli.ts + src/evolution.ts', maturity: 'lab', emittedBy: ['osc eval init', 'osc evolve record --evaluation'], shape: 'Acceptance-criteria evaluation envelope scaffold or external scorer summary reference; structural input only, not approval evidence.' }, { id: 'open-scaffold.evolution-loop.v1', owner: 'src/evolution.ts', maturity: 'lab', emittedBy: ['osc evolve init'], shape: 'Loop directory state for objective, subject, strategy, created time, boundary flags, and stop conditions.' }, { id: 'open-scaffold.evolution-attempt.v1', owner: 'src/evolution.ts', maturity: 'lab', emittedBy: ['osc evolve record'], shape: 'Append-only attempt journal entries with run/evidence refs, decision, score, rationale, optional usage, and retry repair-hypothesis control fields.' }, diff --git a/tests/framework-cleanup-metric.test.ts b/tests/framework-cleanup-metric.test.ts index 0d34cb73..117052f5 100644 --- a/tests/framework-cleanup-metric.test.ts +++ b/tests/framework-cleanup-metric.test.ts @@ -30,8 +30,11 @@ const cleanupBaselineLoc = 20_890; // 170 review hardening: unknown token totals stay null, transcript captures stay not-spawned, and split final text is concatenated (+5 LOC) // 170 review hardening: default captures move to gitignored .osc/state/ambient and help text follows (+1 LOC) // Public readiness hardening adds first-run proof-boundary guidance (+6 LOC). -// 173 Codex token-efficiency proof review registers schema and handles zero/near-threshold ratios (+3 LOC). -const cleanupTargetLoc = 15_151; +// 173 evidence-battery hardening: fail-closed manifest rows for fixture/replication/ablation status and packet-contract gating (+31 LOC). +// 173 Codex review fix: empty evidence battery reports not_evaluated (nullable pass) instead of overstated pass (+1 LOC). +// 173 Codex review fix: required_evidence manifest list rejects omitted required rows (+10 LOC). +// 173 Codex review fix: render required_evidence rows as required in the proof-battery table (+1 LOC). +const cleanupTargetLoc = 15_194; const cleanupTargetFiles = 40; interface MaintainedSourceFile { @@ -74,7 +77,7 @@ describe('framework cleanup maintained-source metric', () => { expect(files.map((file) => file.path)).toContain('src/cli.ts'); expect(files.map((file) => file.path)).toContain('packages/runtime-omx/src/index.ts'); expect(files.every((file) => maintainedRoots.some((root) => file.path === root || file.path.startsWith(`${root}/`)))).toBe(true); - expect(cleanupTargetLoc).toBe(15_151); + expect(cleanupTargetLoc).toBe(15_194); expect(totalLoc).toBeLessThanOrEqual(cleanupTargetLoc); expect(totalLoc).toBeLessThanOrEqual(cleanupBaselineLoc); expect(files.length).toBeLessThanOrEqual(cleanupTargetFiles); diff --git a/tests/package-payload.test.ts b/tests/package-payload.test.ts index 216e73a5..d40be5d8 100644 --- a/tests/package-payload.test.ts +++ b/tests/package-payload.test.ts @@ -89,6 +89,8 @@ describe('npm package payload', () => { expect(paths).toContain('examples/proof/codex-token-efficient-resume/manifest.json'); expect(paths).toContain('examples/proof/codex-token-efficient-resume/receipts/aggregate.json'); expect(paths).toContain('examples/proof/codex-token-efficient-resume/prompts/scaffolded-resume-capsule-prompt.md'); + expect(paths).toContain('examples/proof/codex-token-efficient-resume/evidence/human-reviewer-replication-boundary.md'); + expect(paths).toContain('examples/proof/codex-token-efficient-resume/evidence/controlled-ablations-boundary.md'); // resume-demo fixture must ship so the zero-context-resume demo works from a fresh install if (existsSync(join(repoRoot, 'examples', 'resume-demo'))) { diff --git a/tests/proof.test.ts b/tests/proof.test.ts index 388458a0..7844353e 100644 --- a/tests/proof.test.ts +++ b/tests/proof.test.ts @@ -74,6 +74,17 @@ function manifest(root: string, overrides: Record = {}) { source_refs: ['evidence/scaffolded.json'], }, ], + evidence_battery: [ + { + id: 'cold-resume-fixture', + kind: 'cold_resume_fixture', + status: 'demonstrated', + required_for_pass: true, + claim: 'This test fixture has source-labeled comparison receipts.', + boundary: 'Synthetic unit-test fixture only; not a product benchmark.', + source_refs: ['evidence/control.json', 'evidence/scaffolded.json'], + }, + ], caveats: [ 'Bounded fixture proof only; not a universal benchmark.', 'Receipts are source-labeled instead of inferred.', @@ -106,6 +117,10 @@ describe('proof comparison harness', () => { expect(result.summary.categories.speed).toBe('improved'); expect(result.summary.categories.evolution).toBe('improved'); expect(result.summary.boundedProof).toBe(true); + expect(result.summary.evidenceBatteryPass).toBe(true); + expect(result.evidenceBattery.map((item) => [item.id, item.status, item.requiredForPass])).toEqual([ + ['cold-resume-fixture', 'demonstrated', true], + ]); expect(result.summary.verdict).toContain('bounded'); }); @@ -202,6 +217,46 @@ describe('proof comparison harness', () => { } }); + it('source-labels the Codex fixture evidence battery and adjacent non-claims', () => { + const fixture = resolve(repoRoot, 'examples/proof/codex-token-efficient-resume'); + const result = compareProofManifest(resolve(fixture, 'manifest.json')); + const aggregate = JSON.parse(readFileSync(resolve(fixture, 'receipts/aggregate.json'), 'utf8')) as { + cold_resume_packet_contract: { id: string; required_fields: string[]; source_refs: string[]; fail_closed_if_missing: boolean }; + }; + + expect(result.summary.boundedProof).toBe(true); + expect(result.summary.evidenceBatteryPass).toBe(true); + expect(result.evidenceBattery.filter((item) => item.requiredForPass).map((item) => item.status)).toEqual([ + 'demonstrated', + 'demonstrated', + ]); + expect(result.evidenceBattery.find((item) => item.kind === 'human_reviewer_replication')).toMatchObject({ + status: 'not_demonstrated', + requiredForPass: false, + }); + expect(result.evidenceBattery.find((item) => item.kind === 'controlled_ablation')).toMatchObject({ + status: 'mixed_not_proven', + requiredForPass: false, + }); + expect(aggregate.cold_resume_packet_contract.id).toBe('cold-resume-packet-v2-readable-closeout'); + expect(aggregate.cold_resume_packet_contract.fail_closed_if_missing).toBe(true); + expect(aggregate.cold_resume_packet_contract.required_fields).toEqual([ + 'plan', + 'objective', + 'action', + 'resume_current_frontier_evaluation', + 'acceptance_summary', + 'reasons', + 'required_next_fields', + 'evidence_refs', + 'authority_boundary', + ]); + expect(aggregate.cold_resume_packet_contract.source_refs).toEqual([ + 'prompts/scaffolded-resume-capsule-prompt.md', + 'generate-fixture.mjs', + ]); + }); + it('enforces declared minimum improvement ratios before passing a bounded proof', () => { const root = fixtureRoot(); const passing = compareProofManifest(manifest(root, { @@ -343,6 +398,214 @@ describe('proof comparison harness', () => { expect(zeroThreshold.metrics.find((metric) => metric.id === 'evolution.frontier_delta')).toMatchObject({ improvementRatio: 'unbounded', minimumRatioPassed: true }); }); + it('fails closed when required proof-battery evidence is blocked or missing source refs', () => { + const root = fixtureRoot(); + const blockedPath = manifest(root, { + evidence_battery: [ + { + id: 'human-reviewer-replication', + kind: 'human_reviewer_replication', + status: 'not_demonstrated', + required_for_pass: true, + claim: 'Human reviewers replicated the result.', + boundary: 'This claim must stay blocked until real human-reviewer receipts exist.', + source_refs: ['evidence/control.json'], + }, + ], + }); + + const blocked = compareProofManifest(blockedPath); + expect(blocked.summary.evidenceBatteryPass).toBe(false); + expect(blocked.summary.boundedProof).toBe(false); + expect(blocked.summary.evidenceBatteryBlocking).toEqual(['human-reviewer-replication']); + expect(blocked.summary.verdict).toContain('evidence-battery'); + + const missingSourcePath = manifest(root, { + evidence_battery: [ + { + id: 'controlled-ablation', + kind: 'controlled_ablation', + status: 'demonstrated', + required_for_pass: false, + claim: 'A controlled ablation exists.', + boundary: 'Synthetic fixture only.', + source_refs: ['evidence/not-committed.json'], + }, + ], + }); + expect(validateProofManifestFile(missingSourcePath).failures.map((failure) => failure.code)).toContain('missing-source-ref'); + + const privateSourcePath = manifest(root, { + evidence_battery: [ + { + id: 'private-human-review', + kind: 'human_reviewer_replication', + status: 'demonstrated', + required_for_pass: false, + claim: 'Private human-review notes exist.', + boundary: 'Synthetic fixture only.', + source_refs: ['.osc-dev/human-notes.json'], + }, + ], + }); + expect(validateProofManifestFile(privateSourcePath).failures.map((failure) => failure.code)).toContain('private-source-ref'); + }); + + it('renders required_evidence rows as required in the proof-battery table', () => { + const root = fixtureRoot(); + const path = manifest(root, { + required_evidence: ['human-reviewer-replication'], + evidence_battery: [ + { + id: 'human-reviewer-replication', + kind: 'human_reviewer_replication', + status: 'demonstrated', + required_for_pass: false, + claim: 'Replicated.', + boundary: 'Bounded fixture only.', + source_refs: ['evidence/control.json'], + }, + { + id: 'controlled-ablations', + kind: 'controlled_ablation', + status: 'mixed_not_proven', + required_for_pass: false, + claim: 'Disclosure only.', + boundary: 'Bounded fixture only.', + source_refs: ['evidence/control.json'], + }, + ], + }); + const result = compareProofManifest(path); + const markdown = renderProofComparison(result, 'markdown'); + // The required_evidence row is labeled required even though required_for_pass is false. + expect(markdown).toContain('| human-reviewer-replication | human_reviewer_replication | demonstrated | required |'); + // A row that is neither required_for_pass nor in required_evidence stays disclosure-only. + expect(markdown).toContain('| controlled-ablations | controlled_ablation | mixed_not_proven | disclosure-only |'); + }); + + it('normalizes whitespace in required_evidence IDs before gating', () => { + const root = fixtureRoot(); + const path = manifest(root, { + required_evidence: [' human-reviewer-replication '], + evidence_battery: [ + { + id: 'human-reviewer-replication', + kind: 'human_reviewer_replication', + status: 'not_demonstrated', + required_for_pass: false, + claim: 'Not yet replicated.', + boundary: 'Bounded fixture only.', + source_refs: ['evidence/control.json'], + }, + ], + }); + const result = compareProofManifest(path); + expect(result.summary.evidenceBatteryRequired).toEqual(['human-reviewer-replication']); + expect(result.summary.evidenceBatteryBlocking).toContain('human-reviewer-replication'); + expect(result.summary.evidenceBatteryStatus).toBe('fail'); + expect(result.summary.boundedProof).toBe(false); + }); + + it('blocks a required_evidence ID whose row is present but not demonstrated, even when required_for_pass is false', () => { + const root = fixtureRoot(); + const path = manifest(root, { + required_evidence: ['human-reviewer-replication'], + evidence_battery: [ + { + id: 'human-reviewer-replication', + kind: 'human_reviewer_replication', + status: 'not_demonstrated', + required_for_pass: false, + claim: 'Not yet replicated.', + boundary: 'Bounded fixture only.', + source_refs: ['evidence/control.json'], + }, + ], + }); + const result = compareProofManifest(path); + expect(result.summary.evidenceBatteryBlocking).toContain('human-reviewer-replication'); + expect(result.summary.evidenceBatteryStatus).toBe('fail'); + expect(result.summary.boundedProof).toBe(false); + + // Same row flipped to demonstrated passes the gate. + const passingPath = manifest(root, { + required_evidence: ['human-reviewer-replication'], + evidence_battery: [ + { + id: 'human-reviewer-replication', + kind: 'human_reviewer_replication', + status: 'demonstrated', + required_for_pass: false, + claim: 'Replicated.', + boundary: 'Bounded fixture only.', + source_refs: ['evidence/control.json'], + }, + ], + }); + const passing = compareProofManifest(passingPath); + expect(passing.summary.evidenceBatteryStatus).toBe('pass'); + expect(passing.summary.boundedProof).toBe(true); + }); + + it('rejects a manifest that declares required_evidence but omits the row', () => { + const root = fixtureRoot(); + const missingRowPath = manifest(root, { + required_evidence: ['cold-resume-packet-contract'], + evidence_battery: [ + { + id: 'codex-2x-cold-resume-replicates', + kind: 'cold_resume_fixture', + status: 'demonstrated', + required_for_pass: true, + claim: 'Replicates demonstrated.', + boundary: 'Bounded fixture only.', + source_refs: ['evidence/control.json'], + }, + ], + }); + // Omission is a structural validation failure: osc prove check reports it, + // and osc prove compare throws before computing a verdict. + expect(validateProofManifestFile(missingRowPath).failures.map((failure) => failure.code)).toContain('missing-required-evidence-row'); + expect(() => compareProofManifest(missingRowPath)).toThrow('missing-required-evidence-row'); + + const presentPath = manifest(root, { + required_evidence: ['codex-2x-cold-resume-replicates'], + evidence_battery: [ + { + id: 'codex-2x-cold-resume-replicates', + kind: 'cold_resume_fixture', + status: 'demonstrated', + required_for_pass: true, + claim: 'Replicates demonstrated.', + boundary: 'Bounded fixture only.', + source_refs: ['evidence/control.json'], + }, + ], + }); + const present = compareProofManifest(presentPath); + expect(present.summary.evidenceBatteryRequired).toEqual(['codex-2x-cold-resume-replicates']); + expect(present.summary.evidenceBatteryStatus).toBe('pass'); + expect(present.summary.boundedProof).toBe(true); + + const invalidPath = manifest(root, { required_evidence: 'not-an-array' }); + expect(validateProofManifestFile(invalidPath).failures.map((failure) => failure.code)).toContain('invalid-required-evidence'); + }); + + it('reports evidence battery as not evaluated when a manifest supplies no battery rows', () => { + const result = compareProofManifest(resolve(repoRoot, 'examples/proof/scaffold-vs-naked-codex/manifest.json')); + const markdown = renderProofComparison(result, 'markdown'); + + expect(result.summary.evidenceBatteryPresent).toBe(false); + expect(result.summary.evidenceBatteryPass).toBeNull(); + expect(result.summary.evidenceBatteryStatus).toBe('not_evaluated'); + expect(result.summary.evidenceBatteryBlocking).toEqual([]); + expect(result.summary.boundedProof).toBe(true); + expect(markdown).toContain('Evidence battery: not evaluated'); + expect(markdown).toContain('no evidence_battery rows supplied'); + expect(markdown).not.toContain('passes required evidence-battery items'); + }); + it('renders an honest markdown report with source refs and non-universal caveats', () => { const root = fixtureRoot(); const result = compareProofManifest(manifest(root)); @@ -353,6 +616,9 @@ describe('proof comparison harness', () => { expect(markdown).toContain('quality.acceptance_passes'); expect(markdown).toContain('evidence/control.json'); expect(markdown).toContain('evidence/scaffolded.json'); + expect(markdown).toContain('Proof battery'); + expect(markdown).toContain('cold-resume-fixture'); + expect(markdown).toContain('required'); expect(markdown).toContain('not a universal benchmark'); expect(markdown).not.toMatch(/universally better|proves.*anything/i); }); diff --git a/tests/section-parser.test.ts b/tests/section-parser.test.ts index 605717f0..2d67a79a 100644 --- a/tests/section-parser.test.ts +++ b/tests/section-parser.test.ts @@ -251,7 +251,8 @@ This sample must not count as the real section. // 172 release closeout: moved public-readiness package-sync plan to done after npm/GitHub Release publication proof. // 173 planning: added active Codex token-efficiency proof plan. // v1-public-positioning-polish closeout: added done public-positioning polish plan. - expect(hash(planIssueSnapshot)).toBe('55dd29a6c867d789ee41042e20af0f8d720bdc15d394aa086a93a3012b08dcf7'); + // 173 closeout: moved token-efficiency proof plan to done with checked AC evidence and evidence-battery release note. + expect(hash(planIssueSnapshot)).toBe('dd9d7ea7049481bd1e8ad6f397e6c145b6e2c3c6e1a7ea2bd01897fa82a8b23b'); expect(scaffold.failures).toEqual([]); // 168: evidence note 2026-06-12-168-dollar-verb-retirement.md added to .osc/releases. // 170 review hardening: evidence note refreshed with PR URL, final test count, and Codex-reported total_tokens. @@ -260,7 +261,8 @@ This sample must not count as the real section. // 172 release closeout: finalized v0.32.1 publication evidence note; release-warning set unchanged. // 173 planning plus proof provenance/rubric updates changed the live scaffold corpus hash. // v1-public-positioning-polish closeout added local evidence note; release-warning set unchanged. - expect(hash({ failures: scaffold.failures, releases: releaseOutcomeSnapshot })).toBe('6e07f75f90220e2b484338bb75520b751202dae6b29a58d226e4ef603db46f24'); + // 173 closeout: added source-labeled proof-battery evidence note with close decision. + expect(hash({ failures: scaffold.failures, releases: releaseOutcomeSnapshot })).toBe('30d1492a0e5f3c0a29a1d9a202dcf97f5808a680b78ff32bd7d9b504180af323'); expect(realPlanFiles().every((path) => statSync(path).isFile())).toBe(true); }); });