Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,12 @@ every commit is world-readable and every merge to `main` is a publication act.
- The calibration protocol family now has separate E4 grader-validation, E1
Lane A A/A session-position, and E2 operating-characteristic candidate SAPs.
They are not preregistered, authorize no observations, and select no powered
method. Their counts, thresholds, grids, and consequences remain review
candidates until separately frozen and registered.
method. The 2026-07-17 design decisions are incorporated: E1 uses 120
cross-arm pairs with conservative stage-localized falsification, E4 certifies
overall adjudicated-positive/negative populations under asymmetric thresholds,
and E2 uses rule-relative nuisance control with exact or simultaneous-bound
admission. Exact corpus strata, immutable implementations, schemas, and other
named freeze inputs remain registration blockers.
- Contract B `2.0.0-draft.1` is an immutable, repository-only normative draft
beside v1. It binds synthetic registration, reveal, schedule, attempts,
attrition, copied observation provenance, analysis, decision, and report
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,10 @@ Neither is a current statistical result. The smoke artifact retains summary valu
raw per-attempt outcomes needed for independent interval recomputation. Their recorded values are
historical artifacts, not evidence for model selection.

The candidate calibration family incorporates the 2026-07-17 statistical design decisions but is
still unregistered and non-authorizing. Its remaining freeze fields and operational prerequisites
must be completed outside any observation before a study can begin.

### Verify locally

Install the exact Quarto release named in `.quarto-version` (on macOS,
Expand Down
8 changes: 4 additions & 4 deletions STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ ready for a powered scientific promotion decision or production use.

| Item | Evidence | Status |
|---|---|---|
| Public site | Render run `29614595319` and Pages run `29614595347` passed for current published baseline `6aabf7f` | published |
| Public site | Render run `29657510137` and Pages run `29657510147` passed for current published baseline `5423caa` | published |
| Core version | `pyproject.toml` and frozen core identify `0.2.0`; wheel containment is tested | repository-versioned, not distributed — the version names code in this repository, not a release anyone can install from a registry |
| Experimental calculations | Committed synthetic results regenerate byte-for-byte | decision input only — the numbers inform, but do not make, the pending method decision |
| Calibration SAP family | Separate E4, E1, and E2 candidate protocols define reviewable objectives, counts, failure rules, and claim boundaries | not preregistered; no observations authorized; numerical choices remain candidates |
| Calibration SAP family | Separate E4, E1, and E2 candidate protocols incorporate the 2026-07-17 statistical decisions and retain explicit freeze blockers | not preregistered; no observations authorized; exact corpus strata, implementations, schemas, and operational bindings remain incomplete |
| Contract B v2 draft | Exact schema, verifier, renderer, safety scanner, fixture, report, and delivery manifest are digest-bound (every implementation byte is hash-pinned) | synthetic conformance only — the draft evidence contract is validated against invented data, not real campaigns |
| Independent refusal | The public verifier derives `NOT_EVALUABLE` from the synthetic fixture and rejects mutations — it recomputes the result from frozen evidence bytes and refuses any altered copy | verified for the fixture |
| Publication scanning | Evidence inputs are recursively scanned in the parameterized report path; Contract B checks its bound public payload separately | path-scoped; no whole-repository scan — leak scanning covers the report inputs, not every file in the repository |
Expand Down Expand Up @@ -73,5 +73,5 @@ conformance vector are unchanged.

## Provenance

- Status date: 2026-07-17
- Baseline reviewed for this candidate update: `6aabf7f32e3d824ee63f2eef3a761f29be4e2d4c`
- Status date: 2026-07-18
- Baseline reviewed for this candidate update: `5423caa370664f1e34101e8e812397f6a9608e1a`
62 changes: 39 additions & 23 deletions docs/preregistrations/calibration-sap.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,43 +2,59 @@

## Metadata

- **Date:** 2026-07-17
- **Version:** 1.1.0-candidate.1
- **Status:** Candidate protocol — not preregistered; no observations authorized.
- **Decision date:** 2026-07-17
- **Last updated:** 2026-07-18
- **Version:** 1.2.0-candidate.2
- **Status:** Candidate protocol — design decisions incorporated; not preregistered; no observations authorized.

## Purpose

This document is the index and common claim boundary for three separate candidate statistical
This document is the index and common public claim boundary for three separate statistical
analysis plans:

1. [E4 grader validation](e4-grader-validation-sap.md)
2. [E1 Lane A A/A session-position study](e1-lane-a-aa-sap.md)
3. [E2 operating-characteristics simulation](e2-operating-characteristics-sap.md)

The former umbrella draft combined three different studies and incorrectly described the E1
objective as estimation of within-session correlation. The replacement family uses Lane A `m = 1`
by construction: one task pair per fresh session. E1 is therefore a determinism/falsification study,
not a correlation-estimation study.
Lane A uses `m = 1` by construction: one task pair per fresh session. E1 is therefore an
exact-identity falsification study, not a correlation-estimation study. E4 validates the grader
against independently adjudicated human labels. E2 evaluates candidate decision rules and later
inherits accepted E4 and E1 inputs for sensitivity analysis.

## Common rules
## Common public rules

- E4, E1, and E2 require separate reviewed registrations and separate result artifacts.
- Planned values in these documents are design candidates, not produced observations.
- Every initiated E1 attempt is retained as exactly one terminal attempt row. Failure, timeout,
resource exhaustion, receipt failure, and harness failure are outcomes; none is silently dropped.
- No whole-attempt automatic retry is allowed. A separately authorized rerun must use a new attempt
ordinal and a new registration rather than replacing the original record.
- Each gateway call has a planned provider retry limit of zero. A client invocation may contain
multiple disclosed calls; all calls must be included in its closure record.
- No fallback model, direct provider path, or unregistered schedule substitution is allowed.
- A missing registration, byte binding, assignment row, owner receipt, invocation closure, terminal
attempt row, or required analysis artifact makes the affected study `NOT_EVALUABLE`.
- Values in these documents are frozen design inputs only after their remaining named fields are
completed and registered. They are not produced observations.
- Every initiated attempt is retained as exactly one terminal attempt row. Failure, timeout,
resource exhaustion, provenance failure, and harness failure are outcomes; none is silently
dropped.
- No whole-attempt automatic retry is allowed. A separately authorized rerun uses a new attempt
ordinal under a new registration rather than replacing the original record.
- No unregistered schedule, rule, corpus, or serving-profile substitution is allowed.
- A missing registration, byte binding, assignment row, required provenance record, terminal row,
or analysis artifact makes the affected study `NOT_EVALUABLE`.
- Every study binds one versioned serving-profile digest. Public analysis treats that profile as an
opaque registered input; deployment-specific values, credentials, and authority remain outside
this public methodology.
- Calibration results cannot promote a system, select a powered method, or settle a minimum
practically important benefit. Those require a later coupled ruling and a new protocol.

## Dependency order and phasing

E4 and E1 may proceed independently after their own registrations. E2 has two phases:

1. the base operating-characteristics grid uses frozen synthetic distributions and may execute
without E4 or E1 observations; and
2. grader-error and apparatus-sensitivity cells execute only after accepted E4 and E1 result
packets exist.

The sensitivity transformation and input schema must be frozen before the E4/E1 results are
observed. This prevents later results from tuning the sensitivity analysis.

## Current readiness

These documents are reviewable design inputs only. Before any study can be registered, reviewers
must resolve the candidate decisions called out in each SAP, freeze the machine-readable inputs and
implementations by digest, approve the evidence/report contracts, and establish operational
registration and observation authority.
The statistical decisions from the 2026-07-17 review are incorporated below, but the family is not
registered. Registration remains blocked on the incomplete E4 corpus-stratum fields, exact
machine-readable artifacts and implementations, reviewed evidence/report schemas, and separately
established operational authority. Nothing in this public family authorizes execution.
Loading
Loading