Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 23 additions & 13 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,18 +34,24 @@ every commit is world-readable and every merge to `main` is a publication act.
- The committed results artifact is regenerated and compared byte-for-byte in
CI. Exact computation is therefore reproducible; inferential validity still
depends on the stated design and assumptions.
- No enforcing test, replicate policy, or minimum practically important
benefit has been selected for the next contract. Powered promotion remains
unavailable until that coupled decision is made from a complete package.
- E2 now has a repository-frozen equal-class estimand and
`exact_conditional_sign_v1` leading candidate, plus a content-free exact
base-grid manifest. No enforcing test has been admitted or selected for a
powered campaign; replicate policy and minimum practically important benefit
also remain open. Powered promotion remains unavailable until that coupled
decision is made from a complete package.
- The calibration protocol family now has separate E4 grader-validation, E1
Lane A A/A session-position, and E2 operating-characteristic candidate SAPs.
They are not preregistered, authorize no observations, and select no powered
method. The 2026-07-17 design decisions are incorporated: E1 uses 120
cross-arm pairs with conservative stage-localized falsification, E4 certifies
overall adjudicated-positive/negative populations under asymmetric thresholds,
and E2 uses rule-relative nuisance control with exact or simultaneous-bound
admission. Exact corpus strata, immutable implementations, schemas, and other
named freeze inputs remain registration blockers.
admission. The E4 construction schema/checks and E2 leading-rule/base-grid
generator are now repository-versioned and byte-recomputed as synthetic
design inputs. Actual corpus/reviewer bindings, result-producing E2
convolution and certified maximization, report schemas, and operational
inputs remain registration blockers.
- Contract B `2.0.0-draft.1` is an immutable, repository-only normative draft
beside v1. It binds synthetic registration, reveal, schedule, attempts,
attrition, copied observation provenance, analysis, decision, and report
Expand Down Expand Up @@ -77,16 +83,18 @@ every commit is world-readable and every merge to `main` is a publication act.
The experimental note narrows the problem but deliberately leaves these items
open:

1. **Estimand and design.** State whether the target is a fixed finite suite or
a task superpopulation; define the four-class weighting; define arm-order
assignment; and state the unit and source of replication.
1. **Estimand and design.** E2's candidate calibration estimand now uses equal
four-class weighting for a fixed horizon. A later coupled ruling must decide
whether that rule is valid and selected for the powered finite-suite claim,
and must still define arm-order assignment and the unit and source of
replication.
2. **Applicable validity argument.** Establish a theorem or calibrated design
for the actual class-balanced, paired, potentially non-identically
distributed task setting. Results for randomized treatment assignment do not
automatically validate arm-order randomization.
3. **Nuisance-boundary control.** Replace the current finite `0.01` grid
judgment with a justified continuous-boundary argument or an explicitly
conservative bound.
3. **Nuisance-boundary control.** Implement and verify the frozen certified
branch-and-bound design over the full class-specific Cartesian product; the
earlier finite `0.01` grid is not admissible.
4. **Dependence.** For Lane A, use one task pair per fresh session and attempt to
falsify session-position invariance rather than estimate an intra-session
correlation. Retain actual session position and task-pair identifiers; treat
Expand All @@ -99,8 +107,10 @@ open:
deployment-grounded replicate interpretation.
7. **Bounded-mean alternative.** Complete the authorized bounded-mean/betting
interval frontier and compare its assumptions and operating characteristics.
8. **Outcome and missingness.** Fix the grader definition, measurement-error
study, retry/error/timeout policy, attrition table, and protected-task rule.
8. **Outcome and missingness.** E2 now separates gradeable terminal failure
from missing evidence and freezes the leading-rule refusal policy. The
grader definition, measurement-error result, retry/error/timeout policy,
attrition table, and protected-task rule still require registered bindings.
9. **Practical benefit.** Justify any nonzero margin from calibration,
feasibility, and cost/benefit evidence; a preregistered arbitrary constant is
not a scientific justification.
Expand Down
22 changes: 14 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,9 @@ model leaderboard, or deployment service.
- Evidence Contract `1.1.0` and a parameterized Quarto report for historical and synthetic examples.
- A frozen identity-domain v1 field set, canonicalization rule, and cross-implementation vector.
- An executable experimental methods note with committed, byte-checked synthetic results.
- A content-free E4 corpus-construction contract and E2 exact leading-rule/base-grid manifest that
regenerate byte-for-byte without authoring records or producing operating-characteristic
results.
- Evidence contract `2.0.0-draft.1` — a draft evidence and reporting contract that validates a
synthetic registration-to-report join, independently recomputes the result, and refuses
mismatched or incomplete evidence.
Expand Down Expand Up @@ -80,11 +83,9 @@ order to trust or challenge an evaluation.

## What it does not demonstrate

- No accepted successor estimand, enforcing test, replicate policy, or minimum practically
important benefit — in plain terms, the statistical method that will decide future comparisons,
the test that determines whether evidence supports a decision, how many repeated trials are
required, and the minimum improvement worth changing models for are all still awaiting an
explicit decision.
- No admitted successor enforcing test, replicate policy, or minimum practically important benefit.
E2 has an equal-class estimand and exact leading-rule candidate for calibration, but that is not
a powered-method selection or evidence that the rule controls the intended heterogeneous null.
- No validated operating characteristics for the intended class-balanced design.
- No operational registration signer, trusted timestamp path, or production owner-receipt join.
- No validated grader study, no public-safe validity evidence for the governed 40-task suite, no
Expand All @@ -99,6 +100,7 @@ order to trust or challenge an evaluation.
- [Identity-domain specification](identity-domain.qmd)
- [Delivery manifest](evidence/contract-v2/delivery-manifest.v1.json)
- [Candidate calibration protocol family](docs/preregistrations/calibration-sap.md)
- [E4/E2 offline calibration freeze](analysis/calibration-freeze.v1.json)
- [Roadmap](PLAN.md)

The repository contains two historical worked examples:
Expand All @@ -110,9 +112,12 @@ Neither is a current statistical result. The smoke artifact retains summary valu
raw per-attempt outcomes needed for independent interval recomputation. Their recorded values are
historical artifacts, not evidence for model selection.

The candidate calibration family incorporates the 2026-07-17 statistical design decisions but is
still unregistered and non-authorizing. Its remaining freeze fields and operational prerequisites
must be completed outside any observation before a study can begin.
The candidate calibration family incorporates the reviewed statistical design decisions. Its E4
construction schema/checks and E2 leading-rule/base-grid generator are repository-versioned as a
synthetic, content-free design artifact. The family remains unregistered and non-authorizing: no
corpus, observation, grader admission, operating-characteristic result, or scientific verdict is
present, and remaining result-producing and operational prerequisites must be completed before a
study can begin.

### Verify locally

Expand All @@ -137,6 +142,7 @@ validation, and site/report rendering before publication.
| Statistical core `0.2.0` | Repository-versioned; wheel builds locally; not tagged or published to a package registry | Preserves historical estimators and a superiority component; not authorized for a new powered claim |
| Experimental inference note | Published synthetic decision input | No successor method is selected |
| Evidence contract `2.0.0-draft.1` | Immutable repository-only draft with a frozen delivery manifest | Synthetic conformance only; recomputes `NOT_EVALUABLE`; no operational authority |
| E4/E2 offline calibration freeze | Repository-only schema, exact-rule candidate, scenario generator, and byte-checked synthetic manifest | No authored corpus or result; unregistered and non-authorizing |
| Identity domain v1 | Frozen structural specification and conformance vector | Matching hashes are necessary, not sufficient, for numerical comparability |
| Real campaign evidence | One historical four-task smoke summary with a transport confound; raw attempt records are not published | Not independently recomputable, not a benchmark, and not a scientifically sound, adequately powered comparison |

Expand Down
8 changes: 4 additions & 4 deletions STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ ready for a powered scientific promotion decision or production use.

| Item | Evidence | Status |
|---|---|---|
| Public site | Render run `29657510137` and Pages run `29657510147` passed for current published baseline `5423caa` | published |
| Public site | Render run `29667132085` and Pages run `29667132097` passed for current published baseline `c382065` | published |
| Core version | `pyproject.toml` and frozen core identify `0.2.0`; wheel containment is tested | repository-versioned, not distributed — the version names code in this repository, not a release anyone can install from a registry |
| Experimental calculations | Committed synthetic results regenerate byte-for-byte | decision input only — the numbers inform, but do not make, the pending method decision |
| Calibration SAP family | Separate E4, E1, and E2 candidate protocols incorporate the 2026-07-17 statistical decisions and retain explicit freeze blockers | not preregistered; no observations authorized; exact corpus strata, implementations, schemas, and operational bindings remain incomplete |
| Calibration SAP family | Separate E4, E1, and E2 candidate protocols incorporate the reviewed decisions; the content-free E4 construction contract and E2 exact leading-rule/base-grid manifest regenerate byte-for-byte | not preregistered; no observations authorized; no corpus, grader result, operating-characteristic result, or scientific verdict exists |
| Contract B v2 draft | Exact schema, verifier, renderer, safety scanner, fixture, report, and delivery manifest are digest-bound (every implementation byte is hash-pinned) | synthetic conformance only — the draft evidence contract is validated against invented data, not real campaigns |
| Independent refusal | The public verifier derives `NOT_EVALUABLE` from the synthetic fixture and rejects mutations — it recomputes the result from frozen evidence bytes and refuses any altered copy | verified for the fixture |
| Publication scanning | Evidence inputs are recursively scanned in the parameterized report path; Contract B checks its bound public payload separately | path-scoped; no whole-repository scan — leak scanning covers the report inputs, not every file in the repository |
Expand All @@ -31,7 +31,7 @@ therefore identifies repository and wheel metadata, not a distributed release.

| Requirement | Current gap |
|---|---|
| Scientific design | Estimand, enforcing test, replicate policy, and practical-benefit rule remain undecided — what is measured, the test that determines whether evidence supports a decision, how many repeated trials are required, and the minimum improvement worth changing models for are all still awaiting an explicit decision |
| Scientific design | E2 now has a frozen equal-class estimand and leading exact-rule candidate for calibration, but no enforcing test has been admitted or selected for a powered campaign; the replicate policy and minimum practically important benefit also remain undecided |
| Operating characteristics | Type I error (how often the rule would be wrong), power (how often it would detect a real difference), dependence sensitivity, and feasibility are not established for the intended design |
| Measurement validity | Grader validation, protected-task policy, and governed task selection are incomplete — the scoring instrument itself is not yet validated |
| Operational provenance | Registration signing/timestamp verification and runtime owner receipts are not operational — runtime evidence is not yet signed and independently verified in production |
Expand Down Expand Up @@ -74,4 +74,4 @@ conformance vector are unchanged.
## Provenance

- Status date: 2026-07-18
- Baseline reviewed for this candidate update: `5423caa370664f1e34101e8e812397f6a9608e1a`
- Baseline reviewed for this candidate update: `c382065fe19439140e2d9ad7210845748e3f94a2`
1 change: 1 addition & 0 deletions analysis/calibration-freeze.v1.json

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions analysis/calibration/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
"""Repository-only calibration design contracts.

These modules freeze public, synthetic design inputs. They do not authorize an
observation, admit a grader, or establish operating characteristics.
"""
Loading
Loading