KL-001 STATUS: record the v0.3 outcome, which the file still denied existed - #58
Merged
Conversation
…xisted
STATUS.json said state=preregistered, resultStatus=none, after v0.3 had run and
met its registered primary endpoint. The machine-readable status asserted no
results existed while the results sat beside it in the same directory. Anything
reading status rather than prose would have concluded the experiment had not run.
Recorded:
primary false-clean rate, registered strictly below 0.0426, observed 0.0213
MET
secondary recall at least 0.753, observed 0.7926, MET -- but it could not have
failed, since both arms run the identical scanner and recall is equal
by construction (BL-062)
With what the number is worth attached to it, rather than left for a reader to
infer: two false cleans became one, n=1, and of the four repositories moved to
not_established, one was defective and three were clean. The registered endpoint
counts defective repositories only and is structurally blind to the three
refusals.
claimAllowed now separates the two claims the design note separated. ESTABLISHED
by exhaustive enumeration rather than sampling: the absence verdict is a total
function of two bits, four cells verified, each input load-bearing. NOT
ESTABLISHED: that the dual ledger reduces false cleans in practice.
nextGate is None on a synthetic corpus, with the reason -- frozen-v3 is cancelled
because every rate over an authored population is a generator setting read back.
Two blockers discharged, one honestly rather than cleanly: the recall blocker was
discharged by measuring the baseline and then RETIRED, because the target turned
out unfalsifiable for a different reason than the one the blocker named. The
mapping layer exists. One blocker added that this experiment cannot solve: a
real-repository run needs defect ground truth for real repositories.
The previous commit set state to 'results-reported', which is not in the kernel ladder. The program test caught it -- the test named test_no_experiment_claims_progress_without_the_evidence_for_it, which exists precisely to stop a status field being edited into a claim. I had pushed with it failing; the monitor flagged both the test and the unexpected kernel state. The correct rung is fixture-passed. v0.3 ran against a frozen fixture corpus and met its registered primary endpoint at 0.0213 against strictly-below-0.0426. That is what the fixture gate is. The rung records which gate was passed. claimAllowed records what may be concluded, and it forbids the practical claim, because the registration was afterwards found defective: the endpoint is one-sided and every rate over an authored corpus is a generator setting read back. Keeping those two separate is why the schema has both fields, and it is the honest way to advance a rung on a result whose interpretation has since narrowed. 323 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STATUS.jsonsaidstate: preregistered,resultStatus: none— after v0.3 had run and met its registered primary endpoint. Anything reading status rather than prose would have concluded the experiment never ran.Recorded
With what the number is worth attached rather than left to inference: two false cleans became one, n=1, and of the four repos moved to
not_established, one was defective and three were clean. The endpoint counts defective repos only and is structurally blind to those three.claimAllowednow splits the two claimsnextGate: none on a synthetic corpusfrozen-v3cancelled — every rate over an authored population is a generator setting read back.Blockers
Two discharged, one of them honestly rather than cleanly: the recall blocker was discharged by measuring the baseline and then retired, because the target turned out unfalsifiable for a different reason than the blocker named. The mapping layer exists.
One added that this experiment cannot solve: a real-repository run needs defect ground truth for real repositories.