feat(pipeline): finalize on best iteration, not last#28
Merged
Conversation
Accuracy can regress across retune passes (e.g. 0.39 at iter 3 but 0.37 at the forced proceed), yet each pass overwrote predictions/eval/classifier, so explain + finalize ran on the regressed artifacts. - evaluate node snapshots PREDS/EVAL/CODE to outputs/best/ on each new best - new select_best node on the proceed branch restores that snapshot and redraws sample_for_explanation.csv when the last iteration wasn't the best - extract Manager's sampling into shared write_sample() so the sample format keeps a single definition Gate semantics unchanged: the Manager still decides from the last iteration's report; restore happens after the verdict.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Accuracy can regress across retune passes — a real run showed history [0.21, 0.22, 0.24, 0.39, 0.37, 0.37]: iteration 3 hit 0.39, but the loop force-proceeded at 0.37, and each pass overwrites predictions_test.csv / evaluation_report.json / classifier.py. So explain + finalize ran on the regressed artifacts and the best iteration was lost.
Change
evaluatenode snapshots PREDS/EVAL/CODE tooutputs/best/on each new accuracy best (tracked viabest_accuracy/last_accuracyin graph state).select_bestnode on the proceed branch:gate ─(proceed)→ select_best → explain. If the last iteration wasn't the best, it restores the snapshot over the canonical paths and redrawssample_for_explanation.csvfrom the restored predictions.write_sample()so the sample format keeps a single definition.Gate semantics unchanged: the Manager still decides from the last iteration's report; the restore happens after the verdict.
outputs/best/is internal to the loop, not a new contract handoff — no data_contracts.md change.Testing
New test
test_best_iteration_restored_when_accuracy_regresses: fake Sabina emits a peak-then-regress accuracy sequence, fake Nadi stamps each pass into the predictions file; asserts the surviving artifacts are the best pass's andfinal_report.final_accuracyreflects it. Full suite: 34 passed.Note
Stacked on #27 (
feature/finetune-finbert) — merge that first; this PR then contains only f29a14a.