Complete the DPO training track: gate orchestration + miner prep - #277
Open
ai-hpc wants to merge 1 commit into
Open
Complete the DPO training track: gate orchestration + miner prep#277ai-hpc wants to merge 1 commit into
ai-hpc wants to merge 1 commit into
Conversation
Makes DPO submissions work end-to-end at the gate (the reward-gate pin selection
and per-track frontier were already merged). SFT stays byte-identical.
- eval/training_track_gate.py: track-aware throughout.
- pr_training_track(): a PR with a `rl: dpo` recipe competes on the DPO track.
- validate_pr_body_canonical_pin(track=): DPO PRs cite the canonical preference
URL + pref_sha256; fails closed when no preference pin is configured.
- _canonical_pref_sha256s_for_pr_window(): preference merge-base grace window
(generalized the sft window; sft/pref wrappers preserve callers).
- verify_remote_proof_bundle + _scores + _download_and_verify_bundle thread
acceptable_pref_shas into verify_submission (which already selects the
preference pin + the <arch>::dpo frontier).
- gate_training_pr detects the track and routes; report carries track +
acceptable_pref_shas.
- eval/prepare_mining_dpo.py + scripts/prepare_mining_dpo.sh: download the canonical
preference dataset, write chosen/rejected jsonl, verify against pref_sha256.
- eval/canonical_dataset.py: canonical_pref_repo_id, pref_sha256_matches_canonical_export.
- Tests: track detection, DPO body-pin citation (accept/reject/fail-closed), prep export.
Deferred activation (documented): publish the canonical preference dataset + write the
pref_manifest pin; ship dpo.yaml once should_enforce distinguishes a new recipe file from
a submission.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the DPO submission path. With the reward-gate pin selection and the per-track frontier already merged, this makes the training gate accept a DPO PR end-to-end — while keeping SFT byte-identical.
What's now track-aware (
eval/training_track_gate.py)pr_training_track(head_ref, recipe_paths)— a PR whose changed recipe isrl: dpocompetes on the DPO track.validate_pr_body_canonical_pin(pr_body, track=...)— a DPO PR must cite the canonical preference dataset URL +pref_sha256(SFT PRs unchanged); fails closed if no preference pin is configured._canonical_pref_sha256s_for_pr_window— preference merge-base grace window (generalized the SFT window; SFT/pref wrappers keep all existing callers).verify_remote_proof_bundle/_scores/_download_and_verify_bundlethreadacceptable_pref_shasintoverify_submission, which already selects the preference pin (check_canonical_dataset_claim) and the<arch>::dpofrontier.gate_training_prdetects the track and routes both pin checks; the report carriestrack+acceptable_pref_shas.Miner prep
eval.prepare_mining_dpo(scripts/prepare_mining_dpo.sh) — downloads the canonical preference dataset and writes the local chosen/rejected jsonl, byte-verified againstpref_sha256.eval/canonical_dataset.py:canonical_pref_repo_id,pref_sha256_matches_canonical_export.Tests
Track detection; DPO body-pin citation (accept / wrong-sha reject / no-pin fail-closed); preference export writes chosen/rejected & rejects incomplete pairs. SFT paths untouched (existing tests green).
Verification
ruff/formatclean ·pyright0 errors ·pytest344 passed local / 335 + 9 skipped CI-emulated.Activation — deliberately deferred (one maintainer step + one policy fix)
A DPO bundle still fails closed until:
pref_manifest.pref_sha256pin is written intodatasets/canonical.json(a maintainer publishing step — the pref-pin writer /update_canonical_pin --dpoand the actual dataset are the remaining piece), anddpo.yamlexample recipe is shipped — which needsshould_enforce_training_gateto distinguish a new recipe file from a proof-bundle submission (otherwise adding the file auto-closes the PR, as with DPO training-track foundation (rl: dpo recipe + track-aware canonical gate) #268).Everything above is safe to land now: SFT is byte-identical and DPO still fails closed without the pin.