Skip to content

Add xToken-aligned cross-tokenizer OPD - #13

Draft
artem-osmosis wants to merge 132 commits into
devfrom
agent/cross-tokenizer-opd
Draft

Add xToken-aligned cross-tokenizer OPD#13
artem-osmosis wants to merge 132 commits into
devfrom
agent/cross-tokenizer-opd

Conversation

@artem-osmosis

Copy link
Copy Markdown

Summary

  • port Slime's OPD feature parity into Miles, including sampled-token and top-k training data/loss paths, validation, metrics, and eval reward dispatch
  • add xToken-style cross-tokenizer alignment with independent student/teacher chat templates, strict/retried teacher scoring, zero-signal masking, and alignment coverage telemetry
  • preserve OPD fields through multi-turn, CP, and multimodal preprocessing, including response-side media expansion safeguards
  • add the single-node 8×H200 Llama-3.2-3B / Nemotron-3-Super-120B launcher with scoped Ray/teacher cleanup, safe path guards, and a shared sglang-miles API preflight
  • document the new modes and add focused unit coverage

Validation

  • 8×H200 end-to-end run from checkpoint iteration 79 using sglang-miles-v0.5.13-final at f8cfad35bdfeeed09b259857e738bf833aeb70e0
  • native /begin_weight_update and /end_weight_update transactions returned 200; zero begin 404s and zero legacy /post_process_weights calls
  • 10 optimizer steps (80–89), 10 alignment batches, zero alignment fallbacks
  • prescribed reverse-KL trend check:
    • start: 0.3680919334
    • end: 0.2980055362
    • delta: -0.0700863972
    • first-half mean: 0.3562976375
    • second-half mean: 0.3052670628
    • slope: -0.0089484361 / step
    • start-in-range: true
  • W&B: https://wandb.ai/yatsenkoa/miles-opd/runs/7ow8s81i
  • relevant suite: 149 passed, 1 skipped
  • CI registration suite: 60 passed
  • ruff check, git diff --check, and launcher bash -n pass
  • incompatible SGLang source, overlong Ray socket path, unsafe/pre-existing Ray temp directory, and BSHD+CP multimodal cases fail closed before unsafe work

gongyisheng and others added 30 commits April 4, 2026 16:45
…eight when colocate, sync lora+base for disaggregate mode at first time then only sync lora after that
mathewjhan and others added 29 commits June 4, 2026 23:33
Sync osmosis/dev with upstream radixark/miles main (254091f, 2026-07-06).
51 upstream commits; 212 files (+10066/-1205). 8 content conflicts resolved.

Conflict resolutions (strategy: preserve osmosis behavior, port to upstream radixark#1329 API):
- session/sessions.py: took upstream thin-adapter refactor (radixark#1510/radixark#1518/radixark#1563/radixark#1569);
  osmosis custom features (routing/indexer replay, R3 stripping) already live in core.py.
- ray/rollout/train_data_conversion.py: kept BOTH osmosis adapter_slots + upstream opd_reverse_kl.
- update_weight/common.py: dropped duplicate _check_weight_sync_results + orphaned
  post_process_weights tail (removed upstream by radixark#1329).
- update_weight/update_weight_from_tensor.py: adopted begin/end_weight_update import;
  ported update_multi_lora_weights post_process_weights() -> end_weight_update().
- update_weight/update_weight_from_distributed/{broadcast,mixin,p2p}.py: kept osmosis
  multi-LoRA, never-sync-base, NCCL contiguity fix, and p2p LoRA support.
- tests/.../test_lora_weight_sync_validation.py: took upstream signature (mock_common_ray
  for the new common.ray patch).

Local static checks: py_compile + ruff (update_weight/ + session/) pass; all common
imports resolve. Dynamic pytest tests/fast requires the miles container (torch/ray) — run in CI.
Pre-existing black (24.3.0, line-length 119) violations inherited from dev,
surfaced by `pre-commit run --all-files` on the PR. Not touched by the upstream
merge — formatting only, no logic change:
- miles/ray/rollout/metrics.py: add blank line before log_rollout_data
- miles/rollout/multi_lora_data_source.py: collapse _create_adapter_source signature
Sync upstream (radixark/miles main) into dev
* fp8: block-fp8 storage of the frozen LoRA base with per-layer free

* gated canonical lora: gate-aware q sizing, TP-safe qkv interleave, no MTP adapters

* example: Qwen3.5-35B-A3B MoE-LoRA colocate with the fp8 frozen-base store

* ci: LoRA e2e test with --fp8-frozen-base-store enabled

---------

Co-authored-by: MuuSeoTia <tiahi.m@northeastern.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants