Reorg to training/{sft,rl} + binary-parquet verl loader (patch)#6
Open
bsabri wants to merge 7 commits into
Open
Reorg to training/{sft,rl} + binary-parquet verl loader (patch)#6bsabri wants to merge 7 commits into
bsabri wants to merge 7 commits into
Conversation
Group the training pipelines under training/ (sft/ -> training/sft, grpo/ -> training/rl), rename misc_eval_utils/ -> evaluation/, and move OMNI_TRAINING.md, PARQUET_TRAINING.md + new CODE_STRUCTURE.md under docs/. Update all path references in run_grpo.sh/run_tarpo.sh, README, docs, and .gitignore. The verl/ submodule stays at the repo root.
The verl/ submodule is pinned to DDVD233/verl@hba_public_release, which lacks the embedded-binary-parquet loader (lazy byte decode of audio/video/image, modality-batching sampler fast-path, Qwen2.5-Omni model detection). Carry those changes as patches/0001-hba-binary-parquet-loader.patch and apply them on top of the pinned submodule via scripts/setup_verl.sh (idempotent) — no write access to the verl fork required. Wire the setup step into README, OMNI_TRAINING, and CODE_STRUCTURE.
HARPO (adv_estimator=harpo): port compute_harpo_outcome_advantage into the pinned verl submodule and activate the ray_trainer branch, shipped as patches/0002-hba-harpo-advantage.patch and applied by scripts/setup_verl.sh (validated with a 5-step run on 4x H20). Rename the RL launcher + reward to HARPO (run_tarpo.sh -> run_harpo.sh, human_behaviour_tarpo.py -> human_behaviour_harpo.py). All training launchers now use the FULL dataset: GRPO/HARPO train on every train shard and validate on the full validation split (glob instead of a 32..292 range + single val shard); SFT configs use the full train/validation splits (not [:10%]/[:5%]). setup_verl.sh applies every patches/*.patch; 0001 (loader) regenerated with the full-data launcher. Docs updated (README, OMNI_TRAINING, CODE_STRUCTURE).
Remove the duplicate GRPO launcher: the loader patch now deletes the submodule's examples/grpo_trainer/_train_grpo_hba.sh, leaving training/rl/run_grpo.sh as the single GRPO entry point (sibling of run_harpo.sh). Repoint the README RL section to training/rl/run_grpo.sh / run_harpo.sh.
…a -> train) Consolidate the scattered setup steps into one quickstart with the verl patching step (scripts/setup_verl.sh + the two patches) front-and-center, plus a sanity check. Link it from the README Training section.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reorganizes the repo into a clearer top-level layout and makes the omni / fork GRPO+SFT path runnable on the parquet distribution of HBA — without needing write access to the verl fork.
1. Folder reorg (no behavior change)
sft/→training/sft/grpo/→training/rl/misc_eval_utils/→evaluation/OMNI_TRAINING.md,PARQUET_TRAINING.md, and a newCODE_STRUCTURE.md→docs/run_grpo.sh/run_tarpo.sh, the README, docs, and.gitignore. Theverl/submodule stays at the repo root.2. Binary-parquet data loader (the missing piece)
The HBA parquet embeds media (audio/video/image) as raw bytes (
list<binary>). The pinnedverl/submodule (DDVD233/verl@hba_public_release) only handles string-path media, so training directly on the parquet fails withpyarrow.lib.ArrowInvalid: Invalid UTF8 payload.Rather than modifying the pinned submodule, this PR carries the loader as an applyable patch:
patches/0001-hba-binary-parquet-loader.patch— lazy binary decode (_detect_binary_media/_read_files_lazy/_lazy_get_row), raw-bytesbranches inaudio_utils/vision_utils, the newcount_mm_tokens.py, the modality-sampler lazy fast-path inmain_ppo.create_rl_sampler, and Qwen2.5-Omni model detection (_is_qwen_omni, byconfig.jsonmodel_type).scripts/setup_verl.sh— idempotently inits the submodule and applies the patch. No write access to the verl fork required.Run once after cloning:
Validation (4× NVIDIA H20)
Notes
verl/submodule pointer is unchanged (still pinned athba_public_release); the fix is applied on top via the patch.Invalid UTF8 payload.docs/CODE_STRUCTURE.mdfor the full layout and a description of the loader additions.