[data, rollout, worker] feat: add Open-R1 multimodal and TinyLLaVA-Video-R1 preprocessing and training scripts

[lihanwen7]

What does this PR do?

Add GRPO training support for two multimodal datasets:

• lmms-lab/multimodal-open-r1-8k-verified — image math reasoning, <think>/<answer> format
• Zhang199/TinyLLaVA-Video-R1-training-data — video multiple-choice QA

bugfixes included:

• verl/workers/engine_workers.py — add maybe_fix_3d_position_ids inside train_mini_batch loop to fix _ragged_idx corruption

Checklist Before Starting

• Search for similar PRs:
  https://github.com/verl-project/verl/pulls?q=is%3Apr+openr1mm
  https://github.com/verl-project/verl/pulls?q=is%3Apr+tinyllava

Test

Both datasets trained on GPU and NPU. NPU training curves:

lmms-lab/multimodal-open-r1-8k-verified:

Zhang199/TinyLLaVA-Video-R1-training-data:

API and Usage Example

# Preprocess
python examples/data_preprocess/openr1mm.py --local_save_dir ~/data/openr1mm
python examples/data_preprocess/tinyllava_video_r1.py \
    --data_dir ~/data/tinyllava-video-r1 --local_save_dir ~/data/tinyllava_video_r1

# Train
bash examples/grpo_trainer/run_qwen3_5_2b_openr1_fsdp.sh
bash examples/grpo_trainer/run_qwen3_5_2b_video_fsdp.sh

Design & Code Changes

• examples/data_preprocess/openr1mm.py — HF dataset → verl parquet, image bytes preserved
• examples/data_preprocess/tinyllava_video_r1.py — JSONL → verl parquet, video file paths
• examples/grpo_trainer/run_qwen3_5_2b_openr1_fsdp.sh — training script for image dataset
• examples/grpo_trainer/run_qwen3_5_2b_video_fsdp.sh — training script for video dataset
• verl/workers/engine_workers.py — maybe_fix_3d_position_ids in train_mini_batch loop

Checklist Before Submitting

• Read the Contribute Guide
• Apply pre-commit checks
• Request CI in ci-request

[gemini-code-assist]

Code Review

This pull request adds preprocessing scripts and GRPO training configurations for the Open-R1 multimodal math and TinyLLaVA-Video-R1 datasets, along with related fixes in the agent loop and engine workers. The review feedback highlights critical issues: the referenced custom reward score scripts are missing from the repository, and the Open-R1 preprocessing script needs to disable automatic image decoding and explicitly select columns to prevent saving large, decoded images in the output Parquet files.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.