Smoke run for OAPL

OAPL (Optimal Advantage-based Policy Optimization with a Lagged Inference Policy)

OAPL is an off-policy algorithm for LLM RL post-training, which uses the closed-form solution to KL-regularized RL to explicitly minimize the divergence between the trainer and inference policies, allowing for effective training with highly off-policy data. This repository contains the code for the offline training/code-generation experiments replicating the performance of DeepCoder, and is built on VeRL.

Smoke run

Here we add capability to do smoke run of OAPL on CPU or single machine GPU to test the algorithm.

---

Installation

uv sync
# activate
source .venv/bin/activate

Offline dataset Prepare

./scripts/data/smoke_generate_offline_dataset_cpu.sh

Training

AUTO_GENERATE=0 REFRESH_GENERATED=0 ./scripts/smoke_oapl_offline_cpu.sh