build(deps): bump trl from 0.24.0 to 1.4.0

[dependabot]

Bumps trl from 0.24.0 to 1.4.0.

Release notes

Sourced from trl's releases.

v1.4.0

Features

Chunked cross-entropy loss for SFT (up to –50% VRAM)

A new loss_type="chunked_nll" option drastically reduces peak activation memory in SFT by avoiding the full [batch × seq × vocab] logits tensor. Ignored-label tokens are dropped before the lm_head matmul, and the cross-entropy is computed over the remaining tokens in checkpointed chunks (default chunk_size=256, the sweet spot consistent across model sizes and sequence lengths).

from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen3-4B",
args=SFTConfig(loss_type="chunked_nll"),
train_dataset=dataset,
)
trainer.train()

Peak GPU memory, AdamW fp32:

Model	Hardware	Seq	nll	chunked_nll
Qwen3-1.7B + LoRA	1×H100 80GB	2048	47.9 GB	12.3 GB (3.9× less)
Qwen3-4B	1×H100 80GB	16384	OOM	63.8 GB
Qwen3-14B	8×H100 FSDP2	16384	58.9 GB	38.9 GB (1.5× less)
Qwen3-32B	8×H100 FSDP2	8192	OOM	71.2 GB

End-to-end, chunked NLL is consistently as fast or faster than nll — and it unlocks sequence lengths that don't fit at all under the standard path.

The chunked path also supports VLMs (huggingface/trl#5684).

by @​qgallouedec in huggingface/trl#5575, huggingface/trl#5676 and huggingface/trl#5684

OpenReward Standard environment adapter (experimental)

A new trl.experimental.openreward adapter plugs any environment speaking the Open Reward Standard (ORS) protocol into any TRL trainer accepting an environment_factory (GRPOTrainer, AsyncGRPOTrainer). One identifier wires all three trainer slots — dataset, factory, reward_func:

from trl import GRPOConfig, GRPOTrainer
from trl.experimental.openreward import OpenRewardEnv
env = OpenRewardEnv("Eigent/SETA")  # or "http://localhost:8000"
trainer = GRPOTrainer(
model="Qwen/Qwen3-4B",
args=GRPOConfig(...),
train_dataset=env.dataset,
environment_factory=env.factory,
</tr></table>

... (truncated)

Commits

• 42d8291 Release: v1.4 (#5732)
• 63a78d9 [GKD] Fix Liger fused JSD path computing wrong loss (#5731)
• b05330a Enable chunked NLL loss with VLM in SFT (#5684)
• 0b89e28 Add Qwen2.5 response schema (#5728)
• 3a76294 [GKD] Fix seq kd wasted teacher forward (#5726)
• 47b3778 Add MFU helpers (#5698)
• 4601166 Align tiny Qwen3 MoE config with Qwen/Qwen3-30B-A3B (#5716)
• 8a6cc03 fix: prevent 5 GB+ CUDA memory leak in activation offloading by syncing strea...
• acbd53f Align KTO with DPO: Align _precompute_ref_logps (#5714)
• de7efdc Revert VLM support in parse_response (#5561)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Labels

The following labels could not be found: dependencies, python. Please create them before Dependabot can add them to a pull request.

Please fix the above issues or remove invalid values from dependabot.yml.