TRL

v1.12.0 was accidentally auto-published to PyPI with identical code to v1.11.0, so skip it — the next feature release is v1.13.0.

trl vllm-serve now delegates to vLLM's own server, cutting code from 1218 to ~130 lines and delivering 1.4–1.6× faster GRPO server-mode steps with bit-identical completions. A new experimental AsyncDistillationTrainer supports multi-teacher on-policy distillation, plus support for several new models and templates. Old examples/scripts and examples/notebooks links now 404 after the folder reorganization.

DistillationTrainer and DistillationConfig move from trl.experimental.distillation to the top-level trl package, with the old path emitting a FutureWarning until removal in v2.0.0. AsyncGRPOTrainer gains an experimental loop-owning path for training external agents like opencode, plus VLM support in DistillationTrainer and new DiffusionGemma SFT example.

Fixed quantization tests for bitsandbytes 0.50.0 and reverted an xfail skip for Nemotron H GRPO/RLOO tests after a transformers kernel bug fix.

Fixes a crash when training with Liger kernel on pre-Ampere (Fermi, Kepler, Maxwell, Pascal) GPUs and corrects DAPO, CISPO, and VESPO loss normalization when steps_per_generation differs from gradient_accumulation_steps. Also fixes vLLM server-mode communicator initialization, queue wait time metric in AsyncGRPOTrainer, and prepare_deepspeed crash with CPU offload optimizer.

GRPO and RLOO trainers now support iterable datasets via a new repeat_iterable_dataset generator, and environments can own the data (train_dataset becomes optional). Message-level rollouts land in AsyncGRPO, and the DistillationTrainer received a major refactor including a fix for num_items_in_batch that counted the wrong tokens and NaN'd on prompt-only datasets.

KTO trainer graduates from experimental to the top-level trl package with the same API as DPO/GRPO/SFT, and the experimental import path still works with a FutureWarning. Environment-owned rewards let agentic RL environments define their own reward via a reserved get_reward() method, and multi-environment support allows a single training run to handle multiple environments with environment-specific tool schemas. GRPO now supports both static and adaptive entropy regularization to encourage exploration and prevent policy collapse.

Fixed a hang in GRPO + vLLM colocate + PEFT on non-NVLink hardware and corrected dataset fingerprinting in DPO/SFT tokenization. Also integrated the new response parsing API, added a prompt-learning guard for PEFT with Liger in GRPO, and fixed activation offload storage deduplication.

The default SFT loss_type is now "chunked_nll", delivering ~30% less peak VRAM on average with neutral or slightly faster wall-clock time. Also introduces experimental GMPO trainer, transformers continuous batching, AsyncGRPO weight sync with vLLM 0.22+, and paddding-free AsyncGRPO.

The release introduces a new experimental A2POTrainer for optimal advantage regression and grants KTO trainer support for vision-language models. The AsyncRolloutWorker now runs in a separate process to avoid GIL contention and potential NCCL watchdog timeouts, along with fixes for aiohttp retries and all-NaN reward columns. Gold distillation trainer now aligns tokens via byte offsets, and SDFT/SDPO leverage the vLLM server for live teacher logprobs. Other features include bidirectional masked importance sampling for IcePop, support for NemotronH and Nemotron 3 Ultra, additional training chat templates, and decoupled self-distillation trainers.

Trainer telemetry is now gated on an explicit class-name allowlist, restricting which trainer classes can send telemetry.

Fixed an exponential backtracking bug in Qwen3/Qwen3.5/GLM4MoE response parsing that caused GRPOTrainer to hang indefinitely on truncated tool-call blocks, reducing worst-case complexity from O(2ⁿ) to O(n). Also fixed a CUDA memory leak in BNB dequantization buffers and stale state in OffloadActivations. Added training chat templates for Phi-3.5, Qwen3-VL, and Qwen3.5 Think/NoThink, and final logits softcapping support for AsyncGRPOTrainer on models like Gemma 2.

A new loss_type="chunked_nll" option for SFT drastically reduces peak activation memory by computing cross-entropy over tokens in checkpointed chunks instead of materializing the full [batch × seq × vocab] logits tensor, unlocking sequence lengths that previously caused out-of-memory errors. Also added OpenReward Standard environment adapter support, length-normalized DPO sigmoid loss, training chat templates for Cohere, Cohere2, Gemma 3, Qwen3, and Qwen2.5, and a training-invariance test suite to catch numerical drift across trainer configurations.

Features

Qwen 3.6 integration

TRL v1.3 ships training support for the new Qwen 3.6…

Features

New SSDTrainer — Simple Self-Distillation

A new experimental SSDTrainer implements the…

Features

DistillationTrainer for efficient on-policy distillation

Read the blog post: https://huggingface.co/spaces/HuggingFaceTB/trl-distillation-trainer

![off_vs_on_policy_distillation…

Read our blog post for an overview of TRL v1.

Features

Asynchronous GRPO

Asynchronous GRPO…

Features

Variational Sequence-Level Soft Policy Optimization (VESPO)

<img width="465" height="279" alt="Screenshot 2026-03-20 at 5 49 50 PM" src="https://github.com/user-attachments/assets/b60c9697-6eb7-498e-95b3-df78c367f5fa"…

What's Changed

• Handle mm_token_type_ids in SFT/GRPO/RLOO to fix IndexError by @albertvillanova in https://github.com/huggingface/trl/pull/5178
• Fix prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in…