{"id":"src_g_3xbu2oRX52tM8mwXJzq","slug":"trl","name":"TRL","type":"github","url":"https://github.com/huggingface/trl","orgId":"org_GDdYeYynEgCEBNBwy-m6s","productId":"prod_pCEM29no4janpSW6GK3TC","productSlug":"fine-tuning","org":{"id":"org_GDdYeYynEgCEBNBwy-m6s","slug":"hugging-face","name":"Hugging Face"},"isPrimary":false,"isHidden":false,"discovery":"curated","fetchPriority":"normal","metadata":"{\"evaluatedMethod\":\"github\",\"evaluatedAt\":\"2026-04-07T17:19:15.726Z\",\"changelogDetectedAt\":\"2026-04-07T17:28:23.856Z\",\"wellKnownSweptAt\":\"2026-10-31T06:00:36.365Z\"}","notice":null,"kind":"sdk","stars":19681,"starsFetchedAt":"2026-11-06T17:24:07.745Z","releaseCount":103,"releasesLast30Days":4,"avgReleasesPerWeek":0.9,"latestVersion":"v1.17.1","latestDate":"2026-10-30T14:15:28.000Z","changelogUrl":null,"hasChangelogFile":false,"lastFetchedAt":"2026-11-06T17:24:07.745Z","lastPolledAt":"2026-11-06T17:23:48.401Z","changeDetectedAt":null,"trackingSince":"2023-01-25T14:04:19.000Z","releases":[{"id":"rel_GDI4J5YyX8kJnXdY8-y6k","version":"v1.17.1","type":"feature","title":"v1.17.1","summary":"Fixed LFM2.5 tool-call parsing so a parenthesis inside an argument no longer closes the call prematurely.","titleGenerated":"TRL v1.17.1 keeps parentheses in LFM2.5 tool-call arguments from closing calls","titleShort":"LFM2.5 tool calls survive parentheses in arguments","breaking":"none","importance":1,"content":"## What's Changed\r\n\r\n* Keep a parenthesis inside an LFM2.5 tool-call argument from closing the call by @albertvillanova in https://github.com/huggingface/trl/pull/7942\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.17.0...v1.17.1","publishedAt":"2026-10-30T14:15:28.000Z","fetchedAt":"2026-10-30T17:23:20.757Z","url":"https://github.com/huggingface/trl/releases/tag/v1.17.1","media":[],"coverageCount":0},{"id":"rel_-8FUEc-qSLV5FkBKAOdLq","version":"v1.17.0","type":"feature","title":"v1.17.0","summary":"Adds a LoRA SFT example for mistralai/Mistral-Large-4-1T-A52B that fine-tunes the 1T-parameter model on a single 8×B300 node on top of its FP8 weights, requiring transformers>=5.22.0, peft>=0.21.0 and accelerate>=1.15.0. GRPO and AsyncGRPO now renormalize log-probs over the top-k/top-p candidate set, which raises an error when top_p<1.0 or min_p>0.0 is set without top_k>0, and score centering is now the default AsyncGRPO objective.","titleGenerated":"TRL v1.17.0 adds Mistral Large 4 single-node fine-tuning and unbiased importance sampling","titleShort":"Mistral Large 4 fine-tunes on one node; GRPO sampling unbiased","breaking":"major","importance":4,"content":"<!-- TODO: banner image for Mistral Large 4 -->\r\n\r\nhttps://github.com/user-attachments/assets/5aa86ec7-6444-4bed-bfdc-0445e99bea0d\r\n\r\n## Features\r\n\r\n### 🔥 Mistral Large 4: fine-tune a 1T-parameter model on one node\r\n\r\nTRL now supports [`mistralai/Mistral-Large-4-1T-A52B`](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B): 1T parameters, 52B active. The new [LoRA example](https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4) fine-tunes it with `SFTTrainer` **on a single 8×B300 node, on top of its FP8 weights**: the experts are split 8 ways with token dispatch, everything else is sharded across the 8 GPUs at load time by transformers' `DistributedConfig`, and LoRA trains attention and the dense and shared-expert MLPs.\r\n\r\nMeasured: **32.7k tokens per step in ~8 s, 142 GiB per GPU**, with packing and assistant-only loss on reasoning conversations.\r\n\r\n```python\r\ntraining_args = SFTConfig(\r\n    model_init_kwargs={\r\n        \"dtype\": \"auto\",  # keep the FP8 weights\r\n        \"distributed_config\": DistributedConfig(fsdp_size=8, ep_size=8),\r\n    },\r\n    packing=True,\r\n    assistant_only_loss=True,\r\n    ...\r\n)\r\ntrainer = SFTTrainer(model=\"mistralai/Mistral-Large-4-1T-A52B\", args=training_args, peft_config=LoraConfig(...), ...)\r\n```\r\n\r\n`assistant_only_loss=True` works out of the box: TRL ships a training chat template that matches the released checkpoint's, including its `reasoning_effort` levels (`low`, `medium`, `high`). Needs `transformers>=5.22.0`, `peft>=0.21.0` and `accelerate>=1.15.0`.\r\n\r\n* Add Mistral Large 4 by @qgallouedec in https://github.com/huggingface/trl/pull/8017\r\n* Sync the Mistral Large 4 chat template with the released checkpoint by @albertvillanova in https://github.com/huggingface/trl/pull/8018\r\n\r\n### 🎯 Unbiased importance sampling under top-k, top-p and min-p\r\n\r\nvLLM normalizes its log-probabilities over the tokens left after `top_k`, `top_p` and `min_p` filtering, while the trainer normalized over the whole vocabulary. Every sampled token's log-ratio was off by the log of the probability mass the filter kept, so the vLLM importance sampling ratio was biased towards 0 (around 1e-4 instead of 1 with `top_p=0.9`). The trainers now normalize over the candidate set each token was sampled from, following Keep Sampling Mask from [DeepSeek-V3.2](https://huggingface.co/papers/2512.02556).\r\n\r\n* AsyncGRPO: renormalize log-probs over the top-k/top-p candidate set by @AmineDiro in https://github.com/huggingface/trl/pull/7627\r\n* GRPO: renormalize log-probs over the top-k/top-p candidate set (vLLM colocate mode, vLLM >= 0.28; server mode will follow) by @albertvillanova in https://github.com/huggingface/trl/pull/7866\r\n* Require `top_k` with `min_p` in AsyncGRPO so the importance sampling ratio stays unbiased by @albertvillanova in https://github.com/huggingface/trl/pull/7937\r\n* Track the support maximum in the chunked log-prob kernel, so a far-below-max candidate set no longer gives a log-probability of 0 and non-finite gradients, by @kashif in https://github.com/huggingface/trl/pull/7983\r\n\r\n**💥 What this changes for you.** vLLM bounds the candidate set it returns with `top_k`, so with vLLM and the importance sampling correction on (the default), `GRPOConfig` now raises when `top_p < 1.0` or `min_p > 0.0` is set without `top_k > 0`. Set `top_k` (e.g. `top_k=512`), or `vllm_importance_sampling_correction=False`.\r\n\r\n### ⚖️ Score centering is the default AsyncGRPO objective\r\n\r\nAsyncGRPO's old log-probs are the vLLM sampler's, so its ratio mixes staleness with the numerical mismatch between the server and the trainer. With an FP8 sampler and a bf16 trainer, the clipped objective collapses: reward goes from 0.60 to 0.34 (FP8 weights) or 0.10 (FP8 weights + KV cache) by step 300, against 0.71 with a bf16 sampler.\r\n\r\n[Score centering](https://huggingface.co/papers/2609.20807) with masked importance sampling is now the default (`score_centering=True`, `importance_sampling_min=0.5`, `importance_sampling_max=5.0`). With it, the FP8 sampler trains exactly like the bf16 one (0.72). It costs 15–20% of generation throughput.\r\n\r\n**💥 What this changes for you.** AsyncGRPO runs now train with a different objective by default, and tokens whose ratio to the sampler falls outside `[0.5, 5.0]` are dropped. Set `score_centering=False` to keep the clipped objective.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/7760\r\n\r\n### 🚀 Faster AsyncGRPO rollouts and packing\r\n\r\n* **Adaptive in-flight cap from the vLLM load**: the rollout worker follows the server's KV cache usage instead of admitting a fixed `max_inflight_tasks`, so requests stop queueing and being preempted on the server. No new config: everything is read from the server. By @AmineDiro in https://github.com/huggingface/trl/pull/7610\r\n* **Filter zero-advantage groups** (DAPO's dynamic sampling): groups whose rollouts all got the same reward, or none of which could be scored, no longer cost a forward and backward. By @AmineDiro in https://github.com/huggingface/trl/pull/7609\r\n* **Pack the sampling mask ragged and pad it on the device**: `TreeRow.pack` goes from 250 s to 31 s on rank 0 over a 15-step run, and from 1040 ms to 66 ms on a 28k-token multi-turn row. By @AmineDiro in https://github.com/huggingface/trl/pull/7798\r\n* Keep filling a micro-batch when another row has free room by @clementtisseau in https://github.com/huggingface/trl/pull/7793\r\n* Use FlashAttention-4 on Blackwell GPUs in async GRPO and async distillation by @lewtun in https://github.com/huggingface/trl/pull/7792\r\n\r\n### 🌳 Tree attention for linear-attention hybrids\r\n\r\nTree packing (v1.16) now supports the Gated DeltaNet hybrids: Qwen3.5, Qwen3.6, Qwen3-Next and OLMo-Hybrid. Since v1.16 the trainer refused them at init; before it, sequence packing accepted them but was silently wrong on them. The causal convolution and the delta-rule recurrence now follow the tree instead of the row, through FLA's chunked kernels.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/7686\r\n\r\n### 🎚️ Precise chunked log-probabilities\r\n\r\nThe chunked log-probability and KL/JSD heads get the two precision fixes from [A Fix for Softmax and Cross-Entropy](https://huggingface.co/papers/2610.31642): confident targets keep their small gradients, and matching teachers produce zero gradients. [Entropy-preserving reinforcement learning](https://huggingface.co/papers/2603.11682) (Appendix B.4) reports the same high-confidence policy-gradient issue.\r\n\r\n* Fix chunked loss precision by @kashif in https://github.com/huggingface/trl/pull/7882\r\n* Fix the fused log-prob/entropy backward on strided inputs (shifted views from SFT/DPO) by @winglian in https://github.com/huggingface/trl/pull/7890\r\n\r\n### 🔀 X-Token cross-tokenizer distillation in GOLD\r\n\r\n`GOLDTrainer` gets off-policy cross-tokenizer distillation based on [X-Token](https://huggingface.co/papers/2605.21699): set `lmbda=0.0` and `xtoken_loss_type=\"p_kl\"` or `\"h_kl\"` with a precomputed projection matrix.\r\n\r\n* Add X-Token cross-tokenizer KD to GOLD trainer by @kashif in https://github.com/huggingface/trl/pull/6126\r\n* Support byte-fallback tokenizers (Gemma) in GOLD byte-offset alignment by @kashif in https://github.com/huggingface/trl/pull/8010\r\n\r\n### 🟥 Training on AMD GPUs\r\n\r\nTRL's trainers now run on AMD Instinct GPUs through PyTorch's ROCm build, with a [guide](https://huggingface.co/docs/trl/amd_gpus) and an AMD CI job that runs the test suite on every PR.\r\n\r\n* Document training on AMD GPUs by @qgallouedec in https://github.com/huggingface/trl/pull/7844\r\n* Add AMD/ROCm CI by @kashif in https://github.com/huggingface/trl/pull/5918\r\n\r\n### 🧩 More models\r\n\r\n* **Beam**: tiny model, SFT/DPO/GRPO/RLOO tests, GRPO tool calling, and a LoRA SFT example on one 8×B300 node. Needs `transformers>=5.20.1`. By @qgallouedec in https://github.com/huggingface/trl/pull/7629\r\n* **Mistral Medium 3.5**: training chat template for assistant-only loss by @qgallouedec in https://github.com/huggingface/trl/pull/7940\r\n* **LFM2.5-24B-A2B** (and LFM2.5-2.6B's current template) for tool calling by @qgallouedec in https://github.com/huggingface/trl/pull/7753\r\n* **LFM2.5-1.2B-Instruct and LFM2.5-VL-450M** for tool calling by @albertvillanova in https://github.com/huggingface/trl/pull/7909, https://github.com/huggingface/trl/pull/7910 and https://github.com/huggingface/trl/pull/7911\r\n* **TRL's own Qwen3 training template** for response parsing, so a tokenizer saved by assistant-only SFT works in GRPO, by @edbeeching in https://github.com/huggingface/trl/pull/8007\r\n\r\n### Other\r\n\r\n* Accept instantiated models in `AsyncGRPOTrainer` by @kashif in https://github.com/huggingface/trl/pull/6676, and port instantiated models, MoE aux loss and the injected-worker LoRA name to `AsyncDistillationTrainer` by @kashif in https://github.com/huggingface/trl/pull/7984\r\n* Save AsyncGRPO and AsyncDistillation checkpoints in the mixed-precision dtype instead of fp32 by @AmineDiro in https://github.com/huggingface/trl/pull/7939\r\n* Back off between unscorable harness sessions and stop the worker when they persist by @albertvillanova in https://github.com/huggingface/trl/pull/7992\r\n* Async Harbor GRPO: PEFT support and fixes by @younesbelkada in https://github.com/huggingface/trl/pull/7849 and https://github.com/huggingface/trl/pull/7948\r\n* Write the logged completions to parquet as JSON by @qgallouedec in https://github.com/huggingface/trl/pull/7696\r\n* Add support for vLLM 0.32.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7799, and allow its patch releases by @albertvillanova in https://github.com/huggingface/trl/pull/7850\r\n\r\n### Removals and requirement bumps\r\n\r\n* **Remove the experimental TPO trainer** by @qgallouedec in https://github.com/huggingface/trl/pull/7807\r\n* **Require `top_k` with `top_p < 1.0` or `min_p > 0.0` in GRPO with vLLM importance sampling** (see above) by @albertvillanova in https://github.com/huggingface/trl/pull/7866\r\n* Require `torch>=2.6.0` and drop the pre-2.6 FSDP2 and accelerator fallbacks by @albertvillanova in https://github.com/huggingface/trl/pull/7642\r\n* Require `accelerate>=1.11.0` and drop the FSDP2 ignored-modules fallback by @albertvillanova in https://github.com/huggingface/trl/pull/7797\r\n* Require `peft>=0.17.0` and drop the `target_parameters` guards by @albertvillanova in https://github.com/huggingface/trl/pull/7640\r\n* Raise the bitsandbytes floor to 0.45.2 by @albertvillanova in https://github.com/huggingface/trl/pull/7864\r\n* Raise the math-verify floor to 0.8.0 by @albertvillanova in https://github.com/huggingface/trl/pull/7870\r\n* Drop liger-kernel from the dev dependencies, the Docker images and the tests by @qgallouedec in https://github.com/huggingface/trl/pull/7843\r\n\r\n## Fixes\r\n\r\n* Prevent VLM policies from generating the image placeholder token with vLLM (#7698), in GOLD (#7699) and in Online DPO (#7708), all by @albertvillanova\r\n* Place vLLM image features at the model config's image token by @albertvillanova in https://github.com/huggingface/trl/pull/7773\r\n* Pass `mm_token_type_ids` to the model in Online DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7859\r\n* Cast FSDP2 weight-transfer payloads to the vLLM manifest dtype in AsyncGRPO and AsyncDistillation by @sfc-gh-truwase in https://github.com/huggingface/trl/pull/7131\r\n* Load the tokenizer for models that take no images by @qgallouedec in https://github.com/huggingface/trl/pull/7774\r\n* Keep a single LoRA target module from the CLI or a config file as a list by @albertvillanova in https://github.com/huggingface/trl/pull/7796\r\n* Read and write text files as UTF-8 by @albertvillanova in https://github.com/huggingface/trl/pull/7755\r\n\r\n### Clearer errors\r\n\r\nFailure modes made explicit instead of failing silently or deep in the stack, all by @albertvillanova:\r\n\r\n* Raise when LoRA would skip the routed experts of MoE models on transformers v5 with peft<0.19 (#7769), and warn when it skips their dense and shared expert layers with peft 0.19 to 0.21.2 (#7771)\r\n* Reject a `PeftModel` passed as `ref_model` in DPO and KTO (#7808)\r\n* Reject prompt learning in distillation before wrapping the model (#7718)\r\n\r\n## Documentation\r\n\r\n* Document the math-verify install for `accuracy_reward` by @albertvillanova in https://github.com/huggingface/trl/pull/7875\r\n* Fix the default IS mode in the `GRPOConfig` docstring (#7751) and in the GRPO vLLM tip (#7800) by @sergiopaniego\r\n* Update OpenEnv links to openenv/OpenEnv by @sergiopaniego in https://github.com/huggingface/trl/pull/7970\r\n* Drop ORPO and CPO examples whose prompt alone fills `max_length` by @qgallouedec in https://github.com/huggingface/trl/pull/7913\r\n* Fix the license header of `tree_gated_delta_rule.py` by @albertvillanova in https://github.com/huggingface/trl/pull/7938\r\n\r\n## CI\r\n\r\n**AMD**, all by @albertvillanova: relax the confident-target tolerances for AMD (#7981, #8005)\r\n\r\n**Minimum versions**, all by @albertvillanova: run `tests_min_versions` on Python 3.11 and pin every dependency at its floor (#7741), including bitsandbytes (#7746), liger-kernel (#7846) and deepspeed (#7855); document it (#7745) and when to add, drop and change the default Python version (#7743)\r\n\r\n**Docker images**, all by @albertvillanova: install Python with uv (#7738), use Python 3.13 (#7739), install kernels through the TRL `kernels` extra (#7815), remove redundant installs (#7813), skip the uv cache in the release image (#7817), require vLLM >= 0.31.0 (#7820) and explain the floor (#7862), bump uv and upgrade ca-certificates (#7880)\r\n\r\n**Real objects, not mocks**, all by @albertvillanova: real generation outputs in the GOLD and GKD tests (#7902), real config files in the TrlParser tests (#7903), tiny models instead of full-size checkpoints in the GOLD fixtures (#7904), with a new tiny SmolLM3 (#7905, #7906), a real trainer in the IW-OPD tests (#7976), real `DistillationLogits` in the SDFT tests (#7977), and real objects in the OpenEnv tests (#7985, #7986, #7989, #7990)\r\n\r\n**MoE LoRA coverage**, all by @albertvillanova: test that LoRA on an MoE model trains its routed experts (#7636) and its dense and shared expert layers (#7768), and the error and warning for PEFT versions that skip expert layers (#7860)\r\n\r\n**Other**, all by @albertvillanova: run the test workflows on push to main only (#7848), use the latest Python patch release in pip workflows (#7881), test the math-verify rewards in CI (#7928), check the LFM2.5-2.6B chat template against the Hub (#7907), skip the async GRPO tests on transformers < 5.2.0 (#7750), skip the Liger fixture and tests without an accelerator (#7868), import trl first in the tests so peft 0.17 imports with transformers v5 (#7873), cap the KV cache in the continuous batching tests (#7825), fix a port race in the OpenReward fixture (#7886), pass `enable_thinking` only to templates that use it (#7898), UTF-8 in tests (#7764), plus GOLD SmolVLM test handling against transformers 5.20.1 (#7728, #7791, #7810, #7929)\r\n\r\n## New Contributors\r\n\r\n* @sfc-gh-truwase made their first contribution in https://github.com/huggingface/trl/pull/7131\r\n* @clementtisseau made their first contribution in https://github.com/huggingface/trl/pull/7793\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.16.0...v1.17.0","publishedAt":"2026-10-29T15:51:15.000Z","fetchedAt":"2026-10-29T17:23:16.251Z","url":"https://github.com/huggingface/trl/releases/tag/v1.17.0","media":[],"coverageCount":0},{"id":"rel_sYn3OMOJV8sislSKeJ_20","version":"v1.16.0","type":"feature","title":"v1.16.0","summary":"TRL now trains on AWS Trainium with a new guide and pad_to_multiple_of defaulting to \"auto\" (128 on Neuron, None elsewhere), and AsyncGRPO packs micro-batch rows as a prefix tree, forwarding shared-prompt tokens once (up to 3.1x faster with peak memory dropping from 35.2 to 13.5 GiB). Adds Muse Spark Open support, AMD GPUs in async trainers, and JSON-schema-typed tool-call arguments.","titleGenerated":"TRL v1.16.0 brings AWS Trainium training and tree attention for AsyncGRPO","titleShort":"AWS Trainium support; AsyncGRPO gets tree attention","breaking":"major","importance":4,"content":"## Features\r\n\r\n### 🛠️ TRL now trains on AWS Trainium\r\n\r\n<img width=\"1672\" height=\"941\" alt=\"Neon Warehouse_ AWS and Trainium Support\" src=\"https://github.com/user-attachments/assets/dcb6e2a1-0f7c-4635-819b-a1ec6f5e143b\" />\r\n\r\nThe fused LM head from v1.15 is what made Neuron tractable; the remaining gaps are closed here. There is a [guide](https://huggingface.co/docs/trl/neuron) to go with it.\r\n\r\nNeuron recompiles once per input shape, so an unpadded run stalls about a minute on every new sequence length. `pad_to_multiple_of` now defaults to `\"auto\"`: 128 on Neuron, `None` elsewhere, so nothing changes on GPU. Trainium2, Qwen3-0.6B SFT, 20 steps: **130 s with the default against 260 s without**.\r\n\r\n* Add a guide for training on AWS Trainium by @qgallouedec in https://github.com/huggingface/trl/pull/7729\r\n* Default `pad_to_multiple_of` to `\"auto\"` by @qgallouedec in https://github.com/huggingface/trl/pull/7724\r\n* Run the fused LM head and distillation kernels on devices without Triton by @qgallouedec in https://github.com/huggingface/trl/pull/7669\r\n* Report the Neuron device in `trl env` and telemetry by @qgallouedec in https://github.com/huggingface/trl/pull/7672\r\n* Support any accelerator in activation offloading by @qgallouedec in https://github.com/huggingface/trl/pull/7670\r\n\r\n### 🌳 Tree attention for AsyncGRPO\r\n\r\nAsyncGRPO now packs every micro-batch row as a **prefix tree** instead of a flat sequence. Multi-turn and multi-generation rollouts repeat a lot of prefix: on a shared prompt, those tokens used to be forwarded once per row. Tree packing forwards them once.\r\n\r\n```text\r\nrollout 0:  [ prompt ] [ completion A ]              prompt ──┬── completion A ── later turn\r\nrollout 1:  [ prompt ] [ completion B ]        =>             └── completion B\r\nrollout 2:  [ prompt ] [ completion A ] [ … ]\r\n```\r\n\r\nChains without a fork collapse into segments, and each segment's queries attend to the K/V gathered along its root-to-segment path in a single FA3 varlen call. No mask tensor: every layer except attention runs once per unique token, and only K/V is duplicated along shared prefixes.\r\n\r\nQwen3-1.7B, 32K tokens per rollout, forward + backward:\r\n\r\n| layout | packing ratio | sequence packing | tree packing | peak GiB |\r\n| --- | --- | --- | --- | --- |\r\n| shared prompt | 1.00 | 8677 ms | 8737 ms (0.99×) | 35.2 → 35.2 |\r\n| shared prompt | 2.29 | 8266 ms | **4341 ms (1.90×)** | 35.2 → 18.0 |\r\n| shared prompt | 3.37 | 8179 ms | **2755 ms (2.97×)** | 35.2 → 13.6 |\r\n| nested fork | 1.60 | 8557 ms | 6217 ms (1.38×) | 35.2 → 23.8 |\r\n| nested fork | 3.56 | 8172 ms | **2635 ms (3.10×)** | 35.2 → 13.5 |\r\n\r\nThere is no `packing` option to choose: with nothing shared a tree row is exactly the old sequence packing, at a ~1% cost.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/7328\r\n\r\n### Muse Spark Open\r\n\r\nSupport for [`meta-models/Muse-Spark-Open-816B-A42B`](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B): the tiny-model script, coverage in the VLM tests (SFT, DPO, KTO, chat-template utils, fused LM head), and a LoRA SFT example on 8 nodes of 8×H100 with `DistributedConfig(fsdp_size=64, ep_size=8)`, token dispatch and packing. Measured at **14.2 s/step, 131k tokens/step, 40.2 GB per GPU**.\r\n\r\nNeeds `transformers>=5.20.0`; the adapter save additionally needs [transformers#49500](https://github.com/huggingface/transformers/pull/49500) with peft 0.21.2.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/7536\r\n\r\n### AMD GPUs in the async trainers\r\n\r\n* Support AMD GPUs in the async GRPO and async distillation trainers by @Abdennacer-Badaoui in https://github.com/huggingface/trl/pull/7689\r\n* Hide the GPU from the async rollout worker on ROCm by @Abdennacer-Badaoui in https://github.com/huggingface/trl/pull/7685\r\n\r\n### Tool-call arguments typed by the tool's JSON schema\r\n\r\n`parse_response` now receives the tools, so tool-call arguments come back typed from the tool's own schema instead of whatever the raw parse produced.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/7650\r\n\r\n### Other\r\n\r\n* Async GRPO tool metrics by @AmineDiro in https://github.com/huggingface/trl/pull/7624\r\n* Store async GRPO and distillation rollout sample tokens as numpy arrays by @AmineDiro in https://github.com/huggingface/trl/pull/7626\r\n* Save the final step when async trainers stop at `num_train_epochs` by @lewtun in https://github.com/huggingface/trl/pull/7628\r\n* Pass sequence boundaries from the padding-free collator by @qgallouedec in https://github.com/huggingface/trl/pull/7599\r\n* Build the reference model with the policy's dtype and attention implementation by @qgallouedec in https://github.com/huggingface/trl/pull/7597, and the SDFT/SDPO teacher with the student's by @qgallouedec in https://github.com/huggingface/trl/pull/7616\r\n* Find unused parameters under DDP for vision-language models by @qgallouedec in https://github.com/huggingface/trl/pull/7598\r\n* Pad the generation prompts to `pad_to_multiple_of` by @qgallouedec in https://github.com/huggingface/trl/pull/7671\r\n* Drop the client-side observe loop from the CARLA examples by @sergiopaniego in https://github.com/huggingface/trl/pull/7721\r\n* Use `None` for unscored `env.reward` in OpenReward by @albertvillanova in https://github.com/huggingface/trl/pull/7541\r\n\r\n### Removals and requirement bumps\r\n\r\n* **Remove the experimental SSD trainer** by @qgallouedec in https://github.com/huggingface/trl/pull/7706\r\n* **Refuse `nn.DataParallel` in `RewardTrainer`** by @albertvillanova in https://github.com/huggingface/trl/pull/7571\r\n* Drop vLLM 0.21.0 support by @qgallouedec in https://github.com/huggingface/trl/pull/7700\r\n* Require `accelerate>=1.10.1` and drop the `ParallelismConfig` shim by @albertvillanova in https://github.com/huggingface/trl/pull/7643\r\n* Declare a minimum bitsandbytes version by @albertvillanova in https://github.com/huggingface/trl/pull/7641\r\n* Declare the direct dependencies TRL gets transitively by @albertvillanova in https://github.com/huggingface/trl/pull/7713\r\n\r\n## Fixes\r\n\r\n* **Fix silently wrong `torch.compile` results in the Triton loss kernels on torch < 2.11.** `torch_compile=True` trained with `loss=0.0` and `grad_norm=nan` at every step. Inductor passes scalar kernel args as `constexpr` before torch 2.11, the `.to(tl.float32)` casts raised on those, and Inductor then assumed every input was mutated and miscompiled. By @albertvillanova in https://github.com/huggingface/trl/pull/7688\r\n* **Fix FSDP2 vLLM colocate weight sync for tied-embedding models** by @Abdennacer-Badaoui in https://github.com/huggingface/trl/pull/7647\r\n* **Prevent VLM policies from generating the image placeholder token** in GRPO, RLOO and Distillation by @albertvillanova in https://github.com/huggingface/trl/pull/7695\r\n* **Fix Online DPO VLM training** when the prompt exceeds `max_length` (https://github.com/huggingface/trl/pull/7703) and for processors with 5D pixel values (https://github.com/huggingface/trl/pull/7704), both by @albertvillanova\r\n* **Fix activation offloading on torch < 2.4** by @albertvillanova in https://github.com/huggingface/trl/pull/7748\r\n* Fix `torch.compile(fullgraph=True)` of `selective_log_softmax_and_entropy` on torch 2.6 by @albertvillanova in https://github.com/huggingface/trl/pull/7719\r\n* Keep the Triton loss kernels in fp32 under `torch.compile` by @qgallouedec in https://github.com/huggingface/trl/pull/7600\r\n* Read `is_triton_available` once in the fused LM head kernels so Dynamo does not warn by @qgallouedec in https://github.com/huggingface/trl/pull/7731\r\n* Fix Distillation for models split across devices by @albertvillanova in https://github.com/huggingface/trl/pull/7573\r\n* Restore the gradient checkpointing kwargs after generation by @qgallouedec in https://github.com/huggingface/trl/pull/7668\r\n* Drop the FlashAttention warnings for padding-free and packing in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/7620\r\n* Restore the distributed env vars that vLLM colocate tests leave behind by @Abdennacer-Badaoui in https://github.com/huggingface/trl/pull/7723\r\n\r\n### Clearer errors\r\n\r\nA sweep making failure modes explicit instead of crashing deep in the stack, all by @albertvillanova:\r\n\r\n* Raise a clear error when the fused LM head is used without Triton (#7639)\r\n* Raise a clear error for prefix tuning on peft versions that cannot support it (#7646)\r\n* Raise a clear error when the Distillation student and teacher heads are on different devices (#7605)\r\n* Reject a text-only teacher in GOLD VLM distillation instead of crashing (#7581)\r\n* Name the missing vLLM server setting when the server cannot serve TRL (#7611)\r\n* Raise early for `use_kernels` with transformers 5.2.0 and 5.3.0 (#7577), and if kernels is installed with transformers 5.3.0 in the tiny NemotronH scripts (#7575)\r\n\r\n## Documentation\r\n\r\n* Add a guide for training on AWS Trainium by @qgallouedec in https://github.com/huggingface/trl/pull/7729\r\n* Add guidance on testing through the public API to the contributing guide and agent instructions by @albertvillanova in https://github.com/huggingface/trl/pull/7606\r\n* Document the criterion for raising a dependency version floor by @albertvillanova in https://github.com/huggingface/trl/pull/7638\r\n\r\n## CI\r\n\r\n**Real objects, not mocks.** The bulk of this cycle, applying the repo's testing rule across the suite, all by @albertvillanova: run the GRPO entropy mask tests on a real trainer instead of a Mock (#7566), the teacher logprob tests on a real trainer instead of a MagicMock (#7579), build real GOLD trainers in the init (#7582), input preparation (#7614), on-policy generation (#7613, #7615), output decoding (#7633), ULD helper (#7634) and no-backend tokenizer (#7635) tests, check GOLD dataset preparation through the trainer (#7632), and build real async trainers in the rollout state checkpoint tests (#7583)\r\n\r\n**Coverage**, all by @albertvillanova: test SFT (#7552) and DPO/KTO/Reward/GRPO/RLOO (#7601) on a model split across devices, test fp16 training across six trainers (#7572)\r\n\r\n**Experimental suite on main**, all by @albertvillanova: run the experimental tests on pushes to main (#7675), nightly (#7681), on demand (#7678) and when their Makefile target or dependencies change (#7680), and report failures to Slack (#7676)\r\n\r\n**Other**: use Python 3.13 by default in CI by @qgallouedec in https://github.com/huggingface/trl/pull/7648, remove the slow test infrastructure (#7604), rename the distributed smoke tests to distributed tests (#7631), skip the kernel tests when Triton is not installed (#7563), guard the aiohttp import in the experimental async rollout workers (#7578), drop `use_cpu` and `bf16` settings the CPU fixture already covers (#7612), re-record the `sft_fa2` invariant reference after the fused LM head by @qgallouedec (#7591), plus warning filters in #7586, #7588, #7590, #7645, #7727\r\n\r\n## New Contributors\r\n* @Abdennacer-Badaoui made their first contribution in https://github.com/huggingface/trl/pull/7647\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.15.0...v1.16.0","publishedAt":"2026-10-15T18:00:33.000Z","fetchedAt":"2026-10-15T21:19:30.200Z","url":"https://github.com/huggingface/trl/releases/tag/v1.16.0","media":[{"type":"image","url":"https://github.com/user-attachments/assets/dcb6e2a1-0f7c-4635-819b-a1ec6f5e143b","r2Key":"releases/b75e058f04b9babf6c8d3078cdc6abc2dfb74dd416ad00799b005cd93202701b.png","r2Url":"https://media.releases.sh/releases/b75e058f04b9babf6c8d3078cdc6abc2dfb74dd416ad00799b005cd93202701b.png"}],"coverageCount":0},{"id":"rel_-QlXNUAbicPpBnhrHxbmg","version":"v1.15.0","type":"feature","title":"v1.15.0","summary":"A new fused LM head, on by default in SFT, DPO, KTO, GRPO, RLOO and Distillation, projects hidden states through the LM head in Triton-kernel tiles to avoid building the full logits tensor, raising max trainable sequence length up to 6.9x and cutting peak memory 52% to 82%. The full-logits scoring path, use_liger_kernel chunked path and _forward_redirection are removed, so a PEFT adapter on lm_head now raises and use_liger_kernel=True is deprecated for v2.0.0 removal.","titleGenerated":"TRL v1.15.0 fused LM head boosts max trainable sequence length up to 6.9x","titleShort":"Fused LM head: up to 6.9x longer sequences, 82% less memory","breaking":"major","importance":4,"content":"<img width=\"1672\" height=\"941\" alt=\"Neon-Gripped Industrial Valve\" src=\"https://github.com/user-attachments/assets/329793d4-cef1-49ae-9ab6-5b1387d13071\" />\r\n\r\n## Features\r\n\r\n### ⚡ Fused LM head: up to 6.9× longer sequences on the same GPU\r\n\r\nSFT, DPO, KTO, GRPO, RLOO and Distillation now score tokens with a **fused LM head**: a Triton kernel projects the hidden states through the LM head in tiles and reduces to per-token log-probs and entropy directly, so the `[batch, seq, vocab]` logits tensor is never built. It is **on by default**, there is nothing to turn on.\r\n\r\n**Max trainable sequence length** (gemma-3-1b, 262k vocabulary, 79 GiB of GPU memory, bf16, batch size 1, gradient checkpointing, sdpa):\r\n\r\n| Trainer | v1.14.2 | v1.15.0 | |\r\n| --- | --- | --- | --- |\r\n| DPO | 10,240 | **59,392** | **5.80×** |\r\n| KTO (batch size 2) | 9,216 | **63,488** | **6.89×** |\r\n| GRPO (scoring only) | 28,672 | **114,688** | **4.00×** |\r\n| RLOO (scoring only) | 23,552 | **100,352** | **4.26×** |\r\n| SFT (`loss_type=\"nll\"`) | 20,480 | **107,520** | **5.25×** |\r\n| SFT (default `chunked_nll`) | 107,520 | 107,520 | 1.00× |\r\n\r\n**At 8,192 tokens**, peak memory drops 52% to 82% and steps are 2.3% to 10.9% faster:\r\n\r\n| Trainer | Peak GiB v1.14.2 → v1.15.0 | Tokens/s |\r\n| --- | --- | --- |\r\n| DPO | 48.97 → **12.00** (-75.5%) | 1.035× |\r\n| KTO | 64.91 → **11.98** (-81.5%) | 1.045× |\r\n| GRPO | 24.15 → **10.30** (-57.4%) | 1.023× |\r\n| RLOO | 29.28 → **14.02** (-52.1%) | 1.029× |\r\n| SFT (`nll`) | 30.33 → **8.87** (-70.7%) | 1.029× |\r\n\r\nThe gain is largest for small models with large vocabularies. SFT's previous default (`chunked_nll`) already avoided full logits, so it is unchanged. Scoring needs Triton on a GPU (Linux with CUDA, ROCm or XPU).\r\n\r\n**💥 What this changes for you.** The full-logits scoring path, the `use_liger_kernel` chunked path and `_forward_redirection` are gone from GRPO, RLOO, DPO and KTO:\r\n\r\n* A PEFT adapter **on `lm_head`** now raises. Use `modules_to_save=[\"lm_head\"]` instead.\r\n* `use_liger_kernel=True` is deprecated in these four trainers and will be removed in v2.0.0. Liger's layer kernels still apply, but `fused_linear_cross_entropy` is forced to `False` (it would replace the fused head) and setting it explicitly raises. Use `model_init_kwargs={\"use_kernels\": True}`.\r\n* In DPO and KTO, `compute_metrics` and `compute_loss(..., return_outputs=True)` still receive the full logits, from an extra forward pass taken only when they are used. Both warn that this is deprecated and will be removed in v2.0.0.\r\n\r\nNothing else the trainers logged is lost: the kernel gained `log_sum_sq_probs` (WPO weighting), `mean_logits` (`logits/chosen`, `logits/rejected`) and `is_top1` (DPO's `mean_token_accuracy`) as opt-in outputs, each checked against the full logits in tests.\r\n\r\n<details>\r\n<summary>Benchmark details</summary>\r\n\r\n1× B300 with the PyTorch allocator capped at 79 GiB (about an H100 80GB's usable memory), `google/gemma-3-1b-pt` built from its config with random bf16 weights, lr 1e-6, AdamW, synthetic token ids at exact length. Max length: one fresh process per attempt, 2 training steps, doubling from 8k then bisection to 1024 tokens, arms interleaved round-robin in the same job. Throughput: 12 steps, first 2 discarded, median of 3 A/B/A/B repeats in the same job. Noise floor (A vs A) is 0.1% to 0.8%, below every gap reported. GRPO/RLOO replace generation with fixed random completions, so only scoring is measured.\r\n\r\nNot measured: H100 hardware itself, flash-attention (sdpa only), real generation, multi-GPU.\r\n\r\n</details>\r\n\r\n* Compute the chunked log-probabilities with a Triton kernel by @qgallouedec in https://github.com/huggingface/trl/pull/7386\r\n* Add an opt-in fused LM head to the patched forward by @qgallouedec in https://github.com/huggingface/trl/pull/7387\r\n* 💥 Score GRPO and RLOO tokens with the fused LM head by @qgallouedec in https://github.com/huggingface/trl/pull/7389\r\n* 💥 Score DPO and KTO tokens with the fused LM head by @qgallouedec in https://github.com/huggingface/trl/pull/7390\r\n* Score SFT tokens with the fused LM head by @qgallouedec in https://github.com/huggingface/trl/pull/7466\r\n* Compute the distillation divergence with a Triton kernel by @qgallouedec in https://github.com/huggingface/trl/pull/7465\r\n* Compute only the requested fused LM head outputs by @qgallouedec in https://github.com/huggingface/trl/pull/7406\r\n* Rename `patch_fused_lm_head` to `add_fused_lm_head` by @qgallouedec in https://github.com/huggingface/trl/pull/7415\r\n* Honor `logits_scaling` and `lm_head_multiplier` in the fused LM head by @YaseenBashaT in https://github.com/huggingface/trl/pull/7439\r\n* Fix the fused LM head for models split across devices by @albertvillanova in https://github.com/huggingface/trl/pull/7479\r\n\r\n### Selective activation checkpointing in SFT\r\n\r\nWith gradient checkpointing on, the attention output is now saved during the forward instead of being recomputed in the backward, recovering most of the checkpointing slowdown at long context for one extra hidden-state-sized tensor per layer. Same eager SAC approach torchtitan uses under FSDP2, no `torch.compile` needed.\r\n\r\n```python\r\nSFTConfig(\r\n    gradient_checkpointing=True,\r\n    gradient_checkpointing_kwargs={\"selective\": True},\r\n)\r\n```\r\n\r\nPEFT + DeepSpeed ZeRO-3 is rejected: that combination needs reentrant checkpointing, which SAC cannot use.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/7194\r\n\r\n### Assistant-only loss on vision datasets in SFT\r\n\r\n`assistant_only_loss=True` now works on vision datasets. `SFTTrainer` used to refuse it because processors did not return usable assistant masks once image placeholders were expanded; [transformers#48793](https://github.com/huggingface/transformers/pull/48793) fixed that, and it ships in `transformers` 5.18.\r\n\r\n```python\r\nSFTConfig(assistant_only_loss=True)  # now fine with an image dataset\r\n```\r\n\r\nThe vision collator tokenizes conversational examples through the processor's `apply_chat_template` and requests the assistant masks to mask non-assistant tokens in the labels. Older `transformers` still raises, now with the required version in the message.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/7377\r\n\r\n### Conversations in the completions table\r\n\r\nPrompts and completions are logged as raw conversation lists instead of `batch_decode`'d flat strings, so the table renders turn by turn. Much more readable for multi-turn and tool-calling setups, and for VLMs.\r\n\r\n<img width=\"1627\" alt=\"Completions table rendered turn by turn\" src=\"https://github.com/user-attachments/assets/649cf786-d4e5-4955-b486-ff7cd8d7439b\" />\r\n\r\nThe rendered system prompt (including tool definitions injected by the chat template) is no longer visible, since the original messages are shown rather than the fully-templated string.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5309\r\n\r\n### AsyncGRPO and AsyncDistillation\r\n\r\n* AsyncGRPO: train OpenEnv harnesses from validated token captures by @adithya-s-k in https://github.com/huggingface/trl/pull/6947\r\n* Add PEFT support to `AsyncDistillationTrainer` by @kashif in https://github.com/huggingface/trl/pull/7476\r\n* Fix duplicate completions in AsyncGRPO groups with data-parallel vLLM by @lewtun in https://github.com/huggingface/trl/pull/7549\r\n\r\n### Other\r\n\r\n* Add SmolVLM original and training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5868\r\n* Declare gradient accumulation loss scaling with `loss_is_scaled_for_ga` by @qgallouedec in https://github.com/huggingface/trl/pull/7508\r\n* Keep SFT's MoE aux loss independent of world size and gradient accumulation by @qgallouedec in https://github.com/huggingface/trl/pull/7570\r\n* Discover environment tools on the class, not the instance by @Rome-1 in https://github.com/huggingface/trl/pull/7360\r\n* Accept an IPv6 `base_url` in `VLLMClient` by @qgallouedec in https://github.com/huggingface/trl/pull/7462\r\n* Pass processor kwargs via `processor_kwargs` in the VLM SFT collator by @qgallouedec in https://github.com/huggingface/trl/pull/7547\r\n* Call `trainer.end()` at the end of the scripts and examples by @qgallouedec in https://github.com/huggingface/trl/pull/7384\r\n* Install vLLM in the TRL Docker images by @qgallouedec in https://github.com/huggingface/trl/pull/7464\r\n* Train harnesses through Harbor: drop the standalone opencode example by @sergiopaniego in https://github.com/huggingface/trl/pull/7457\r\n* Add support for vLLM 0.31.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7567\r\n\r\n## Breaking changes and removals\r\n\r\n* **Drop Python 3.10 support** by @qgallouedec in https://github.com/huggingface/trl/pull/7497\r\n* **Refuse `nn.DataParallel`** (and drop the multi-GPU slow test job) by @qgallouedec in https://github.com/huggingface/trl/pull/7407\r\n* **Remove the experimental MiniLLM trainer** by @qgallouedec in https://github.com/huggingface/trl/pull/7500\r\n* **`use_liger_kernel` is deprecated** in `DPOTrainer`, `KTOTrainer`, `GRPOTrainer` and `RLOOTrainer` and will be removed in v2.0.0. These trainers now use the fused LM head, so only Liger's layer kernels apply; use `model_init_kwargs={\"use_kernels\": True}` instead. Docs updated by @qgallouedec in https://github.com/huggingface/trl/pull/7535\r\n* Drop vLLM 0.20.0 / 0.20.1 / 0.20.2 support by @qgallouedec in https://github.com/huggingface/trl/pull/7435, https://github.com/huggingface/trl/pull/7499 and https://github.com/huggingface/trl/pull/7568\r\n* Remove the unreachable `prepare_multimodal_messages_vllm` by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6946\r\n\r\n## Fixes\r\n\r\n* **Fix `SFTTrainer` not appending EOS when `dataset_text_field` is not `\"text\"`** by @JoeyTan21 in https://github.com/huggingface/trl/pull/7446\r\n* **End completions on every eos id the model declares** in GRPO, RLOO and Distillation by @albertvillanova in https://github.com/huggingface/trl/pull/7505, and in the experimental trainers in https://github.com/huggingface/trl/pull/7506\r\n* **Start the completion where the tokenized prompt and prompt+completion diverge** by @qgallouedec in https://github.com/huggingface/trl/pull/7463\r\n* **Fix the server-mode crash at the first weight sync on vLLM 0.20 to 0.25** by @albertvillanova in https://github.com/huggingface/trl/pull/7410\r\n* **Generate one completion per sample after tool calls in vLLM server mode** by @qgallouedec in https://github.com/huggingface/trl/pull/7418\r\n* **Fix `precompute_ref_log_probs` under FSDP with no reference model** by @qgallouedec in https://github.com/huggingface/trl/pull/7507\r\n* **Keep the DFT loss finite on a batch with no trainable tokens** by @albertvillanova in https://github.com/huggingface/trl/pull/7471\r\n* **Guard the `requests` and `urllib3` imports in the vLLM client** by @qgallouedec in https://github.com/huggingface/trl/pull/7545, and in the experimental async vLLM clients by @albertvillanova in https://github.com/huggingface/trl/pull/7561\r\n* `fix(cpo, orpo)`: truncate responses independently to prevent empty completions by @RohanMali2003 in https://github.com/huggingface/trl/pull/6588\r\n* Fix FLOPs overcount for models with untied word embeddings by @jayzuccarelli in https://github.com/huggingface/trl/pull/7450\r\n* `[MFU]` Use device peak flops instead of hardcoded H100's by @dwarez in https://github.com/huggingface/trl/pull/7197\r\n* Fix segfault on Apple Silicon when loading bf16 checkpoints from a model ID by @albertvillanova in https://github.com/huggingface/trl/pull/7455\r\n* Keep the DoRA magnitude vector in float32 under QLoRA by @qgallouedec in https://github.com/huggingface/trl/pull/7342\r\n* Fix missing OpenReward outcome rewards by @adithya-s-k in https://github.com/huggingface/trl/pull/7467\r\n* Keep the next turn's first token out of the Llava-Next assistant mask by @albertvillanova in https://github.com/huggingface/trl/pull/7375\r\n* Tie weights after loading remote-code models with transformers 5.3.0 by @albertvillanova in https://github.com/huggingface/trl/pull/7559\r\n* Make SDFT importable without peft and align SSD by @albertvillanova in https://github.com/huggingface/trl/pull/7562\r\n* Preserve explicit `model_init_kwargs` in CLI scripts by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/7394\r\n* Honor `resume_from_checkpoint` in CLI scripts by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/7391\r\n* Seed before creating the model so the PEFT adapter init is reproducible by @qgallouedec in https://github.com/huggingface/trl/pull/7385, and in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7401\r\n* Drop the redundant spawn start method from `trl vllm-serve` by @albertvillanova in https://github.com/huggingface/trl/pull/7379\r\n* `FIX`: `test_offloading_with_peft_models` on L40, register model params in activation offloading tests by @AmineDiro in https://github.com/huggingface/trl/pull/7380\r\n* Drop the dead `gate_proj` LoRA target from the Nemotron 3 SFT example by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6866\r\n* Target the Mamba `in_proj` in the Nemotron 3 LoRA example by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6870\r\n* Replace the introspection guard in `sft_qwen3_8b_1m_context.py` with a version check by @kumarrah2002 in https://github.com/huggingface/trl/pull/7537\r\n\r\n## Documentation\r\n\r\n* Expand the Jobs training guide by @davanstrien in https://github.com/huggingface/trl/pull/7441\r\n* Document `expandable_segments` allocator config for memory tuning by @akshansh47 in https://github.com/huggingface/trl/pull/5794\r\n* Update \"What's New\" for the fused LM head by @qgallouedec in https://github.com/huggingface/trl/pull/7593\r\n* Document every stored chat template and check it in CI by @albertvillanova in https://github.com/huggingface/trl/pull/7502\r\n* Document what goes into a patch release by @albertvillanova in https://github.com/huggingface/trl/pull/7440\r\n* Document input validation guidelines by @albertvillanova in https://github.com/huggingface/trl/pull/7472\r\n* Unwrap hard-wrapped paragraphs in the docs by @qgallouedec in https://github.com/huggingface/trl/pull/7405\r\n* Discourage `SimpleNamespace`, `object.__new__` and mocks in tests in the agent instructions by @qgallouedec in https://github.com/huggingface/trl/pull/7459\r\n* Require first-time contributors to link an issue assigned to them by @qgallouedec in https://github.com/huggingface/trl/pull/7313\r\n* Extend the AI usage policy to issues by @albertvillanova in https://github.com/huggingface/trl/pull/7473\r\n* Require bug reports to state how the problem was hit by @albertvillanova in https://github.com/huggingface/trl/pull/7474\r\n* Label CONTRIBUTING.md changes as documentation by @albertvillanova in https://github.com/huggingface/trl/pull/7475\r\n\r\n## CI\r\n\r\n**Chat template coverage** — the stored templates are now documented, checked in CI, and watched weekly against the reference Hub repos, all by @albertvillanova:\r\n\r\n* Document every stored chat template and check it in CI (#7502), add a script checking that reference Hub repos still ship a stored chat template (#7523), check it weekly (#7521)\r\n* Recognize the current LFM2 (#7494) and LFM2.5 (#7493) chat templates, the other Gemma 4 revisions (#7495), parse the current LFM2 tool calls (#7522), keep the earlier LFM2 revision under test (#7527), read templates shipped as named variants by their default variant (#7530)\r\n\r\n**Tiny-model config alignment**, continuing the sweep, all by @albertvillanova: DeepSeek-V3 (#7398) and V3-0528 (#7399), GPT-OSS (#7424), Qwen2-VL (#7425) and Qwen2.5-VL (#7426) mrope sections, Nemotron-3-Nano (#7443), Nemotron-3.5-Lightning (#7444), Nemotron-3-Super (#7447), Nemotron-3-Ultra (#7448), LFM2 (#7400), LFM2.5 (#7532)\r\n\r\n**Test quality**, following the \"real objects, not mocks\" rule, all by @albertvillanova: build real trainers instead of `object.__new__` in the GRPO and SDFT tests (#7531), build real configs instead of `SimpleNamespace` in the GOLD and GKD tests (#7564), build the async rollout loops through `__init__` (#7555), run the async GRPO rollout loop tests on the real tokenizer and response parser (#7556), align the DFT loss tests on a real model output (#7481)\r\n\r\n**Suite hygiene**, all by @albertvillanova unless noted: move the vLLM training (#7488), continuous batching (#7487) and SFT activation offloading (#7477) tests to the regular suite, unmark the vLLM client/server tests as slow (#7490), run them on a single xdist worker (#7489), replace the SFT slow tests with an fp16 test (#7550), remove the always-skipped Gemma 3n tests (#7485), drop the no-op slow and low_priority markers (#7486), stop piping the vLLM test server output (#7414), run the VLM test server on the last visible accelerator (#7412), test the vLLM client against the documented `vllm serve` command (#7378), inject `MODEL_REVISIONS` into the Auto loaders (#7528), mirror the pad token onto the model config in the callback tests (#7383), remove the DataParallel gather warning filter (#7554), tie the mamba-ssm autocast warning filter removal to transformers 5.6 (#7431), remove the nightly cache cleanup workflow (#7395), scope `GITHUB_TOKEN` permissions per job by @hf-security-analysis (#7381), bump the doc-builder workflow pin by @paulinebm (#7542), Dependabot group bump (#7411)\r\n\r\n**Hotfixes**: xfail the GRPO continuous batching test under DataParallel (#7422), xfail Qwen3.5-MoE VLM tests against a broken transformers dev build (#7437) and revert once fixed (#7460)\r\n\r\n## New Contributors\r\n* @dwarez made their first contribution in https://github.com/huggingface/trl/pull/7197\r\n* @Rome-1 made their first contribution in https://github.com/huggingface/trl/pull/7360\r\n* @JoeyTan21 made their first contribution in https://github.com/huggingface/trl/pull/7446\r\n* @jayzuccarelli made their first contribution in https://github.com/huggingface/trl/pull/7450\r\n* @RohanMali2003 made their first contribution in https://github.com/huggingface/trl/pull/6588\r\n* @kumarrah2002 made their first contribution in https://github.com/huggingface/trl/pull/7537\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.14.0...v1.15.0\r\n","publishedAt":"2026-10-08T19:26:48.000Z","fetchedAt":"2026-10-08T21:18:56.412Z","url":"https://github.com/huggingface/trl/releases/tag/v1.15.0","media":[{"type":"image","url":"https://github.com/user-attachments/assets/329793d4-cef1-49ae-9ab6-5b1387d13071","r2Key":"releases/5018e7c1f394cb3e3ea542b77c448f9f4f8f88e3ae80f66965526184ef7ae059.png","r2Url":"https://media.releases.sh/releases/5018e7c1f394cb3e3ea542b77c448f9f4f8f88e3ae80f66965526184ef7ae059.png"},{"type":"image","url":"https://github.com/user-attachments/assets/649cf786-d4e5-4955-b486-ff7cd8d7439b","r2Key":"releases/e88323bb69b7998dd7e0ed6ebc74d276302e05db4874891056c8dda536564eaf.png","r2Url":"https://media.releases.sh/releases/e88323bb69b7998dd7e0ed6ebc74d276302e05db4874891056c8dda536564eaf.png"}],"coverageCount":0},{"id":"rel_H7vNfWdQk6TovPDaKnm17","version":"v1.14.2","type":"feature","title":"v1.14.2","summary":"GRPO, RLOO, and Distillation on the default non-vLLM generation path now stop completions at every eos id the model declares, fixing silent training on tokens generated past the end of the turn that showed up as completions/clipped_ratio near 1. Also fixes DFT loss on batches with no trainable tokens, precompute_ref_log_probs under FSDP with no reference model, completion offsets where the tokenized prompt and prompt+completion diverge, and guards the requests and urllib3 imports in the vLLM client.","titleGenerated":"TRL v1.14.2 fixes GRPO/RLOO training on tokens past end-of-turn","titleShort":"GRPO/RLOO stop at end-of-turn ids; DFS loss stays finite","breaking":"none","importance":3,"content":"Patch release fixing two cases of silently wrong training and three crashes.\r\n\r\nWorth calling out: **#7505** affects GRPO, RLOO and Distillation on the default generation path (`use_vllm=False`). Models that end a turn with an id only their generation config declares (Gemma 3/4, Phi-3.5, ...) did not stop at the end of the turn, and everything generated after it was trained on, up to `max_completion_length`. The only visible sign was `completions/clipped_ratio` close to 1. With vLLM, generation stopped correctly but the metrics were still wrong, and `mask_truncated_completions=True` dropped every finished completion. Runs on models whose tokenizer eos is their only end-of-turn id are unchanged.\r\n\r\n## What's Changed\r\n\r\n* Keep the DFT loss finite on a batch with no trainable tokens by @albertvillanova in https://github.com/huggingface/trl/pull/7471\r\n* Fix `precompute_ref_log_probs` under FSDP with no reference model by @qgallouedec in https://github.com/huggingface/trl/pull/7507\r\n* Fix: Start the completion where the tokenized prompt and prompt+completion diverge by @qgallouedec in https://github.com/huggingface/trl/pull/7463\r\n* End completions on every eos id the model declares in GRPO, RLOO and Distillation by @albertvillanova in https://github.com/huggingface/trl/pull/7505\r\n* Guard the requests and urllib3 imports in the vLLM client by @qgallouedec in https://github.com/huggingface/trl/pull/7545\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.14.1...v1.14.2\r\n","publishedAt":"2026-10-06T20:30:35.000Z","fetchedAt":"2026-10-06T21:18:34.933Z","url":"https://github.com/huggingface/trl/releases/tag/v1.14.2","media":[],"coverageCount":0},{"id":"rel_vPHZ9rpgU-8rpBiQcc-Kj","version":"v1.14.1","type":"feature","title":"v1.14.1","summary":"Fixed a server-mode crash at the first weight sync on vLLM 0.20 through 0.25, and vLLM server mode now generates one completion per sample after tool calls. CLI scripts again honor resume_from_checkpoint and preserve explicit model_init_kwargs.","titleGenerated":"TRL v1.14.1 fixes server-mode crash on first weight sync in vLLM","titleShort":"vLLM server-mode weight sync crash fixed","breaking":"none","importance":2,"content":"## What's Changed\r\n\r\n* Fix the server-mode crash at the first weight sync on vLLM 0.20 to 0.25 by @albertvillanova in https://github.com/huggingface/trl/pull/7410\r\n* [Fix] Generate one completion per sample after tool calls in vLLM server mode by @qgallouedec in https://github.com/huggingface/trl/pull/7418\r\n* Preserve explicit model_init_kwargs in CLI scripts by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/7394\r\n* Honor resume_from_checkpoint in CLI scripts by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/7391\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.14.0...v1.14.1","publishedAt":"2026-09-29T13:41:37.000Z","fetchedAt":"2026-09-29T17:17:59.180Z","url":"https://github.com/huggingface/trl/releases/tag/v1.14.1","media":[],"coverageCount":0},{"id":"rel_oGhz1Hr_wFgUSzwDM751v","version":"v1.14.0","type":"feature","title":"v1.14.0","summary":"The trl.losses module introduced in v1.13 is removed, with DPO, KTO, and GRPO now streaming their own per-token log-probs through _ChunkedLogProbFunction, which also unlocks DPO with precomputed reference log-probs that did not exist before. A fused Triton logprob and entropy kernel now ships in-tree at trl.kernels and is on by default on CUDA, ROCm, and XPU, and six experimental trainers were removed from trl.experimental.","titleGenerated":"TRL v1.14.0 removes trl.losses, ships fused Triton logprob kernel","titleShort":"trl.losses gone; fused Triton logprob kernel on by default","breaking":"major","importance":4,"content":"## Features\r\n\r\n### `trl.losses` is gone: DPO, KTO and GRPO stream their own log-probs\r\n\r\n> [!WARNING]\r\n> `from trl.losses import FusedLinearDPOLoss` (or `FusedLinearKTOLoss`, `FusedLinearGRPOLoss`, `FusedLinearJSDLoss`) no longer works. The module introduced in v1.13 has been removed.\r\n\r\nv1.13 vendored Liger's `chunked_loss` into `trl.losses` as a holding action, not a destination (#7063). The problem it was holding: those classes reimplement each trainer's loss math, so TRL carried two copies of every formula, and copies drift. That drift is where the bugs were. GRPO's `grad_norm` differed from the default path, KTO ignored the reference model under PEFT, KTO class weights and DPO `label_smoothing` were silently dropped. Worst of all, under plain DDP the fused path ran the loss on the unwrapped model, so `DistributedDataParallel`'s reducer was never armed and gradients were never all-reduced: every rank silently kept its own.\r\n\r\nThe fix is to stop reimplementing. All DPO, KTO and GRPO need from the fused path is per-token log-probs of the selected tokens, and TRL already had `_ChunkedLogProbFunction` for exactly that: it streams the vocabulary with an online logsumexp and recomputes in the backward pass. Each trainer now runs `backbone -> _ChunkedLogProbFunction -> its own existing loss code`, unchanged. One implementation of each loss, and the memory win is kept, because full logits are still never materialized.\r\n\r\nThirteen restrictions existed only because the loss had been rewritten in a form that could not express those options. Most are gone: DPO with `use_liger_kernel=True` now accepts mixed loss types, f-divergences and precomputed reference log-probs, and GRPO gains entropy and off-policy masking. Still refused: `use_weighting`, `compute_metrics`, a PEFT adapter on `lm_head`, prompt-learning PEFT, and the MoE auxiliary loss.\r\n\r\nSingle H100, Qwen3-0.6B, `bs=4`, seq 512:\r\n\r\n| config | median step | peak |\r\n| --- | --- | --- |\r\n| v1.13 fused, inline ref | 0.2243 s | 7.05 GB |\r\n| v1.14 chunked, inline ref | 0.2457 s (+9.5%) | 7.05 GB |\r\n| v1.14 chunked, `precompute_ref_log_probs=True` | **0.1971 s (-12.1%)** | **4.83 GB (-31%)** |\r\n\r\nThe third row is the point: that configuration did not exist before, because the fused path rejected precomputed reference log-probs outright.\r\n\r\n`use_liger_kernel=True` still works and still enables Liger's model kernels through transformers. In DPO, GRPO and KTO it now selects TRL's chunked log-probability path instead of Liger's fused loss.\r\n\r\n* Stream KTO log probabilities by @kashif in https://github.com/huggingface/trl/pull/7075\r\n* Stream GRPO log probabilities by @kashif in https://github.com/huggingface/trl/pull/7077\r\n* Move DPO to chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7243\r\n* Drop the fused Liger JSD path from the experimental distillation trainers by @kashif in https://github.com/huggingface/trl/pull/7301\r\n* Skip frozen chunked log probability gradients by @kashif in https://github.com/huggingface/trl/pull/7076\r\n* Redirect the KTO liger loss through the distributed wrapper under DDP by @qgallouedec in https://github.com/huggingface/trl/pull/7247\r\n* Document what use_liger_kernel actually does in each trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7361\r\n\r\n### A fused Triton logprob + entropy kernel now ships in TRL\r\n\r\nThe loss head is the only hot path TRL owns: transformers already kernelizes the model internals, and nothing on the Hub covers what happens after the decoder. On one H100 (bf16, `V=151936`, `H=4096`, 8192 tokens), `selective_log_softmax` plus `entropy_from_logits` cost 12.8 ms and 2.32 GiB over `[8, 1024, 151936]`. A fused Triton kernel does the same in 0.89 ms, roughly 14x, and lands closer to the fp32 reference than the bf16 path it replaces. GRPO runs this two to four times per step.\r\n\r\nIt now lives in-tree at `trl.kernels` and is on by default on CUDA, ROCm and XPU, with the PyTorch implementation as fallback. It was briefly loaded from the Hub; that was reversed because this kernel is pure Triton and compiles at runtime, so the Hub bought nothing while costing a version gate unrelated to whether the kernel works, a download that fails offline, `trust_remote_code=True` for anyone with `kernels` installed, a silent `except Exception: pass` wrapped around all of it, and 13 of 13 tests skipped in CI.\r\n\r\n* Use fused kernels for logprobs and entropy by @kashif in https://github.com/huggingface/trl/pull/7253\r\n* Move TRL loss kernels into TRL by @kashif in https://github.com/huggingface/trl/pull/7266\r\n* Ship the logprob and entropy kernel in trl instead of the Hub by @qgallouedec in https://github.com/huggingface/trl/pull/7363\r\n\r\n### Other\r\n\r\n* Ensure device agnostic for examples/docs/templates by @kaixuanliu in https://github.com/huggingface/trl/pull/6770\r\n* AsyncGRPO: PEFT/LoRA support with adapter-only vLLM sync by @AmineDiro in https://github.com/huggingface/trl/pull/7017\r\n* Add support for vLLM 0.29.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7178\r\n* [GOLD] Recommend LFM2.5 and Gemma 4 students by @kashif in https://github.com/huggingface/trl/pull/6388\r\n* Add TailSFT example by @kashif in https://github.com/huggingface/trl/pull/7212\r\n* [AsyncGRPO] Run sync tools on a thread pool and support async tools by @AmineDiro in https://github.com/huggingface/trl/pull/7175\r\n\r\n## Breaking\r\n\r\n### Six experimental trainers removed\r\n\r\n`GRPOWithReplayBufferTrainer`, `BCOTrainer`, `PRMTrainer`, `XPOTrainer`, `NashMDTrainer` and the GSPO-token trainer are gone from `trl.experimental`.\r\n\r\nWhat changed is not the guarantee, which was always that `trl.experimental` has none, but the fact that the decision is now written down. Removals used to be argued one pull request at a time, with every thread relitigating the policy before it got to the feature. #7182 settled what the judgment weighs, and #7254 put it in the docs: usage, external issues and pull requests that come from people actually running the feature, whether a stable trainer already covers it, maintenance cost, downstream consumers, owner, and age. Deliberately inputs to a judgment rather than a test: no threshold, no notice cycle, and the numbers stay in the removal pull request where you can check them. A paper index entry outlives the code it described.\r\n\r\nThe numbers are what make the case. `BCOTrainer` was 1,600 lines carrying its own reference-model handling, DeepSpeed preparation, tokenization and eval loop, for 11 genuine trainings in two months; it was also the only reason TRL depended on scikit-learn and joblib at all, and both are now gone from `pyproject.toml`. `GRPOWithReplayBufferTrainer` had hand-copied `_generate_and_score_completions` from GRPO and drifted to 385 of 616 lines different, for 8 trainings.\r\n\r\n* Remove the experimental BCO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7136\r\n* Remove the experimental GRPO-with-replay-buffer trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7132\r\n* Remove the experimental PRM trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7133\r\n* Remove the experimental Nash-MD trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7255\r\n* Remove the experimental XPO trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7256\r\n* Remove the experimental GSPO-token trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7331\r\n\r\nAlso breaking, and covered above: `trl.losses` was removed along the way.\r\n\r\n* Move DPO to chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7243\r\n\r\n## Fixes\r\n\r\n* **Gradients were never all-reduced for KTO under plain DDP.** `compute_loss` redirected through the distributed wrapper only for ZeRO-3 and FSDP, so on plain DDP the loss ran on the unwrapped model, `DistributedDataParallel.forward()` never ran, its reducer was never armed, and every rank silently kept its own gradients. By @qgallouedec in https://github.com/huggingface/trl/pull/7247\r\n* **Wrapped packing duplicated data when the input table was sliced.** `_pack_wrapped` sized the output from the sliced view but read from the raw child buffer, so it re-read from the start of the buffer instead of from the slice, and `datasets.map(batched=True)` passes sliced tables. By @qgallouedec in https://github.com/huggingface/trl/pull/6670\r\n* **A reward function logging on only some ranks hung or crashed the run.** GRPO and RLOO looped over the keys each rank held and called one collective per key, so ranks called different collectives: `log_metric` on rank 0 alone hung on NCCL, and `log_extra` crashed inside `gather_object` with a 1EB allocation. By @behroozazarkhalili in https://github.com/huggingface/trl/pull/7382\r\n* **Special token ids now reach the config the `Trainer` actually reads**, so it no longer realigns them at train time and logs a change the user did not make (part of #7093): eos in the trainers (#7127), written to the text config for composite models (#7315), the reward model pad token (#7318), and the requested pad token applied to vision datasets (#7316, #7352)\r\n* Do not quantize the teacher by default in the distillation and GOLD entry points by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6769\r\n* Prepare the dataset once per node when the datasets cache is node-local by @qgallouedec in https://github.com/huggingface/trl/pull/7241\r\n* Read the MoE auxiliary loss coefficient from the model config by @qgallouedec in https://github.com/huggingface/trl/pull/7248\r\n* Await async reward functions written as callable classes by @albertvillanova in https://github.com/huggingface/trl/pull/7213\r\n* Fix two stream races in OffloadActivations by @kashif in https://github.com/huggingface/trl/pull/7195\r\n* Only dedupe offloaded activation storages in streams mode by @kashif in https://github.com/huggingface/trl/pull/7193\r\n* Pass the FSDP2 ignored_params kwarg only on torch 2.7 or later by @albertvillanova in https://github.com/huggingface/trl/pull/7259\r\n* Guard the torch 2.6 APIs used by FSDP2 and online DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7260\r\n* Reduce connection-pool warnings in the vLLM client by @vladbataev in https://github.com/huggingface/trl/pull/7199\r\n\r\n## Documentation\r\n\r\n* **Document how a feature leaves `trl.experimental`** by @albertvillanova in https://github.com/huggingface/trl/pull/7254, the rule the six removals above were made under\r\n* Document what use_liger_kernel actually does in each trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7361\r\n* Document that AsyncGRPO tools must be module-level functions by @albertvillanova in https://github.com/huggingface/trl/pull/7226\r\n* Remove outdated \"under construction\" banners from docs by @qgallouedec in https://github.com/huggingface/trl/pull/7177\r\n* docs: fix 'allows to scale' grammar in DeepSpeed guide by @mrchatam in https://github.com/huggingface/trl/pull/7196\r\n\r\n## CI and maintenance\r\n\r\n**Tiny models**, a sweep bringing every tiny test model in line with its reference, all by @albertvillanova:\r\n\r\n* Aligned with their references: Qwen2.5-Coder (#7134), DeepSeek-R1-Distill (#7135), FalconMamba (#7139), Mistral v0.1 (#7203) and v0.2 (#7204), Gemma (#7217), Gemma2 (#7228), OPT (#7233), Phi-3 (#7257), Phi-3.5 (#7276), plus `tie_word_embeddings` across the board (#7130)\r\n* Fields mirrored in a second pass: `use_mrope` (#7332), `use_mambapy` (#7333), `rope_scaling` (#7334), `cache_implementation` and `hidden_act` (#7335), `activation_dropout` and `prefix` (#7336), `attention_bias` (#7337, #7338)\r\n* Generators: seeded so regeneration is idempotent and pushes nothing (#7258), pointed at a FalconMamba reference (#7138), `head_dim` reduced for Gemma (#7216) and Gemma2 (#7227), `ffn_dim` passed for OPT (#7232), the Phi3 leftover replaced by the 3 and 3.5 variants (#7239), three with no test consumer retired (#7186)\r\n* Coverage: the canonical GPTNeoX sequence-classification id (#7185), the tiny DeepSeek-R1-Distill trained in the SFT test (#7187), the dtype check skipped for non-safetensors references (#7238), the chat-template tests off tiny Bloom (#7262)\r\n\r\n**Tests run only when they can be affected**, all by @albertvillanova:\r\n\r\n* Skipped when nothing they cover changed (#7171, #7281), and run when test data or shipped package data does (#7283), or when the experimental workflow itself changes (#7278); code quality moved to its own workflow (#7277)\r\n* Superseded pull request runs cancelled (#7170), the doc build keyed on the pull request number (#7174). The same grouping was applied to `main` (#7172) and then reverted (#7201), because GitHub buckets `cancelled` with failure and `main` looked broken.\r\n\r\n**Release automation.** Publishing now happens on a tag push, and refuses unless `VERSION` in the tagged commit matches the tag (#7165). Release notes come out sectioned, from a label-to-section map (#7166) with documentation and maintenance labels applied automatically from the changed paths (#7205, #7210). Plus the dev version bump (#7160).\r\n\r\n**Test reliability**, all by @albertvillanova: the rerun filter matches the out-of-memory message rather than the exception type (#7292) and needs `pytest-rerunfailures>=16.7` to see wrapped exceptions (#7325); `ACCELERATE_MIXED_PRECISION` is restored between tests (#7321); the Llava assistant-mask tests are un-xfailed now that transformers fixed them (#7373); non-identity adapters in the `use_adapter` tests (#7275); `MODEL_REVISIONS` injected into the PEFT loaders (#7288); the reward model pad token covered (#7350).\r\n\r\n**Workflow hardening.** Token permissions narrowed in the test workflows (#7287), the workflows hardened (#7300), and the upstream issue behind the trufflehog digest pin linked (#7214).\r\n\r\n**Other.** Dependabot bumps (#7192, #7299), the latest-release tests moved from T4 to L40S (#7184) with the Slack action checkout workaround dropped (#7183), the kernel publish job run under a personal namespace (#7343), the token sync comments reworded (#7167) and all three generation-config eos shapes covered (#7168), LM-head gradient buffers and GEMMs skipped when its parameters are frozen (#7076), and the end-of-turn loss mask warning made actionable (#7230).\r\n\r\n## Dependencies\r\n\r\nThe supported vLLM window moves from `>=0.19.1,<=0.28.0` to `>=0.20.0,<=0.30.0`.\r\n\r\n* Add support for vLLM 0.30.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7365\r\n* Drop vLLM 0.19.1 support by @qgallouedec in https://github.com/huggingface/trl/pull/7366\r\n\r\n## New Contributors\r\n\r\n* @mrchatam made their first contribution in https://github.com/huggingface/trl/pull/7196\r\n* @itwangwang made their first contribution in https://github.com/huggingface/trl/pull/7230\r\n* @SuryanshSS1011 made their first contribution in https://github.com/huggingface/trl/pull/6670\r\n* @vladbataev made their first contribution in https://github.com/huggingface/trl/pull/7199\r\n\r\n## What's Changed\r\n\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/7160\r\n* Sync the model config eos token id with the tokenizer in the trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7127\r\n* Align tie_word_embeddings with the reference models by @albertvillanova in https://github.com/huggingface/trl/pull/7130\r\n* Ensure device agnostic for examples/docs/templates by @kaixuanliu in https://github.com/huggingface/trl/pull/6770\r\n* Remove the experimental GRPO-with-replay-buffer trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7132\r\n* Align tiny Qwen2.5-Coder config with Qwen/Qwen2.5-Coder-0.5B by @albertvillanova in https://github.com/huggingface/trl/pull/7134\r\n* Reword the token sync comments to say what the line does by @albertvillanova in https://github.com/huggingface/trl/pull/7167\r\n* Cover the three generation config eos token shapes in the sync test by @albertvillanova in https://github.com/huggingface/trl/pull/7168\r\n* Align tiny DeepSeek-R1-Distill config with deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B by @albertvillanova in https://github.com/huggingface/trl/pull/7135\r\n* Point the tiny FalconMamba generator at a FalconMamba reference by @albertvillanova in https://github.com/huggingface/trl/pull/7138\r\n* Align tiny FalconMamba config with tiiuae/falcon-mamba-7b-instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7139\r\n* AsyncGRPO: PEFT/LoRA support with adapter-only vLLM sync by @AmineDiro in https://github.com/huggingface/trl/pull/7017\r\n* Stream KTO log probabilities by @kashif in https://github.com/huggingface/trl/pull/7075\r\n* Skip frozen chunked log probability gradients by @kashif in https://github.com/huggingface/trl/pull/7076\r\n* Cancel superseded PR runs in the test workflows by @albertvillanova in https://github.com/huggingface/trl/pull/7170\r\n* Add support for vLLM 0.29.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7178\r\n* Skip the tests on pushes that change no tested file by @albertvillanova in https://github.com/huggingface/trl/pull/7171\r\n* Cancel superseded runs on main by @albertvillanova in https://github.com/huggingface/trl/pull/7172\r\n* docs: fix 'allows to scale' grammar in DeepSpeed guide by @mrchatam in https://github.com/huggingface/trl/pull/7196\r\n* Remove outdated \"under construction\" banners from docs by @qgallouedec in https://github.com/huggingface/trl/pull/7177\r\n* Move the latest release tests from T4 to L40S by @albertvillanova in https://github.com/huggingface/trl/pull/7184\r\n* Use the canonical id for the tiny GPTNeoX sequence classification model by @albertvillanova in https://github.com/huggingface/trl/pull/7185\r\n* Train the tiny DeepSeek-R1-Distill model in the SFT test by @albertvillanova in https://github.com/huggingface/trl/pull/7187\r\n* Fix two stream races in OffloadActivations by @kashif in https://github.com/huggingface/trl/pull/7195\r\n* Only dedupe offloaded activation storages in streams mode by @kashif in https://github.com/huggingface/trl/pull/7193\r\n* Stop cancelling superseded runs on main by @albertvillanova in https://github.com/huggingface/trl/pull/7201\r\n* Key the doc build concurrency group on the pull request number by @albertvillanova in https://github.com/huggingface/trl/pull/7174\r\n* Drop the Slack action checkout workaround from the latest release tests by @albertvillanova in https://github.com/huggingface/trl/pull/7183\r\n* Retire three tiny model generators with no test consumer by @albertvillanova in https://github.com/huggingface/trl/pull/7186\r\n* Publish to PyPI on tag push and guard VERSION against the tag by @albertvillanova in https://github.com/huggingface/trl/pull/7165\r\n* Section the auto-generated release notes by label by @albertvillanova in https://github.com/huggingface/trl/pull/7166\r\n* Automatically label documentation and maintenance PRs by @albertvillanova in https://github.com/huggingface/trl/pull/7205\r\n* chore: update labeler.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/7210\r\n* Align tiny Mistral v0.1 config with mistralai/Mistral-7B-Instruct-v0.1 by @albertvillanova in https://github.com/huggingface/trl/pull/7203\r\n* Align tiny Mistral v0.2 config with mistralai/Mistral-7B-Instruct-v0.2 by @albertvillanova in https://github.com/huggingface/trl/pull/7204\r\n* [GOLD] Recommend LFM2.5 and Gemma 4 students by @kashif in https://github.com/huggingface/trl/pull/6388\r\n* Bump the \"github-actions-and-pre-commit\" group with 2 updates across multiple ecosystems by @dependabot[bot] in https://github.com/huggingface/trl/pull/7192\r\n* Link the upstream issue behind the trufflehog digest pin by @albertvillanova in https://github.com/huggingface/trl/pull/7214\r\n* Await async reward functions written as callable classes by @albertvillanova in https://github.com/huggingface/trl/pull/7213\r\n* Reduce head_dim in the tiny Gemma generator by @albertvillanova in https://github.com/huggingface/trl/pull/7216\r\n* Do not quantize the teacher by default in the distillation and GOLD entry points by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6769\r\n* Align tiny Gemma config with google/gemma-7b-it by @albertvillanova in https://github.com/huggingface/trl/pull/7217\r\n* Reduce head_dim in the tiny Gemma2 generator by @albertvillanova in https://github.com/huggingface/trl/pull/7227\r\n* Replace the tiny-Phi3ForCausalLM leftover with the 3 and 3.5 variants by @albertvillanova in https://github.com/huggingface/trl/pull/7239\r\n* Make the end-of-turn loss mask warning actionable by @itwangwang in https://github.com/huggingface/trl/pull/7230\r\n* Fix wrapped packing duplicating data when the input table is sliced by @SuryanshSS1011 in https://github.com/huggingface/trl/pull/6670\r\n* Skip the dtype check when the reference is not a safetensors repo by @albertvillanova in https://github.com/huggingface/trl/pull/7238\r\n* Document that AsyncGRPO tools must be module-level functions by @albertvillanova in https://github.com/huggingface/trl/pull/7226\r\n* Pass ffn_dim instead of intermediate_size in the tiny OPT generator by @albertvillanova in https://github.com/huggingface/trl/pull/7232\r\n* Align tiny OPT config with facebook/opt-1.3b by @albertvillanova in https://github.com/huggingface/trl/pull/7233\r\n* Add TailSFT example by @kashif in https://github.com/huggingface/trl/pull/7212\r\n* [AsyncGRPO] Run sync tools on a thread pool and support async tools by @AmineDiro in https://github.com/huggingface/trl/pull/7175\r\n* Prepare the dataset once per node when the datasets cache is node-local by @qgallouedec in https://github.com/huggingface/trl/pull/7241\r\n* Remove the experimental BCO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7136\r\n* Remove the experimental PRM trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7133\r\n* Document how a feature leaves trl.experimental by @albertvillanova in https://github.com/huggingface/trl/pull/7254\r\n* Remove the experimental XPO trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7256\r\n* Remove the experimental Nash-MD trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7255\r\n* Seed the tiny model generation scripts by @albertvillanova in https://github.com/huggingface/trl/pull/7258\r\n* Align tiny Gemma2 config with google/gemma-2-2b-it by @albertvillanova in https://github.com/huggingface/trl/pull/7228\r\n* Align tiny Phi-3 config with microsoft/Phi-3-mini-4k-instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7257\r\n* Move TRL loss kernels into TRL by @kashif in https://github.com/huggingface/trl/pull/7266\r\n* Narrow the token permissions in the test workflows by @albertvillanova in https://github.com/huggingface/trl/pull/7287\r\n* Redirect the KTO liger loss through the distributed wrapper under DDP by @qgallouedec in https://github.com/huggingface/trl/pull/7247\r\n* Inject MODEL_REVISIONS into the PEFT loaders in tests by @albertvillanova in https://github.com/huggingface/trl/pull/7288\r\n* Run the experimental tests when their workflow changes by @albertvillanova in https://github.com/huggingface/trl/pull/7278\r\n* Move the code quality check to its own workflow by @albertvillanova in https://github.com/huggingface/trl/pull/7277\r\n* Stop running the tests for changes that cannot affect them by @albertvillanova in https://github.com/huggingface/trl/pull/7281\r\n* Stream GRPO log probabilities by @kashif in https://github.com/huggingface/trl/pull/7077\r\n* Restore ACCELERATE_MIXED_PRECISION between tests by @albertvillanova in https://github.com/huggingface/trl/pull/7321\r\n* Bump the \"github-actions-and-pre-commit\" group with 2 updates across multiple ecosystems by @dependabot[bot] in https://github.com/huggingface/trl/pull/7299\r\n* fix(ci): harden GitHub Actions workflows (#7299) by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/7300\r\n* Stop depending on the tiny Bloom model in the chat template tests by @albertvillanova in https://github.com/huggingface/trl/pull/7262\r\n* Use non identity adapters in the use_adapter tests by @albertvillanova in https://github.com/huggingface/trl/pull/7275\r\n* Run the tests when test data or shipped package data changes by @albertvillanova in https://github.com/huggingface/trl/pull/7283\r\n* Match the out-of-memory message instead of the exception type in the rerun filter by @albertvillanova in https://github.com/huggingface/trl/pull/7292\r\n* Raise the pytest-rerunfailures floor to 16.7 so the rerun filter matches wrapped exceptions by @albertvillanova in https://github.com/huggingface/trl/pull/7325\r\n* Pass the FSDP2 ignored_params kwarg only on torch 2.7 or later by @albertvillanova in https://github.com/huggingface/trl/pull/7259\r\n* Guard the torch 2.6 APIs used by FSDP2 and online DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7260\r\n* Align tiny Phi-3.5 config with microsoft/Phi-3.5-mini-instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7276\r\n* Mirror use_mrope in the tiny DeepSeek-R1-Distill config by @albertvillanova in https://github.com/huggingface/trl/pull/7332\r\n* Mirror use_mambapy in the tiny FalconMamba config by @albertvillanova in https://github.com/huggingface/trl/pull/7333\r\n* Use fused kernels for logprobs and entropy by @kashif in https://github.com/huggingface/trl/pull/7253\r\n* Run the kernel publish job under a personal namespace by @qgallouedec in https://github.com/huggingface/trl/pull/7343\r\n* Mirror rope_scaling in the tiny Gemma config by @albertvillanova in https://github.com/huggingface/trl/pull/7334\r\n* Mirror cache_implementation and hidden_act in the tiny Gemma2 config by @albertvillanova in https://github.com/huggingface/trl/pull/7335\r\n* Mirror activation_dropout and prefix in the tiny OPT config by @albertvillanova in https://github.com/huggingface/trl/pull/7336\r\n* Mirror attention_bias in the tiny Phi-3 config by @albertvillanova in https://github.com/huggingface/trl/pull/7337\r\n* Mirror attention_bias in the tiny Phi-3.5 config by @albertvillanova in https://github.com/huggingface/trl/pull/7338\r\n* Set the reward model pad token on the text config by @albertvillanova in https://github.com/huggingface/trl/pull/7318\r\n* Write the aligned token ids to the text config by @albertvillanova in https://github.com/huggingface/trl/pull/7315\r\n* Drop the fused Liger JSD path from the experimental distillation trainers by @kashif in https://github.com/huggingface/trl/pull/7301\r\n* Remove the experimental GSPO-token trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7331\r\n* Move DPO to chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7243\r\n* Apply the requested pad token to vision datasets in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7352\r\n* Reduce connection-pool warnings in the vLLM client by @vladbataev in https://github.com/huggingface/trl/pull/7199\r\n* Read the MoE auxiliary loss coefficient from the model config by @qgallouedec in https://github.com/huggingface/trl/pull/7248\r\n* Hotfix CI: Expect the Llava assistant masks tests to pass now that transformers fixed them by @albertvillanova in https://github.com/huggingface/trl/pull/7373\r\n* Test that the reward model pad token is set on the text config by @albertvillanova in https://github.com/huggingface/trl/pull/7350\r\n* Set the pad token for vision datasets too by @albertvillanova in https://github.com/huggingface/trl/pull/7316\r\n* Drop vLLM 0.19.1 support by @qgallouedec in https://github.com/huggingface/trl/pull/7366\r\n* Document what use_liger_kernel actually does in each trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7361\r\n* Add support for vLLM 0.30.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7365\r\n* Ship the logprob and entropy kernel in trl instead of the Hub by @qgallouedec in https://github.com/huggingface/trl/pull/7363\r\n* Agree on logged keys across ranks before flushing GRPO and RLOO metrics by @behroozazarkhalili in https://github.com/huggingface/trl/pull/7382\r\n* Release: v1.14 by @albertvillanova in https://github.com/huggingface/trl/pull/7392\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.13.0...v1.14.0\r\n","publishedAt":"2026-09-25T06:40:07.000Z","fetchedAt":"2026-09-25T09:16:39.252Z","url":"https://github.com/huggingface/trl/releases/tag/v1.14.0","media":[],"coverageCount":0},{"id":"rel_uJYz2MFBk1vhv7Ecg7aa0","version":"v1.13.0","type":"feature","title":"v1.13.0","summary":"PPOTrainer, PPOConfig, and the value-head modeling modules are removed. The default chunked_nll lm_head projection now runs on bf16 tensor cores instead of fp32 SIMT, cutting a 23.37 ms fwd+bwd chunk to 3.86 ms (−6×) and peak memory from 5.99 GB to 3.03 GB, with end-to-end tokens/s gains of 1.2–1.7× across models. Vendored fused linear losses (DPO, KTO, GRPO, JSD) into trl.losses, added a long-context guide for 1M-token training, raised dependency floors (peft, deepspeed, vLLM), and added QLoRA test coverage.","titleGenerated":"TRL v1.13.0 drops PPO trainer and accelerates chunked-NLL lm_head projection 6x","titleShort":"PPO trainer removed; chunked-CE lm_head 6× faster","breaking":"major","importance":4,"content":"## Features\r\n\r\n### Training beyond 1M tokens\r\n\r\n<img width=\"1672\" height=\"941\" alt=\"image\" src=\"https://github.com/user-attachments/assets/806f9153-d098-4a45-bef4-5d4119b7ef25\" />\r\n\r\n\r\nA new [long-context guide](https://huggingface.co/docs/trl/long_context_training) plus a runnable example that trains a book-length sequence per step on a single 8×H100 node.\r\n\r\nMeasured: **Qwen3-8B at 1,048,576 tokens, 380 s/step, 56.2 GB per GPU** (bf16, `per_device_train_batch_size=1`, `loss_type=\"chunked_nll\"`). Runs end to end and saves. The guide documents the levers in the order you hit them (`chunked_nll`, gradient-checkpointing `offload`, YaRN RoPE) and the constraints (full attention only, no packing). Needs `transformers >= 5.16` for checkpointing `offload`.\r\n\r\n* Document 1M-token training and add a context-parallelism example by @qgallouedec in https://github.com/huggingface/trl/pull/6846\r\n* Rewrite the long context guide by @qgallouedec in https://github.com/huggingface/trl/pull/7003\r\n\r\n\r\n### Chunked-CE `lm_head` projection now runs on tensor cores\r\n\r\nThe inner loop of the default `loss_type=\"chunked_nll\"` was doing `h.float() @ w.float().t()`. Both operands are already bf16, so the upcast bought no information: it moved the GEMM off the tensor cores onto the fp32 SIMT path and materialized an fp32 copy of the whole `lm_head` weight (2.03 GB for a 248k vocabulary), rebuilt for every chunk and again on every gradient-checkpoint recompute. In an 8×H100 profile of `trl sft` on Qwen3.6-35B-A3B those two fp32 SIMT GEMMs were **21.6% of all GPU kernel time**.\r\n\r\nOne chunk, 256 tokens × vocab 248,320 × hidden 2048, 1×H100, bf16, fwd+bwd:\r\n\r\n| | fwd+bwd | peak mem |\r\n| --- | --- | --- |\r\n| before | 23.37 ms | 5.99 GB |\r\n| after | **3.86 ms (6.0×)** | **3.03 GB** |\r\n\r\nEnd to end, 16,384 tokens per step, tokens/s/GPU:\r\n\r\n| model | mode | before | after | |\r\n| --- | --- | --- | --- | --- |\r\n| gemma-3-270m (vocab 262k) | full FT, 1×H100 | 24036 | **31609** | 1.32× |\r\n| Qwen3-0.6B (vocab 152k) | full FT, 1×H100 | 21276 | **26641** | 1.25× |\r\n| Qwen3-8B | full FT, 2×H100 FSDP2 | 3554 | **6009** | **1.69×** |\r\n| Qwen3-8B | LoRA r16, 2×H100 FSDP2 | 4531 | **7125** | **1.57×** |\r\n| Qwen3-30B-A3B (MoE) | LoRA attn, 2×H100 FSDP2 | 4251 | **5101** | 1.20× |\r\n\r\nSame fix applied to the distillation trainers (which paid it twice per chunk, student and teacher). Under accelerate mixed precision the numerics are bit-identical (autocast was already casting the operands back down); without autocast over the projection the GEMM moves to bf16 tensor cores, which is exactly what `loss_type=\"nll\"` already does.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6863\r\n\r\n### Fused linear losses vendored into `trl.losses`\r\n\r\nTRL's only import from `liger-kernel` was `liger_kernel.chunked_loss` (the fused linear DPO / KTO / GRPO / JSD losses). That module is pure PyTorch, and most of the GRPO and DPO variants in it were written for TRL. Upstream reviews had stalled, so the code comes home as `trl.losses`: `FusedLinearDPOLoss`, `FusedLinearKTOLoss`, `FusedLinearGRPOLoss`, `FusedLinearJSDLoss` (copied from Liger-Kernel v0.8.2, BSD-2 notice kept, bitwise identical to the installed Liger on random inputs).\r\n\r\nNo new config flag: `use_liger_kernel=True` keeps selecting the fused loss exactly as before, and still needs `liger-kernel` installed because transformers' `Trainer` patches the model kernels with it.\r\n\r\n* Vendor the fused linear losses from Liger-Kernel into `trl.losses` by @qgallouedec in https://github.com/huggingface/trl/pull/7059\r\n* Fix DPO and KTO Liger loss gaps by @kashif in https://github.com/huggingface/trl/pull/7062\r\n* Harden chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7065\r\n* Drop fused JSD from `DistillationTrainer` by @kashif in https://github.com/huggingface/trl/pull/7064\r\n* Test liger vs non-liger GRPO loss parity by @kashif in https://github.com/huggingface/trl/pull/6814\r\n* Run the vendored loss parity suites as slow tests only by @qgallouedec in https://github.com/huggingface/trl/pull/7067\r\n\r\n### Contributor skills at `.agents/skills`\r\n\r\nAgent skills for contributing to TRL now ship in the repo at `.agents/skills`.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6901\r\n\r\n### Docs: stable vs experimental taxonomy, and how to change the objective\r\n\r\n* **Split the docs taxonomy into stable and experimental trainers** by @qgallouedec in https://github.com/huggingface/trl/pull/7022\r\n* **Document how to change the training objective by subclassing a trainer** by @sergiopaniego in https://github.com/huggingface/trl/pull/6918\r\n* **Document how to choose and pin a Hub kernel version** by @albertvillanova in https://github.com/huggingface/trl/pull/6998\r\n* Add missing TRL blog posts to the docs landing page by @qgallouedec in https://github.com/huggingface/trl/pull/7023\r\n\r\n### QLoRA on the `quantization_config` argument\r\n\r\nThe QLoRA test suites move onto the `quantization_config` trainer argument added in v1.8, and gain coverage on the VLM paths.\r\n\r\n* Add a QLoRA test to the GRPO and RLOO VLM test suites by @albertvillanova in https://github.com/huggingface/trl/pull/6909\r\n* Add a QLoRA test to the trainers that support `quantization_config` by @albertvillanova in https://github.com/huggingface/trl/pull/6910\r\n* Convert the QLoRA tests to the `quantization_config` trainer argument by @albertvillanova in https://github.com/huggingface/trl/pull/6927\r\n\r\n### Dependency floors raised\r\n\r\n* Require `peft>=0.13.0` and drop the 0.12 version guards by @qgallouedec in https://github.com/huggingface/trl/pull/7102\r\n* Require `deepspeed>=0.18.6` and drop the 0.16.4 guard by @albertvillanova in https://github.com/huggingface/trl/pull/7118\r\n* Guard `target_parameters` use for `peft<0.17.0` by @albertvillanova in https://github.com/huggingface/trl/pull/7119\r\n* Add support for vLLM 0.28.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7001\r\n* Drop vLLM 0.19.0 support by @qgallouedec in https://github.com/huggingface/trl/pull/7002\r\n\r\n### Other\r\n\r\n* **Count each token once in `num_tokens` under tensor parallelism** by @qgallouedec in https://github.com/huggingface/trl/pull/7101\r\n* **XPO / NashMD perf**: avoid `output_hidden_states` when only `last_hidden_state` is used by @ciaoyizhen in https://github.com/huggingface/trl/pull/4755\r\n* Sync the model config pad token id with the tokenizer in the trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7097 and (experimental) https://github.com/huggingface/trl/pull/7128\r\n* Forward model revision to the auto-loaded `processing_class` by @kashif in https://github.com/huggingface/trl/pull/6978\r\n* Inject the model revision into concrete config loads too by @albertvillanova in https://github.com/huggingface/trl/pull/7099\r\n* Match hub kernels by repo id when checking the attention implementation by @albertvillanova in https://github.com/huggingface/trl/pull/6985\r\n* Catch `AsyncDistillationTrainer` up with recent AsyncGRPO changes by @kashif in https://github.com/huggingface/trl/pull/6823 and https://github.com/huggingface/trl/pull/6987\r\n* Remove the leftover `dataset_formatting` module by @albertvillanova in https://github.com/huggingface/trl/pull/7117\r\n* Remove the unused `FLASH_ATTENTION_VARIANTS` constant from the DPO trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7013\r\n* Document that padding-free is temporarily unavailable in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7014\r\n* Realign duplicated comments across the trainers by @qgallouedec in https://github.com/huggingface/trl/pull/7145\r\n\r\n## Breaking\r\n\r\n### ⚠️ PPO is removed\r\n\r\n`PPOTrainer`, `PPOConfig`, and `modeling_value_head.py` (`PreTrainedModelWrapper`, `AutoModelForCausalLMWithValueHead`, `AutoModelForSeq2SeqLMWithValueHead`) are gone, along with their tests, doc page, and examples.\r\n\r\nBit of a moment for TRL: `PPOTrainer` predates every PR in this repo. It landed in `dfb6a580` on 2020-03-28, the commit that first added the library, back when the package was called `lm_ppo`. It was the oldest thing in TRL and the last piece of the original codebase.\r\n\r\nWhy now: unmaintained (no feature work in over a year, every PPO commit since #4482 a drive-by fix), the only trainer never aligned on the input format (it still took tokenized `input_ids`), near-zero recorded usage, and a magnet for automated bug hunters filing real reports against code nobody runs. Nobody loses anything: `from trl import PPOTrainer` already stopped working in v1.10, so anyone actually running PPO is pinned to an older TRL and those installs keep resolving exactly as they do today.\r\n\r\n`create_reference_model` stays (BCO, A2PO and Online DPO use it).\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/7020\r\n\r\n## Fixes\r\n\r\n* **Fix missing entropy gradient in `_ChunkedLogProbFunction` backward** — `backward` accepted `grad_entropy` but never used it, so any loss backpropagating through the `entropy` output silently got zero gradient from that term, with no error or warning. Only `AsyncGRPOTrainer` read `entropy` (under `no_grad`, for logging), so no user-facing regression today, but this is exported non-experimental code and an entropy-bonus loss term would have hit it silently. By @verma8076 in https://github.com/huggingface/trl/pull/6625\r\n* **vLLM weight sync now fails instead of hanging** by @AmineDiro in https://github.com/huggingface/trl/pull/6913\r\n* **Fail fast on incompatible vLLM for default Async GRPO weight transfer** by @DimensionSTP in https://github.com/huggingface/trl/pull/6830\r\n* **Fix VLM support in AsyncGRPO** by @AmineDiro in https://github.com/huggingface/trl/pull/6839\r\n* Fix variable reference in BCO docs by @albertvillanova in https://github.com/huggingface/trl/pull/7046\r\n* Fix the IW-OPD self-distillation loss assertion in tests by @albertvillanova in https://github.com/huggingface/trl/pull/7123\r\n* Fix typos in help strings and error messages by @qgallouedec in https://github.com/huggingface/trl/pull/7146\r\n* Fix two messages that describe the wrong behaviour by @qgallouedec in https://github.com/huggingface/trl/pull/7147\r\n* Fix three broken doc links by @qgallouedec in https://github.com/huggingface/trl/pull/7155\r\n* Fix the example commands in the script docstrings by @qgallouedec in https://github.com/huggingface/trl/pull/7156\r\n* Correct what the TRL Docker image provides in the Jobs guide by @albertvillanova in https://github.com/huggingface/trl/pull/7054\r\n* Fix `paper_index` paths and add GOLD, IW-OPD, and AsyncDistillation telemetry by @YeonwooSung in https://github.com/huggingface/trl/pull/7051\r\n* Qualify doc references to experimental classes by @qgallouedec in https://github.com/huggingface/trl/pull/7158\r\n\r\n## Documentation\r\n\r\nA docstring and docs consistency pass across the repo, all by @qgallouedec:\r\n\r\n* Fix docstring type specs (#7148), reformat five docstrings to the repo style (#7149), fix docstring content drift (#7150), match config field order to the docstring order (#7151), reflow two docstrings to the doc-builder width (#7153), fix the `DistillationTrainer` signature annotations (#7152), docs formatting pass (#7154), tidy pyproject / Makefile / .gitignore (#7157)\r\n* Document the TRL Docker image tags by @albertvillanova in https://github.com/huggingface/trl/pull/7055\r\n\r\n## CI\r\n\r\n**Docker images** — released and dev builds are now separate workflows, the image version comes from the release tag, and the installed TRL version is pinned:\r\n\r\n* Split the released and dev Docker image builds into separate workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6990\r\n* Read the Docker image version from the release tag by @albertvillanova in https://github.com/huggingface/trl/pull/6991\r\n* Pin the TRL version installed in the Docker image by @albertvillanova in https://github.com/huggingface/trl/pull/6992\r\n* Add the missing trailing newline to the dev Dockerfile by @albertvillanova in https://github.com/huggingface/trl/pull/7030\r\n* chore: update `docker-build.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6997\r\n\r\n**Tiny-model config alignment** — a sweep bringing every tiny test model's config in line with its reference model, all by @albertvillanova:\r\n\r\n* Qwen2.5 (#7098), Qwen3 tiny+small (#7106), Qwen3 Instruct-2507 (#7107), GptOss (#7108), GPTNeoX / pythia-14m (#7109), Llama 3.2 + Remote (#7110), Llama 3 (#7112), Llama 3.1 (#7113)\r\n* Pin `layer_types` so tiny Gemma3 / Olmo3 cover both attention types (#6962) and tiny Cohere2 (#6963)\r\n* Show list lengths and mark truncation in the tiny-model config diff (#7105)\r\n\r\n**Other CI**, mostly by @albertvillanova:\r\n\r\n* Use the doc-builder pre-commit hook (#6799), bump its revision (#6969, #7078)\r\n* Retry tests failing on cuBLAS allocation errors the rerun filter missed (#6916)\r\n* Use ruff `select` instead of `extend-select` to keep the rule set explicit (#6964), stop dependabot from opening ruff pre-commit bumps (#6965), reformat lambda functions for code clarity (#7092)\r\n* Remove the flash-attn2 xfail markers now that the padding-free tests pass (#7057), fix the invariant suite recording instructions (#7058)\r\n* Pin the invariant suite's FA2 hub kernel to the v2 branch by @qgallouedec in https://github.com/huggingface/trl/pull/6979\r\n* Filter the benign torch DataParallel scalar-gather warning (#7019)\r\n* Fix the input metadata of the post-slack composite action (#7029)\r\n* Group Dependabot GitHub Actions and pre-commit updates in a single PR (#7088), merge the two Dependabot github-actions entries into one (#7091)\r\n* Dependabot bumps in #6958, #7000, #7079, #7089\r\n\r\n## New Contributors\r\n* @ciaoyizhen made their first contribution in https://github.com/huggingface/trl/pull/4755\r\n* @DimensionSTP made their first contribution in https://github.com/huggingface/trl/pull/6830\r\n* @YeonwooSung made their first contribution in https://github.com/huggingface/trl/pull/7051\r\n\r\n## What's Changed\r\n* ⬆️ Fix dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6939\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6940\r\n* Add a QLoRA test to the GRPO and RLOO VLM test suites by @albertvillanova in https://github.com/huggingface/trl/pull/6909\r\n* Add a QLoRA test to the trainers that support quantization_config by @albertvillanova in https://github.com/huggingface/trl/pull/6910\r\n* Convert the QLoRA tests to the quantization_config trainer argument by @albertvillanova in https://github.com/huggingface/trl/pull/6927\r\n* Fix missing entropy gradient in _ChunkedLogProbFunction backward by @verma8076 in https://github.com/huggingface/trl/pull/6625\r\n* Document how to change the training objective by subclassing a trainer by @sergiopaniego in https://github.com/huggingface/trl/pull/6918\r\n* Use the doc-builder pre-commit hook by @albertvillanova in https://github.com/huggingface/trl/pull/6799\r\n* Retry tests failing on cuBLAS allocation errors the rerun filter missed by @albertvillanova in https://github.com/huggingface/trl/pull/6916\r\n* Fix:  vLLM weight sync fails instead of hanging by @AmineDiro in https://github.com/huggingface/trl/pull/6913\r\n* Bump the actions group across 1 directory with 5 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6958\r\n* Bump https://github.com/huggingface/doc-builder from 0ab9ea03baf111ed8dd83e88233430b663127368 to 1b16dac5e33043af565fdf4c1b5b0fe81d0891c8 by @dependabot[bot] in https://github.com/huggingface/trl/pull/6969\r\n* Pin layer_types so the tiny Gemma3 and Olmo3 models cover both attention types by @albertvillanova in https://github.com/huggingface/trl/pull/6962\r\n* Pin layer_types so the tiny Cohere2 model covers both attention types by @albertvillanova in https://github.com/huggingface/trl/pull/6963\r\n* Use ruff `select` instead of `extend-select` to keep the rule set explicit by @albertvillanova in https://github.com/huggingface/trl/pull/6964\r\n* fix vlm support asyncgrpo by @AmineDiro in https://github.com/huggingface/trl/pull/6839\r\n* Split the released and dev Docker image builds into separate workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6990\r\n* chore: update docker-build.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6997\r\n* Stop dependabot from opening ruff pre-commit bumps by @albertvillanova in https://github.com/huggingface/trl/pull/6965\r\n* Read the Docker image version from the release tag by @albertvillanova in https://github.com/huggingface/trl/pull/6991\r\n* Pin the TRL version installed in the Docker image by @albertvillanova in https://github.com/huggingface/trl/pull/6992\r\n* Bump trufflesecurity/trufflehog from 3.97.0 to 3.97.1 in the actions group by @dependabot[bot] in https://github.com/huggingface/trl/pull/7000\r\n* Match hub kernels by repo id when checking the attention implementation by @albertvillanova in https://github.com/huggingface/trl/pull/6985\r\n* Remove the unused FLASH_ATTENTION_VARIANTS constant from the DPO trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7013\r\n* Document that padding-free is temporarily unavailable in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7014\r\n* Drop vLLM 0.19.0 support by @qgallouedec in https://github.com/huggingface/trl/pull/7002\r\n* Pin the invariant suite's FA2 hub kernel to the v2 branch by @qgallouedec in https://github.com/huggingface/trl/pull/6979\r\n* Filter the benign torch DataParallel scalar-gather warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/7019\r\n* Test liger vs non-liger GRPO loss parity by @kashif in https://github.com/huggingface/trl/pull/6814\r\n* Add the missing trailing newline to the dev Dockerfile by @albertvillanova in https://github.com/huggingface/trl/pull/7030\r\n* Fix the input metadata of the post-slack composite action by @albertvillanova in https://github.com/huggingface/trl/pull/7029\r\n* Catch AsyncDistillationTrainer up with recent AsyncGRPO changes by @kashif in https://github.com/huggingface/trl/pull/6823\r\n* Document 1M-token training and add a context-parallelism example by @qgallouedec in https://github.com/huggingface/trl/pull/6846\r\n* Rewrite the long context guide by @qgallouedec in https://github.com/huggingface/trl/pull/7003\r\n* Document how to choose and pin a Hub kernel version by @albertvillanova in https://github.com/huggingface/trl/pull/6998\r\n* Fix variable reference in BCO docs by @albertvillanova in https://github.com/huggingface/trl/pull/7046\r\n* Remove PPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/7020\r\n* Correct what the TRL Docker image provides in the Jobs guide by @albertvillanova in https://github.com/huggingface/trl/pull/7054\r\n* Document the TRL Docker image tags by @albertvillanova in https://github.com/huggingface/trl/pull/7055\r\n* Do the chunked-CE `lm_head` projection on tensor cores instead of in fp32 by @qgallouedec in https://github.com/huggingface/trl/pull/6863\r\n* Add contributor skills at `.agents/skills` by @qgallouedec in https://github.com/huggingface/trl/pull/6901\r\n* Split the docs taxonomy into stable and experimental trainers by @qgallouedec in https://github.com/huggingface/trl/pull/7022\r\n* Vendor the fused linear losses from Liger-Kernel into `trl.losses` by @qgallouedec in https://github.com/huggingface/trl/pull/7059\r\n* Drop fused JSD from DistillationTrainer by @kashif in https://github.com/huggingface/trl/pull/7064\r\n* Fix DPO and KTO Liger loss gaps by @kashif in https://github.com/huggingface/trl/pull/7062\r\n* Harden chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7065\r\n* Run the vendored loss parity suites as slow tests only by @qgallouedec in https://github.com/huggingface/trl/pull/7067\r\n* XPO/NashMD perf: avoid `output_hidden_states` when only `last_hidden_state` is used by @ciaoyizhen in https://github.com/huggingface/trl/pull/4755\r\n* Bump https://github.com/huggingface/doc-builder from 1b16dac5e33043af565fdf4c1b5b0fe81d0891c8 to c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27 by @dependabot[bot] in https://github.com/huggingface/trl/pull/7078\r\n* Bump the actions group with 5 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/7079\r\n* Group Dependabot GitHub Actions and pre-commit updates in a single PR by @albertvillanova in https://github.com/huggingface/trl/pull/7088\r\n* Fix the invariant suite recording instructions by @albertvillanova in https://github.com/huggingface/trl/pull/7058\r\n* Remove the flash-attn2 xfail markers now that the padding-free tests pass by @albertvillanova in https://github.com/huggingface/trl/pull/7057\r\n* Bump the \"github-actions-and-pre-commit\" group with 2 updates across multiple ecosystems by @dependabot[bot] in https://github.com/huggingface/trl/pull/7089\r\n* Merge the two Dependabot github-actions entries into one by @albertvillanova in https://github.com/huggingface/trl/pull/7091\r\n* Reformat lambda functions for code clarity by @albertvillanova in https://github.com/huggingface/trl/pull/7092\r\n* Fail fast on incompatible vLLM for default Async GRPO weight transfer by @DimensionSTP in https://github.com/huggingface/trl/pull/6830\r\n* Inject the model revision into concrete config loads too by @albertvillanova in https://github.com/huggingface/trl/pull/7099\r\n* Align tiny/small Qwen2.5 config with Qwen/Qwen2.5-32B-Instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7098\r\n* Sync the async dist. trainers with recent AsyncGRPO changes by @kashif in https://github.com/huggingface/trl/pull/6987\r\n* Align tiny and small Qwen3 configs with their reference models by @albertvillanova in https://github.com/huggingface/trl/pull/7106\r\n* Align tiny Qwen3 Instruct-2507 config with Qwen/Qwen3-4B-Instruct-2507 by @albertvillanova in https://github.com/huggingface/trl/pull/7107\r\n* Require peft>=0.13.0 and drop the 0.12 version guards by @qgallouedec in https://github.com/huggingface/trl/pull/7102\r\n* Align tiny GptOss config with openai/gpt-oss-20b by @albertvillanova in https://github.com/huggingface/trl/pull/7108\r\n* Require deepspeed>=0.18.6 and drop the 0.16.4 guard by @albertvillanova in https://github.com/huggingface/trl/pull/7118\r\n* Remove the leftover dataset_formatting module by @albertvillanova in https://github.com/huggingface/trl/pull/7117\r\n* Align tiny GPTNeoX config with EleutherAI/pythia-14m by @albertvillanova in https://github.com/huggingface/trl/pull/7109\r\n* Align tiny Llama 3.2 and Remote configs with meta-llama/Llama-3.2-1B-Instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7110\r\n* Align tiny Llama 3 config with meta-llama/Meta-Llama-3-8B-Instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7112\r\n* Align tiny Llama 3.1 config with meta-llama/Llama-3.1-8B-Instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7113\r\n* Show list lengths and mark truncation in the tiny-model config diff by @albertvillanova in https://github.com/huggingface/trl/pull/7105\r\n* Fix the IW-OPD self-distillation loss assertion in tests by @albertvillanova in https://github.com/huggingface/trl/pull/7123\r\n* Sync the model config pad token id with the tokenizer in the trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7097\r\n* Sync the model config pad token id with the tokenizer in the experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7128\r\n* Guard target_parameters use for peft<0.17.0 by @albertvillanova in https://github.com/huggingface/trl/pull/7119\r\n* Forward model revision to the auto-loaded processing_class by @kashif in https://github.com/huggingface/trl/pull/6978\r\n* Add missing TRL blog posts to the docs landing page by @qgallouedec in https://github.com/huggingface/trl/pull/7023\r\n* Add support for vLLM 0.28.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7001\r\n* Count each token once in num_tokens under tensor parallelism by @qgallouedec in https://github.com/huggingface/trl/pull/7101\r\n* Realign duplicated comments across the trainers by @qgallouedec in https://github.com/huggingface/trl/pull/7145\r\n* Fix typos in help strings and error messages by @qgallouedec in https://github.com/huggingface/trl/pull/7146\r\n* Fix two messages that describe the wrong behaviour by @qgallouedec in https://github.com/huggingface/trl/pull/7147\r\n* Fix docstring type specs by @qgallouedec in https://github.com/huggingface/trl/pull/7148\r\n* Reformat five docstrings to the repo style by @qgallouedec in https://github.com/huggingface/trl/pull/7149\r\n* Fix docstring content drift by @qgallouedec in https://github.com/huggingface/trl/pull/7150\r\n* Match config field order to the docstring order by @qgallouedec in https://github.com/huggingface/trl/pull/7151\r\n* Reflow two docstrings to the doc-builder width by @qgallouedec in https://github.com/huggingface/trl/pull/7153\r\n* Fix the DistillationTrainer signature annotations by @qgallouedec in https://github.com/huggingface/trl/pull/7152\r\n* Docs formatting pass by @qgallouedec in https://github.com/huggingface/trl/pull/7154\r\n* Tidy pyproject, Makefile and .gitignore by @qgallouedec in https://github.com/huggingface/trl/pull/7157\r\n* Fix the example commands in the script docstrings by @qgallouedec in https://github.com/huggingface/trl/pull/7156\r\n* Fix three broken doc links by @qgallouedec in https://github.com/huggingface/trl/pull/7155\r\n* Fix paper_index paths and add GOLD, IW-OPD, and AsyncDistillation telemetry by @YeonwooSung in https://github.com/huggingface/trl/pull/7051\r\n* Qualify doc references to experimental classes by @qgallouedec in https://github.com/huggingface/trl/pull/7158\r\n* Release: v1.13 by @qgallouedec in https://github.com/huggingface/trl/pull/7159\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.12.0...v1.13.0\r\n","publishedAt":"2026-09-10T00:50:26.000Z","fetchedAt":"2026-09-10T01:14:29.976Z","url":"https://github.com/huggingface/trl/releases/tag/v1.13.0","media":[{"type":"image","url":"https://github.com/user-attachments/assets/806f9153-d098-4a45-bef4-5d4119b7ef25","r2Key":"releases/efcaa3f83f5f796e611964a3a89ee3bddb5f86eb99f34afaa4e68bd02f52732d.png","r2Url":"https://media.releases.sh/releases/efcaa3f83f5f796e611964a3a89ee3bddb5f86eb99f34afaa4e68bd02f52732d.png"}],"coverageCount":0},{"id":"rel_GJEoPJemdKuwPynV7CzX1","version":"v1.12.0","type":"feature","title":"v1.12.0","summary":"v1.12.0 was accidentally auto-published to PyPI with identical code to v1.11.0, so skip it — the next feature release is v1.13.0.","titleGenerated":"TRL v1.12.0 is accidental duplicate of v1.11.0; next feature release is v1.13.0","titleShort":"v1.12.0 duplicates v1.11.0; skip to v1.13.0","breaking":"none","importance":3,"content":"## ⚠️ v1.12.0 is an accidental duplicate of v1.11.0\r\n\r\nDuring the v1.11.0 release, `VERSION` was bumped to `1.12.0` instead of `1.12.0.dev0`. Our publish workflow fires on any push to `main` that touches `VERSION` and only skips the upload when the version contains `\"dev\"`, so **`trl 1.12.0` was auto-published to PyPI at 19:46 on 2026-08-26, 90 seconds after `1.11.0`, with identical code**.\r\n\r\n**`trl==1.12.0` on PyPI is bit-identical to `trl==1.11.0`.** There are no features, fixes, or behavior changes in this version.\r\n\r\nThe version number is burned (PyPI does not allow yanking + re-uploading the same version), so we skip `1.12` entirely for real development. The next feature release is **`v1.13.0`**.\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.11.0...v1.12.0 (empty)\r\n","publishedAt":"2026-08-26T20:05:50.000Z","fetchedAt":"2026-08-26T21:12:42.831Z","url":"https://github.com/huggingface/trl/releases/tag/v1.12.0","media":[],"coverageCount":0},{"id":"rel_lK_Rd3S5nhpIYWvV4peDt","version":"v1.11.0","type":"feature","title":"v1.11.0","summary":"trl vllm-serve now delegates to vLLM's own server, cutting code from 1218 to ~130 lines and delivering 1.4–1.6× faster GRPO server-mode steps with bit-identical completions. A new experimental AsyncDistillationTrainer supports multi-teacher on-policy distillation, plus support for several new models and templates. Old examples/scripts and examples/notebooks links now 404 after the folder reorganization.","titleGenerated":"TRL v1.11.0 replaces custom vLLM server and adds AsyncDistillationTrainer","titleShort":"vLLM's own server replaces custom; async distillation trainer new","breaking":"major","importance":4,"content":"## Features\r\n\r\n### Use vLLM's own server instead of TRL's\r\n\r\n`trl vllm-serve` used to run a custom FastAPI app around vLLM's `LLM`, with its own data-parallel fan-out and a worker extension for weight sync. vLLM's own server covers all of that. The custom server is gone: `trl/scripts/vllm_serve.py` drops from **1218 to ~130 lines** and now translates its flags into `vllm serve`, prints the equivalent command, forwards any extra argument, and sets the three settings TRL needs (`--weight-transfer-config`, `--logprobs-mode processed_logprobs`, `--max-logprobs -1`). `fastapi`, `uvicorn` and `pydantic` leave the `vllm` extra.\r\n\r\nWeight sync now uses vLLM's NCCL weight-transfer engine — tensors are announced once and streamed in a single packed broadcast, instead of one HTTP request + broadcast per tensor. Distillation teacher logprobs come from `prompt_logprobs`.\r\n\r\n**Benchmark** (GRPO server-mode on H100, Qwen2.5-1.5B / Qwen2.5-VL-3B, same seed, same data):\r\n\r\n| Setup | before | after | speedup |\r\n| --- | --- | --- | --- |\r\n| Text, TP=1 | 1.08, 0.91 s/step | **0.63, 0.62 s/step** | **1.59×** |\r\n| Text, TP=2 | 0.96, 0.96 s/step | **0.62, 0.60 s/step** | **1.57×** |\r\n| VLM, TP=1 | 2.14, 2.13 s/step | **1.48, 1.48 s/step** | **1.44×** |\r\n\r\nBit-identical completions vs the old server (16/16 over 4 prompts × n=4, logprobs identical to 0.0 over 499 tokens). `trl vllm-serve` still accepts the same flags — it just forwards to `vllm serve` now.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6765\r\n\r\n### New experimental `AsyncDistillationTrainer` (+ multi-teacher MOPD)\r\n\r\nAsync on-policy distillation, architected like `AsyncGRPOTrainer`: a background rollout worker generates the student's own completions and scores them against a teacher served over HTTP, so generation and training overlap instead of alternating. The teacher is never loaded locally — just a vLLM server URL.\r\n\r\nAlso supports **MOPD (multi-teacher on-policy distillation)**: pass more than one entry in `teacher_server_urls` and route each sample to a teacher via a `teacher_id` column (e.g. a math teacher and a code teacher, each served independently).\r\n\r\n```python\r\nfrom trl.experimental.async_distillation import AsyncDistillationConfig, AsyncDistillationTrainer\r\n\r\ntrainer = AsyncDistillationTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=AsyncDistillationConfig(\r\n        teacher_server_urls=[\"http://teacher-math:8000\", \"http://teacher-code:8000\"],\r\n        teacher_top_k=16,\r\n    ),\r\n    train_dataset=dataset,  # rows carry a `teacher_id` column for MOPD\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/6705\r\n\r\n### New model & template support\r\n\r\n* **Muse Glimmer** (+ chat template) by @qgallouedec in https://github.com/huggingface/trl/pull/6738 and https://github.com/huggingface/trl/pull/6739\r\n* **Nemotron 3.5 Lightning** by @qgallouedec in https://github.com/huggingface/trl/pull/6736\r\n* **LFM2.5-VL** by @qgallouedec in https://github.com/huggingface/trl/pull/6737\r\n* **Qwen3.8** by @qgallouedec in https://github.com/huggingface/trl/pull/6764\r\n* **DeepSeek-R1-Distill chat template** by @qgallouedec in https://github.com/huggingface/trl/pull/6766\r\n\r\n### Examples reorganized into per-example folders\r\n\r\n`examples/` was split by format (`scripts/` vs `notebooks/`), which scattered related files (e.g. `sft_nemotron_3` existed as both a script *and* a notebook, in different folders; OpenEnv notebooks lived apart from OpenEnv scripts). Everything is now **one folder per story** — the name says the method and the task (`grpo_wordle`, `sft_gpt_oss`, `ppo_tldr`), and the folder holds every file the example needs (scripts, notebooks, prompts, chat templates, eval code). Thin single-trainer demo scripts are removed (they live in `trl/scripts/` and each trainer doc page has a runnable snippet). Old `examples/scripts/…` and `examples/notebooks/…` GitHub / Colab links will 404 — worth flagging in downstream posts.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6820\r\n\r\n### AsyncGRPO: metrics rework, dtype control, and cleanup\r\n\r\n* **Rework AsyncGRPO metrics + docs** by @AmineDiro in https://github.com/huggingface/trl/pull/6715\r\n* **Training dtype in `AsyncGRPOConfig`** by @AmineDiro in https://github.com/huggingface/trl/pull/6774\r\n* **Simplify `AsyncGRPOTrainer`** — drop redundant metric guards, merge train-begin callbacks by @qgallouedec in https://github.com/huggingface/trl/pull/6378\r\n* **Fix checkpoint resume triggering unnecessary vLLM inference** by @AmmarHassona in https://github.com/huggingface/trl/pull/5911\r\n\r\n### Rewritten TRL skill (Python-API first) + drop `trl skills install`\r\n\r\nThe TRL skill is now written around the Python API — trainer selection, dataset format, load-bearing config fields, GRPO reward-function contract — with the CLI as a footnote. `trl skills install` is dropped and the skill source moves out of the package.\r\n\r\n* Rewrite the TRL skill around the Python API by @qgallouedec in https://github.com/huggingface/trl/pull/6794\r\n* Drop `trl skills install` and move the skill source out of the package by @qgallouedec in https://github.com/huggingface/trl/pull/6793\r\n\r\n### `DistillationTrainer` supports tool calling\r\n\r\nContinuing the graduation cycle from v1.10.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6723\r\n\r\n### Suggest migrating to `DistillationTrainer` when the GKD config is fully covered\r\n\r\n`GKDTrainer` now emits a hint when the config is fully covered by the stable `DistillationTrainer` — no behavior change, just a nudge for users still on the older API.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6847\r\n\r\n### Other\r\n\r\n* Allow dict config fields to be set from the command line by @qgallouedec in https://github.com/huggingface/trl/pull/6792\r\n* Raise when packing is combined with context parallelism by @qgallouedec in https://github.com/huggingface/trl/pull/6843\r\n* Emit `position_ids` from the SFT collator when sequence parallelism is enabled by @qgallouedec in https://github.com/huggingface/trl/pull/6812\r\n* Remove one host sync from SFT `chunked_nll` loss by @qgallouedec in https://github.com/huggingface/trl/pull/6842\r\n* Align the chunked-loss config access between SFT and Distillation by @qgallouedec in https://github.com/huggingface/trl/pull/6744\r\n* Add `dataset_source` to telemetry by @qgallouedec in https://github.com/huggingface/trl/pull/6599\r\n* Raise a clear error when generation returns no completions by @albertvillanova in https://github.com/huggingface/trl/pull/6714\r\n* Move GPU CI from T4 to L40S by @qgallouedec in https://github.com/huggingface/trl/pull/6741\r\n* Add on-policy multimodal GKD regression test by @sergiopaniego in https://github.com/huggingface/trl/pull/6342\r\n\r\n### vLLM version sweep\r\n\r\n* Add support for vLLM 0.27.0 and 0.27.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6742\r\n* Drop vLLM 0.17 by @qgallouedec in https://github.com/huggingface/trl/pull/6722\r\n* Drop vLLM 0.18 by @qgallouedec in https://github.com/huggingface/trl/pull/6844\r\n* Require `liger-kernel` 0.8.2 and drop the SAPO warning filter by @albertvillanova in https://github.com/huggingface/trl/pull/6768\r\n\r\n## Fixes\r\n\r\n* **Fix GRPO/RLOO training with multiple images for VLMs without grid metadata** by @qgallouedec in https://github.com/huggingface/trl/pull/6570\r\n* **Fix GRPO/RLOO with dynamic-tiling VLMs (InternVL)** by @qgallouedec in https://github.com/huggingface/trl/pull/6619\r\n* **Handle `None` token logprobs from vLLM in GRPO importance sampling** by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6693\r\n* **`[GRPO]` Match the Liger `dapo` / `cispo` / `vespo` normalizer to the non-Liger path** — completes the fix from v1.10 (#6024) on the Liger path. By @kashif in https://github.com/huggingface/trl/pull/5890\r\n* **`[SFT]` Treat `past_key_values` and `attentions` as optional on the model output** by @Hakureirm in https://github.com/huggingface/trl/pull/6686\r\n* **Fix `AsyncGRPOTrainer` crashing on environment-owned datasets** by @albertvillanova in https://github.com/huggingface/trl/pull/6824\r\n* **Fix SDPO `scale_rewards` CLI parsing** and warn on ignored feedback by @sergiopaniego in https://github.com/huggingface/trl/pull/6341\r\n* Fix `opencode.py` proxy trace / log paths after OpenEnv moved them to functions by @sergiopaniego in https://github.com/huggingface/trl/pull/6672\r\n\r\n## Documentation and Examples\r\n\r\n* Rework AsyncGRPO docs alongside the metrics rework (@AmineDiro, #6715)\r\n* Reorganize examples into self-contained per-example folders (see above)\r\n* Suggest migrating to `DistillationTrainer` when GKD config is covered (see above)\r\n\r\n## CI\r\n\r\nBig CI-hygiene cycle by @albertvillanova. Highlights:\r\n\r\n* Test one Python version on PRs, matrix nightly by @qgallouedec in https://github.com/huggingface/trl/pull/6763\r\n* Notify Slack when the tests of the latest release fail by @albertvillanova in https://github.com/huggingface/trl/pull/6900\r\n* Post the Docker image build results to the shared CI Slack channel by @albertvillanova in https://github.com/huggingface/trl/pull/6903\r\n* Factor the Slack notification into a local composite action by @albertvillanova in https://github.com/huggingface/trl/pull/6895 (+ updates in #6896, #6897, #6908, #6934, #6936)\r\n* Move slow-test job status handling, dependencies, and reporting to Slack by @albertvillanova in https://github.com/huggingface/trl/pull/6752, https://github.com/huggingface/trl/pull/6754, https://github.com/huggingface/trl/pull/6756\r\n* Pin the TruffleHog scanner image by digest + tighten secret scanning by @albertvillanova in https://github.com/huggingface/trl/pull/6677, https://github.com/huggingface/trl/pull/6757, https://github.com/huggingface/trl/pull/6758, https://github.com/huggingface/trl/pull/6759\r\n* Various hotfix pins and reverts across the cycle (bitsandbytes / liger-kernel / peft dev builds / transformers dev) by @albertvillanova in https://github.com/huggingface/trl/pull/6517, https://github.com/huggingface/trl/pull/6835, https://github.com/huggingface/trl/pull/6840, https://github.com/huggingface/trl/pull/6841, https://github.com/huggingface/trl/pull/6838, https://github.com/huggingface/trl/pull/6915, https://github.com/huggingface/trl/pull/6919\r\n* Bump the actions group across 1 directory with 8 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6727 and https://github.com/huggingface/trl/pull/6902\r\n* Restore Liger Kernel's global patches between tests by @albertvillanova in https://github.com/huggingface/trl/pull/6783\r\n* Stub the weight transfer in the AsyncGRPO epoch-stop test by @albertvillanova in https://github.com/huggingface/trl/pull/6826\r\n* Align AsyncDistillationTrainer with AsyncGRPOTrainer on pre-init args by @albertvillanova in https://github.com/huggingface/trl/pull/6825\r\n* Use Flash Attention compatible models in the continuous batching test / async trainer tests by @albertvillanova in https://github.com/huggingface/trl/pull/6853 and https://github.com/huggingface/trl/pull/6854\r\n* Derive `head_dim` when the model config does not declare it by @albertvillanova in https://github.com/huggingface/trl/pull/6855\r\n* Build the GRPO VLM test images as PIL images to avoid a datasets downcast warning by @albertvillanova in https://github.com/huggingface/trl/pull/6889\r\n* Remove the slow GRPO VLM training test by @albertvillanova in https://github.com/huggingface/trl/pull/6912\r\n* Ignore benign NumPy `__array_wrap__` deprecation and torch inductor TF32 advisory warnings in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6850 and https://github.com/huggingface/trl/pull/6851\r\n* Stop excluding the TruffleHog `lob` / `postgres` detectors by @albertvillanova in https://github.com/huggingface/trl/pull/6920, https://github.com/huggingface/trl/pull/6921 (+ #6926)\r\n* Fix misleading SFT step names, misaligned failed-test report headers, use built-in `GITHUB_ACTIONS` by @albertvillanova in https://github.com/huggingface/trl/pull/6726, https://github.com/huggingface/trl/pull/6753, https://github.com/huggingface/trl/pull/6749\r\n* Pin Python version in the publish workflow / align Python version in the experimental tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6883, https://github.com/huggingface/trl/pull/6881, https://github.com/huggingface/trl/pull/6882, https://github.com/huggingface/trl/pull/6884\r\n* Bump doc-builder pre-commit revision + drop ruff workaround by @albertvillanova in https://github.com/huggingface/trl/pull/6784, https://github.com/huggingface/trl/pull/6790\r\n* Remove the redundant `VERSION` env from the Docker build step / fix the Slack title of the TRL Docker image build job by @albertvillanova in https://github.com/huggingface/trl/pull/6931, https://github.com/huggingface/trl/pull/6932\r\n* Remove unused `RUN_SLOW` env variable from the slow tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6748\r\n* Remove unnecessary `docstyle-ignore` tag from docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6785\r\n* Inline the pre-commit action in the code-quality jobs by @albertvillanova in https://github.com/huggingface/trl/pull/6886\r\n\r\n## New Contributors\r\n* @Hakureirm made their first contribution in https://github.com/huggingface/trl/pull/6686\r\n* @AmmarHassona made their first contribution in https://github.com/huggingface/trl/pull/5911\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6725\r\n* Hotfix CI: Skip transformers continuous batching test on pre-Ampere GPUs by @albertvillanova in https://github.com/huggingface/trl/pull/6712\r\n* Raise a clear error when generation returns no completions by @albertvillanova in https://github.com/huggingface/trl/pull/6714\r\n* Pin the TruffleHog scanner image by digest by @albertvillanova in https://github.com/huggingface/trl/pull/6677\r\n* Handle None token logprobs from vLLM in GRPO importance sampling by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6693\r\n* Drop vLLM 0.17 support by @qgallouedec in https://github.com/huggingface/trl/pull/6722\r\n* Fix misleading SFT step names in slow tests CI workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6726\r\n* Bump the actions group across 1 directory with 8 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6727\r\n* [SFT] Treat past_key_values and attentions as optional on the model output by @Hakureirm in https://github.com/huggingface/trl/pull/6686\r\n* Add support for vLLM 0.27.0 and 0.27.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6742\r\n* Remove unused RUN_SLOW env variable from slow tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6748\r\n* Do not run slow tests when the setup steps failed by @albertvillanova in https://github.com/huggingface/trl/pull/6752\r\n* Fix misaligned headers in the failed tests report table by @albertvillanova in https://github.com/huggingface/trl/pull/6753\r\n* Use built-in GITHUB_ACTIONS variable instead of custom IS_GITHUB_CI by @albertvillanova in https://github.com/huggingface/trl/pull/6749\r\n* Fix opencode.py proxy trace/log paths after OpenEnv moved them to functions by @sergiopaniego in https://github.com/huggingface/trl/pull/6672\r\n* Add DeepSeek-R1-Distill chat template by @qgallouedec in https://github.com/huggingface/trl/pull/6766\r\n* Report slow tests results to Slack based on the job status by @albertvillanova in https://github.com/huggingface/trl/pull/6754\r\n* Declare the test report dependencies in the test extra by @albertvillanova in https://github.com/huggingface/trl/pull/6756\r\n* Pass the pytest report log flag from the workflow instead of the Makefile by @albertvillanova in https://github.com/huggingface/trl/pull/6750\r\n* Add Muse Glimmer support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6738\r\n* Add Nemotron 3.5 Lightning support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6736\r\n* Add LFM2.5-VL support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6737\r\n* Add Qwen3.8 support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6764\r\n* Add Muse Glimmer chat template by @qgallouedec in https://github.com/huggingface/trl/pull/6739\r\n* Restore Liger Kernel's global patches between tests by @albertvillanova in https://github.com/huggingface/trl/pull/6783\r\n* Bump doc-builder pre-commit revision by @albertvillanova in https://github.com/huggingface/trl/pull/6784\r\n* Restrict token access in the secret scanning workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6757\r\n* Fail the secret scanning job when the scan errors out by @albertvillanova in https://github.com/huggingface/trl/pull/6758\r\n* Remove unnecessary docstyle-ignore tag from docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6785\r\n* Rework AsyncGRPO metrics metrics and docs by @AmineDiro in https://github.com/huggingface/trl/pull/6715\r\n* Scan the pushed commits in the secret scanning workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6759\r\n* Drop ruff workaround from doc-builder pre-commit hook by @albertvillanova in https://github.com/huggingface/trl/pull/6790\r\n* Require liger-kernel 0.8.2 and drop the SAPO warning filter by @albertvillanova in https://github.com/huggingface/trl/pull/6768\r\n* AsyncGRPOConfig : dtype asyncgrpo training by @AmineDiro in https://github.com/huggingface/trl/pull/6774\r\n* Add AsyncDistillationTrainer by @kashif in https://github.com/huggingface/trl/pull/6705\r\n* Simplify AsyncGRPOTrainer: drop redundant metric guards, merge train-begin callbacks by @qgallouedec in https://github.com/huggingface/trl/pull/6378\r\n* [GRPO] Match the Liger dapo/cispo/vespo normalizer to the non-Liger path by @kashif in https://github.com/huggingface/trl/pull/5890\r\n* Fix AsyncGRPO checkpoint resume triggering unnecessary vLLM inference by @AmmarHassona in https://github.com/huggingface/trl/pull/5911\r\n* Fix AsyncGRPOTrainer crashing on environment-owned datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6824\r\n* Move GPU CI from T4 to L40S by @qgallouedec in https://github.com/huggingface/trl/pull/6741\r\n* Align the chunked-loss config access between SFT and Distillation by @qgallouedec in https://github.com/huggingface/trl/pull/6744\r\n* Allow dict config fields to be set from the command line by @qgallouedec in https://github.com/huggingface/trl/pull/6792\r\n* Hotfix CI: xfail the transformers continuous batching test by @albertvillanova in https://github.com/huggingface/trl/pull/6835\r\n* Hotfix CI: skip the offloading test when several accelerators are visible by @albertvillanova in https://github.com/huggingface/trl/pull/6840\r\n* Hotfix CI: xfail the tests that DataParallel breaks on multiple GPUs by @albertvillanova in https://github.com/huggingface/trl/pull/6841\r\n* Add `dataset_source` to telemetry by @qgallouedec in https://github.com/huggingface/trl/pull/6599\r\n* Test one Python version on PRs, matrix nightly by @qgallouedec in https://github.com/huggingface/trl/pull/6763\r\n* Fix GRPO/RLOO training with multiple images for VLMs without grid metadata by @qgallouedec in https://github.com/huggingface/trl/pull/6570\r\n* Hotfix CI: xfail the experimental async trainer training tests by @albertvillanova in https://github.com/huggingface/trl/pull/6838\r\n* Fix GRPO/RLOO with dynamic-tiling VLMs (InternVL) by @qgallouedec in https://github.com/huggingface/trl/pull/6619\r\n* Ignore benign NumPy `__array_wrap__` deprecation warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6850\r\n* Ignore benign torch inductor TF32 advisory warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6851\r\n* Use Flash Attention compatible models in the continuous batching test by @albertvillanova in https://github.com/huggingface/trl/pull/6853\r\n* Stub the weight transfer in the AsyncGRPO epoch-stop test by @albertvillanova in https://github.com/huggingface/trl/pull/6826\r\n* Derive head_dim when the model config does not declare it by @albertvillanova in https://github.com/huggingface/trl/pull/6855\r\n* Use a Flash Attention compatible model in the async trainer tests by @albertvillanova in https://github.com/huggingface/trl/pull/6854\r\n* Fix SDPO scale_rewards CLI parsing and warn on ignored feedback by @sergiopaniego in https://github.com/huggingface/trl/pull/6341\r\n* Add on-policy multimodal GKD regression test by @sergiopaniego in https://github.com/huggingface/trl/pull/6342\r\n* Align AsyncDistillationTrainer with AsyncGRPOTrainer on pre-init args by @albertvillanova in https://github.com/huggingface/trl/pull/6825\r\n* Align Python version in the experimental tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6881\r\n* Quote python-version value in the tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6882\r\n* Set up Python explicitly in the slow tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6884\r\n* Pin the mutable post-slack action reference to a commit SHA by @albertvillanova in https://github.com/huggingface/trl/pull/6891\r\n* Bump the pinned post-slack action SHA to the current main by @albertvillanova in https://github.com/huggingface/trl/pull/6892\r\n* Emit position_ids from the SFT collator when sequence parallelism is enabled by @qgallouedec in https://github.com/huggingface/trl/pull/6812\r\n* Inline the pre-commit action in the code quality jobs by @albertvillanova in https://github.com/huggingface/trl/pull/6886\r\n* Pin Python version in the publish workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6883\r\n* Notify Slack when the tests of the latest release fail by @albertvillanova in https://github.com/huggingface/trl/pull/6900\r\n* Reorganize examples into self-contained per-example folders by @qgallouedec in https://github.com/huggingface/trl/pull/6820\r\n* [DistillationTrainer refactor] Support tool calling by @qgallouedec in https://github.com/huggingface/trl/pull/6723\r\n* Factor the Slack notification into a local composite action by @albertvillanova in https://github.com/huggingface/trl/pull/6895\r\n* Update slack-github-action from v1 to v4 in the local Slack action by @albertvillanova in https://github.com/huggingface/trl/pull/6896\r\n* Let Dependabot update actions pinned in local composite actions by @albertvillanova in https://github.com/huggingface/trl/pull/6897\r\n* Build the GRPO VLM test images as PIL images to avoid a datasets downcast warning by @albertvillanova in https://github.com/huggingface/trl/pull/6889\r\n* Hotfix CI: Xfail MoE LoRA tests against a broken peft dev build by @albertvillanova in https://github.com/huggingface/trl/pull/6915\r\n* Bump the actions group across 1 directory with 8 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6902\r\n* Revert xfail for MoE LoRA tests now that peft#3602 fixed the autocast dtype bug by @albertvillanova in https://github.com/huggingface/trl/pull/6919\r\n* Stop excluding the trufflehog lob detector by @albertvillanova in https://github.com/huggingface/trl/pull/6920\r\n* Stop excluding the trufflehog postgres detector by @albertvillanova in https://github.com/huggingface/trl/pull/6921\r\n* Remove the stale detector exclusion comment by @albertvillanova in https://github.com/huggingface/trl/pull/6926\r\n* Remove the slow GRPO VLM training test by @albertvillanova in https://github.com/huggingface/trl/pull/6912\r\n* Post the Docker image build results to the shared CI Slack channel by @albertvillanova in https://github.com/huggingface/trl/pull/6903\r\n* Fix the Slack title of the TRL Docker image build job by @albertvillanova in https://github.com/huggingface/trl/pull/6931\r\n* Remove the redundant VERSION env from the Docker build step by @albertvillanova in https://github.com/huggingface/trl/pull/6932\r\n* Read the job status from the job context in the local Slack action by @albertvillanova in https://github.com/huggingface/trl/pull/6908\r\n* Drop vLLM 0.18 support by @qgallouedec in https://github.com/huggingface/trl/pull/6844\r\n* Use vLLM's own server instead of TRL's by @qgallouedec in https://github.com/huggingface/trl/pull/6765\r\n* Drop `trl skills install` and move the skill source out of the package by @qgallouedec in https://github.com/huggingface/trl/pull/6793\r\n* Make the Slack CI report readable at a glance by @qgallouedec in https://github.com/huggingface/trl/pull/6934\r\n* Fix the Slack report posting twice by @qgallouedec in https://github.com/huggingface/trl/pull/6936\r\n* Rewrite the TRL skill around the Python API by @qgallouedec in https://github.com/huggingface/trl/pull/6794\r\n* Raise when packing is combined with context parallelism by @qgallouedec in https://github.com/huggingface/trl/pull/6843\r\n* Suggest migrating to DistillationTrainer when the GKD config is fully covered by @qgallouedec in https://github.com/huggingface/trl/pull/6847\r\n* Remove one host sync from SFT chunked_nll loss by @qgallouedec in https://github.com/huggingface/trl/pull/6842\r\n* Release: v1.11 by @qgallouedec in https://github.com/huggingface/trl/pull/6937\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.10.0...v1.11.0\r\n","publishedAt":"2026-08-26T20:02:08.000Z","fetchedAt":"2026-08-26T21:12:42.831Z","url":"https://github.com/huggingface/trl/releases/tag/v1.11.0","media":[],"coverageCount":0},{"id":"rel_XU019-yMF4AdzT2wUZEAp","version":"v1.10.0","type":"feature","title":"v1.10.0","summary":"DistillationTrainer and DistillationConfig move from trl.experimental.distillation to the top-level trl package, with the old path emitting a FutureWarning until removal in v2.0.0. AsyncGRPOTrainer gains an experimental loop-owning path for training external agents like opencode, plus VLM support in DistillationTrainer and new DiffusionGemma SFT example.","titleGenerated":"TRL v1.10.0 graduates DistillationTrainer and adds loop-owning AsyncGRPO training","titleShort":"DistillationTrainer graduates; AsyncGRPO trains loop-owning agents","breaking":"minor","importance":4,"content":"## Features\r\n\r\n### 🎓 `DistillationTrainer` is now a stable trainer\r\n\r\nAfter a ~30-PR refactor that reshaped its data contract, loss path, generation stack, config surface, and tests, `DistillationTrainer` and `DistillationConfig` graduate from `trl.experimental.distillation` to the top-level `trl` package. Same import surface as SFT / DPO / GRPO / KTO. The old experimental path still works and emits a `FutureWarning` (removal in v2.0.0).\r\n\r\n```python\r\n# Before\r\nfrom trl.experimental.distillation import DistillationConfig, DistillationTrainer\r\n\r\n# Now\r\nfrom trl import DistillationConfig, DistillationTrainer\r\n```\r\n\r\nAlso lands a `trl distillation` CLI and moves the tests to `tests/`. The full refactor arc: switched signature columns to `prompt` (deprecated `messages`-format), pinned the generation stack to GRPO's, wired the chunked JSD loss and deleted the full-logit path, cleaned the Liger path to share extraction with the chunked path, adopted GRPO's `log()` / `training_step` timing / `_save_checkpoint`, rebuilt the test suite to GRPO shape, and much more.\r\n\r\nby @qgallouedec across ~30 PRs (#6479, #6480, #6481, #6482, #6484, #6487, #6497, #6508, #6509, #6510, #6511, #6512, #6513, #6521, #6522, #6523, #6524, #6525, #6526, #6530, #6537, #6604, #6605, #6606, #6607, #6609, #6610, #6611, #6612, #6613, #6614, #6629, #6630, #6632, #6633, #6634, #6639, #6640, #6641, #6642, #6643, #6644, #6645, #6647, #6653).\r\n\r\n### `DistillationTrainer` supports Vision Language Models\r\n\r\nAlongside the graduation, VLMs work end-to-end in `DistillationTrainer`.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6650\r\n\r\n### AsyncGRPO: loop-owning agent training (OpenEnv harness)\r\n\r\nNew experimental **loop-owning (black-box)** path for `AsyncGRPOTrainer` — for training external agents like [`opencode`](https://opencode.ai) that run their own tool loop, rather than TRL driving each turn as in the `environment_factory` white-box path.\r\n\r\nThe agent runs in an OpenEnv session in `transparent_proxy` mode; an in-sandbox proxy captures each turn's token ids and logprobs. On completion, TRL reads the trace, rebuilds per-turn training rows, and scores the workspace with the session's `verify()`. Ships with three caller-supplied policy hooks (`rollout_reward_fn`, `train_turn_fn`, `agent_turn_fn`) so you can drop framework aux calls (title generator, context summarizer) and reinforce only the turns you want.\r\n\r\nIncludes a self-contained `examples/scripts/openenv/opencode.py` (local subprocess sandbox + DeepCoder held-out stdin/stdout verifier), validated end-to-end on Qwen3-4B.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6420, plus HF-sandbox variant by @sergiopaniego in https://github.com/huggingface/trl/pull/6565\r\n\r\n### AsyncGRPO: observability + sampling controls\r\n\r\n* **Record rollout traces to trackio** by @AmineDiro in https://github.com/huggingface/trl/pull/6491\r\n* **Step-time metric** by @AmineDiro in https://github.com/huggingface/trl/pull/6490\r\n* **Fix queue wait time metric** by @AmineDiro in https://github.com/huggingface/trl/pull/6489\r\n* **`top_p` / `top_k` / `min_p` / `repetition_penalty` in `AsyncGRPOConfig`** by @AmineDiro in https://github.com/huggingface/trl/pull/6608\r\n* **Fork-independent epoch counting** — landed in v1.9 window\r\n* Close in-flight harness sessions concurrently on shutdown by @sergiopaniego in https://github.com/huggingface/trl/pull/6564\r\n\r\n### DiffusionGemma block-diffusion SFT example\r\n\r\nA new SFT example for [`google/diffusiongemma-26B-A4B-it`](https://huggingface.co/google/diffusiongemma-26B-A4B-it) implementing the reference recipe: one response block per step, uniform random token corruption with `t ~ U(eps, 1)`, two-pass self-conditioning, flat cross-entropy over the whole canvas plus an autoregressive co-loss on the encoder. LoRA targets attention + dense MLP linears. Requires `transformers >= 5.12.0`.\r\n\r\nShips with `diffusion_gemma.jinja` / `diffusion_gemma_training.jinja` chat templates (with `{% generation %}` markers) so `assistant_only_loss=True` works out of the box.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/6003\r\n\r\n### More model & template support\r\n\r\n* **LFM2 and LFM2.5** — day-zero training support by @qgallouedec in https://github.com/huggingface/trl/pull/6428\r\n* **Gemma 4 response template** by @qgallouedec in https://github.com/huggingface/trl/pull/6535\r\n\r\n### Default changes\r\n\r\nTwo default flips this cycle — pinning them explicitly is recommended if you want the old behavior.\r\n\r\n* **`SFTConfig.max_completion_length` / `GRPOConfig.max_completion_length`**: default bumped from `256` to `512`. By @dhruvnigam93 in https://github.com/huggingface/trl/pull/6264\r\n* **`GRPOConfig.use_bias_correction_kl`** now defaults to `True`. By @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6503\r\n\r\n### Packing and padding-free on VLMs with text-only data\r\n\r\nVLM configurations no longer reject packing / padding-free when the batch happens to be text-only.\r\n\r\nby @cris96spa in https://github.com/huggingface/trl/pull/6547\r\n\r\n### Other\r\n\r\n* Use a reference adapter with `target_parameters` on `peft>=0.20.0` by @albertvillanova in https://github.com/huggingface/trl/pull/6591\r\n* Reject unsupported `train_dataset` types in core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6493 and https://github.com/huggingface/trl/pull/6494\r\n* Reject unsupported `f_divergence_type` / loss type combinations in DPO by @qgallouedec in https://github.com/huggingface/trl/pull/6559\r\n* Validate that reward functions return one reward per completion by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6534\r\n* Validate YAML config root type by @RerankerGuo in https://github.com/huggingface/trl/pull/6656\r\n* Respect `TQDM_DISABLE` in DPO/KTO/BCO reference log-prob loops by @qgallouedec in https://github.com/huggingface/trl/pull/6507\r\n* Respect `use_cpu` in `create_model_from_path` `device_map` default by @Strongich in https://github.com/huggingface/trl/pull/6295\r\n* Use `dtype` instead of `torch_dtype` (deprecated) by @qgallouedec in https://github.com/huggingface/trl/pull/6560\r\n* Add missing logged metrics to the GRPO and RLOO docs by @PoojanTa in https://github.com/huggingface/trl/pull/6624\r\n* Add `test_train_moe_peft_model` for KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6589\r\n\r\n### vLLM version sweep\r\n\r\n* Add support for vLLM 0.26.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6569\r\n\r\n## Fixes\r\n\r\n* **Fix DAPO / CISPO / VESPO loss normalization** when `steps_per_generation != gradient_accumulation_steps` — gradients were silently mis-scaled by `gradient_accumulation_steps / steps_per_generation` (e.g. **0.5×** or **4×** the intended token-mean in common configs). Default-equivalent configs are unaffected. By @0xadvait in https://github.com/huggingface/trl/pull/6024\r\n* **Fix Liger kernel crash on pre-Ampere GPUs** by @albertvillanova in https://github.com/huggingface/trl/pull/6518\r\n* **Fix vLLM weight synchronization** when `vllm_enable_sleep_mode=True` by @muupan in https://github.com/huggingface/trl/pull/5313\r\n* **Fix vLLM server-mode communicator initialization** to use the current accelerator device by @sywangyi in https://github.com/huggingface/trl/pull/6417\r\n* **Use accelerator device instead of hardcoded `cuda`** in FSDP2 vLLM weight sync by @verma8076 in https://github.com/huggingface/trl/pull/6592\r\n* **Fix `prepare_deepspeed` crash with CPU offload optimizer** by @roycho96 in https://github.com/huggingface/trl/pull/5916\r\n* **`fix(ppo)`: exclude padding tokens from entropy calculation** by @mukund1985 in https://github.com/huggingface/trl/pull/6121\r\n* **`fix`: avoid `CopySlices` when scaling policy logits** by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6554\r\n* **Fix multimodal field handling in GRPO tool-call loop** by @HariomJangra in https://github.com/huggingface/trl/pull/6286\r\n* **`[GRPO]` Apply the completion mask elementwise in the LuSPO loss aggregation** by @YaseenBashaT in https://github.com/huggingface/trl/pull/6654\r\n* **`[GRPO]` Fix entropy bonus normalization inconsistency across loss types** by @YaseenBashaT in https://github.com/huggingface/trl/pull/6648\r\n* Fix incorrect example script path in Online DPO docs by @latent-9 in https://github.com/huggingface/trl/pull/6598\r\n* Point the opencode example at `huggingface/OpenEnv` by @sergiopaniego in https://github.com/huggingface/trl/pull/6529\r\n* Remove symlink support from skills installation by @albertvillanova in https://github.com/huggingface/trl/pull/6665\r\n\r\n## Documentation and Examples\r\n\r\n* docs: complete `GRPOConfig` `loss_type` help by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6477\r\n* Fix inaccuracies in the release instructions by @albertvillanova in https://github.com/huggingface/trl/pull/6566\r\n* Fix patch release instructions: pull release branch instead of main by @albertvillanova in https://github.com/huggingface/trl/pull/6563\r\n* Add `opencode_hf_sandbox.py`: loop-owning training in remote HF sandboxes by @sergiopaniego in https://github.com/huggingface/trl/pull/6565\r\n\r\n## CI\r\n\r\n* `test`: don't hard-code `bf16=True` on devices that lack bf16 support by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6036\r\n* Cleanup xpu cache memory after each test by @kaixuanliu in https://github.com/huggingface/trl/pull/5771\r\n* Force `use_cpu` in tests when no CUDA device is available by @albertvillanova in https://github.com/huggingface/trl/pull/6504\r\n* Silence transformers `LOAD REPORT` tables in test output by @albertvillanova in https://github.com/huggingface/trl/pull/6502\r\n* Require vision for VLM response parsing tests by @albertvillanova in https://github.com/huggingface/trl/pull/6602\r\n* Require `jmespath` only on transformers below 5.13.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6597\r\n* Add regression tests for evaluating init-time eval datasets after training with precomputed reference log-probs by @albertvillanova in https://github.com/huggingface/trl/pull/6488\r\n* Test `train_dataset=None` raises for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6492\r\n* Remove unnecessary `loss_type` in KTO precompute ref log-probs test by @albertvillanova in https://github.com/huggingface/trl/pull/6501\r\n* Fix `test_peft_with_quantization` tests for bitsandbytes 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6546\r\n* Ignore benign bitsandbytes 4-bit blocksize alignment warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6582\r\n* Suppress deprecated `torch.get_autocast_gpu_dtype()` warning from mamba-ssm kernel by @albertvillanova in https://github.com/huggingface/trl/pull/6556\r\n* Document removal criteria for pytest warning filters by @albertvillanova in https://github.com/huggingface/trl/pull/6595\r\n* Fix test: don't call `add_response_schema` when the model already ships a response template by @qgallouedec in https://github.com/huggingface/trl/pull/6574\r\n* Fix CI quality: fix copyright year in `openenv_harness.py` by @albertvillanova in https://github.com/huggingface/trl/pull/6531\r\n* Hotfix CI: temporarily pin `liger-kernel < 0.8.1` by @albertvillanova in https://github.com/huggingface/trl/pull/6517\r\n* Hotfix CI: temporarily pin `bitsandbytes < 0.50.0` by @albertvillanova in https://github.com/huggingface/trl/pull/6544\r\n* Hotfix CI: xfail NemotronH GRPO/RLOO tests against a broken transformers dev build by @albertvillanova in https://github.com/huggingface/trl/pull/6542 (reverted in https://github.com/huggingface/trl/pull/6558)\r\n* `ci`: exclude the buggy TruffleHog LOB detector by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6674\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6540\r\n* Revert transformers rotary device deprecation warning filter by @albertvillanova in https://github.com/huggingface/trl/pull/6596\r\n\r\n## New Contributors\r\n* @gowtham-sai-yadav made their first contribution in https://github.com/huggingface/trl/pull/6477\r\n* @dhruvnigam93 made their first contribution in https://github.com/huggingface/trl/pull/6264\r\n* @cris96spa made their first contribution in https://github.com/huggingface/trl/pull/6547\r\n* @mukund1985 made their first contribution in https://github.com/huggingface/trl/pull/6121\r\n* @latent-9 made their first contribution in https://github.com/huggingface/trl/pull/6598\r\n* @verma8076 made their first contribution in https://github.com/huggingface/trl/pull/6592\r\n* @PoojanTa made their first contribution in https://github.com/huggingface/trl/pull/6624\r\n* @YaseenBashaT made their first contribution in https://github.com/huggingface/trl/pull/6654\r\n* @RerankerGuo made their first contribution in https://github.com/huggingface/trl/pull/6656\r\n* @HariomJangra made their first contribution in https://github.com/huggingface/trl/pull/6286\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6496\r\n* Test train_dataset=None raises for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6492\r\n* Add regression tests for evaluating init-time eval datasets after training with precomputed reference log-probs by @albertvillanova in https://github.com/huggingface/trl/pull/6488\r\n* docs: complete GRPOConfig loss_type help by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6477\r\n* test: don't hard-code bf16=True on devices that lack bf16 support by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6036\r\n* [DistillationTrainer refactor]  Switch tests, docs, and example to prompt-only datasets by @qgallouedec in https://github.com/huggingface/trl/pull/6479\r\n* cleanup xpu cache memory after each test by @kaixuanliu in https://github.com/huggingface/trl/pull/5771\r\n* Fix vLLM server-mode communicator initialization to use the current A… by @sywangyi in https://github.com/huggingface/trl/pull/6417\r\n* [DistillationTrainer refactor] Remove messages-format support and prompt-length config by @qgallouedec in https://github.com/huggingface/trl/pull/6480\r\n* [DistillationTrainer refactor] Pin signature columns to `[\"prompt\", \"image\", \"images\"]` by @qgallouedec in https://github.com/huggingface/trl/pull/6481\r\n* [DistillationTrainer refactor] Emit `completion_mask` alongside `labels` by @qgallouedec in https://github.com/huggingface/trl/pull/6482\r\n* [DistillationTrainer refactor] Loss consumes `completion_mask` by @qgallouedec in https://github.com/huggingface/trl/pull/6484\r\n* Fix : queue wait time metric AsyncGRPOTrainer by @AmineDiro in https://github.com/huggingface/trl/pull/6489\r\n* [DistillationTrainer refactor] Emit prompt_ids / prompt_mask / completion_ids by @qgallouedec in https://github.com/huggingface/trl/pull/6487\r\n* Remove unnecessary loss_type in KTO precompute ref log-probs test by @albertvillanova in https://github.com/huggingface/trl/pull/6501\r\n* Force use_cpu in tests when no CUDA device is available by @albertvillanova in https://github.com/huggingface/trl/pull/6504\r\n* Hotfix CI: Temporarily pin liger-kernel < 0.8.1 by @albertvillanova in https://github.com/huggingface/trl/pull/6517\r\n* Silence transformers LOAD REPORT tables in test output by @albertvillanova in https://github.com/huggingface/trl/pull/6502\r\n* Change default `max_completion_length` from 256 to 512 by @dhruvnigam93 in https://github.com/huggingface/trl/pull/6264\r\n* Add DiffusionGemma block-diffusion SFT example by @kashif in https://github.com/huggingface/trl/pull/6003\r\n* Record rollout traces to trackio  by @AmineDiro in https://github.com/huggingface/trl/pull/6491\r\n* [DistillationTrainer refactor] Switch the loss to GRPO's keys by @qgallouedec in https://github.com/huggingface/trl/pull/6497\r\n* [DistillationTrainer refactor] Freeze ServerDistillationTrainer self-contained by @qgallouedec in https://github.com/huggingface/trl/pull/6508\r\n* Add LFM2 and LFM2.5 support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6428\r\n* [DistillationTrainer refactor] Delete `_compute_prompt_length` by @qgallouedec in https://github.com/huggingface/trl/pull/6509\r\n* Async grpo OpenEnv harness rollout by @AmineDiro in https://github.com/huggingface/trl/pull/6420\r\n* [DistillationTrainer refactor] Add `_tokenize_prompts` (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6510\r\n* [DistillationTrainer refactor] Add the shared generation config fields (add-only) by @qgallouedec in https://github.com/huggingface/trl/pull/6511\r\n* Fix crash when training with Liger kernel on pre-Ampere GPUs by @albertvillanova in https://github.com/huggingface/trl/pull/6518\r\n* [DistillationTrainer refactor] Add the ctor generation-supporting state (add-only) by @qgallouedec in https://github.com/huggingface/trl/pull/6512\r\n* Fix prepare_deepspeed crash with cpu offload optimizer by @roycho96 in https://github.com/huggingface/trl/pull/5916\r\n* Fix CI quality: Fix copyright year in trl/experimental/async_grpo/openenv_harness.py by @albertvillanova in https://github.com/huggingface/trl/pull/6531\r\n* Reject unsupported train_dataset types in core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6493\r\n* [DistillationTrainer refactor] Add `_generate_single_turn` (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6513\r\n* [DistillationTrainer refactor] Add `_generate` + `_generate_and_score_completions` (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6521\r\n* Test train_dataset types at init for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6494\r\n* Add Gemma4 response template by @qgallouedec in https://github.com/huggingface/trl/pull/6535\r\n* Validate that reward functions return one reward per completion by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6534\r\n* fix(grpo): correct DAPO/CISPO/VESPO loss normalization when steps_per_generation != gradient_accumulation_steps by @0xadvait in https://github.com/huggingface/trl/pull/6024\r\n* Hotfix CI: Temporarily pin bitsandbytes < 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6544\r\n* Hotfix CI: Xfail NemotronH GRPO/RLOO tests against a broken transformers dev build by @albertvillanova in https://github.com/huggingface/trl/pull/6542\r\n* added step time metric to AsyncGRPOTrainer by @AmineDiro in https://github.com/huggingface/trl/pull/6490\r\n* Allow packing and padding-free on VLMs with text-only data by @cris96spa in https://github.com/huggingface/trl/pull/6547\r\n* fix(ppo): exclude padding tokens from entropy calculation by @mukund1985 in https://github.com/huggingface/trl/pull/6121\r\n* Respect `TQDM_DISABLE` in DPO/KTO/BCO reference log-prob loops by @qgallouedec in https://github.com/huggingface/trl/pull/6507\r\n* Revert xfail for NemotronH GRPO/RLOO tests now that transformers#47569 fixed the kernels bug by @albertvillanova in https://github.com/huggingface/trl/pull/6558\r\n* fix: avoid CopySlices when scaling policy logits by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6554\r\n* Fix a bug in vLLM weight synchronization when `vllm_enable_sleep_mode=True` by @muupan in https://github.com/huggingface/trl/pull/5313\r\n* Point the opencode example at huggingface/OpenEnv by @sergiopaniego in https://github.com/huggingface/trl/pull/6529\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6540\r\n* Suppress deprecated torch.get_autocast_gpu_dtype() warning from mamba-ssm kernel by @albertvillanova in https://github.com/huggingface/trl/pull/6556\r\n* Fix test_peft_with_quantization tests for bitsandbytes 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6546\r\n* Fix patch release instructions: Pull release branch instead of main by @albertvillanova in https://github.com/huggingface/trl/pull/6563\r\n* Fix inaccuracies in the release instructions by @albertvillanova in https://github.com/huggingface/trl/pull/6566\r\n* Use `dtype` instead of `torch_dtype` (deprecated) by @qgallouedec in https://github.com/huggingface/trl/pull/6560\r\n* Fix test: don't call `add_response_schema` when the model already ships a response template by @qgallouedec in https://github.com/huggingface/trl/pull/6574\r\n* [DistillationTrainer refactor] Switch generation to GRPO's stack; delete the buffer by @qgallouedec in https://github.com/huggingface/trl/pull/6522\r\n* [DistillationTrainer refactor] Align the generation_kwargs dict with GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/6523\r\n* [DistillationTrainer refactor] Align the vLLM ctor block with GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/6524\r\n* [DistillationTrainer refactor] Add `_get_last_hidden_state` (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6525\r\n* Default `use_bias_correction_kl=True` in GRPOConfig by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6503\r\n* Ignore benign bitsandbytes 4-bit blocksize alignment warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6582\r\n* Respect use_cpu in create_model_from_path device_map default by @Strongich in https://github.com/huggingface/trl/pull/6295\r\n* Ignore transformers rotary `device` deprecation warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6585\r\n* Add `test_train_moe_peft_model` for KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6589\r\n* Revert transformers rotary device deprecation warning filter by @albertvillanova in https://github.com/huggingface/trl/pull/6596\r\n* Document removal criteria for pytest warning filters by @albertvillanova in https://github.com/huggingface/trl/pull/6595\r\n* Reject unsupported `f_divergence_type` / loss type combinations in DPO by @qgallouedec in https://github.com/huggingface/trl/pull/6559\r\n* Add support for vLLM 0.26.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6569\r\n* Use a reference adapter with `target_parameters` on `peft>=0.20.0` by @albertvillanova in https://github.com/huggingface/trl/pull/6591\r\n* Fix incorrect example script path in Online DPO docs by @latent-9 in https://github.com/huggingface/trl/pull/6598\r\n* Add top_p/top_k/min_p/repetition_penalty to AsyncGRPOConfig by @AmineDiro in https://github.com/huggingface/trl/pull/6608\r\n* Use accelerator device instead of hardcoded cuda in FSDP2 vLLM weight sync by @verma8076 in https://github.com/huggingface/trl/pull/6592\r\n* [DistillationTrainer refactor] Add the chunked JSD loss (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6526\r\n* [DistillationTrainer refactor] Wire compute_loss to the chunked JSD path; delete the full-logit loss by @qgallouedec in https://github.com/huggingface/trl/pull/6530\r\n* [DistillationTrainer refactor] Heavy-clean the Liger path: share extraction with the chunked path by @qgallouedec in https://github.com/huggingface/trl/pull/6537\r\n* [DistillationTrainer refactor] Default `disable_dropout` to `False` by @qgallouedec in https://github.com/huggingface/trl/pull/6604\r\n* [DistillationTrainer refactor] Config finish by @qgallouedec in https://github.com/huggingface/trl/pull/6605\r\n* [DistillationTrainer refactor] Constructor cleanup: safe internal cleanups by @qgallouedec in https://github.com/huggingface/trl/pull/6606\r\n* [DistillationTrainer refactor] Config cosmetic parity by @qgallouedec in https://github.com/huggingface/trl/pull/6607\r\n* [DistillationTrainer refactor] Constructor cleanup (40b): drop eval-metric params, require `model` by @qgallouedec in https://github.com/huggingface/trl/pull/6609\r\n* [DistillationTrainer refactor] Constructor cleanup (40c): quantization + PEFT/QLoRA parity by @qgallouedec in https://github.com/huggingface/trl/pull/6610\r\n* [DistillationTrainer refactor] Make `advantages` optional in `print_prompt_completions_sample` by @qgallouedec in https://github.com/huggingface/trl/pull/6611\r\n* [DistillationTrainer refactor] Use the shared `print_prompt_completions_sample` by @qgallouedec in https://github.com/huggingface/trl/pull/6612\r\n* [DistillationTrainer refactor] Adopt GRPO's `log()` + bounded `_logs` deques by @qgallouedec in https://github.com/huggingface/trl/pull/6613\r\n* [DistillationTrainer refactor] Adopt GRPO's `training_step` timing + `_save_checkpoint` by @qgallouedec in https://github.com/huggingface/trl/pull/6614\r\n* Require vision for VLM response parsing tests by @albertvillanova in https://github.com/huggingface/trl/pull/6602\r\n* Require jmespath only on transformers below 5.13.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6597\r\n* Add missing logged metrics to the GRPO and RLOO docs by @PoojanTa in https://github.com/huggingface/trl/pull/6624\r\n* Close in-flight harness sessions concurrently on shutdown by @sergiopaniego in https://github.com/huggingface/trl/pull/6564\r\n* Add opencode_hf_sandbox.py: loop-owning training in remote HF sandboxes by @sergiopaniego in https://github.com/huggingface/trl/pull/6565\r\n* Remove symlink support from skills installation by @albertvillanova in https://github.com/huggingface/trl/pull/6665\r\n* ci: exclude the buggy TruffleHog lob detector by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6674\r\n* [GRPO] Apply the completion mask elementwise in the luspo loss aggregation by @YaseenBashaT in https://github.com/huggingface/trl/pull/6654\r\n* [GRPO] Fix entropy bonus normalization inconsistency across loss types by @YaseenBashaT in https://github.com/huggingface/trl/pull/6648\r\n* Validate YAML config root type by @RerankerGuo in https://github.com/huggingface/trl/pull/6656\r\n* Fix multimodal field handling in GRPO tool call loop by @HariomJangra in https://github.com/huggingface/trl/pull/6286\r\n* [DistillationTrainer refactor] Reject `use_liger_kernel` with `logit_scale` / `final_logit_softcapping` by @qgallouedec in https://github.com/huggingface/trl/pull/6629\r\n* [DistillationTrainer refactor] Convention/cosmetic sweep by @qgallouedec in https://github.com/huggingface/trl/pull/6630\r\n* [DistillationTrainer refactor] Config: GRPO parity drift fixes by @qgallouedec in https://github.com/huggingface/trl/pull/6632\r\n* [DistillationTrainer refactor] Log the student entropy metric by @qgallouedec in https://github.com/huggingface/trl/pull/6633\r\n* [DistillationTrainer refactor] Rewrite the test suite to GRPO shape by @qgallouedec in https://github.com/huggingface/trl/pull/6634\r\n* [DistillationTrainer refactor] Docs: align distillation_trainer.md with stable-trainer conventions by @qgallouedec in https://github.com/huggingface/trl/pull/6639\r\n* [DistillationTrainer refactor] Promote DistillationTrainer to the stable API by @qgallouedec in https://github.com/huggingface/trl/pull/6640\r\n* [DistillationTrainer refactor] Move DistillationTrainer to the Stable telemetry group by @qgallouedec in https://github.com/huggingface/trl/pull/6641\r\n* [DistillationTrainer refactor] Add the `trl distillation` CLI by @qgallouedec in https://github.com/huggingface/trl/pull/6642\r\n* [DistillationTrainer refactor] Move the distillation tests to `tests/` by @qgallouedec in https://github.com/huggingface/trl/pull/6647\r\n* [DistillationTrainer refactor] Docs: de-namespace DistillationTrainer for the stable API by @qgallouedec in https://github.com/huggingface/trl/pull/6643\r\n* [DistillationTrainer refactor] Update the example script for the stable API by @qgallouedec in https://github.com/huggingface/trl/pull/6644\r\n* [DistillationTrainer refactor] Re-point ServerDistillationTrainer at the stable base by @qgallouedec in https://github.com/huggingface/trl/pull/6645\r\n* [DistillationTrainer refactor] Support Vision Language Models by @qgallouedec in https://github.com/huggingface/trl/pull/6650\r\n* [DistillationTrainer refactor] Docs: announce the DistillationTrainer promotion in What's New by @qgallouedec in https://github.com/huggingface/trl/pull/6653\r\n* Release: v1.10 by @qgallouedec in https://github.com/huggingface/trl/pull/6724\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.9.0...v1.10.0\r\n","publishedAt":"2026-08-13T01:36:04.000Z","fetchedAt":"2026-08-13T05:10:27.521Z","url":"https://github.com/huggingface/trl/releases/tag/v1.10.0","media":[],"coverageCount":0},{"id":"rel_PupF6nL1tk3deaqBRAazI","version":"v1.9.2","type":"feature","title":"v1.9.2","summary":"Fixed quantization tests for bitsandbytes 0.50.0 and reverted an xfail skip for Nemotron H GRPO/RLOO tests after a transformers kernel bug fix.","titleGenerated":"TRL v1.9.2 fixes quantization test and unblocks Nemotron kernels","titleShort":"Quantization tests fixed; Nemotron kernels unblocked","breaking":"none","importance":2,"content":"## What's Changed\r\n\r\n* Revert xfail for NemotronH GRPO/RLOO tests now that transformers#47569 fixed the kernels bug by @albertvillanova in https://github.com/huggingface/trl/pull/6558\r\n* Fix test_peft_with_quantization tests for bitsandbytes 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6546\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.9.1...v1.9.2","publishedAt":"2026-07-28T10:39:01.000Z","fetchedAt":"2026-07-28T13:08:06.823Z","url":"https://github.com/huggingface/trl/releases/tag/v1.9.2","media":[],"coverageCount":0},{"id":"rel_WJkG0IXM71QVvkGnKuG6s","version":"v1.9.1","type":"feature","title":"v1.9.1","summary":"Fixes a crash when training with Liger kernel on pre-Ampere (Fermi, Kepler, Maxwell, Pascal) GPUs and corrects DAPO, CISPO, and VESPO loss normalization when steps_per_generation differs from gradient_accumulation_steps. Also fixes vLLM server-mode communicator initialization, queue wait time metric in AsyncGRPOTrainer, and prepare_deepspeed crash with CPU offload optimizer.","titleGenerated":"TRL v1.9.1 fixes Liger kernel crash on pre-Ampere GPUs and DAPO/VESPO loss normalization","titleShort":"Liger kernel crash on pre-Ampere GPUs fixed; DAPO/VESPO loss normalization corrected","breaking":"none","importance":2,"content":"## What's Changed\r\n\r\n* Fix vLLM server-mode communicator initialization to use the current A… by @sywangyi in https://github.com/huggingface/trl/pull/6417\r\n* Fix : queue wait time metric AsyncGRPOTrainer by @AmineDiro in https://github.com/huggingface/trl/pull/6489\r\n* Hotfix CI: Temporarily pin liger-kernel < 0.8.1 by @albertvillanova in https://github.com/huggingface/trl/pull/6517\r\n* Fix crash when training with Liger kernel on pre-Ampere GPUs by @albertvillanova in https://github.com/huggingface/trl/pull/6518\r\n* Fix prepare_deepspeed crash with cpu offload optimizer by @roycho96 in https://github.com/huggingface/trl/pull/5916\r\n* Add Gemma4 response template by @qgallouedec in https://github.com/huggingface/trl/pull/6535\r\n* fix(grpo): correct DAPO/CISPO/VESPO loss normalization when steps_per_generation != gradient_accumulation_steps by @0xadvait in https://github.com/huggingface/trl/pull/6024\r\n* Hotfix CI: Temporarily pin bitsandbytes < 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6544\r\n* Hotfix CI: Xfail NemotronH GRPO/RLOO tests against a broken transformers dev build by @albertvillanova in https://github.com/huggingface/trl/pull/6542\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.9.0...v1.9.1\r\n","publishedAt":"2026-07-26T16:49:29.000Z","fetchedAt":"2026-07-26T17:07:50.027Z","url":"https://github.com/huggingface/trl/releases/tag/v1.9.1","media":[],"coverageCount":0},{"id":"rel_1Rmxks9DRP1V0tp4Z4gjz","version":"v1.9.0","type":"feature","title":"v1.9.0","summary":"GRPO and RLOO trainers now support iterable datasets via a new repeat_iterable_dataset generator, and environments can own the data (train_dataset becomes optional). Message-level rollouts land in AsyncGRPO, and the DistillationTrainer received a major refactor including a fix for num_items_in_batch that counted the wrong tokens and NaN'd on prompt-only datasets.","titleGenerated":"TRL v1.9.0 supports streaming datasets in GRPO/RLOO and environment-owned data","titleShort":"Streaming datasets in GRPO/RLOO; environments own data without dummy dataset","breaking":"minor","importance":3,"content":"## Features\r\n\r\n### Iterable / streaming datasets in GRPO and RLOO\r\n\r\nLong-standing request finally landed. GRPO and RLOO rely on `RepeatSampler` to repeat each prompt `num_generations` times and group them across processes — but samplers can't attach to iterable datasets (no length, no indexing), and the old behavior silently bypassed the sampler, quietly corrupting advantage computation.\r\n\r\nA new `repeat_iterable_dataset` generator reproduces `RepeatSampler`'s exact ordering by transforming the stream itself instead of reordering indices. Trainers wrap iterable datasets via `IterableDataset.from_generator`, force `dispatch_batches=False`, and shard the stream per-process to preserve prompt groups after cross-process gathering.\r\n\r\n```python\r\nfrom datasets import load_dataset\r\nfrom trl import GRPOConfig, GRPOTrainer\r\n\r\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\", streaming=True)\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=GRPOConfig(max_steps=1000),  # required for iterable datasets\r\n    train_dataset=dataset,\r\n    reward_funcs=accuracy_reward,\r\n)\r\n```\r\n\r\n`max_steps` is required for iterable datasets (no length). A clear error also fires if `dispatch_batches=True` is combined with an iterable dataset.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6351\r\n\r\n### Environment-owned datasets\r\n\r\nWhen an `environment_factory` is provided, the environment can now own the data. `train_dataset` becomes optional: `reset()` returns the prompt. No more fabricating a dummy dataset just to drive the loop.\r\n\r\n```python\r\nclass WordleEnv:\r\n    def reset(self, **kwargs) -> str:      # returns the prompt\r\n        self._target = sample(words)\r\n        return f\"Guess a {len(self._target)}-letter word.\"\r\n    def get_reward(self) -> float:\r\n        return 1.0 if self._solved else 0.0\r\n    def guess(self, word: str) -> str:     # exposed tool\r\n        self._solved = word == self._target\r\n        return _feedback(word, self._target)\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=GRPOConfig(max_steps=1000),\r\n    environment_factory=WordleEnv,          # no train_dataset needed\r\n)\r\n```\r\n\r\nApplies to both `GRPOTrainer` and `AsyncGRPOTrainer`. A provided `train_dataset` still works exactly as before; multi-environment routing datasets can now be **`environment`-only** (no `prompt` column required).\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6349\r\n\r\n### Message-level rollouts in AsyncGRPO\r\n\r\nAn opt-in way to build training rows from a multi-turn conversation. Message mode keeps the conversation as messages and **re-tokenizes the whole thing each turn**, then checks whether the fresh tokens still start with the tokens held so far. If yes → append the new part (same as token mode). If no → a rewrite happened → close the row and open a new one that matches what the model actually read.\r\n\r\n```python\r\nAsyncGRPOConfig(\r\n    rollout_protocol=\"message\",   # \"token\" (default) | \"message\"\r\n    fork_threshold_tokens=1024,   # message-mode only\r\n)\r\n```\r\n\r\nPer-turn drift is classified as **CLEAN** (append), **REALIGN** (last-answer tail wobble → overwrite same row), or **FORK** (real rewrite → new row). One advantage per conversation, stamped on every row it produced; under token-mean loss a fork is invisible. Sets up the plumbing for future tree-trajectory support — scoring is already branch-agnostic (groups by `rollout_id`).\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6250\r\n\r\n### AsyncGRPO: `VLLMClient` and decoupled weight sync\r\n\r\nA follow-up to v1.8's rollout worker split. A small `VLLMClient` becomes the single place that speaks HTTP to the vLLM server (named methods for `wait_for_server_ready`, `get_max_model_len`, `pause`/`resume`, weight-update endpoints — stateless, so picklable). Generation deliberately stays on its own async `aiohttp` session in the rollout child.\r\n\r\nWeight sync is now built from config independent of `rollout_worker` (previously a custom rollout worker force-set it to `None`), and is injectable via a new `WeightTransferProtocol`. `token_budget` now waits for `/health` before defaulting to `get_max_model_len()`, so a slow-starting vLLM waits instead of failing the run.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6269\r\n\r\n### `DistillationTrainer` refactor\r\n\r\nA ~13-PR sweep that reshapes the experimental `DistillationTrainer` for future stability:\r\n\r\n- **Extract `ServerDistillationTrainer`** for the teacher-server path (#6454)\r\n- **Move IW-OPD** to a dedicated experimental trainer (#6453)\r\n- **Remove the local sparse top-1 loss path** from the base trainer (#6455)\r\n- **Remove top-k support** from the base generalized JSD loss (#6456)\r\n- **Validate the teacher by vocab size** instead of tokenizer identity (#6457)\r\n- **Deprecate & remove `lmbda`** on/off-policy mixing and the off-policy training branch (#6458, #6460); GKD paper reproduction now points at `GKDTrainer` (#6459)\r\n- **Accept a `prompt` column** alongside `messages` (#6461); **deprecate messages-format datasets** (#6474)\r\n- ⚠️ **Fix `num_items_in_batch`** to count generated completion tokens — the loss denominator was wrong for on-policy training (counted dataset completions, not generated ones) and **NaN'd on prompt-only datasets**. This changes loss values for anyone on the old path (#6478)\r\n- Preparatory pins on the objective and on/off-policy behavior end to end (#6450, #6451, #6452)\r\n\r\nAll by @qgallouedec.\r\n\r\n### New IW-OPD distillation objective\r\n\r\nImportance-Weighted On-Policy Distillation lands as an optional objective on the experimental `DistillationTrainer` — detached IW-OPD advantages built from sampled-token teacher and rollout logprobs (with cached vLLM rollout logprobs when `use_vllm=True`).\r\n\r\n```python\r\nDistillationConfig(\r\n    distillation_objective=\"iw_opd\",\r\n    iw_opd_gamma=1.0,\r\n    iw_opd_epsilon=0.2,\r\n    lmbda=1.0, reverse_kl_top_1_mode=\"sampled\",  # required\r\n)\r\n```\r\n\r\nNow split into its own experimental trainer as part of the refactor above.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/6191\r\n\r\n### `log_multimodal` for GRPO / RLOO\r\n\r\nNew config toggle to control whether images from multimodal completions are logged to trackers. Also applied to experimental DPPO and GRPO-with-replay-buffer.\r\n\r\nby @apardyl in https://github.com/huggingface/trl/pull/5408 and by @albertvillanova in https://github.com/huggingface/trl/pull/6352\r\n\r\n### vLLM version sweep\r\n\r\n* Support vLLM 0.24.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6355\r\n* Support vLLM 0.25.0 and 0.25.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6406\r\n* Drop vLLM 0.16 by @qgallouedec in https://github.com/huggingface/trl/pull/6404\r\n\r\n### Other\r\n\r\n* Simplify KTO KL completion construction into a single batched map by @albertvillanova in https://github.com/huggingface/trl/pull/6335\r\n* Add security policy to the repo by @albertvillanova in https://github.com/huggingface/trl/pull/6362\r\n* Removed `generate_rollout_completions` by @sergiopaniego in https://github.com/huggingface/trl/pull/5870\r\n* Set explicit neutral `top_p` in GKD on-policy generation by @sergiopaniego in https://github.com/huggingface/trl/pull/6337\r\n* Support `DatasetDict` and `IterableDatasetDict` as `eval_dataset` in `evaluate` by @albertvillanova in https://github.com/huggingface/trl/pull/6326\r\n* AsyncGRPO fork-independent epoch counting by @AmineDiro in https://github.com/huggingface/trl/pull/6385\r\n* Align `AsyncGRPOTrainer` args handling with stable trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6377\r\n* Reject `auto_find_batch_size` in GRPO and RLOO by @qgallouedec in https://github.com/huggingface/trl/pull/6411\r\n* Drop fully-masked examples after truncation in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/6320\r\n* Drop examples with no completion left after truncation in DPO and KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6321\r\n* Add `GMPOTrainer` to the telemetry allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/6410\r\n* `[GRPO]` Treat explicit `max_tool_calling_iterations=0` as zero, not unlimited by @Strongich in https://github.com/huggingface/trl/pull/6423\r\n* Add a tip to the Paper Index on reading papers with the `hf` CLI by @davanstrien in https://github.com/huggingface/trl/pull/6422\r\n\r\n## Deprecations and removals\r\n\r\n* **Remove the experimental DPPO trainer** by @qgallouedec in https://github.com/huggingface/trl/pull/6402\r\n* **Deprecate `messages`-format datasets in `DistillationTrainer`** — use `prompt` column instead — by @qgallouedec in https://github.com/huggingface/trl/pull/6474\r\n* **Deprecate `lmbda` on/off-policy mixing** in `DistillationTrainer` — off-policy path fully removed by @qgallouedec in https://github.com/huggingface/trl/pull/6458 and https://github.com/huggingface/trl/pull/6460\r\n\r\n## Fixes\r\n\r\n* **Fix biased cross-rank aggregation of token-weighted metrics** in GRPO / RLOO — mean-of-means was biased when per-rank token counts differ; now reduces numerator + count jointly for a true global mean. Also fixes `clip_ratio/low_min` / `high_max` to be per-completion instead of per-rank-batch-mean (DP-layout-independent). Logging-only, no loss change. By @qgallouedec in https://github.com/huggingface/trl/pull/6380\r\n* **Fix DPO/KTO `use_liger_kernel` under DeepSpeed ZeRO-3** by @qgallouedec in https://github.com/huggingface/trl/pull/6372\r\n* **Force `use_reentrant=True` for PEFT + ZeRO-3 + gradient checkpointing** across all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6356\r\n* **Fix `precompute_ref_log_probs=True` with DeepSpeed in DPO and KTO** by @qgallouedec in https://github.com/huggingface/trl/pull/6403\r\n* **Fix `cast_lm_head_to_fp32` under FSDP2 / DeepSpeed ZeRO-3** by @qgallouedec in https://github.com/huggingface/trl/pull/6330\r\n* **Fix corrupted reasoning blocks in Qwen3-family training chat templates** — the both-tags guard broke replay of self-generated thinking traces (SFT / distillation on model outputs); reverted to the original `</think>`-only check. By @qgallouedec in https://github.com/huggingface/trl/pull/6363\r\n* **Fix EOS being masked when `pad_token_id == eos_token_id` in GKD and GOLD** by @roycho96 in https://github.com/huggingface/trl/pull/6357\r\n* **Fix GRPO truncated-completion loss normalization** by @Functionhx in https://github.com/huggingface/trl/pull/6439\r\n* **Fix `apply_chat_template` dropping leading tokens from `chosen`** when the prompt common prefix shrinks against `rejected` by @sohumt123 in https://github.com/huggingface/trl/pull/6433\r\n* Fix crash when reward function is a `functools.partial` or callable instance by @qgallouedec in https://github.com/huggingface/trl/pull/6313\r\n* Fix QLoRA + FSDP1 mixed-dtype crash in `prepare_peft_model` by @sergiopaniego in https://github.com/huggingface/trl/pull/6343\r\n* Truncate GOLD on-policy prompts before generation, keeping the prompt end by @roycho96 in https://github.com/huggingface/trl/pull/6359\r\n* Fix `FileNotFoundError` in DPO/KTO `precompute_ref_log_probs` by @Functionhx in https://github.com/huggingface/trl/pull/6348\r\n* Fix `GeometricMixtureWrapper.generate()` crash after transformers' MoE decode optimization change by @albertvillanova in https://github.com/huggingface/trl/pull/6413\r\n* Fix GKD `vocab_size` check for multimodal model configs by @sergiopaniego in https://github.com/huggingface/trl/pull/6340\r\n* Fix OpenEnv docker-image example bootstrap by @sergiopaniego in https://github.com/huggingface/trl/pull/6329\r\n* Fix error type for `IterableDataset` in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6324\r\n* Fix KTO evaluate crash on streaming datasets with unpaired-preference data by @albertvillanova in https://github.com/huggingface/trl/pull/6325\r\n* Fix DPO/KTO evaluate crash with a dict `eval_dataset` and `precompute_ref_log_probs` by @albertvillanova in https://github.com/huggingface/trl/pull/6370\r\n* Fix DPO/KTO double-preprocessing `eval_dataset` dict passed to `evaluate()` by @albertvillanova in https://github.com/huggingface/trl/pull/6434\r\n* Raise when DPO/KTO cannot precompute correct reference log-probs at eval time by @albertvillanova in https://github.com/huggingface/trl/pull/6443\r\n* Avoid in-place mutation of input prompts in GRPO and DPPO environment loops by @albertvillanova in https://github.com/huggingface/trl/pull/6364 and https://github.com/huggingface/trl/pull/6366\r\n* Align KTO with DPO: drop `hasattr` guards on `lm_head.bias` in the Liger path by @qgallouedec in https://github.com/huggingface/trl/pull/6425\r\n* Align KTO with DPO: fix `logits/chosen` and `logits/rejected` metrics by @qgallouedec in https://github.com/huggingface/trl/pull/6424\r\n* Fix missing `eos_token_id` in non-vLLM on-policy GOLD test stubs by @kashif in https://github.com/huggingface/trl/pull/6394\r\n* Fix `BEMACallback` `bias_power` docstring by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6390\r\n\r\n## Documentation and Examples\r\n\r\n* Document `max_steps` requirement for iterable datasets in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6432\r\n* Align `AsyncGRPOConfig.max_completion_length` description with `GRPOConfig` by @qgallouedec in https://github.com/huggingface/trl/pull/6374\r\n* Align `AsyncGRPOTrainer.processing_class` docstring with `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6375\r\n* Align `AsyncGRPOTrainer` clip-ratio comment with `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6376\r\n* Fix formatting of error message for `loss_type` by @qgallouedec in https://github.com/huggingface/trl/pull/6373\r\n* Fix misleading name of `BEMACallback` `bias_power=0.0` test by @albertvillanova in https://github.com/huggingface/trl/pull/6464\r\n\r\n## CI\r\n\r\n* Destroy vLLM colocate mode process group after each CI test by @albertvillanova in https://github.com/huggingface/trl/pull/6353\r\n* Test KTO training on iterable datasets with the KL loss by @albertvillanova in https://github.com/huggingface/trl/pull/6338\r\n* Test all supported `eval_dataset` input types in core trainer `init` / `evaluate()` tests by @albertvillanova in https://github.com/huggingface/trl/pull/6336 and https://github.com/huggingface/trl/pull/6371\r\n* Add `eval_dataset` init tests for experimental GRPO / Online-DPO / SFT & distillation / preference / BCO+CPO+ORPO+PRM trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6395, https://github.com/huggingface/trl/pull/6396, https://github.com/huggingface/trl/pull/6397, https://github.com/huggingface/trl/pull/6398 and https://github.com/huggingface/trl/pull/6399\r\n* Silence experimental `environment_factory` warning in GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6445\r\n* Silence PEFT `ensure_weight_tying` warning on newer PEFT in GRPO tests / DPO+KTO liger tests by @albertvillanova in https://github.com/huggingface/trl/pull/6446 and https://github.com/huggingface/trl/pull/6463\r\n* Ignore benign `bitsandbytes` `_check_is_size` `FutureWarning` in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6448\r\n* Ignore benign `IndexPutBackward0` warning from Liger SAPO loss by @albertvillanova in https://github.com/huggingface/trl/pull/6436\r\n* Filter reworded `torch.jit.script_method` warning on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6465\r\n* Filter triton `AnnAssign` `DeprecationWarning` on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6467\r\n* Fix `create_reference_model` deprecation warning by @albertvillanova in https://github.com/huggingface/trl/pull/6418\r\n* Fix uncaptured cross-architecture VLM warning in GOLD trainer CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/6419\r\n* Re-record the `dpo` invariant reference after #6321 by @qgallouedec in https://github.com/huggingface/trl/pull/6421\r\n* Remove redundant `torch.tensor` re-wrapping of batch labels in KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6431\r\n* Bump the actions group across 1 directory with 10 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6442\r\n\r\n## New Contributors\r\n* @Functionhx made their first contribution in https://github.com/huggingface/trl/pull/6348\r\n* @sohumt123 made their first contribution in https://github.com/huggingface/trl/pull/6433\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6347\r\n* Fix FileNotFoundError in DPO/KTO precompute_ref_log_probs by @Functionhx in https://github.com/huggingface/trl/pull/6348\r\n* Add `log_multimodal` param to GRPOConfig and RLOOConfig to control image logging by @apardyl in https://github.com/huggingface/trl/pull/5408\r\n* Set explicit neutral top_p in GKD on-policy generation by @sergiopaniego in https://github.com/huggingface/trl/pull/6337\r\n* Fix GKD vocab_size check for multimodal model configs by @sergiopaniego in https://github.com/huggingface/trl/pull/6340\r\n* Fix OpenEnv docker-image example bootstrap by @sergiopaniego in https://github.com/huggingface/trl/pull/6329\r\n* Align log_multimodal in the DPPO and GRPO-with-replay-buffer experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6352\r\n* Fix QLoRA + FSDP1 mixed-dtype crash in prepare_peft_model by @sergiopaniego in https://github.com/huggingface/trl/pull/6343\r\n* Destroy vLLM colocate mode process group after each CI test by @albertvillanova in https://github.com/huggingface/trl/pull/6353\r\n* Fix error type for IterableDataset in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6324\r\n* Fix KTO evaluate crash on streaming datasets with unpaired-preference data by @albertvillanova in https://github.com/huggingface/trl/pull/6325\r\n* Support DatasetDict and IterableDatasetDict as eval_dataset in evaluate by @albertvillanova in https://github.com/huggingface/trl/pull/6326\r\n* Simplify KTO KL completion construction into a single batched map by @albertvillanova in https://github.com/huggingface/trl/pull/6335\r\n* implement message level rollout with linear trajectories by @AmineDiro in https://github.com/huggingface/trl/pull/6250\r\n* Add security policy to the repo by @albertvillanova in https://github.com/huggingface/trl/pull/6362\r\n* Removed `generate_rollout_completions` by @sergiopaniego in https://github.com/huggingface/trl/pull/5870\r\n* Fix crash when reward function is a `functools.partial` or callable instance by @qgallouedec in https://github.com/huggingface/trl/pull/6313\r\n* Drop fully-masked examples after truncation in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/6320\r\n* Fix `cast_lm_head_to_fp32` under FSDP2 / DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/6330\r\n* Add support for vLLM 0.24.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6355\r\n* Fix DPO/KTO `use_liger_kernel` under DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/6372\r\n* Test KTO training on iterable datasets with the KL loss by @albertvillanova in https://github.com/huggingface/trl/pull/6338\r\n* Avoid in-place mutation of input prompts in GRPO environments loop by @albertvillanova in https://github.com/huggingface/trl/pull/6364\r\n* Avoid in-place mutation of input prompts in DPPO environments loop by @albertvillanova in https://github.com/huggingface/trl/pull/6366\r\n* Fix DPO/KTO evaluate crash with a dict eval_dataset and precompute_ref_log_probs by @albertvillanova in https://github.com/huggingface/trl/pull/6370\r\n* Test all supported eval_dataset input types in core trainer init tests by @albertvillanova in https://github.com/huggingface/trl/pull/6336\r\n* Test all supported eval_dataset input types in core trainer evaluate() tests by @albertvillanova in https://github.com/huggingface/trl/pull/6371\r\n* Drop examples with no completion left after truncation in DPO and KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6321\r\n* Fix EOS being masked when pad_token_id equals eos_token_id in GKD and GOLD by @roycho96 in https://github.com/huggingface/trl/pull/6357\r\n* Truncate GOLD on-policy prompts before generation, keeping the prompt end by @roycho96 in https://github.com/huggingface/trl/pull/6359\r\n* Fix missing eos_token_id in non-vLLM on-policy GOLD test stubs by @kashif in https://github.com/huggingface/trl/pull/6394\r\n* add iw-opd distillation by @kashif in https://github.com/huggingface/trl/pull/6191\r\n* Force `use_reentrant=True` for PEFT + ZeRO-3 + gradient checkpointing in all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6356\r\n* Fix GeometricMixtureWrapper.generate() crash after transformers' MoE decode optimization change by @albertvillanova in https://github.com/huggingface/trl/pull/6413\r\n* Add eval_dataset init tests for experimental GRPO-family trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6395\r\n* Add eval_dataset init tests for experimental Online-DPO-family trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6396\r\n* Add eval_dataset init tests for experimental SFT and distillation trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6397\r\n* Add eval_dataset init tests for experimental preference trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6398\r\n* Add eval_dataset init tests for experimental BCO, CPO, ORPO and PRM trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6399\r\n* Fix corrupted reasoning blocks in Qwen3-family training chat templates by @qgallouedec in https://github.com/huggingface/trl/pull/6363\r\n* Fix uncaptured cross-architecture VLM warning in GOLD trainer CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/6419\r\n* Fix create_reference_model deprecation warning by @albertvillanova in https://github.com/huggingface/trl/pull/6418\r\n* Add `GMPOTrainer` to the telemetry allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/6410\r\n* Async grpo vllm client by @AmineDiro in https://github.com/huggingface/trl/pull/6269\r\n* Re-record the `dpo` invariant reference after #6321 by @qgallouedec in https://github.com/huggingface/trl/pull/6421\r\n* Add a tip to the Paper Index on reading papers with the hf CLI by @davanstrien in https://github.com/huggingface/trl/pull/6422\r\n* [GRPO] Treat explicit max_tool_calling_iterations=0 as zero, not unlimited  by @Strongich in https://github.com/huggingface/trl/pull/6423\r\n* Align KTO with DPO: Drop `hasattr` guards on `lm_head.bias` in the Liger path by @qgallouedec in https://github.com/huggingface/trl/pull/6425\r\n* Align KTO with DPO: Fix `logits/chosen` and `logits/rejected` metrics by @qgallouedec in https://github.com/huggingface/trl/pull/6424\r\n* Support iterable datasets in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6351\r\n* Align `AsyncGRPOTrainer` clip-ratio comment with `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6376\r\n* Ignore benign IndexPutBackward0 warning from Liger SAPO loss by @albertvillanova in https://github.com/huggingface/trl/pull/6436\r\n* Align `AsyncGRPOConfig.max_completion_length` description with `GRPOConfig` by @qgallouedec in https://github.com/huggingface/trl/pull/6374\r\n* Fix formatting of error message for loss_type by @qgallouedec in https://github.com/huggingface/trl/pull/6373\r\n* Align `AsyncGRPOTrainer.processing_class` docstring with `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6375\r\n* Remove redundant torch.tensor re-wrapping of batch labels in KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6431\r\n* Document max_steps requirement for iterable datasets in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6432\r\n* Fix DPO/KTO double-preprocessing eval_dataset dict passed to evaluate() by @albertvillanova in https://github.com/huggingface/trl/pull/6434\r\n* Bump the actions group across 1 directory with 10 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6442\r\n* Fix GRPO truncated-completion loss normalization by @Functionhx in https://github.com/huggingface/trl/pull/6439\r\n* Environment-owned dataset: make `train_dataset` optional by @qgallouedec in https://github.com/huggingface/trl/pull/6349\r\n* Align `AsyncGRPOTrainer` args handling with stable trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6377\r\n* Fix biased cross-rank aggregation of token-weighted metrics (GRPO, RLOO) by @qgallouedec in https://github.com/huggingface/trl/pull/6380\r\n* Silence experimental environment_factory warning in GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6445\r\n* Ignore benign bitsandbytes _check_is_size FutureWarning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6448\r\n* Silence PEFT ensure_weight_tying warning on newer PEFT in GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6446\r\n* Fix BEMACallback bias_power docstring, which described the opposite of the actual behavior by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6390\r\n* AsyncGRPO fork-independent epoch counting by @AmineDiro in https://github.com/huggingface/trl/pull/6385\r\n* Add support for vLLM 0.25.0 and 0.25.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6406\r\n* Drop vLLM 0.16 support by @qgallouedec in https://github.com/huggingface/trl/pull/6404\r\n* Remove the experimental DPPO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6402\r\n* Reject `auto_find_batch_size` in GRPO and RLOO by @qgallouedec in https://github.com/huggingface/trl/pull/6411\r\n* Fix `precompute_ref_log_probs=True` with DeepSpeed in DPO and KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6403\r\n* [DistillationTrainer refactor] Pin the distillation objective before refactoring by @qgallouedec in https://github.com/huggingface/trl/pull/6450\r\n* [DistillationTrainer refactor] Pin on-policy and off-policy training end to end by @qgallouedec in https://github.com/huggingface/trl/pull/6451\r\n* [DistillationTrainer refactor] Document the num_items_in_batch denominator bug by @qgallouedec in https://github.com/huggingface/trl/pull/6452\r\n* Fix `apply_chat_template` dropping leading tokens from `chosen` when the prompt common prefix shrinks against `rejected` by @sohumt123 in https://github.com/huggingface/trl/pull/6433\r\n* [DistillationTrainer refactor] Move IW-OPD to a dedicated experimental trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6453\r\n* [DistillationTrainer refactor] Extract the teacher-server path into ServerDistillationTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6454\r\n* [DistillationTrainer refactor] Remove the local sparse top-1 loss path from the base trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6455\r\n* [DistillationTrainer refactor] Remove top-k support from the base generalized JSD loss by @qgallouedec in https://github.com/huggingface/trl/pull/6456\r\n* [DistillationTrainer refactor] Validate the teacher by vocab size instead of tokenizer identity by @qgallouedec in https://github.com/huggingface/trl/pull/6457\r\n* [DistillationTrainer refactor] Deprecate `lmbda` on/off-policy mixing by @qgallouedec in https://github.com/huggingface/trl/pull/6458\r\n* [DistillationTrainer refactor] Point the GKD paper reproduction at `GKDTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6459\r\n* [DistillationTrainer refactor] Remove `lmbda` and the off-policy training branch by @qgallouedec in https://github.com/huggingface/trl/pull/6460\r\n* Raise when DPO/KTO cannot precompute correct reference log-probs at eval time by @albertvillanova in https://github.com/huggingface/trl/pull/6443\r\n* Silence PEFT ensure_weight_tying warning in DPO and KTO liger tests by @albertvillanova in https://github.com/huggingface/trl/pull/6463\r\n* Fix misleading name of BEMACallback bias_power=0.0 test by @albertvillanova in https://github.com/huggingface/trl/pull/6464\r\n* Filter reworded torch.jit.script_method warning on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6465\r\n* Filter triton AnnAssign DeprecationWarning on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6467\r\n* [DistillationTrainer refactor] Accept a `prompt` column alongside `messages` by @qgallouedec in https://github.com/huggingface/trl/pull/6461\r\n* [DistillationTrainer refactor] Deprecate `messages`-format datasets by @qgallouedec in https://github.com/huggingface/trl/pull/6474\r\n* [DistillationTrainer refactor] Fix `num_items_in_batch` to count generated completion tokens ⚠️ changes loss values by @qgallouedec in https://github.com/huggingface/trl/pull/6478\r\n* Release: v1.9 by @qgallouedec in https://github.com/huggingface/trl/pull/6495\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.8.0...v1.9.0\r\n","publishedAt":"2026-07-21T20:36:12.000Z","fetchedAt":"2026-07-21T21:07:15.942Z","url":"https://github.com/huggingface/trl/releases/tag/v1.9.0","media":[],"coverageCount":0},{"id":"rel_YYu6ID45bA0Ojq05mBYK8","version":"v1.8.0","type":"feature","title":"v1.8.0","summary":"KTO trainer graduates from experimental to the top-level trl package with the same API as DPO/GRPO/SFT, and the experimental import path still works with a FutureWarning. Environment-owned rewards let agentic RL environments define their own reward via a reserved get_reward() method, and multi-environment support allows a single training run to handle multiple environments with environment-specific tool schemas. GRPO now supports both static and adaptive entropy regularization to encourage exploration and prevent policy collapse.","titleGenerated":"TRL v1.8.0 graduates KTO trainer to stable, adds environment-owned rewards and entropy regularization","titleShort":"KTO now stable; environment rewards and GRPO entropy regularization added","breaking":"none","importance":4,"content":"## Features\r\n\r\n### 🎓 KTO is now a stable trainer\r\n\r\nAfter many cycles of `KTOTrainer` ↔ `DPOTrainer` alignment work, KTO graduates from `trl.experimental.kto` to the top-level `trl` package. Same API as DPO/GRPO/SFT — imports move from experimental, tests move to the main test tree, docs no longer flag it as experimental. The experimental path still works and emits a `FutureWarning` (removal in v2.0.0).\r\n\r\n```python\r\n# Before\r\nfrom trl.experimental.kto import KTOConfig, KTOTrainer\r\n\r\n# Now\r\nfrom trl import KTOConfig, KTOTrainer\r\n```\r\n\r\nPer our telemetry, KTO is the 4th most used trainer in TRL — this graduation was overdue.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6175, https://github.com/huggingface/trl/pull/6287 and https://github.com/huggingface/trl/pull/6345\r\n\r\n### Environment-owned rewards & multi-environment support\r\n\r\nThree interrelated changes make agentic RL training with environments substantially more ergonomic.\r\n\r\n**Environment-owned reward.** If your `environment_factory` env defines a reserved `get_reward()` method (no args → `float`), it's called once per completed rollout and treated as a reward source. `reward_funcs` becomes optional — no more leaking env state back out to trainer-owned reward funcs.\r\n\r\n```python\r\nclass WordleEnv:\r\n    def reset(self, **kwargs):\r\n        self._target = sample(words); self._solved = False\r\n\r\n    def get_reward(self) -> float:       # optional, reserved (not a tool)\r\n        return 1.0 if self._solved else 0.0\r\n\r\n    def guess(self, word: str) -> str:   # exposed as a tool\r\n        self._solved = word == self._target; ...\r\n\r\ntrainer = GRPOTrainer(\r\n    model=model,\r\n    train_dataset=dataset,\r\n    environment_factory=WordleEnv,       # no reward_funcs needed\r\n)\r\n```\r\n\r\n**Multi-environment support.** `environment_factory` now accepts `dict[str, factory]` in addition to a single callable. Each dataset row selects its environment via an `environment` column, and **only that env's tools are exposed in that row's prompt** — so a coding task and a game can train together in one run without leaking each other's tool schemas. Single-callable usage is unchanged.\r\n\r\nSame wiring lands in GRPO, AsyncGRPO, DPPO, and GRPO-with-replay-buffer.\r\n\r\nEnv-owned reward by @qgallouedec in https://github.com/huggingface/trl/pull/6238; multi-env in https://github.com/huggingface/trl/pull/6001 and https://github.com/huggingface/trl/pull/6002\r\n\r\n### Entropy regularization for GRPO\r\n\r\n`GRPOConfig` now supports both static and adaptive entropy regularization ([Skywork-OR1](https://huggingface.co/papers/2505.22617)). The bonus encourages exploration and helps prevent premature policy collapse.\r\n\r\n```python\r\nGRPOConfig(\r\n    entropy_coef=0.01,          # static\r\n    # or:\r\n    use_adaptive_entropy=True,  # adjust to target entropy\r\n    entropy_target=1.5,\r\n    entropy_coef_delta=0.01,\r\n    entropy_coef_min=0.0,\r\n    entropy_coef_max=0.1,\r\n)\r\n```\r\n\r\nAdaptive mode adjusts the coefficient once per optimizer step from window-aggregated entropy (gradient accumulation-aware) and persists `entropy_coef` in the checkpoint for resume. Not compatible with the Liger kernel.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6140\r\n\r\n### `quantization_config` trainer argument (streamlined QLoRA)\r\n\r\nQLoRA no longer requires reaching into `model_init_kwargs` or pre-loading the model manually.\r\n\r\n```python\r\nSFTTrainer(\r\n    model=\"meta-llama/Llama-2-7b-hf\",\r\n    quantization_config=BitsAndBytesConfig(load_in_4bit=True),\r\n    peft_config=LoraConfig(),\r\n    train_dataset=dataset,\r\n)\r\n```\r\n\r\nAdded to `SFTTrainer`, `DPOTrainer`, `GRPOTrainer`, `RLOOTrainer`, `RewardTrainer`, and `KTOTrainer`. Sits next to `peft_config` (the other non-serializable QLoRA ingredient), flows into `from_pretrained`, and raises if also set in `args.model_init_kwargs`. Also drops the redundant `get_kbit_device_map()` line — QLoRA trains identically without it across all tested configurations.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6157 and https://github.com/huggingface/trl/pull/6276\r\n\r\n### MoE aux loss extends to DPO and KTO\r\n\r\nv1.7 added the router load-balancing auxiliary loss to `GRPOTrainer` / `RLOOTrainer` / `AsyncGRPOTrainer`. It's now on `DPOTrainer` and `KTOTrainer` too, so post-training MoE models with preference data keeps experts balanced.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6208 and https://github.com/huggingface/trl/pull/6275\r\n\r\n### Neuron-friendly `chunked_nll` via static-shape token packing\r\n\r\nThe chunked NLL path had data-dependent indexing that broke XLA/Neuron compilation. This PR reworks it to pack valid tokens to the front via a stable `argsort` on the validity mask, iterate over `ceil(n_valid / chunk_size) * chunk_size` whole chunks (a tensor, no Python-int sync), and use `ignore_index=-100` inside each chunk. GPU behavior is unchanged; Neuron now works too — same tiny memory footprint (~1.33 GiB vs the naive 5.94 GiB alternative) with no per-mask recompilation.\r\n\r\nby @michaelbenayoun in https://github.com/huggingface/trl/pull/6314\r\n\r\n### Packing-aware dynamic batching in AsyncGRPO\r\n\r\nAsyncGRPO micro-batching becomes packing-aware and token-bounded on top of the padding-free path from v1.7:\r\n\r\n- **Σ Lᵢ² row balancing** — a greedy longest-first assignment gives every DP row the same Σ Lᵢ² instead of the same token count (attention is O(L²), FFN is O(L), so equal token counts don't equalize wall-clock). **Cross-rank stragglers vanish; +19% MFU at 4B in the benchmark.**\r\n- **Token-budget packing** (opt-in) — cap each row at `token_budget` tokens with a variable sample count, decoupling peak memory from `per_device_train_batch_size`. Useful in memory-bound regimes (no gradient checkpointing, long context, very large models).\r\n\r\nBoth ride HF Trainer's existing gradient accumulation — no training-loop surgery, FSDP/EP collectives stay in lockstep.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6092\r\n\r\n### VLM support in `GOLDTrainer`\r\n\r\n`GOLDTrainer` now supports vision-language models end-to-end.\r\n\r\nby @Strongich in https://github.com/huggingface/trl/pull/5969\r\n\r\n### Support tool calling in KTO\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6259\r\n\r\n### Support PEFT + Liger in DPO and KTO\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6159 and https://github.com/huggingface/trl/pull/6277\r\n\r\n### Per-dataset `fraction` in dataset mixtures\r\n\r\nYou can now weight a `DatasetMixtureConfig` by fraction instead of only by explicit counts.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6199\r\n\r\n### SFT: truncate during dataset preparation\r\n\r\nContinuing the label-refactor from v1.7 (#6037), truncation moves out of the collator and into dataset preparation. `SFTTrainer` and `DPOTrainer` now truncate consistently at the same phase.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6155\r\n\r\n### Simplify tokenization [1-5/N]\r\n\r\nA big refactor pass merging DPO / SFT / Reward / KTO tokenization into one shared implementation:\r\n\r\n* Remove redundant `is_vlm` parameter — https://github.com/huggingface/trl/pull/6298\r\n* Make `_tokenize` a module-level function — https://github.com/huggingface/trl/pull/6301\r\n* Factor `_tokenize` into a single shared function — https://github.com/huggingface/trl/pull/6302\r\n* Bundle `apply_chat_template` kwargs — https://github.com/huggingface/trl/pull/6305\r\n* Pass `chat_template` as `apply_chat_template_kwargs` — https://github.com/huggingface/trl/pull/6315\r\n\r\nAll by @albertvillanova. Related: align collators across DPO / SFT / Reward / KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6178.\r\n\r\n### Other\r\n\r\n* Support `DatasetDict` and `IterableDatasetDict` as `eval_dataset` in trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6322\r\n* Auto-load `processing_class` in CPO/ORPO trainers when omitted by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6087\r\n* Add `trust_remote_code` to `GRPOConfig` by @muupan in https://github.com/huggingface/trl/pull/4186\r\n* Integrate the new response parsing API by @qgallouedec in https://github.com/huggingface/trl/pull/5791\r\n* Raise a clear error when the ChatML collator truncates the whole prompt by @qgallouedec in https://github.com/huggingface/trl/pull/6310\r\n* Raise a clear error when GKD student and teacher vocab sizes differ by @sergiopaniego in https://github.com/huggingface/trl/pull/6252\r\n* Add prompt-learning guard for PEFT with Liger in GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6186\r\n* `[async GRPO]` Per-generation `reset()` observation by @qgallouedec in https://github.com/huggingface/trl/pull/6072\r\n* Default AsyncGRPO `token_budget` to vLLM `max_model_len` by @AmineDiro in https://github.com/huggingface/trl/pull/6218\r\n* Drop vLLM 0.14 support by @qgallouedec in https://github.com/huggingface/trl/pull/6209\r\n* Drop vLLM 0.15 support by @qgallouedec in https://github.com/huggingface/trl/pull/6239\r\n* Log `entropy` and `num_tokens` metrics in KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6257 and https://github.com/huggingface/trl/pull/6256\r\n* Standardize `TrainerCallback` import to the public top-level path by @qgallouedec in https://github.com/huggingface/trl/pull/6249\r\n* Remove redundant `get_kbit_device_map()` by @qgallouedec in https://github.com/huggingface/trl/pull/6158\r\n* Factorize `get_dataset_column_names` from core and experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6272 and https://github.com/huggingface/trl/pull/6273\r\n\r\n## Deprecations and removals\r\n\r\n* **Remove `GFPOTrainer`** by @qgallouedec in https://github.com/huggingface/trl/pull/6309 — no known usage, and its behavior can be reproduced by adjusting `reward_funcs`.\r\n* **Remove `PAPOTrainer`** by @qgallouedec in https://github.com/huggingface/trl/pull/6235\r\n* **Remove post-training-toolkit integration** by @qgallouedec in https://github.com/huggingface/trl/pull/6308\r\n* **Remove `sft_video_llm.py` script** by @qgallouedec in https://github.com/huggingface/trl/pull/6193\r\n\r\n## Fixes\r\n\r\n* **Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware** by @albertvillanova in https://github.com/huggingface/trl/pull/6139 and https://github.com/huggingface/trl/pull/6187\r\n* **Align CPO / ORPO / BCO / Distillation / TPO with DPO: fix ZeRO-3 + PEFT mixed-dtype error** by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6192\r\n* **Fix `chunked_nll` patch hiding VLM kwargs from `generate`** by @Strongich in https://github.com/huggingface/trl/pull/6156\r\n* **Fix vLLM server-mode generation in `OnlineDPOTrainer`** by @qgallouedec in https://github.com/huggingface/trl/pull/6228\r\n* **Fix activation offload storage dedupe reuse** by @winglian in https://github.com/huggingface/trl/pull/6241\r\n* **Update merged probability computation to follow Bayes' rule** by @vman049 in https://github.com/huggingface/trl/pull/5905\r\n* Fix dataset fingerprinting in DPO/SFT tokenization by @qgallouedec in https://github.com/huggingface/trl/pull/6206\r\n* Fix data types for PPO models by @albertvillanova in https://github.com/huggingface/trl/pull/6202\r\n* Fix `precompute_ref_log_probs` guard for iterable datasets nested in a dict by @albertvillanova in https://github.com/huggingface/trl/pull/6307\r\n* Fix teacher quantization kwargs and guard eval callback in GKD example by @sergiopaniego in https://github.com/huggingface/trl/pull/6251\r\n* Raise on `quantization_config` + already-instantiated model in `DPOTrainer` / `KTOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6312\r\n* Fix incorrect examples in distillation docs by @sergiopaniego in https://github.com/huggingface/trl/pull/6334\r\n* Fix `KTOTrainer._prepare_dataset` type hint by @qgallouedec in https://github.com/huggingface/trl/pull/6207\r\n* Fix PEFT `ensure_weight_tying` warning in liger + PEFT GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6188\r\n* Fix spurious liger token_accuracy CI warnings in SFT tests by @albertvillanova in https://github.com/huggingface/trl/pull/6189\r\n* Fix `add_response_schema` tests for the new `parse_response` prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236\r\n* Fix tiny DeepSeek-V3 configs for MLA: `num_key_value_heads = num_attention_heads` by @albertvillanova in https://github.com/huggingface/trl/pull/6268\r\n* Raise `ValueError` instead of `NotImplementedError` for Liger DPO with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6225\r\n\r\n## Documentation and Examples\r\n\r\n* Docs: fix Jobs requirement — positive credit balance, not a Pro/Team/Enterprise plan by @davanstrien in https://github.com/huggingface/trl/pull/6306\r\n* Docs: clarify dtype defaults between transformers v5 and TRL by @casinca in https://github.com/huggingface/trl/pull/5457\r\n* Document that `max_steps` is required for iterable train datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6333\r\n* Align KTO doc with DPO and fix Logged-metrics wording by @qgallouedec in https://github.com/huggingface/trl/pull/6258\r\n* Align `epsilon` help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014 (v1.7 window)\r\n* Fix KTO `max_length` docstring (truncation direction) by @qgallouedec in https://github.com/huggingface/trl/pull/6254\r\n* Fix KTO `compute_metrics` type hint (`EvalLoopOutput` → `EvalPrediction`) by @qgallouedec in https://github.com/huggingface/trl/pull/6253\r\n\r\n## CI\r\n\r\n* Liger teardown in experimental tests by @cmpatino in https://github.com/huggingface/trl/pull/6203\r\n* Sort conditional imports by @qgallouedec in https://github.com/huggingface/trl/pull/6180\r\n* Align format of versions in CI workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6223\r\n* Update CI `huggingface/doc-builder` by @albertvillanova in https://github.com/huggingface/trl/pull/6304\r\n* Fix Upload PR Documentation: bump `doc-builder` pin by @mishig25 in https://github.com/huggingface/trl/pull/6317\r\n* Fix / set explicit owner in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6200 and https://github.com/huggingface/trl/pull/6201\r\n* Fix CI `test_chatml_collator_truncates_keeping_completion_end` by @albertvillanova in https://github.com/huggingface/trl/pull/6318\r\n* Test DPO / KTO init fails with `compute_metrics` and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6231 and https://github.com/huggingface/trl/pull/6232\r\n* Add `TestKTOTrainerSlow` with `test_train_vlm_gemma_3n` by @albertvillanova in https://github.com/huggingface/trl/pull/6162\r\n* Test `DataCollatorForChatML` raises when completion fills window by @albertvillanova in https://github.com/huggingface/trl/pull/6319\r\n* Improve consistency of Liger Kernel initialization in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6242\r\n* Align Liger loss naming across core and experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6243, https://github.com/huggingface/trl/pull/6245 and https://github.com/huggingface/trl/pull/6244\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6214, https://github.com/huggingface/trl/pull/6285\r\n\r\n## New Contributors\r\n* @vman049 made their first contribution in https://github.com/huggingface/trl/pull/5905\r\n* @Strongich made their first contribution in https://github.com/huggingface/trl/pull/5969\r\n* @michaelbenayoun made their first contribution in https://github.com/huggingface/trl/pull/6314\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6185\r\n* Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware by @albertvillanova in https://github.com/huggingface/trl/pull/6139\r\n* Remove RUNNING_NAME from KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6179\r\n* Align CPO/ORPO/BCO/Distillation/TPO with DPO: fix ZeRO-3 + PEFT mixed-dtype error by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6192\r\n* Align KTO collator keys with the DPO convention by @qgallouedec in https://github.com/huggingface/trl/pull/6182\r\n* Move `_get_kl_completion_ids` into `_get_kl_dataset` by @qgallouedec in https://github.com/huggingface/trl/pull/6181\r\n* Align experimental KTOTrainer docstring and signature with DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6183\r\n* Add packing-aware dynamic batching to AsyncGRPO by @AmineDiro in https://github.com/huggingface/trl/pull/6092\r\n* Liger Teardown in Experimental Tests by @cmpatino in https://github.com/huggingface/trl/pull/6203\r\n* Sort conditional imports by @qgallouedec in https://github.com/huggingface/trl/pull/6180\r\n* Align TPO with DPO: Add evaluate() override by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6195\r\n* Auto-load `processing_class` in CPO/ORPO trainers when omitted by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6087\r\n* Fix dataset fingerprinting in DPO/SFT tokenization by @qgallouedec in https://github.com/huggingface/trl/pull/6206\r\n* Pass GPU device_ids to barrier fix in GRPO + vLLM colocate + PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6187\r\n* Fix PEFT ensure_weight_tying warning in liger + PEFT GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6188\r\n* Fix spurious liger token_accuracy CI warnings in SFT tests by @albertvillanova in https://github.com/huggingface/trl/pull/6189\r\n* Fix deprecation in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6200\r\n* Set explicit owner in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6201\r\n* Fix data types for PPO models by @albertvillanova in https://github.com/huggingface/trl/pull/6202\r\n* Fix: update merged probability computation to follow Bayes' rule by @vman049 in https://github.com/huggingface/trl/pull/5905\r\n* Bump the actions group across 1 directory with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6214\r\n* Remove sft_video_llm.py script by @qgallouedec in https://github.com/huggingface/trl/pull/6193\r\n* Drop vLLM 0.14 support by @qgallouedec in https://github.com/huggingface/trl/pull/6209\r\n* Integrate the new response parsing API by @qgallouedec in https://github.com/huggingface/trl/pull/5791\r\n* [docs] Clarify dtype defaults between trf v5 and TRL by @casinca in https://github.com/huggingface/trl/pull/5457\r\n* Align format of versions in CI workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6223\r\n* Align KTO with DPO: Align validation of liger and compute_metrics by @albertvillanova in https://github.com/huggingface/trl/pull/6224\r\n* Raise ValueError instead of NotImplementedError for Liger DPO with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6225\r\n* Support multiple environments [1/2]: Pool and build environment tool dicts at batch time by @qgallouedec in https://github.com/huggingface/trl/pull/6001\r\n* Test DPO init fails with compute_metrics and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6231\r\n* Align KTO with DPO: Test init fails with compute_metrics and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6232\r\n* [async GRPO] Per-generation reset() observation by @qgallouedec in https://github.com/huggingface/trl/pull/6072\r\n* Fix `add_response_schema` tests for the new `parse_response` prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236\r\n* Align KTO with DPO: Add TestKTOTrainerSlow with test_train_vlm_gemma_3n by @albertvillanova in https://github.com/huggingface/trl/pull/6162\r\n* Support PEFT with Liger in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6159\r\n* Drop `policy_` prefix from KTO loss-internal logps/logits by @qgallouedec in https://github.com/huggingface/trl/pull/6204\r\n* Fix KTOTrainer._prepare_dataset type hint by @qgallouedec in https://github.com/huggingface/trl/pull/6207\r\n* Improve consistency of Liger Kernel initialization in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6242\r\n* Align Liger loss naming across core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6243\r\n* Align Liger loss naming across experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6245\r\n* Align KTO with DPO: Align _tokenize by @albertvillanova in https://github.com/huggingface/trl/pull/6212\r\n* Align KTO with DPO: Align error for Liger with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6226\r\n* Remove redundant `get_kbit_device_map()` by @qgallouedec in https://github.com/huggingface/trl/pull/6158\r\n* Fix vLLM server-mode generation in `OnlineDPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6228\r\n* Use trl's guarded `is_liger_kernel_available` in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6247\r\n* Default AsyncGRPO token_budget to vLLM max_model_len by @AmineDiro in https://github.com/huggingface/trl/pull/6218\r\n* Fix tiny DeepSeek-V3 configs for MLA: num_key_value_heads =num_attention_heads by @albertvillanova in https://github.com/huggingface/trl/pull/6268\r\n* Align KTO with DPO: Align Liger loss naming by @albertvillanova in https://github.com/huggingface/trl/pull/6244\r\n* Raise a clear error when GKD student and teacher vocab sizes differ by @sergiopaniego in https://github.com/huggingface/trl/pull/6252\r\n* SFT: Truncate during dataset preparation, not collation by @qgallouedec in https://github.com/huggingface/trl/pull/6155\r\n* Add prompt-learning guard for PEFT with Liger in GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6186\r\n* Add MoE load-balancing auxiliary loss to `DPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6208\r\n* Add `quantization_config` trainer argument (streamline QLoRA) by @qgallouedec in https://github.com/huggingface/trl/pull/6157\r\n* Align KTO with DPO: Support tool calling by @qgallouedec in https://github.com/huggingface/trl/pull/6259\r\n* Align KTO with DPO: Align error for Liger with precompute_ref_logps by @albertvillanova in https://github.com/huggingface/trl/pull/6270\r\n* Remove unused get_dataset_column_names from DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6271\r\n* Factorize get_dataset_column_names from core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6272\r\n* Factorize get_dataset_column_names from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6273\r\n* Align KTO with DPO: Log `entropy` metric by @qgallouedec in https://github.com/huggingface/trl/pull/6257\r\n* Align KTO doc with DPO and fix Logged metrics wording by @qgallouedec in https://github.com/huggingface/trl/pull/6258\r\n* Align KTO with DPO: Validate Liger kernel configuration before trainer initialization by @albertvillanova in https://github.com/huggingface/trl/pull/6227\r\n* Align KTO with DPO: Remove stray misplaced comment in KTO `_get_train_sampler` by @qgallouedec in https://github.com/huggingface/trl/pull/6255\r\n* Align KTO with DPO: Fix KTO `max_length` docstring (truncation direction) by @qgallouedec in https://github.com/huggingface/trl/pull/6254\r\n* Align KTO with DPO: Fix KTO `compute_metrics` type hint (`EvalLoopOutput` → `EvalPrediction`) by @qgallouedec in https://github.com/huggingface/trl/pull/6253\r\n* Align KTO with DPO: Align models import by @qgallouedec in https://github.com/huggingface/trl/pull/6248\r\n* Align KTO with DPO: Align `F` import with the rest of the repo by @qgallouedec in https://github.com/huggingface/trl/pull/6246\r\n* Fix teacher quantization kwargs and guard eval callback in GKD example by @sergiopaniego in https://github.com/huggingface/trl/pull/6251\r\n* Align experimental TPO _tokenize with core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6213\r\n* Align KTO with DPO: Support PEFT with Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6277\r\n* Align KTO with DPO: clarify dtype default in `model` docstring by @qgallouedec in https://github.com/huggingface/trl/pull/6278\r\n* Align KTO with DPO: Log `num_tokens` metric by @qgallouedec in https://github.com/huggingface/trl/pull/6256\r\n* Standardize `TrainerCallback` import to the public top-level path by @qgallouedec in https://github.com/huggingface/trl/pull/6249\r\n* Fix activation offload storage dedupe reuse by @winglian in https://github.com/huggingface/trl/pull/6241\r\n* Align KTO with DPO: MoE load-balancing auxiliary loss by @qgallouedec in https://github.com/huggingface/trl/pull/6275\r\n* Align data collators across DPO / SFT / Reward / KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6178\r\n* Drop vLLM 0.15 support by @qgallouedec in https://github.com/huggingface/trl/pull/6239\r\n* Add entropy regularization to GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6140\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6285\r\n* Remove KTO shims from stable by @albertvillanova in https://github.com/huggingface/trl/pull/6287\r\n* Promote KTO to stable API by @albertvillanova in https://github.com/huggingface/trl/pull/6175\r\n* [GOLD] VLM support for GOLDTrainer by @Strongich in https://github.com/huggingface/trl/pull/5969\r\n* Docs: fix Jobs requirement — positive credit balance, not a Pro/Team/Enterprise plan by @davanstrien in https://github.com/huggingface/trl/pull/6306\r\n* Simplify tokenization [1/N]: Remove redundant is_vlm parameter by @albertvillanova in https://github.com/huggingface/trl/pull/6298\r\n* Update CI huggingface/doc-builder by @albertvillanova in https://github.com/huggingface/trl/pull/6304\r\n* Simplify tokenization [2/N]: Make _tokenize a module-level function by @albertvillanova in https://github.com/huggingface/trl/pull/6301\r\n* Raise clear error when ChatML collator truncates the whole prompt by @qgallouedec in https://github.com/huggingface/trl/pull/6310\r\n* Remove the PAPO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6235\r\n* Add trust_remote_code to GRPOConfig by @muupan in https://github.com/huggingface/trl/pull/4186\r\n* Simplify tokenization [3/N]: Factor _tokenize into a single shared function by @albertvillanova in https://github.com/huggingface/trl/pull/6302\r\n* Simplify tokenization [4/N]: Bundle apply_chat_template kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/6305\r\n* Fix precompute_ref_log_probs guard for iterable datasets nested in a dict by @albertvillanova in https://github.com/huggingface/trl/pull/6307\r\n* Fix Upload PR Documentation: bump doc-builder pin by @mishig25 in https://github.com/huggingface/trl/pull/6317\r\n* Removes GFPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6309\r\n* Fix CI test_chatml_collator_truncates_keeping_completion_end: no prompt tokens left after truncation by @albertvillanova in https://github.com/huggingface/trl/pull/6318\r\n* Add per-dataset `fraction` to dataset mixtures by @qgallouedec in https://github.com/huggingface/trl/pull/6199\r\n* Test DataCollatorForChatML raises when completion fills window by @albertvillanova in https://github.com/huggingface/trl/pull/6319\r\n* Simplify tokenization [5/N]: Pass chat_template as apply_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/6315\r\n* Support multiple environments [2/2]: Per-example environment selection by @qgallouedec in https://github.com/huggingface/trl/pull/6002\r\n* Remove post-training-toolkit integration by @qgallouedec in https://github.com/huggingface/trl/pull/6308\r\n* Environment-owned reward by @qgallouedec in https://github.com/huggingface/trl/pull/6238\r\n* Align KTO with DPO: `quantization_config` trainer argument by @qgallouedec in https://github.com/huggingface/trl/pull/6276\r\n* Support DatasetDict and IterableDatasetDict as eval_dataset in trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6322\r\n* Document that max_steps is required for iterable train datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6333\r\n* Raise on `quantization_config` + already-instantiated model in `DPOTrainer`/ `KTOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6312\r\n* Fix chunked_nll patch hiding VLM kwargs from generate by @Strongich in https://github.com/huggingface/trl/pull/6156\r\n* Neuron-friendly `chunked_nll` via static-shape token packing by @michaelbenayoun in https://github.com/huggingface/trl/pull/6314\r\n* Fix incorrect examples in distillation docs by @sergiopaniego in https://github.com/huggingface/trl/pull/6334\r\n* Docs: treat KTO as a stable trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6345\r\n* Release: v1.8 by @qgallouedec in https://github.com/huggingface/trl/pull/6346\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.7.0...v1.8.0\r\n","publishedAt":"2026-07-09T18:41:28.000Z","fetchedAt":"2026-07-09T21:05:25.211Z","url":"https://github.com/huggingface/trl/releases/tag/v1.8.0","media":[],"coverageCount":0},{"id":"rel_m-2DKXJ2DPtKCbQ7eiKwY","version":"v1.7.1","type":"feature","title":"v1.7.1","summary":"Fixed a hang in GRPO + vLLM colocate + PEFT on non-NVLink hardware and corrected dataset fingerprinting in DPO/SFT tokenization. Also integrated the new response parsing API, added a prompt-learning guard for PEFT with Liger in GRPO, and fixed activation offload storage deduplication.","titleGenerated":"TRL v1.7.1 fixes GRPO + vLLM hang and dataset fingerprinting","titleShort":"GRPO + vLLM hang fixed on non-NVLink; dataset fingerprinting corrected","breaking":"none","importance":2,"content":"## What's Changed\r\n\r\n* Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware by @albertvillanova in https://github.com/huggingface/trl/pull/6139\r\n* Fix dataset fingerprinting in DPO/SFT tokenization by @qgallouedec in https://github.com/huggingface/trl/pull/6206\r\n* Pass GPU device_ids to barrier fix in GRPO + vLLM colocate + PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6187\r\n* Integrate the new response parsing API by @qgallouedec in https://github.com/huggingface/trl/pull/5791\r\n* Fix `add_response_schema` tests for the new `parse_response` prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236\r\n* Add prompt-learning guard for PEFT with Liger in GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6186\r\n* Fix activation offload storage dedupe reuse by @winglian in https://github.com/huggingface/trl/pull/6241\r\n\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.7.0...v1.7.1","publishedAt":"2026-07-04T04:05:54.000Z","fetchedAt":"2026-07-04T05:03:01.742Z","url":"https://github.com/huggingface/trl/releases/tag/v1.7.1","media":[],"coverageCount":0},{"id":"rel_b-tcAO3CgHjd-FDiEqan7","version":"v1.7.0","type":"feature","title":"v1.7.0","summary":"The default SFT loss_type is now \"chunked_nll\", delivering ~30% less peak VRAM on average with neutral or slightly faster wall-clock time. Also introduces experimental GMPO trainer, transformers continuous batching, AsyncGRPO weight sync with vLLM 0.22+, and paddding-free AsyncGRPO.","titleGenerated":"TRL v1.7.0 makes chunked cross-entropy loss default for SFT and adds GMPO trainer","titleShort":"SFT default loss flips to chunked_nll; GMPO trainer arrives","breaking":"major","importance":null,"content":"## Features\r\n\r\n### SFT default `loss_type` is now `\"chunked_nll\"`\r\n\r\nThe flip announced in v1.6 has landed. Setting `loss_type` is optional, and the default now resolves to `\"chunked_nll\"` — giving every `SFTTrainer` run **~30% less peak VRAM on average (up to ~50% on large-vocab models)** with wall-clock time neutral or slightly faster. No action needed.\r\n\r\nThe auto-resolve falls back to `\"nll\"` when `use_liger_kernel=True` (the two paths are incompatible). If you want the old behavior — e.g. for custom heads — pin it explicitly:\r\n\r\n```python\r\nSFTConfig(loss_type=\"nll\")\r\n```\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5846\r\n\r\n### MoE auxiliary loss in GRPO / RLOO / AsyncGRPO\r\n\r\nPost-training MoE models now correctly include the router load-balancing auxiliary loss, matching the model's own reference forward and `SFTTrainer`. Enable via `model_init_kwargs`:\r\n\r\n```python\r\nGRPOConfig(\r\n    ...,\r\n    model_init_kwargs={\"output_router_logits\": True, \"router_aux_loss_coef\": 0.001},\r\n)\r\n```\r\n\r\nPlumbed through `_get_per_token_logps_and_entropies` (now returns a 3-tuple including `aux_loss`), folded into the policy loss with grad-accum scaling matched per trainer, and logged as `aux_loss`. AsyncGRPO recomputes it via `load_balancing_loss_func` in the chunked LM-head path (same as SFT's chunked path).\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6083, plus `router_aux_loss_coef` config wiring by @qgallouedec in https://github.com/huggingface/trl/pull/6085\r\n\r\n### New experimental GMPO trainer\r\n\r\n[Geometric-Mean Policy Optimization](https://huggingface.co/papers/2507.20673) lands as an experimental trainer. Replaces GRPO's per-token arithmetic mean of importance ratios with a **sequence-level geometric mean** (mean of clipped log-ratios, then `exp`); clipping is one-sided by advantage sign and applied in log space. Default `epsilon=0.4` per the paper.\r\n\r\n```python\r\nfrom trl.experimental.gmpo import GMPOConfig, GMPOTrainer\r\n\r\ntrainer = GMPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=GMPOConfig(epsilon=0.4),\r\n    reward_funcs=accuracy_reward,\r\n    train_dataset=dataset,\r\n)\r\n```\r\n\r\nby @raghulchandramouli in https://github.com/huggingface/trl/pull/6078\r\n\r\n### Transformers continuous batching in GRPO / RLOO\r\n\r\n`use_transformers_paged` was deprecated in v1.4; it's now replaced with proper [transformers continuous batching](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationMixin.generate_batch). The old branch silently bypassed importance-sampling correction (`logprobs = None`); the new path captures logprobs from `output.logprobs` and exposes a `ContinuousBatchingConfig` for KV-cache tuning.\r\n\r\n```python\r\nGRPOConfig(\r\n    ...,\r\n    use_transformers_continuous_batching=True,\r\n    transformers_continuous_batching_config={\r\n        \"use_cuda_graph\": False,\r\n        \"max_memory_percent\": 0.4,  # leave headroom for training\r\n    },\r\n)\r\n```\r\n\r\nBenchmark (Llama-3.2-1B-Instruct, A100 80GB, GSM8K): **1.25× faster at N=64 generations** with **-16 GB peak VRAM** vs default `generate()`. Use when N ≥ 32 with variable completion lengths.\r\n\r\n`use_transformers_paged=True` still works and forwards to the new flag with a `FutureWarning`. Requires `transformers>=5.8.0`.\r\n\r\nby @sergiopaniego in https://github.com/huggingface/trl/pull/5765\r\n\r\n### AsyncGRPO: native weight sync with vLLM ≥ 0.22.0\r\n\r\n`WeightTransferClient` now drives vLLM's native [4-phase RL weight-transfer API](https://docs.vllm.ai/en/latest/training/weight_transfer/) instead of the older 2-call flow: `pause(mode=\"keep\")` → `start_weight_update` → threaded `update_weights` + NCCL broadcast → `finish_weight_update` → `resume`. Validated end-to-end on H100 across single-node, FSDP2×4 + TP=4, and 2-node FSDP2×4 + DP=2×TP=4 (weight-sync time ≈ 0.18-0.8 s).\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/5892\r\n\r\n### Padding-free training in AsyncGRPO\r\n\r\nAsyncGRPO now supports the same padding-free path SFT already had. Flattens the batch and uses `position_ids`-based document boundaries instead of right-padding to the longest sequence — meaningful speedup and memory savings on heterogeneous-length workloads.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5854\r\n\r\n### Experimental Harbor integration\r\n\r\nA new `trl.experimental.harbor` adapter plugs [Harbor](https://www.harborframework.com) agentic task suites into `GRPOTrainer` via `environment_factory`. Same pattern as the OpenReward integration — one spec wires all three trainer slots:\r\n\r\n```python\r\nfrom trl import GRPOConfig, GRPOTrainer\r\nfrom trl.experimental.harbor import HarborSpec\r\n\r\nspec = HarborSpec(\"AdithyaSK/data_agent_rl_environment_train\", agent=\"bash\", num_tasks=64)\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=GRPOConfig(num_generations=8, max_steps=50, max_tool_calling_iterations=25),\r\n    train_dataset=spec.train_dataset,\r\n    environment_factory=spec.environment_factory,\r\n    reward_funcs=spec.reward_funcs,\r\n)\r\n```\r\n\r\nBuilt-in `bash` harness, plus `jupyter` and `terminal_notes` example harnesses. Gated by the new `trl[harbor]` extra.\r\n\r\nby @adithya-s-k in https://github.com/huggingface/trl/pull/6018\r\n\r\n### `trust_remote_code` in trainer configs\r\n\r\nA single `trust_remote_code: bool = False` field on the trainer configs now covers the **whole load surface** — model, processor / tokenizer, reference model, reward model, reward tokenizer, teacher — instead of forcing users to thread it through several independent kwarg dicts.\r\n\r\n```python\r\nSFTConfig(trust_remote_code=True)\r\n```\r\n\r\n`ModelConfig.trust_remote_code` is removed to avoid duplicate `--trust_remote_code` when combining dataclasses; CLI behavior is unchanged.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5802\r\n\r\n### KTO ↔ DPO alignment: tests, evaluate, sync_ref_model\r\n\r\nThe last alignment cycle before graduation: KTO now has parity with DPO on `pad_to_multiple_of`, `sync_ref_model`, `evaluate()`, method order/signature, metric placement (all moved into `_compute_loss`), and a real text + VLM (including multi-image) test suite.\r\n\r\nPRs all by @albertvillanova: #6029, #6030, #6033, #6034, #6035, #6080, #6093, #6148, #6149, #6150, #6152, #6160, #6163.\r\n\r\n### LFM2-VL multimodal inputs in GRPO / RLOO\r\n\r\nGRPO and RLOO now support [LFM2-VL](https://huggingface.co/LiquidAI/LFM2-VL-1.6B) multimodal inputs end-to-end.\r\n\r\nby @zwischenraum in https://github.com/huggingface/trl/pull/6114\r\n\r\n### New built-in reward helpers\r\n\r\n* `get_repetition_penalty_reward` by @qgallouedec in https://github.com/huggingface/trl/pull/6058\r\n* `get_cosine_scaled_reward` by @qgallouedec in https://github.com/huggingface/trl/pull/6066\r\n\r\n### SFT refactor: build labels during dataset preparation\r\n\r\nLabel construction moves out of the collator and into dataset preparation, so \"what's trainable\" is defined in exactly one place. A single batched `map` produces a `labels` column where each token keeps its ID when every applicable mask is 1, else `-100`. Plain LM stays storage-neutral; pre-tokenized datasets with mask columns now go through the same path. Step 1 toward fixing #3927.\r\n\r\nby @0xadvait in https://github.com/huggingface/trl/pull/6037\r\n\r\n### Idefics3 chat template\r\n\r\n`{% generation %}`-marker training template for Idefics3, enabling `assistant_only_loss=True`.\r\n\r\nby @aazizyan in https://github.com/huggingface/trl/pull/5871\r\n\r\n### vLLM version sweep\r\n\r\n* Support vLLM 0.19.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6107\r\n* Support vLLM 0.20.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6108\r\n* Support vLLM 0.22.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6119\r\n* Support vLLM 0.23.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6153\r\n* Drop vLLM 0.12 by @qgallouedec in https://github.com/huggingface/trl/pull/6109\r\n* Drop vLLM 0.13 by @qgallouedec in https://github.com/huggingface/trl/pull/6154\r\n\r\n### Other\r\n\r\n* Normalize JSD distillation loss by `num_items_in_batch` for gradient accumulation by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6006\r\n* `[AsyncGRPO]` Rollout worker: set aiohttp limit to `max(100, max_inflight_tasks)` by @ggcr in https://github.com/huggingface/trl/pull/5861\r\n* Align SDPO with GRPO/RLOO: drop NaN values when averaging logged metrics by @anshulkulhari7 in https://github.com/huggingface/trl/pull/6055\r\n* Make `evaluate()` accept the same dataset types as the trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6116\r\n* Keep extra columns in `unpair_preference_dataset` by @albertvillanova in https://github.com/huggingface/trl/pull/6161\r\n* Warn when sequence-level importance sampling is combined with a token-summed loss type by @discobot in https://github.com/huggingface/trl/pull/6042\r\n* `fix(profiling)`: log `ProfilingContext` metrics to Trackio backend by @Anai-Guo in https://github.com/huggingface/trl/pull/5979\r\n* Move experimental example scripts out of the packaged tree by @sergiopaniego in https://github.com/huggingface/trl/pull/6141\r\n* Remove redundant `.contiguous()` calls by @qgallouedec in https://github.com/huggingface/trl/pull/6045 and https://github.com/huggingface/trl/pull/6046\r\n* Harmonize logger imports by @qgallouedec in https://github.com/huggingface/trl/pull/6142\r\n\r\n## Fixes\r\n\r\n* **Share frozen layers with reference model instead of duplicating in memory** — `create_reference_model` with `num_shared_layers` was double-allocating the \"shared\" frozen layers because the loop never assigned `_ref_param` back. Now it does, so shared layers are held once. By @behroozazarkhalili in https://github.com/huggingface/trl/pull/6053\r\n* **Fix `chunked_nll` mixed Tensor/DTensor error under FSDP2 + PEFT** by @albertvillanova in https://github.com/huggingface/trl/pull/6065\r\n* **Fix per-chunk `lm_head.weight` all-gathers under FSDP2 + `chunked_nll`** by @albertvillanova in https://github.com/huggingface/trl/pull/6077\r\n* **Fix ZeRO-3 + PEFT mixed-dtype error** for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6091 and KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6093\r\n* Fix ref adapter creation when the LoRA config uses `target_parameters` by @discobot in https://github.com/huggingface/trl/pull/6043\r\n* Raise when `use_liger_kernel` is combined with a PEFT adapter on `lm_head` by @akshansh47 in https://github.com/huggingface/trl/pull/5977\r\n* `[fix]` GLM-4-MoE template: turn-terminating token to the turn itself by @qgallouedec in https://github.com/huggingface/trl/pull/6044\r\n* Fix `unpair_preference_dataset` dropping extra columns by @albertvillanova in https://github.com/huggingface/trl/pull/6059\r\n* `fix(gold)`: preserve vllm prompt special tokens by @he-yufeng in https://github.com/huggingface/trl/pull/6063\r\n* `fix(sft)`: reject transformed datasets during preparation by @he-yufeng in https://github.com/huggingface/trl/pull/6054\r\n* Fix broken light-mode banner image in README by @strickvl in https://github.com/huggingface/trl/pull/6112\r\n* Fix BrowserGym OpenEnv example dependency and task wiring by @burtenshaw in https://github.com/huggingface/trl/pull/6117\r\n* Fix signature of `_unpair_row` by @albertvillanova in https://github.com/huggingface/trl/pull/6062\r\n* Remove silently-ignored W&B/Hub fields from GOLD and Distillation configs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6023\r\n\r\n## Documentation and Examples\r\n\r\n* docs: sync experimental trainer docstrings with their `__init__` signatures by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6011\r\n* docs: fix stale default values in config docstrings by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6015\r\n* docs: fix function docstrings that drifted from their signatures by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6022\r\n* Fix broken doc links in `paper_index` (experimental page was split) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6070\r\n* Fix broken import in GOLD trainer docs (GOLD is experimental) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6073\r\n* Fix broken section anchors in docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6071\r\n* Fix broken example link in BCO trainer docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6084\r\n* `docs(rloo)`: add clarifying comment on KL penalty formula divergence from GRPO by @abderahmane-ai in https://github.com/huggingface/trl/pull/6096\r\n* Fix async GRPO docs to require `vllm>=0.22.0` by @sergiopaniego in https://github.com/huggingface/trl/pull/6101\r\n* Fix broken internal doc link to GKD Trainer in MiniLLM docs by @ShamSaleem in https://github.com/huggingface/trl/pull/6131\r\n* Align `epsilon` help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014\r\n* Fix style of optional parameters in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6081\r\n* Align format of code examples in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6147\r\n* Add license to Harbor examples by @qgallouedec in https://github.com/huggingface/trl/pull/6104\r\n\r\n## CI\r\n\r\n* Extract VLM tests into dedicated classes for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6033\r\n* Add raises test with vision dataset and text model for DPO and SFT by @albertvillanova in https://github.com/huggingface/trl/pull/6026\r\n* Extract `_push_param_to_vllm` helper in `VLLMGeneration` by @albertvillanova in https://github.com/huggingface/trl/pull/6004\r\n* Use relative imports in `async_grpo` to match the rest of `trl/experimental` by @qgallouedec in https://github.com/huggingface/trl/pull/6012\r\n* Align AsyncGRPO with GRPO: `num_completions_to_print`, `epsilon_high` fallback, `logging_steps` docstring, loss variable names, clip-ratio metrics by @qgallouedec in https://github.com/huggingface/trl/pull/6020, https://github.com/huggingface/trl/pull/6019, https://github.com/huggingface/trl/pull/6016, https://github.com/huggingface/trl/pull/6013 and https://github.com/huggingface/trl/pull/6021\r\n* Add vision requirement marker to Idefics3 test parameter by @qgallouedec in https://github.com/huggingface/trl/pull/6106\r\n* Update distributed SFT test after default `chunked_nll` loss by @albertvillanova in https://github.com/huggingface/trl/pull/6074\r\n* `test`: bound memory in `test_gkd_trainer_with_liger` to avoid OOM on shared runners by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6103\r\n* Fix `sft_fa2` invariant test by @qgallouedec in https://github.com/huggingface/trl/pull/6069\r\n* Pin SHA instead of version tag for CI `actions/checkout` / `astral-sh/setup-uv` / `actions/setup-python` / `pre-commit/action` by @albertvillanova in https://github.com/huggingface/trl/pull/6097, https://github.com/huggingface/trl/pull/6098, https://github.com/huggingface/trl/pull/6099 and https://github.com/huggingface/trl/pull/6100\r\n* Replace `parse_version` with `Version` by @albertvillanova in https://github.com/huggingface/trl/pull/6164\r\n* Hotfix CI: temporarily pin `deepspeed < 0.19.2` by @albertvillanova in https://github.com/huggingface/trl/pull/6090\r\n* `chore`: update `tests_transformers_branch.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6051\r\n* `chore`: update `clear_cache.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6047\r\n* `chore`: update `docker-build.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6048\r\n* Delete CI `pr_style_bot` workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6082\r\n* Remove issue labeller by @qgallouedec in https://github.com/huggingface/trl/pull/6052\r\n* Bump the actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6041\r\n* Change `_get_train_sampler` comment to mention `num_iterations > 1` by @anidoesdev in https://github.com/huggingface/trl/pull/6125\r\n\r\n## New Contributors\r\n* @Anai-Guo made their first contribution in https://github.com/huggingface/trl/pull/5979\r\n* @ggcr made their first contribution in https://github.com/huggingface/trl/pull/5861\r\n* @he-yufeng made their first contribution in https://github.com/huggingface/trl/pull/6063\r\n* @discobot made their first contribution in https://github.com/huggingface/trl/pull/6043\r\n* @anshulkulhari7 made their first contribution in https://github.com/huggingface/trl/pull/6055\r\n* @abderahmane-ai made their first contribution in https://github.com/huggingface/trl/pull/6096\r\n* @akshansh47 made their first contribution in https://github.com/huggingface/trl/pull/5977\r\n* @strickvl made their first contribution in https://github.com/huggingface/trl/pull/6112\r\n* @ShamSaleem made their first contribution in https://github.com/huggingface/trl/pull/6131\r\n* @0xadvait made their first contribution in https://github.com/huggingface/trl/pull/6037\r\n* @anidoesdev made their first contribution in https://github.com/huggingface/trl/pull/6125\r\n* @zwischenraum made their first contribution in https://github.com/huggingface/trl/pull/6114\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6010\r\n* docs: sync experimental trainer docstrings with their __init__ signatures by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6011\r\n* docs: fix stale default values in config docstrings by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6015\r\n* fix(profiling): log ProfilingContext metrics to Trackio backend by @Anai-Guo in https://github.com/huggingface/trl/pull/5979\r\n* docs: fix function docstrings that drifted from their signatures by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6022\r\n* Add raises test with vision dataset and text model for DPO and SFT by @albertvillanova in https://github.com/huggingface/trl/pull/6026\r\n* Align KTO with DPO: Unwrap VLM batch dimension for text-only data in _tokenize by @albertvillanova in https://github.com/huggingface/trl/pull/6029\r\n* Extract VLM tests into dedicated classes for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6033\r\n* Extract _push_param_to_vllm helper in VLLMGeneration by @albertvillanova in https://github.com/huggingface/trl/pull/6004\r\n* Align KTO with DPO: Add tests for text data collator by @albertvillanova in https://github.com/huggingface/trl/pull/6034\r\n* Align KTO with DPO: Support pad_to_multiple_of by @albertvillanova in https://github.com/huggingface/trl/pull/6035\r\n* Align KTO with DPO: Add VLM tests by @albertvillanova in https://github.com/huggingface/trl/pull/6030\r\n* Normalize JSD distillation loss by num_items_in_batch for gradient accumulation by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6006\r\n* Bump the actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6041\r\n* [fix] GLM-4-MoE template: turn-terminating token to the turn itself by @qgallouedec in https://github.com/huggingface/trl/pull/6044\r\n* fix: share frozen layers with reference model instead of duplicating in memory by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6053\r\n* Default SFT loss to chunked_nll by @qgallouedec in https://github.com/huggingface/trl/pull/5846\r\n* Remove redundant `.contiguous()` calls by @qgallouedec in https://github.com/huggingface/trl/pull/6045\r\n* Remove silently-ignored W&B/Hub fields from GOLD and Distillation configs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6023\r\n* [AsyncGRPO] Rollout worker: set aiohttp limit to max(100, max_inflight_tasks) by @ggcr in https://github.com/huggingface/trl/pull/5861\r\n* async grpo native weight sync with vllm>=0.22.0 by @AmineDiro in https://github.com/huggingface/trl/pull/5892\r\n* Fix `unpair_preference_dataset` dropping extra columns by @albertvillanova in https://github.com/huggingface/trl/pull/6059\r\n* fix(gold): preserve vllm prompt special tokens by @he-yufeng in https://github.com/huggingface/trl/pull/6063\r\n* Fix broken doc links in paper_index (experimental page was split) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6070\r\n* chore: update tests_transformers_branch.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6051\r\n* Fix chunked_nll mixed Tensor/DTensor error under FSDP2 + PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6065\r\n* Fix signature of _unpair_row by @albertvillanova in https://github.com/huggingface/trl/pull/6062\r\n* fix(sft): reject transformed datasets during preparation by @he-yufeng in https://github.com/huggingface/trl/pull/6054\r\n* Fix broken import in GOLD trainer docs (GOLD is experimental) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6073\r\n* Fix broken section anchors in docs (slug mismatches, renamed/removed sections) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6071\r\n* Update distributed SFT test after default chunked_nll loss by @albertvillanova in https://github.com/huggingface/trl/pull/6074\r\n* Add GMPO (Geometric-Mean Policy Optimization) experimental trainer by @raghulchandramouli in https://github.com/huggingface/trl/pull/6078\r\n* Fix style of optional parameters in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6081\r\n* Align KTO with DPO: Support config pad_to_multiple_of by @albertvillanova in https://github.com/huggingface/trl/pull/6080\r\n* chore: update clear_cache.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6047\r\n* chore: update docker-build.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6048\r\n* Padding-free training in AsyncGRPO by @qgallouedec in https://github.com/huggingface/trl/pull/5854\r\n* Delete CI pr_style_bot workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6082\r\n* Fix broken example link in BCO trainer docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6084\r\n* Hotfix CI: Temporarily pin deepspeed < 0.19.2 by @albertvillanova in https://github.com/huggingface/trl/pull/6090\r\n* Add MoE auxiliary loss to GRPO, RLOO, and AsyncGRPO trainers by @AmineDiro in https://github.com/huggingface/trl/pull/6083\r\n* Add experimental Harbor integration for GRPO environment training by @adithya-s-k in https://github.com/huggingface/trl/pull/6018\r\n* Fix `sft_fa2` invariant test by @qgallouedec in https://github.com/huggingface/trl/pull/6069\r\n* Use relative imports in async_grpo to match the rest of trl/experimental` by @qgallouedec in https://github.com/huggingface/trl/pull/6012\r\n* Fix ref adapter creation when the LoRA config uses `target_parameters` by @discobot in https://github.com/huggingface/trl/pull/6043\r\n* Remove issue labeller by @qgallouedec in https://github.com/huggingface/trl/pull/6052\r\n* Add Idefics3 original and training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5871\r\n* Align SDPO with GRPO/RLOO: drop NaN values when averaging logged metrics by @anshulkulhari7 in https://github.com/huggingface/trl/pull/6055\r\n* docs(rloo): add clarifying comment on KL penalty formula divergence from GRPO by @abderahmane-ai in https://github.com/huggingface/trl/pull/6096\r\n* feat(grpo): replace deprecated `use_transformers_paged` with transformers continuous batching by @sergiopaniego in https://github.com/huggingface/trl/pull/5765\r\n* test: bound memory in test_gkd_trainer_with_liger to avoid OOM on shared runners by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6103\r\n* Add vision requirement marker to Idefics3 test parameter by @qgallouedec in https://github.com/huggingface/trl/pull/6106\r\n* Pin SHA instead of version tag for CI actions/checkout by @albertvillanova in https://github.com/huggingface/trl/pull/6097\r\n* Pin SHA instead of version tag for CI astral-sh/setup-uv by @albertvillanova in https://github.com/huggingface/trl/pull/6098\r\n* Pin SHA instead of version tag for CI actions/setup-python by @albertvillanova in https://github.com/huggingface/trl/pull/6099\r\n* Pin SHA instead of version tag for CI pre-commit/action by @albertvillanova in https://github.com/huggingface/trl/pull/6100\r\n* Fix ZeRO-3 + PEFT mixed-dtype error for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6091\r\n* Align KTO with DPO: Fix ZeRO-3 + PEFT dtype mismatch for non-quantized models by @albertvillanova in https://github.com/huggingface/trl/pull/6093\r\n* Fix per-chunk lm_head.weight all-gathers under FSDP2 + chunked_nll by @albertvillanova in https://github.com/huggingface/trl/pull/6077\r\n* Raise when use_liger_kernel is combined with a PEFT adapter on lm_head by @akshansh47 in https://github.com/huggingface/trl/pull/5977\r\n* Align `epsilon` help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014\r\n* Fix broken light-mode banner image in README by @strickvl in https://github.com/huggingface/trl/pull/6112\r\n* Drop vLLM 0.12 support by @qgallouedec in https://github.com/huggingface/trl/pull/6109\r\n* Add `router_aux_loss_coef` by @qgallouedec in https://github.com/huggingface/trl/pull/6085\r\n* Add support for vLLM 0.19.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6107\r\n* Add license to Harbor examples by @qgallouedec in https://github.com/huggingface/trl/pull/6104\r\n* Remove redundant `.contiguous()` from the shift logits/labels pattern by @qgallouedec in https://github.com/huggingface/trl/pull/6046\r\n* Fix BrowserGym OpenEnv example dependency and task wiring by @burtenshaw in https://github.com/huggingface/trl/pull/6117\r\n* Fix async GRPO docs to require vllm>=0.22.0 by @sergiopaniego in https://github.com/huggingface/trl/pull/6101\r\n* Add `get_repetition_penalty_reward` by @qgallouedec in https://github.com/huggingface/trl/pull/6058\r\n* Add `trust_remote_code` to trainer configs by @qgallouedec in https://github.com/huggingface/trl/pull/5802\r\n* Align AsyncGRPO num_completions_to_print with GRPO (int | None) by @qgallouedec in https://github.com/huggingface/trl/pull/6020\r\n* Align AsyncGRPO epsilon_high with GRPO (None fallback to epsilon) by @qgallouedec in https://github.com/huggingface/trl/pull/6019\r\n* Fix logging_steps default mentioned in AsyncGRPOConfig docstring by @qgallouedec in https://github.com/huggingface/trl/pull/6016\r\n* Align async GRPO loss variable names with GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6013\r\n* Make `evaluate()` accept the same dataset types as the trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6116\r\n* Add support for vLLM 0.20.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6108\r\n* Add support for vLLM 0.22.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6119\r\n* Fix broken internal doc link to GKD Trainer in MiniLLM docs by @ShamSaleem in https://github.com/huggingface/trl/pull/6131\r\n* Align AsyncGRPO clip-ratio metrics with GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6021\r\n* Add `get_cosine_scaled_reward` by @qgallouedec in https://github.com/huggingface/trl/pull/6066\r\n* Move experimental example scripts out of the packaged tree by @sergiopaniego in https://github.com/huggingface/trl/pull/6141\r\n* Harmonize logger imports by @qgallouedec in https://github.com/huggingface/trl/pull/6142\r\n* refactor(sft): build labels during dataset preparation instead of collation by @0xadvait in https://github.com/huggingface/trl/pull/6037\r\n* Add support for vLLM 0.23.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6153\r\n* Change `_get_train_sampler` comment to mention `num_iterations > 1` by @anidoesdev in https://github.com/huggingface/trl/pull/6125\r\n* Warn when sequence-level importance sampling is combined with a token-summed loss type by @discobot in https://github.com/huggingface/trl/pull/6042\r\n* Drop vLLM 0.13 support by @qgallouedec in https://github.com/huggingface/trl/pull/6154\r\n* Align KTO with DPO: Add evaluate() override by @albertvillanova in https://github.com/huggingface/trl/pull/6148\r\n* Align KTO with DPO: Align order and signature of methods by @albertvillanova in https://github.com/huggingface/trl/pull/6149\r\n* Align KTO with DPO: Move all metrics computation from log to _compute_loss by @albertvillanova in https://github.com/huggingface/trl/pull/6150\r\n* Align KTO with DPO: Support sync_ref_model by @albertvillanova in https://github.com/huggingface/trl/pull/6152\r\n* Replace parse_version with Version by @albertvillanova in https://github.com/huggingface/trl/pull/6164\r\n* Keep extra columns in unpair_preference_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/6161\r\n* Align KTO with DPO: Add tests by @albertvillanova in https://github.com/huggingface/trl/pull/6160\r\n* Align format of code examples in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6147\r\n* Align KTO with DPO: Add VLM multi-image test by @albertvillanova in https://github.com/huggingface/trl/pull/6163\r\n* Support LFM2-VL multimodal inputs in GRPO and RLOO by @zwischenraum in https://github.com/huggingface/trl/pull/6114\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.6.0...v1.7.0\r\n","publishedAt":"2026-06-25T22:52:18.000Z","fetchedAt":"2026-06-25T23:00:34.406Z","url":"https://github.com/huggingface/trl/releases/tag/v1.7.0","media":[],"coverageCount":0},{"id":"rel_4aRgXtA9OXhhfQPQK3xdf","version":"v1.6.0","type":"feature","title":"v1.6.0","summary":"The release introduces a new experimental A2POTrainer for optimal advantage regression and grants KTO trainer support for vision-language models. The AsyncRolloutWorker now runs in a separate process to avoid GIL contention and potential NCCL watchdog timeouts, along with fixes for aiohttp retries and all-NaN reward columns. Gold distillation trainer now aligns tokens via byte offsets, and SDFT/SDPO leverage the vLLM server for live teacher logprobs. Other features include bidirectional masked importance sampling for IcePop, support for NemotronH and Nemotron 3 Ultra, additional training chat templates, and decoupled self-distillation trainers.","titleGenerated":"Fine-tuning TRL v1.6.0 debuts A2PO trainer, supports VLM KTO, and improves async GRPO","titleShort":"A2PO trainer debuts; VLM KTO support; Async GRPO spawns process","breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### AsyncGRPO rollout worker now runs in a separate process\r\n\r\n`AsyncRolloutWorker` is no longer a thread — it's a spawned child process with its own GIL. The trainer's autograd engine no longer competes with `recursive_parse` / `accuracy_reward` for the GIL, which was causing 1-5s stalls in real Qwen3-30B-A3B @ 16k runs and ultimately NCCL watchdog timeouts on other ranks.\r\n\r\nArchitectural changes:\r\n- `AsyncRolloutWorker` (parent) owns the child process + shared `mp.Queue` / `mp.Value` / `mp.Event`.\r\n- `_AsyncRolloutLoop` (child-only) handles tokenization, dataset iteration, reward funcs, and asyncio loops.\r\n- A new `WeightTransferClient` owns the NCCL group with vLLM (`/pause`, `/resume`, `/init_weight_transfer_engine`, `/update_weights`); the rollout child only talks to `/v1/completions`.\r\n\r\nTwo correctness fixes shipped alongside (they would have conflicted otherwise): broader aiohttp retry (now catches `ClientPayloadError`) with bounded exponential backoff, and **all-NaN reward columns** are now preserved — `np.nansum` was silently returning 0, giving unscorable completions a real advantage signal and pushing the policy *away* from correct answers (~30% of DeepMath / OpenR1-Math rows).\r\n\r\n> [!NOTE]\r\n> `reward_funcs` / `tools` / `environment_factory` must now be picklable, and the child runs CPU-only (`CUDA_VISIBLE_DEVICES=\"\"`).\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/5749\r\n\r\n### New experimental A2PO trainer (Optimal Advantage Regression)\r\n\r\nA new `A2POTrainer` implements [A\\*-PO from \"Accelerating RL for LLM Reasoning with Optimal Advantage Regression\"](https://huggingface.co/papers/2505.20686). Two stages: an **offline V\\* estimation pass** from reference policy samples (with optional `filter_all_incorrect` to drop prompts where every reference completion fails), then **on-policy training with one generation per prompt** and a plain least-squares loss on `β₂·log(π/π_ref)` vs `r − V*`. No group, no critic, no clipping, no reward normalization.\r\n\r\n```python\r\nfrom trl.experimental.a2po import A2POConfig, A2POTrainer\r\n\r\ntrainer = A2POTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=A2POConfig(num_value_samples=8, filter_all_incorrect=True),\r\n    train_dataset=dataset,\r\n    reward_funcs=accuracy_reward,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nDesigned for binary verifiable rewards (math/code), not open-ended problems.\r\n\r\nby @raghulchandramouli in https://github.com/huggingface/trl/pull/5940\r\n\r\n### KTO now supports VLMs + big alignment push\r\n\r\nThe biggest KTO ↔ DPO alignment cycle yet — **KTOTrainer now supports vision-language models**, plus a deep restructuring of compute_loss, KL dataset generation, ref-logp precomputation, activation offloading, sampler strategy, metrics, and more. KTO graduation is very close.\r\n\r\n```python\r\nfrom trl.experimental.kto import KTOConfig, KTOTrainer\r\n\r\ntrainer = KTOTrainer(\r\n    model=\"Qwen/Qwen2.5-VL-3B-Instruct\",\r\n    args=KTOConfig(...),\r\n    train_dataset=vision_kto_dataset,\r\n)\r\n```\r\n\r\nVLM support: by @albertvillanova in https://github.com/huggingface/trl/pull/5939. Plus ~20 alignment PRs all by @albertvillanova: #5820, #5849, #5852, #5850, #5866, #5864, #5856, #5872, #5875, #5900, #5901, #5899, #5906, #5909, #5914, #5982, #5936, #5996, #5998, #5999.\r\n\r\n### Cross-tokenizer alignment in GOLD via byte offsets\r\n\r\nThe GOLD distillation trainer used to align student/teacher tokens by extending two decoded strings and flushing on equality. It silently broke on **any byte-level disagreement** — including the common case of one tokenizer prepending BOS while the other doesn't (Llama-3 ↔ Qwen-3). The X-Token paper called this out by name.\r\n\r\nEach side now carries `(start_byte, end_byte)` spans derived once from the fast tokenizer's char offsets, and the walker syncs on cumulative byte boundaries. On the on-policy path, spans come from `piece_byte_len` over the sampled token ids (not from re-encoding the decoded completion — BPE makes that round-trip non-injective).\r\n\r\nTwo related fixes shipped: long rows no longer lose the completion (now keeping the last `max_length` tokens), and the vLLM on-policy `original_prompt_text` is now decoded from the truncated ids the student actually consumed.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/5885\r\n\r\n### SDFT / SDPO: live teacher logprobs from the vLLM server\r\n\r\nWhen `teacher_model_kind=\"live\"` and `vllm_mode=\"server\"`, the vLLM generation server already holds the current student weights (synced every step for rollouts). The new `use_teacher_server=True` flag scores the teacher's log-probs on **that same server** instead of running a separate local teacher forward — removing the teacher from the training step entirely.\r\n\r\nSupported modes: `sampled_token` (reverse KL on the realized token) and `topk_logits`. When buffered batches reuse steps (`num_iterations > 1`), weights are re-synced before scoring so the teacher never scores stale.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/5989\r\n\r\n### Bidirectional masked importance sampling (MIS) for IcePop\r\n\r\nvLLM importance sampling in GRPO now uses a **two-sided band** `[C_min, C_max]` instead of a single upper cap, aligning TIS/MIS with [IcePop's bidirectional handling](https://github.com/huggingface/trl/pull/4732) of train–inference ratio outliers.\r\n\r\n```python\r\nfrom trl import GRPOConfig\r\n\r\nconfig = GRPOConfig(\r\n    vllm_importance_sampling_clip_min=0.5,\r\n    vllm_importance_sampling_clip_max=2.0,\r\n    vllm_importance_sampling_correction=\"mask\",  # or \"truncate\"\r\n)\r\n```\r\n\r\nThe old `vllm_importance_sampling_cap` is deprecated and maps to `clip_max`.\r\n\r\nby @casinca in https://github.com/huggingface/trl/pull/4732\r\n\r\n### NemotronH and Nemotron 3 Ultra support\r\n\r\nDay-zero training support for NVIDIA's new model families.\r\n\r\n* **NemotronH integration** by @qgallouedec in https://github.com/huggingface/trl/pull/5938\r\n* **Nemotron 3 Ultra support** by @qgallouedec in https://github.com/huggingface/trl/pull/5942\r\n* Enable gradient checkpointing in Nemotron 3 SFT example by @sergiopaniego in https://github.com/huggingface/trl/pull/5944\r\n\r\n### Even more training chat templates\r\n\r\nThree more model families with `{% generation %}` markers (assistant-only loss out of the box):\r\n\r\n- **Qwen2.5-VL** by @aazizyan in https://github.com/huggingface/trl/pull/5838\r\n- **Qwen2-VL** by @aazizyan in https://github.com/huggingface/trl/pull/5839\r\n- **Llava-Next** by @aazizyan in https://github.com/huggingface/trl/pull/5959\r\n\r\n### Distributed backend boilerplate, hidden\r\n\r\nA new `trl/distributed.py` introduces a single `DistributedBackend` class that detects ZeRO stage and FSDP version once, then exposes two context managers (`gather_params`, `summon_full_params`) used everywhere. Replaces the scattered `getattr(state, \"fsdp_plugin\", None)` / `gather_if_zero3` / `summon_full_params if ... else nullcontext()` boilerplate spread across `vllm_generation.py`, `models/utils.py`, and the main trainers. Future deprecations land in one place.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6000\r\n\r\n### Decoupled self-distillation trainers\r\n\r\nA two-PR refactor that disentangles SDPO, SDFT, and other self-distillation trainers from their shared base, making each one self-contained and consistent with the rest of the codebase.\r\n\r\nby @LeonEricsson in https://github.com/huggingface/trl/pull/5862 and https://github.com/huggingface/trl/pull/5883\r\n\r\n### Heads-up: SFT default `loss_type` will change in 1.7\r\n\r\nSetting `SFTConfig.loss_type` is now optional, and leaving it unset emits a `FutureWarning`: in TRL **1.7** the default will switch from `\"nll\"` to `\"chunked_nll\"`. No action needed — you'll just get the new default automatically on upgrade — unless you want to pin the current behavior (e.g. for custom models) with `loss_type=\"nll\"`.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5997\r\n\r\n### Other\r\n\r\n* Support `'None'` as CLI value for `Optional[T]` fields by @qgallouedec in https://github.com/huggingface/trl/pull/5843\r\n* Support non-`lm_head` output projections in chunked SFT loss (GPTNeoX) by @qgallouedec in https://github.com/huggingface/trl/pull/5857\r\n* `SFTTrainer`: merge entropy and accuracy computation to eliminate redundant logits copy by @flutist in https://github.com/huggingface/trl/pull/5897\r\n* Remove redundant `.contiguous()` calls in `DPOTrainer` to reduce peak memory by @flutist in https://github.com/huggingface/trl/pull/5926\r\n* Remove unnecessary explicit `.contiguous()` before `entropy_from_logits` by @qgallouedec in https://github.com/huggingface/trl/pull/5930\r\n* Exclude `None` reward completions from GRPO/RLOO advantage baseline by @AmineDiro in https://github.com/huggingface/trl/pull/5902\r\n* Support multimodal config in PPO ValueHead by @albertvillanova in https://github.com/huggingface/trl/pull/5907\r\n* Support vision datasets for Liger in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5943\r\n* Raise if `precompute_ref_log_probs` with vision datasets in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5867\r\n* 🔒 Gate trainer telemetry on an explicit class-name allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/5851\r\n* Update vLLM version support to 0.19.0 by @sergiopaniego in https://github.com/huggingface/trl/pull/5879\r\n* Improve error message when image tokens are truncated by `max_length` by @lxk8998 in https://github.com/huggingface/trl/pull/5927\r\n* Padding-free invariance test by @qgallouedec in https://github.com/huggingface/trl/pull/5842\r\n* Per-field invariance tolerances, calibrated by @qgallouedec in https://github.com/huggingface/trl/pull/5844\r\n\r\n## Fixes\r\n\r\n* **Fix `loss_type=\"chunked_nll\"` under DeepSpeed ZeRO-3** by @qgallouedec in https://github.com/huggingface/trl/pull/5873\r\n* **Fix GRPO `use_liger_kernel` under DeepSpeed ZeRO-3** by @kashif in https://github.com/huggingface/trl/pull/5891\r\n* `async_grpo`: don't return on `queue.Empty` by @AmineDiro in https://github.com/huggingface/trl/pull/5751\r\n* Don't treat ROCm GPUs as Ampere by @kashif in https://github.com/huggingface/trl/pull/5917\r\n* Route liger student forward through DDP wrapper in GKD, GOLD, and Distillation trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5934\r\n* Fix backbone access in GRPO by aligning with SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5949\r\n* Fix priority order in PPO ValueHead and raise ValueError for unsupported config by @albertvillanova in https://github.com/huggingface/trl/pull/5908\r\n* Fix `generate_batch`: inference tensors block inplace ops in background thread by @albertvillanova in https://github.com/huggingface/trl/pull/5818 (cross-listed from v1.5 changelog window)\r\n* Fix SFT padding-free test config by @kashif in https://github.com/huggingface/trl/pull/5923\r\n* Specify `encoding=\"utf-8\"` when reading `.jinja` chat templates on Windows by @ColebyPearson in https://github.com/huggingface/trl/pull/5869\r\n* Fix `ValueError` by pinning `kernels < 0.15.1` by @albertvillanova in https://github.com/huggingface/trl/pull/5880\r\n* Set `kernels` optional dependency via transformers by @albertvillanova in https://github.com/huggingface/trl/pull/5884\r\n* Support `kernels` extra for `transformers < 5.1.0` by @albertvillanova in https://github.com/huggingface/trl/pull/5928\r\n* Add missing `use_liger_kernel` guard to SDPO teacher-server validation by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5994\r\n* Flash Attention capitalization fix by @qgallouedec in https://github.com/huggingface/trl/pull/5855\r\n\r\n## Documentation and Examples\r\n\r\n* Remove NeMo Gym Integration Guide (broken) by @cmunley1 in https://github.com/huggingface/trl/pull/5840\r\n* docs(GRPOTrainer): remove duplicate sentence by @zafstojano in https://github.com/huggingface/trl/pull/5957\r\n* docs(RLOOTrainer): fix blockquote math not rendering by @zafstojano in https://github.com/huggingface/trl/pull/5958\r\n* docs: highlight the role of KL in RLOO compared to GRPO by @zafstojano in https://github.com/huggingface/trl/pull/5966\r\n* docs: clarify PPO entropy metrics in PPO trainer docs by @biefan in https://github.com/huggingface/trl/pull/5289\r\n* docs: update OpenEnv GitHub org references and package name by @sergiopaniego in https://github.com/huggingface/trl/pull/5919\r\n* docs: update OpenEnv doc URLs to `huggingface.co/docs/openenv` by @sergiopaniego in https://github.com/huggingface/trl/pull/5929\r\n* docs: sync SDFT/SDPO config docstrings with their fields by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5992\r\n* docs: sync Distillation/GOLD/OnlineDPO config docstrings with their fields by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5995\r\n* docs: Document `bnb_4bit_quant_storage` and normalize docstring param headers by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5993\r\n* docs: fix rendering typos by @zafstojano in https://github.com/huggingface/trl/pull/5991\r\n* fix(docs): correct broken GKD Trainer link in MiniLLM docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5960 and https://github.com/huggingface/trl/pull/5961\r\n* fix(docs): drop duplicate \"a\" in `online_dpo_vlm` example description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5978\r\n* Fix broken doc links by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5971\r\n* Fix broken code examples in docs (RLOO syntax, `SFTConfig` `max_length`) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5970\r\n* Fix malformed ScaleRL paper link in `GRPOConfig` `epsilon_high` help by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5972\r\n* fix(cli): drop duplicate \"to\" in `trl skills install` description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6008\r\n* Remove invalid `max_prompt_length` argument from GRPO example by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5964\r\n\r\n## CI\r\n\r\n* Refresh `sft.json` / `dpo.json` snapshots after transformers `num_items_in_batch fix` by @qgallouedec in https://github.com/huggingface/trl/pull/5845\r\n* Add testing for Olmo 3 by @qgallouedec in https://github.com/huggingface/trl/pull/5962\r\n* Align trainer train tests by @qgallouedec in https://github.com/huggingface/trl/pull/5963\r\n* Align trainers: Remove redundant else branch by @albertvillanova in https://github.com/huggingface/trl/pull/5983\r\n* [CI] Check that training chat templates keep the stop token in the loss mask by @kashif in https://github.com/huggingface/trl/pull/5988\r\n* Create CI workflow to sync TRL skill with `huggingface/skills` by @albertvillanova in https://github.com/huggingface/trl/pull/5950\r\n* Simplify agent skills target and default to `.agents` by @albertvillanova in https://github.com/huggingface/trl/pull/5987\r\n* chore: enable Dependabot weekly GitHub Actions bumps by @hf-dependantbot-rollout[bot] in https://github.com/huggingface/trl/pull/5910\r\n* Bump the actions group with 9 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/5913\r\n* Bump the actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/5954\r\n* chore: update `docker-build.yml` with version parsing by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5920\r\n* ci: use GitHub App auth for doc preview comment bot by @sergiopaniego in https://github.com/huggingface/trl/pull/5915\r\n\r\n## New Contributors\r\n* @ColebyPearson made their first contribution in https://github.com/huggingface/trl/pull/5869\r\n* @hf-dependantbot-rollout[bot] made their first contribution in https://github.com/huggingface/trl/pull/5910\r\n* @raghulchandramouli made their first contribution in https://github.com/huggingface/trl/pull/5940\r\n* @zafstojano made their first contribution in https://github.com/huggingface/trl/pull/5957\r\n* @DaoyuanLi2816 made their first contribution in https://github.com/huggingface/trl/pull/5964\r\n* @lxk8998 made their first contribution in https://github.com/huggingface/trl/pull/5927\r\n* @biefan made their first contribution in https://github.com/huggingface/trl/pull/5289\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5836\r\n* Add Qwen2.5-VL original and training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5838\r\n* Align KTO with DPO: Simplify metrics from sum/count to direct averages by @albertvillanova in https://github.com/huggingface/trl/pull/5820\r\n* async_grpo don't return on queue.Empty by @AmineDiro in https://github.com/huggingface/trl/pull/5751\r\n* Align KTO with DPO: Refactor forward by @albertvillanova in https://github.com/huggingface/trl/pull/5849\r\n* Per-field invariance tolerances, calibrated by @qgallouedec in https://github.com/huggingface/trl/pull/5844\r\n* Add Qwen2-VL original and training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5839\r\n* Remove NeMo Gym Integration Guide (broken) by @cmunley1 in https://github.com/huggingface/trl/pull/5840\r\n* Align KTO with DPO: Align compute_ref_log_probs by @albertvillanova in https://github.com/huggingface/trl/pull/5852\r\n* Align KTO with DPO: Align precompute_ref_logps by @albertvillanova in https://github.com/huggingface/trl/pull/5850\r\n* Flash Attention capitalization fix by @qgallouedec in https://github.com/huggingface/trl/pull/5855\r\n* 🔒 Gate trainer telemetry on an explicit class-name allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/5851\r\n* Align KTO with DPO: Support remove_unused_columns by @albertvillanova in https://github.com/huggingface/trl/pull/5866\r\n* Raise if precompute_ref_log_probs with vision datasets in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5867\r\n* Support `'None'` as CLI value for `Optional[T]` fields by @qgallouedec in https://github.com/huggingface/trl/pull/5843\r\n* KTO: Replace _get_train_sampler with train_sampling_strategy for transformers >= 5.2.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5864\r\n* Fix: specify encoding=\"utf-8\" when reading .jinja chat templates on Windows by @ColebyPearson in https://github.com/huggingface/trl/pull/5869\r\n* Align KTO with DPO: Align ref log probability names by @albertvillanova in https://github.com/huggingface/trl/pull/5856\r\n* KTO: Support non-sequential train_sampling_strategy for apo_zero_unpaired by @albertvillanova in https://github.com/huggingface/trl/pull/5872\r\n* Align KTO with DPO: Remove null_ref_context by @albertvillanova in https://github.com/huggingface/trl/pull/5875\r\n* Fix ValueError by pinning kernels < 0.15.1 by @albertvillanova in https://github.com/huggingface/trl/pull/5880\r\n* Update vLLM version support to 0.19.0 by @sergiopaniego in https://github.com/huggingface/trl/pull/5879\r\n* Set kernels optional dependency via transformers by @albertvillanova in https://github.com/huggingface/trl/pull/5884\r\n* Support non-lm_head output projections in chunked SFT loss (GPTNeoX) by @qgallouedec in https://github.com/huggingface/trl/pull/5857\r\n* SFTTrainer: merge entropy and accuracy computation to eliminate redundant logits copy by @flutist in https://github.com/huggingface/trl/pull/5897\r\n* Align KTO with DPO: Add disable_gradient_checkpointing to ref model forward passes by @albertvillanova in https://github.com/huggingface/trl/pull/5900\r\n* Align KTO with DPO: Add activation offloading support by @albertvillanova in https://github.com/huggingface/trl/pull/5901\r\n* Align KTO with DPO: Decouple KL dataset generation by @albertvillanova in https://github.com/huggingface/trl/pull/5899\r\n* Fix GRPO use_liger_kernel under DeepSpeed ZeRO-3 by @kashif in https://github.com/huggingface/trl/pull/5891\r\n* Replace custom numpy cache in precompute_ref_logps with native datasets by @albertvillanova in https://github.com/huggingface/trl/pull/5906\r\n* [1/2] refactor: decoupled self distillation trainers (sdpo, sdft, ...) by @LeonEricsson in https://github.com/huggingface/trl/pull/5862\r\n* Align KTO with DPO: Use datasets caching in precompute_ref_logps by @albertvillanova in https://github.com/huggingface/trl/pull/5909\r\n* Support multimodal config in PPO ValueHead by @albertvillanova in https://github.com/huggingface/trl/pull/5907\r\n* Fix priority order in PPO ValueHead and raise ValueError for unsupported config by @albertvillanova in https://github.com/huggingface/trl/pull/5908\r\n* Fix `loss_type=\"chunked_nll\"` under DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/5873\r\n* chore: enable Dependabot weekly GitHub Actions bumps by @hf-dependantbot-rollout[bot] in https://github.com/huggingface/trl/pull/5910\r\n* Exclude None  reward completions  from GRPO/RLOO advantage baseline by @AmineDiro in https://github.com/huggingface/trl/pull/5902\r\n* Don't treat ROCm GPUs as Ampere by @kashif in https://github.com/huggingface/trl/pull/5917\r\n* ci: use GitHub App auth for doc preview comment bot by @sergiopaniego in https://github.com/huggingface/trl/pull/5915\r\n* Bump the actions group with 9 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/5913\r\n* Align KTO with DPO: Replace completion_labels/get_batch_logps with completion_mask by @albertvillanova in https://github.com/huggingface/trl/pull/5914\r\n* chore: update docker-build.yml with version parsing by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5920\r\n* docs: update OpenEnv GitHub org references and package name by @sergiopaniego in https://github.com/huggingface/trl/pull/5919\r\n* Support kernels extra for transformers < 5.1.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5928\r\n* feat: move async rollout worker to separate process by @AmineDiro in https://github.com/huggingface/trl/pull/5749\r\n* Remove redundant .contiguous() calls in DPOTrainer to reduce peak memory by @flutist in https://github.com/huggingface/trl/pull/5926\r\n* docs: update OpenEnv doc URLs from meta-pytorch.org to huggingface.co/docs/openenv by @sergiopaniego in https://github.com/huggingface/trl/pull/5929\r\n* Refresh `sft.json` / `dpo.json` snapshots after transformers `num_items_in_batch fix` by @qgallouedec in https://github.com/huggingface/trl/pull/5845\r\n* NemotronH integration by @qgallouedec in https://github.com/huggingface/trl/pull/5938\r\n* Nemotron 3 Ultra support by @qgallouedec in https://github.com/huggingface/trl/pull/5942\r\n* Enable gradient checkpointing in Nemotron 3 SFT example (transformers>=5.7.0) by @sergiopaniego in https://github.com/huggingface/trl/pull/5944\r\n* Fix SFT padding-free test config by @kashif in https://github.com/huggingface/trl/pull/5923\r\n* Add experimental A2PO trainer (Optimal Advantage Regression) by @raghulchandramouli in https://github.com/huggingface/trl/pull/5940\r\n* Align KTO with DPO: Inline _compute_logps into _compute_loss by @albertvillanova in https://github.com/huggingface/trl/pull/5936\r\n* Fix: Route liger student forward through DDP wrapper in GKD, GOLD, and Distillation trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5934\r\n* Fix backbone access in GRPO by aligning with SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5949\r\n* fix(docs): Remove duplicate sentence in GRPOTrainer docs by @zafstojano in https://github.com/huggingface/trl/pull/5957\r\n* fix(docs): Blockquote math not rendering in RLooTrainer docs by @zafstojano in https://github.com/huggingface/trl/pull/5958\r\n* Remove invalid max_prompt_length argument from GRPO example by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5964\r\n* Bump the actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/5954\r\n* Add Llava-Next training tempalates support with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5959\r\n* fix(docs): correct broken GKD Trainer link in MiniLLM docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5961\r\n* Remove unnecessary explicit `.contiguous()` before `entropy_from_logits` by @qgallouedec in https://github.com/huggingface/trl/pull/5930\r\n* Improve error message when image tokens are truncated by max_length by @lxk8998 in https://github.com/huggingface/trl/pull/5927\r\n* fix(docs): correct broken GKD Trainer link in MiniLLM docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5960\r\n* Cross-tokenizer alignment via byte offsets in GOLD trainer by @kashif in https://github.com/huggingface/trl/pull/5885\r\n* [2/2] refactor: decoupled self distillation trainers; cleanup by @LeonEricsson in https://github.com/huggingface/trl/pull/5883\r\n* Create CI workflow to sync TRL skill with huggingface/skills by @albertvillanova in https://github.com/huggingface/trl/pull/5950\r\n* Support vision datasets for Liger in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5943\r\n* Align trainer train tests by @qgallouedec in https://github.com/huggingface/trl/pull/5963\r\n* Fix malformed ScaleRL paper link in GRPOConfig epsilon_high help by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5972\r\n* Add testing for Olmo3 by @qgallouedec in https://github.com/huggingface/trl/pull/5962\r\n* chore(docs): Highlight the role of KL in RLOO compared to GRPO by @zafstojano in https://github.com/huggingface/trl/pull/5966\r\n* fix(docs): drop duplicate \"a\" in online_dpo_vlm example description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5978\r\n* Fix broken doc links (CONTRIBUTING online DPO paths, async GRPO anchor) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5971\r\n* Align KTO with DPO: Support VLM by @albertvillanova in https://github.com/huggingface/trl/pull/5939\r\n* Fix broken code examples in docs (RLOO syntax, SFTConfig max_length) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5970\r\n* Align KTO with DPO: Improve error message for VLM truncation by @albertvillanova in https://github.com/huggingface/trl/pull/5982\r\n* Align trainers: Remove redundant else branch by @albertvillanova in https://github.com/huggingface/trl/pull/5983\r\n* SDFT/SDPO: live teacher logprobs from the vLLM server by @kashif in https://github.com/huggingface/trl/pull/5989\r\n* docs: sync SDFT/SDPO config docstrings with their fields by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5992\r\n* fix(docs): Fix rendering typos by @zafstojano in https://github.com/huggingface/trl/pull/5991\r\n* Add missing use_liger_kernel guard to SDPO teacher-server validation by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5994\r\n* docs: sync Distillation/GOLD/OnlineDPO config docstrings with their fields by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5995\r\n* feat: Bidirectional masked importance sampling ratio (MIS) for IcePop by @casinca in https://github.com/huggingface/trl/pull/4732\r\n* Simplify agent skills target and default to `.agents` by @albertvillanova in https://github.com/huggingface/trl/pull/5987\r\n* Align KTO with DPO: Remove unused use_dpo_data_collator attribute by @albertvillanova in https://github.com/huggingface/trl/pull/5996\r\n* Align KTO with DPO: Rename kto_loss_fn to liger_loss_fn by @albertvillanova in https://github.com/huggingface/trl/pull/5998\r\n* Align KTO with DPO: Inline kto_loss in _compute_loss by @albertvillanova in https://github.com/huggingface/trl/pull/5999\r\n* Document bnb_4bit_quant_storage and normalize docstring param headers by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5993\r\n* [CI] Check that training chat templates keep the stop token in the loss mask by @kashif in https://github.com/huggingface/trl/pull/5988\r\n* Announce upcoming SFT `loss_type` default change from `'nll'` to `'chunked_nll'` by @qgallouedec in https://github.com/huggingface/trl/pull/5997\r\n* Padding-free invariance test by @qgallouedec in https://github.com/huggingface/trl/pull/5842\r\n* Hide DeepSpeed/FSDP distributed backend boilerplate by @albertvillanova in https://github.com/huggingface/trl/pull/6000\r\n* fix(cli): drop duplicate \"to\" in trl skills install description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6008\r\n* docs: clarify PPO entropy metrics in PPO trainer docs by @biefan in https://github.com/huggingface/trl/pull/5289\r\n* Release: v1.6 by @qgallouedec in https://github.com/huggingface/trl/pull/6009\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.5.0...v1.6.0\r\n","publishedAt":"2026-06-11T22:00:15.000Z","fetchedAt":"2026-06-11T22:00:52.455Z","url":"https://github.com/huggingface/trl/releases/tag/v1.6.0","media":[],"coverageCount":0},{"id":"rel_8_eXU-zwuSdzRk6xMoDTB","version":"v1.5.1","type":"feature","title":"v1.5.1","summary":"Trainer telemetry is now gated on an explicit class-name allowlist, restricting which trainer classes can send telemetry.","titleGenerated":"TRL Fine-tuning v1.5.1 gates trainer telemetry on allowlist","titleShort":"Trainer telemetry now allowlisted","breaking":"unknown","importance":null,"content":"## What's Changed\r\n\r\n* 🔒 Gate trainer telemetry on an explicit class-name allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/5851\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.5.0...v1.5.1","publishedAt":"2026-05-27T15:26:53.000Z","fetchedAt":"2026-05-27T18:01:01.539Z","url":"https://github.com/huggingface/trl/releases/tag/v1.5.1","media":[],"coverageCount":0},{"id":"rel_DGph8QcHlLU1aD0WmSMY3","version":"v1.5.0","type":"feature","title":"v1.5.0","summary":"Fixed an exponential backtracking bug in Qwen3/Qwen3.5/GLM4MoE response parsing that caused GRPOTrainer to hang indefinitely on truncated tool-call blocks, reducing worst-case complexity from O(2ⁿ) to O(n). Also fixed a CUDA memory leak in BNB dequantization buffers and stale state in OffloadActivations. Added training chat templates for Phi-3.5, Qwen3-VL, and Qwen3.5 Think/NoThink, and final logits softcapping support for AsyncGRPOTrainer on models like Gemma 2.","titleGenerated":"TRL v1.5.0 fixes exponential backtracking in response parsing and CUDA memory leak","titleShort":"Response parsing hang fixed; CUDA memory leak patched","breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### Even more training chat templates\r\n\r\nThree more model families gain training-compatible templates with `{% generation %}` markers (so `assistant_only_loss=True` just works):\r\n\r\n- **Phi-3.5** by @DagaBhai in https://github.com/huggingface/trl/pull/5746\r\n- **Qwen3-VL** by @aazizyan in https://github.com/huggingface/trl/pull/5764\r\n- **Qwen3.5 Think / NoThink** by @aazizyan in https://github.com/huggingface/trl/pull/5824\r\n\r\n### Final logits softcapping for async GRPO\r\n\r\nThe chunked LM-head path used by `AsyncGRPOTrainer` now supports models that use `final_logit_softcapping` (notably Gemma 2). `_ChunkedLogProbFunction` applies `logit_scale`, optional tanh-based softcapping, and temperature consistently in both forward and backward — softcapped models are no longer rejected.\r\n\r\nby @mlarnouhet in https://github.com/huggingface/trl/pull/5691\r\n\r\n### KTO ↔ DPO alignment continues\r\n\r\nTwo more cycles closer to KTO graduation:\r\n\r\n* Align `compute_loss` flow by @albertvillanova in https://github.com/huggingface/trl/pull/5810\r\n* Align `_compute_loss_liger` flow by @albertvillanova in https://github.com/huggingface/trl/pull/5816\r\n\r\n### Trainer telemetry (opt-out)\r\n\r\n`_BaseTrainer.__init__` now emits a single anonymous `huggingface_hub.send_telemetry` ping per trainer instantiation, so we can finally see which trainers / model families / distributed backends are actually being used in practice and prioritize accordingly.\r\n\r\nThe payload is intentionally minimal — TRL version, trainer class name, model architecture, PEFT yes/no, distributed backend (`deepspeed`/`fsdp`/`ddp`/`none`), bucketed world size, device type, GPU model when available. No user data, no dataset names, no model paths, no hyperparameter values, never sent in CI / offline / `HF_HUB_DISABLE_TELEMETRY` mode.\r\n\r\nSee [`usage_stats.md`](https://huggingface.co/docs/trl/usage_stats) for what's collected and how to opt out.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5758\r\n\r\n### Other\r\n\r\n* `OpenRewardSpec`: fix omitting task-scoped tools during rollout binding (fixes #5727) by @rycerzes in https://github.com/huggingface/trl/pull/5729\r\n* Add OpenReward example to the list of examples by @sergiopaniego in https://github.com/huggingface/trl/pull/5752\r\n* Add DDP-2 members to invariant test suite by @qgallouedec in https://github.com/huggingface/trl/pull/5736\r\n* Align and simplify the stable training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/5812\r\n* Replace uv installation script with setup action by @qgallouedec in https://github.com/huggingface/trl/pull/5735\r\n\r\n## Fixes\r\n\r\n* **Fix exponential backtracking in qwen3 / qwen3_5 / glm4moe response parsing** — `GRPOTrainer` was hanging indefinitely on truncated `<tool_call>` blocks (a degenerate case that happens naturally when generation hits `max_completion_length` mid-tool-call). Rewrote the regex to be non-backtracking — worst case goes from O(2ⁿ) to O(n). By @xodn348 in https://github.com/huggingface/trl/pull/5798\r\n* **CUDA memory leak: release BNB dequantization buffers & stale state in `OffloadActivations`** — follow-up to v1.4's activation-offloading leak fix. By @butterwecksolutions in https://github.com/huggingface/trl/pull/5730\r\n* Invalidate ZeRO-3 param coordinator trace in `add_hooks` by @roycho96 in https://github.com/huggingface/trl/pull/4693\r\n* Fix nested `vocab_size` for `DistillationTrainer` and `GOLDTrainer` by @Beichen-Ma in https://github.com/huggingface/trl/pull/5592\r\n* Fix MPS support in experimental `empty_cache()` by @jamie-peterson-ml in https://github.com/huggingface/trl/pull/5799\r\n* Fix `metric_for_best_model` for trainer-specific eval metrics by @qgallouedec in https://github.com/huggingface/trl/pull/5811\r\n* Fix `generate_batch`: inference tensors blocking inplace ops in background thread by @albertvillanova in https://github.com/huggingface/trl/pull/5818\r\n* Replace deprecated `torch_dtype` with `dtype` across examples, docs, notebooks, tests, and experimental `distillation` / `gold` trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5717\r\n\r\n## Documentation and Examples\r\n\r\n* docs(grpo): align model to Qwen2.5 and add GRPO OOM tab in quickstart by @xodn348 in https://github.com/huggingface/trl/pull/5740\r\n\r\n## CI\r\n\r\n* Migrate tests to **Qwen3.5 Think/NoThink fixtures** + tiny-model generation scripts by @aazizyan in https://github.com/huggingface/trl/pull/5819 and https://github.com/huggingface/trl/pull/5821\r\n* Align tiny `Glm4MoeForCausalLM` / `Cohere` / `Cohere2` / `Qwen2.5-VL` configs with their reference models by @qgallouedec in https://github.com/huggingface/trl/pull/5638, https://github.com/huggingface/trl/pull/5706, https://github.com/huggingface/trl/pull/5707 and https://github.com/huggingface/trl/pull/5739\r\n* Fix tiny Qwen3-VL `deepstack_visual_indexes` and drop the test skip by @qgallouedec in https://github.com/huggingface/trl/pull/5779\r\n* Fix tiny Qwen2.5-VL `fullatt_block_indexes` out of range for depth=2 by @albertvillanova in https://github.com/huggingface/trl/pull/5805\r\n* Remove non-existent params from tiny Qwen2-VL model by @albertvillanova in https://github.com/huggingface/trl/pull/5795\r\n* Fix vision config `num_heads` key in Qwen VL tiny model scripts by @matdou in https://github.com/huggingface/trl/pull/5792\r\n* Drop unjustified `model.visual.` skip in GRPO/RLOO Qwen2.5-VL tests by @qgallouedec in https://github.com/huggingface/trl/pull/5780\r\n* Make the LLaVA / LLaVA-Next test guard explicit by @qgallouedec in https://github.com/huggingface/trl/pull/5778\r\n* Remove obsolete Gemma 3 vision-head guard from VLM training tests by @qgallouedec in https://github.com/huggingface/trl/pull/5772\r\n* Fix OOM in CI: reduce batch size in VLM SFT / GRPO/RLOO VLM / toolcall tests by @albertvillanova in https://github.com/huggingface/trl/pull/5687, https://github.com/huggingface/trl/pull/5767, https://github.com/huggingface/trl/pull/5801\r\n* Fix OOM in CI by clearing chained exception tracebacks by @albertvillanova in https://github.com/huggingface/trl/pull/5776\r\n* Fix OOM in CI by reducing intermediate_size and image token budget for tiny Gemma 4 by @albertvillanova in https://github.com/huggingface/trl/pull/5760\r\n* Fix CI errors in response parsing for gpt-oss/llama with transformers v5 by @albertvillanova in https://github.com/huggingface/trl/pull/5755\r\n* Fix CI `AttributeError: 'GptOssConfig' object has no attribute 'num_experts'` by @albertvillanova in https://github.com/huggingface/trl/pull/5756\r\n* Fix CI `apply_model_revisions` by removing `_commit_hash` kwarg by @albertvillanova in https://github.com/huggingface/trl/pull/5762\r\n* Fix CI test to avoid skipping `model.visual` params by @albertvillanova in https://github.com/huggingface/trl/pull/5806\r\n* Fix transformers min version for tiny gemma 4 as 5.5.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5763\r\n* Hotfix CI: pin `torch < 2.12.0` (later reverted) by @albertvillanova in https://github.com/huggingface/trl/pull/5769\r\n* Fix catch-all empty string in Makefile `pytest --only-rerun` by @albertvillanova in https://github.com/huggingface/trl/pull/5784\r\n* chore: update `tests_latest.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5733\r\n\r\n## New Contributors\r\n* @hf-security-analysis[bot] made their first contribution in https://github.com/huggingface/trl/pull/5733\r\n* @Beichen-Ma made their first contribution in https://github.com/huggingface/trl/pull/5592\r\n* @DagaBhai made their first contribution in https://github.com/huggingface/trl/pull/5746\r\n* @xodn348 made their first contribution in https://github.com/huggingface/trl/pull/5740\r\n* @mlarnouhet made their first contribution in https://github.com/huggingface/trl/pull/5691\r\n* @matdou made their first contribution in https://github.com/huggingface/trl/pull/5792\r\n* @jamie-peterson-ml made their first contribution in https://github.com/huggingface/trl/pull/5799\r\n* @rycerzes made their first contribution in https://github.com/huggingface/trl/pull/5729\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5734\r\n* chore: update tests_latest.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5733\r\n* fix: CUDA memory leak / release BNB dequantization buffers & stale state in OffloadActivations by @butterwecksolutions in https://github.com/huggingface/trl/pull/5730\r\n* fix: invalidate ZeRO-3 param coordinator trace in add_hooks by @roycho96 in https://github.com/huggingface/trl/pull/4693\r\n* Fix nested vocab_size for DistillationTrainer and GOLDTrainer by @Beichen-Ma in https://github.com/huggingface/trl/pull/5592\r\n* feat:  add Phi-3.5 training chat templates with generation markers by @DagaBhai in https://github.com/huggingface/trl/pull/5746\r\n* docs(grpo): align model to Qwen2.5 and add GRPO OOM tab in quickstart by @xodn348 in https://github.com/huggingface/trl/pull/5740\r\n* `torch_dtype` -> `dtype` by @qgallouedec in https://github.com/huggingface/trl/pull/5717\r\n* Add OpenReward example to the list of examples by @sergiopaniego in https://github.com/huggingface/trl/pull/5752\r\n* Fix CI errors in response parsing for gptoss/llama with transformers v5 by @albertvillanova in https://github.com/huggingface/trl/pull/5755\r\n* Add DDP-2 members to invariant test suite by @qgallouedec in https://github.com/huggingface/trl/pull/5736\r\n* Hotfix CI param not updated AssertionError: Pin torch < 2.12.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5769\r\n* Align tiny-Glm4MoeForCausalLM with GLM-4.5 reference config by @qgallouedec in https://github.com/huggingface/trl/pull/5638\r\n* Align tiny Cohere config with aya-expanse-8b by @qgallouedec in https://github.com/huggingface/trl/pull/5706\r\n* Align tiny Cohere2 config with tiny-aya-earth by @qgallouedec in https://github.com/huggingface/trl/pull/5707\r\n* Fix OOM in CI by reducing intermediate_size and image token budget for tiny Gemma4 by @albertvillanova in https://github.com/huggingface/trl/pull/5760\r\n* Fix CI AttributeError: 'GptOssConfig' object has no attribute 'num_experts' by @albertvillanova in https://github.com/huggingface/trl/pull/5756\r\n* Fix CI apply_model_revisions by removing _commit_hash kwarg by @albertvillanova in https://github.com/huggingface/trl/pull/5762\r\n* Remove obsolete Gemma3 vision-head guard from VLM training tests by @qgallouedec in https://github.com/huggingface/trl/pull/5772\r\n* Replace uv installation script with setup action by @qgallouedec in https://github.com/huggingface/trl/pull/5735\r\n* Fix OOM in CI by clearing chained exception tracebacks by @albertvillanova in https://github.com/huggingface/trl/pull/5776\r\n* Fix transformers min version for tiny gemma4 as 5.5.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5763\r\n* Final logits softcapping support for async GRPO Trainer by @mlarnouhet in https://github.com/huggingface/trl/pull/5691\r\n* Fix vision config num_heads key in Qwen VL tiny model scripts and revert torch pin by @matdou in https://github.com/huggingface/trl/pull/5792\r\n* Drop unjustified `model.visual.` skip in GRPO / RLOO Qwen2.5-VL tests by @qgallouedec in https://github.com/huggingface/trl/pull/5780\r\n* Fix OOM in CI by reducing batch size and sequence length for toolcall tests by @albertvillanova in https://github.com/huggingface/trl/pull/5801\r\n* Fix exponential backtracking in qwen3 / qwen3_5 / glm4moe response parsing by @xodn348 in https://github.com/huggingface/trl/pull/5798\r\n* Add telemetry to trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5758\r\n* Add Qwen3-VL training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5764\r\n* Align tiny Qwen2.5-VL with Qwen/Qwen2.5-VL-3B-Instruct by @qgallouedec in https://github.com/huggingface/trl/pull/5739\r\n* Fix tiny Qwen3-VL `deepstack_visual_indexes` and drop the test skip by @qgallouedec in https://github.com/huggingface/trl/pull/5779\r\n* Fix OOM in CI by reducing batch size in GRPO/RLOO VLM tests by @albertvillanova in https://github.com/huggingface/trl/pull/5767\r\n* Fix catch-all empty string in Makefile pytest --only-rerun by @albertvillanova in https://github.com/huggingface/trl/pull/5784\r\n* Remove non-existent params from tiny Qwen2-VL model by @albertvillanova in https://github.com/huggingface/trl/pull/5795\r\n* Fix tiny Qwen2.5-VL fullatt_block_indexes out of range for depth=2 by @albertvillanova in https://github.com/huggingface/trl/pull/5805\r\n* Make the LLaVA / LLaVA-Next test guard explicit by @qgallouedec in https://github.com/huggingface/trl/pull/5778\r\n* Fix MPS support in experimental empty_cache() by @jamie-peterson-ml in https://github.com/huggingface/trl/pull/5799\r\n* Fix CI test to avoid skipping model.visual params by @albertvillanova in https://github.com/huggingface/trl/pull/5806\r\n* Align KTO with DPO: Align compute_loss flow by @albertvillanova in https://github.com/huggingface/trl/pull/5810\r\n* Fix generate_batch: inference tensors block inplace ops in background thread by @albertvillanova in https://github.com/huggingface/trl/pull/5818\r\n* Fix `metric_for_best_model` for trainer-specific eval metrics by @qgallouedec in https://github.com/huggingface/trl/pull/5811\r\n* Align and simplify the stable training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/5812\r\n* Align KTO with DPO: Align _compute_loss_liger flow by @albertvillanova in https://github.com/huggingface/trl/pull/5816\r\n* Add tiny Qwen3.5 Think/NoThink fixture generation scripts by @aazizyan in https://github.com/huggingface/trl/pull/5819\r\n* Migrate tests to Qwen3.5 Think/NoThink fixtures by @aazizyan in https://github.com/huggingface/trl/pull/5821\r\n* Fix `OpenRewardSpec` omitting task‑scoped tools during rollout binding (fixes #5727) by @rycerzes in https://github.com/huggingface/trl/pull/5729\r\n* Add Qwen3.5 Think/NoThink training chat templates with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5824\r\n* Release: v1.5 by @qgallouedec in https://github.com/huggingface/trl/pull/5835\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.4.0...v1.5.0\r\n","publishedAt":"2026-05-25T15:33:19.000Z","fetchedAt":"2026-05-25T16:01:17.398Z","url":"https://github.com/huggingface/trl/releases/tag/v1.5.0","media":[],"coverageCount":0}],"pagination":{"nextCursor":"2026-05-25T15:33:19.000Z|2026-05-25T16:01:17.398Z|rel_DGph8QcHlLU1aD0WmSMY3","limit":20},"summaries":{"rolling":{"windowDays":90,"summary":"TRL moved toward production-grade reinforcement learning with v1.0.0, marking a transition from prototype frameworks to deployable training systems. Asynchronous GRPO decoupled generation from gradient updates by offloading rollouts to external vLLM servers, eliminating idle GPU time during training. VESPO (Variational Sequence-Level Soft Policy Optimization) replaced heuristic token-level clipping with a principled variational framework that derives smooth importance weighting, addressing training instability from policy staleness and asynchronous updates. Earlier releases hardened the foundation with async reward functions parallelized across GRPO and RLOO, vLLM 0.12.0 compatibility, tool-calling support for agent training, and memory optimizations like forward-masked logits that cut VRAM usage by up to 50 percent during forward passes.","releaseCount":8,"generatedAt":"2026-04-07T17:28:27.620Z"},"monthly":[{"year":2026,"month":3,"summary":"TRL hit v1.0 by shipping asynchronous GRPO, which offloads generation to external vLLM servers to parallelize rollouts and training while eliminating GPU idle time. The release also introduced VESPO, a variational framework that replaces heuristic token-level clipping with a principled Gamma weighting function to stabilize off-policy training. Earlier in the month, v0.29.1 fixed multimodal token handling across SFT/GRPO/RLOO and decoupled rollout dispatch from the vLLM backend to improve compatibility across versions.","releaseCount":3,"generatedAt":"2026-04-07T17:28:30.516Z"}]}}