{"id":"prod_pCEM29no4janpSW6GK3TC","name":"Fine-tuning","slug":"fine-tuning","orgId":"org_GDdYeYynEgCEBNBwy-m6s","url":null,"description":"Libraries for efficient model fine-tuning and alignment","category":"ai","kind":"sdk","avatarUrl":null,"createdAt":"2026-04-10T16:06:59.610Z","embeddedAt":"2026-04-15T16:19:40.805Z","deletedAt":null,"sources":[{"id":"src_p8uVHjEHY1lSK3poHvlWI","slug":"peft","name":"PEFT","type":"github","url":"https://github.com/huggingface/peft","metadata":"{\"evaluatedMethod\":\"github\",\"evaluatedAt\":\"2026-04-07T17:19:21.796Z\",\"changelogDetectedAt\":\"2026-04-07T17:29:08.155Z\",\"wellKnownSweptAt\":\"2026-10-31T06:00:36.365Z\"}","kind":"sdk"},{"id":"src_g_3xbu2oRX52tM8mwXJzq","slug":"trl","name":"TRL","type":"github","url":"https://github.com/huggingface/trl","metadata":"{\"evaluatedMethod\":\"github\",\"evaluatedAt\":\"2026-04-07T17:19:15.726Z\",\"changelogDetectedAt\":\"2026-04-07T17:28:23.856Z\",\"wellKnownSweptAt\":\"2026-10-31T06:00:36.365Z\"}","kind":"sdk"}],"tags":["fine-tuning","python","rlhf"],"aliases":[],"notice":null,"releases":[{"id":"rel_GDI4J5YyX8kJnXdY8-y6k","version":"v1.17.1","type":"feature","title":"v1.17.1","summary":"Fixed LFM2.5 tool-call parsing so a parenthesis inside an argument no longer closes the call prematurely.","titleGenerated":"TRL v1.17.1 keeps parentheses in LFM2.5 tool-call arguments from closing calls","titleShort":"LFM2.5 tool calls survive parentheses in arguments","breaking":"none","importance":1,"content":"## What's Changed\r\n\r\n* Keep a parenthesis inside an LFM2.5 tool-call argument from closing the call by @albertvillanova in https://github.com/huggingface/trl/pull/7942\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.17.0...v1.17.1","publishedAt":"2026-10-30T14:15:28.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.17.1","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":251,"contentTokens":75,"composition":{"bugs":1,"features":0,"enhancements":0}},{"id":"rel_-8FUEc-qSLV5FkBKAOdLq","version":"v1.17.0","type":"feature","title":"v1.17.0","summary":"Adds a LoRA SFT example for mistralai/Mistral-Large-4-1T-A52B that fine-tunes the 1T-parameter model on a single 8×B300 node on top of its FP8 weights, requiring transformers>=5.22.0, peft>=0.21.0 and accelerate>=1.15.0. GRPO and AsyncGRPO now renormalize log-probs over the top-k/top-p candidate set, which raises an error when top_p<1.0 or min_p>0.0 is set without top_k>0, and score centering is now the default AsyncGRPO objective.","titleGenerated":"TRL v1.17.0 adds Mistral Large 4 single-node fine-tuning and unbiased importance sampling","titleShort":"Mistral Large 4 fine-tunes on one node; GRPO sampling unbiased","breaking":"major","importance":4,"content":"<!-- TODO: banner image for Mistral Large 4 -->\r\n\r\nhttps://github.com/user-attachments/assets/5aa86ec7-6444-4bed-bfdc-0445e99bea0d\r\n\r\n## Features\r\n\r\n### 🔥 Mistral Large 4: fine-tune a 1T-parameter model on one node\r\n\r\nTRL now supports [`mistralai/Mistral-Large-4-1T-A52B`](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B): 1T parameters, 52B active. The new [LoRA example](https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4) fine-tunes it with `SFTTrainer` **on a single 8×B300 node, on top of its FP8 weights**: the experts are split 8 ways with token dispatch, everything else is sharded across the 8 GPUs at load time by transformers' `DistributedConfig`, and LoRA trains attention and the dense and shared-expert MLPs.\r\n\r\nMeasured: **32.7k tokens per step in ~8 s, 142 GiB per GPU**, with packing and assistant-only loss on reasoning conversations.\r\n\r\n```python\r\ntraining_args = SFTConfig(\r\n    model_init_kwargs={\r\n        \"dtype\": \"auto\",  # keep the FP8 weights\r\n        \"distributed_config\": DistributedConfig(fsdp_size=8, ep_size=8),\r\n    },\r\n    packing=True,\r\n    assistant_only_loss=True,\r\n    ...\r\n)\r\ntrainer = SFTTrainer(model=\"mistralai/Mistral-Large-4-1T-A52B\", args=training_args, peft_config=LoraConfig(...), ...)\r\n```\r\n\r\n`assistant_only_loss=True` works out of the box: TRL ships a training chat template that matches the released checkpoint's, including its `reasoning_effort` levels (`low`, `medium`, `high`). Needs `transformers>=5.22.0`, `peft>=0.21.0` and `accelerate>=1.15.0`.\r\n\r\n* Add Mistral Large 4 by @qgallouedec in https://github.com/huggingface/trl/pull/8017\r\n* Sync the Mistral Large 4 chat template with the released checkpoint by @albertvillanova in https://github.com/huggingface/trl/pull/8018\r\n\r\n### 🎯 Unbiased importance sampling under top-k, top-p and min-p\r\n\r\nvLLM normalizes its log-probabilities over the tokens left after `top_k`, `top_p` and `min_p` filtering, while the trainer normalized over the whole vocabulary. Every sampled token's log-ratio was off by the log of the probability mass the filter kept, so the vLLM importance sampling ratio was biased towards 0 (around 1e-4 instead of 1 with `top_p=0.9`). The trainers now normalize over the candidate set each token was sampled from, following Keep Sampling Mask from [DeepSeek-V3.2](https://huggingface.co/papers/2512.02556).\r\n\r\n* AsyncGRPO: renormalize log-probs over the top-k/top-p candidate set by @AmineDiro in https://github.com/huggingface/trl/pull/7627\r\n* GRPO: renormalize log-probs over the top-k/top-p candidate set (vLLM colocate mode, vLLM >= 0.28; server mode will follow) by @albertvillanova in https://github.com/huggingface/trl/pull/7866\r\n* Require `top_k` with `min_p` in AsyncGRPO so the importance sampling ratio stays unbiased by @albertvillanova in https://github.com/huggingface/trl/pull/7937\r\n* Track the support maximum in the chunked log-prob kernel, so a far-below-max candidate set no longer gives a log-probability of 0 and non-finite gradients, by @kashif in https://github.com/huggingface/trl/pull/7983\r\n\r\n**💥 What this changes for you.** vLLM bounds the candidate set it returns with `top_k`, so with vLLM and the importance sampling correction on (the default), `GRPOConfig` now raises when `top_p < 1.0` or `min_p > 0.0` is set without `top_k > 0`. Set `top_k` (e.g. `top_k=512`), or `vllm_importance_sampling_correction=False`.\r\n\r\n### ⚖️ Score centering is the default AsyncGRPO objective\r\n\r\nAsyncGRPO's old log-probs are the vLLM sampler's, so its ratio mixes staleness with the numerical mismatch between the server and the trainer. With an FP8 sampler and a bf16 trainer, the clipped objective collapses: reward goes from 0.60 to 0.34 (FP8 weights) or 0.10 (FP8 weights + KV cache) by step 300, against 0.71 with a bf16 sampler.\r\n\r\n[Score centering](https://huggingface.co/papers/2609.20807) with masked importance sampling is now the default (`score_centering=True`, `importance_sampling_min=0.5`, `importance_sampling_max=5.0`). With it, the FP8 sampler trains exactly like the bf16 one (0.72). It costs 15–20% of generation throughput.\r\n\r\n**💥 What this changes for you.** AsyncGRPO runs now train with a different objective by default, and tokens whose ratio to the sampler falls outside `[0.5, 5.0]` are dropped. Set `score_centering=False` to keep the clipped objective.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/7760\r\n\r\n### 🚀 Faster AsyncGRPO rollouts and packing\r\n\r\n* **Adaptive in-flight cap from the vLLM load**: the rollout worker follows the server's KV cache usage instead of admitting a fixed `max_inflight_tasks`, so requests stop queueing and being preempted on the server. No new config: everything is read from the server. By @AmineDiro in https://github.com/huggingface/trl/pull/7610\r\n* **Filter zero-advantage groups** (DAPO's dynamic sampling): groups whose rollouts all got the same reward, or none of which could be scored, no longer cost a forward and backward. By @AmineDiro in https://github.com/huggingface/trl/pull/7609\r\n* **Pack the sampling mask ragged and pad it on the device**: `TreeRow.pack` goes from 250 s to 31 s on rank 0 over a 15-step run, and from 1040 ms to 66 ms on a 28k-token multi-turn row. By @AmineDiro in https://github.com/huggingface/trl/pull/7798\r\n* Keep filling a micro-batch when another row has free room by @clementtisseau in https://github.com/huggingface/trl/pull/7793\r\n* Use FlashAttention-4 on Blackwell GPUs in async GRPO and async distillation by @lewtun in https://github.com/huggingface/trl/pull/7792\r\n\r\n### 🌳 Tree attention for linear-attention hybrids\r\n\r\nTree packing (v1.16) now supports the Gated DeltaNet hybrids: Qwen3.5, Qwen3.6, Qwen3-Next and OLMo-Hybrid. Since v1.16 the trainer refused them at init; before it, sequence packing accepted them but was silently wrong on them. The causal convolution and the delta-rule recurrence now follow the tree instead of the row, through FLA's chunked kernels.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/7686\r\n\r\n### 🎚️ Precise chunked log-probabilities\r\n\r\nThe chunked log-probability and KL/JSD heads get the two precision fixes from [A Fix for Softmax and Cross-Entropy](https://huggingface.co/papers/2610.31642): confident targets keep their small gradients, and matching teachers produce zero gradients. [Entropy-preserving reinforcement learning](https://huggingface.co/papers/2603.11682) (Appendix B.4) reports the same high-confidence policy-gradient issue.\r\n\r\n* Fix chunked loss precision by @kashif in https://github.com/huggingface/trl/pull/7882\r\n* Fix the fused log-prob/entropy backward on strided inputs (shifted views from SFT/DPO) by @winglian in https://github.com/huggingface/trl/pull/7890\r\n\r\n### 🔀 X-Token cross-tokenizer distillation in GOLD\r\n\r\n`GOLDTrainer` gets off-policy cross-tokenizer distillation based on [X-Token](https://huggingface.co/papers/2605.21699): set `lmbda=0.0` and `xtoken_loss_type=\"p_kl\"` or `\"h_kl\"` with a precomputed projection matrix.\r\n\r\n* Add X-Token cross-tokenizer KD to GOLD trainer by @kashif in https://github.com/huggingface/trl/pull/6126\r\n* Support byte-fallback tokenizers (Gemma) in GOLD byte-offset alignment by @kashif in https://github.com/huggingface/trl/pull/8010\r\n\r\n### 🟥 Training on AMD GPUs\r\n\r\nTRL's trainers now run on AMD Instinct GPUs through PyTorch's ROCm build, with a [guide](https://huggingface.co/docs/trl/amd_gpus) and an AMD CI job that runs the test suite on every PR.\r\n\r\n* Document training on AMD GPUs by @qgallouedec in https://github.com/huggingface/trl/pull/7844\r\n* Add AMD/ROCm CI by @kashif in https://github.com/huggingface/trl/pull/5918\r\n\r\n### 🧩 More models\r\n\r\n* **Beam**: tiny model, SFT/DPO/GRPO/RLOO tests, GRPO tool calling, and a LoRA SFT example on one 8×B300 node. Needs `transformers>=5.20.1`. By @qgallouedec in https://github.com/huggingface/trl/pull/7629\r\n* **Mistral Medium 3.5**: training chat template for assistant-only loss by @qgallouedec in https://github.com/huggingface/trl/pull/7940\r\n* **LFM2.5-24B-A2B** (and LFM2.5-2.6B's current template) for tool calling by @qgallouedec in https://github.com/huggingface/trl/pull/7753\r\n* **LFM2.5-1.2B-Instruct and LFM2.5-VL-450M** for tool calling by @albertvillanova in https://github.com/huggingface/trl/pull/7909, https://github.com/huggingface/trl/pull/7910 and https://github.com/huggingface/trl/pull/7911\r\n* **TRL's own Qwen3 training template** for response parsing, so a tokenizer saved by assistant-only SFT works in GRPO, by @edbeeching in https://github.com/huggingface/trl/pull/8007\r\n\r\n### Other\r\n\r\n* Accept instantiated models in `AsyncGRPOTrainer` by @kashif in https://github.com/huggingface/trl/pull/6676, and port instantiated models, MoE aux loss and the injected-worker LoRA name to `AsyncDistillationTrainer` by @kashif in https://github.com/huggingface/trl/pull/7984\r\n* Save AsyncGRPO and AsyncDistillation checkpoints in the mixed-precision dtype instead of fp32 by @AmineDiro in https://github.com/huggingface/trl/pull/7939\r\n* Back off between unscorable harness sessions and stop the worker when they persist by @albertvillanova in https://github.com/huggingface/trl/pull/7992\r\n* Async Harbor GRPO: PEFT support and fixes by @younesbelkada in https://github.com/huggingface/trl/pull/7849 and https://github.com/huggingface/trl/pull/7948\r\n* Write the logged completions to parquet as JSON by @qgallouedec in https://github.com/huggingface/trl/pull/7696\r\n* Add support for vLLM 0.32.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7799, and allow its patch releases by @albertvillanova in https://github.com/huggingface/trl/pull/7850\r\n\r\n### Removals and requirement bumps\r\n\r\n* **Remove the experimental TPO trainer** by @qgallouedec in https://github.com/huggingface/trl/pull/7807\r\n* **Require `top_k` with `top_p < 1.0` or `min_p > 0.0` in GRPO with vLLM importance sampling** (see above) by @albertvillanova in https://github.com/huggingface/trl/pull/7866\r\n* Require `torch>=2.6.0` and drop the pre-2.6 FSDP2 and accelerator fallbacks by @albertvillanova in https://github.com/huggingface/trl/pull/7642\r\n* Require `accelerate>=1.11.0` and drop the FSDP2 ignored-modules fallback by @albertvillanova in https://github.com/huggingface/trl/pull/7797\r\n* Require `peft>=0.17.0` and drop the `target_parameters` guards by @albertvillanova in https://github.com/huggingface/trl/pull/7640\r\n* Raise the bitsandbytes floor to 0.45.2 by @albertvillanova in https://github.com/huggingface/trl/pull/7864\r\n* Raise the math-verify floor to 0.8.0 by @albertvillanova in https://github.com/huggingface/trl/pull/7870\r\n* Drop liger-kernel from the dev dependencies, the Docker images and the tests by @qgallouedec in https://github.com/huggingface/trl/pull/7843\r\n\r\n## Fixes\r\n\r\n* Prevent VLM policies from generating the image placeholder token with vLLM (#7698), in GOLD (#7699) and in Online DPO (#7708), all by @albertvillanova\r\n* Place vLLM image features at the model config's image token by @albertvillanova in https://github.com/huggingface/trl/pull/7773\r\n* Pass `mm_token_type_ids` to the model in Online DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7859\r\n* Cast FSDP2 weight-transfer payloads to the vLLM manifest dtype in AsyncGRPO and AsyncDistillation by @sfc-gh-truwase in https://github.com/huggingface/trl/pull/7131\r\n* Load the tokenizer for models that take no images by @qgallouedec in https://github.com/huggingface/trl/pull/7774\r\n* Keep a single LoRA target module from the CLI or a config file as a list by @albertvillanova in https://github.com/huggingface/trl/pull/7796\r\n* Read and write text files as UTF-8 by @albertvillanova in https://github.com/huggingface/trl/pull/7755\r\n\r\n### Clearer errors\r\n\r\nFailure modes made explicit instead of failing silently or deep in the stack, all by @albertvillanova:\r\n\r\n* Raise when LoRA would skip the routed experts of MoE models on transformers v5 with peft<0.19 (#7769), and warn when it skips their dense and shared expert layers with peft 0.19 to 0.21.2 (#7771)\r\n* Reject a `PeftModel` passed as `ref_model` in DPO and KTO (#7808)\r\n* Reject prompt learning in distillation before wrapping the model (#7718)\r\n\r\n## Documentation\r\n\r\n* Document the math-verify install for `accuracy_reward` by @albertvillanova in https://github.com/huggingface/trl/pull/7875\r\n* Fix the default IS mode in the `GRPOConfig` docstring (#7751) and in the GRPO vLLM tip (#7800) by @sergiopaniego\r\n* Update OpenEnv links to openenv/OpenEnv by @sergiopaniego in https://github.com/huggingface/trl/pull/7970\r\n* Drop ORPO and CPO examples whose prompt alone fills `max_length` by @qgallouedec in https://github.com/huggingface/trl/pull/7913\r\n* Fix the license header of `tree_gated_delta_rule.py` by @albertvillanova in https://github.com/huggingface/trl/pull/7938\r\n\r\n## CI\r\n\r\n**AMD**, all by @albertvillanova: relax the confident-target tolerances for AMD (#7981, #8005)\r\n\r\n**Minimum versions**, all by @albertvillanova: run `tests_min_versions` on Python 3.11 and pin every dependency at its floor (#7741), including bitsandbytes (#7746), liger-kernel (#7846) and deepspeed (#7855); document it (#7745) and when to add, drop and change the default Python version (#7743)\r\n\r\n**Docker images**, all by @albertvillanova: install Python with uv (#7738), use Python 3.13 (#7739), install kernels through the TRL `kernels` extra (#7815), remove redundant installs (#7813), skip the uv cache in the release image (#7817), require vLLM >= 0.31.0 (#7820) and explain the floor (#7862), bump uv and upgrade ca-certificates (#7880)\r\n\r\n**Real objects, not mocks**, all by @albertvillanova: real generation outputs in the GOLD and GKD tests (#7902), real config files in the TrlParser tests (#7903), tiny models instead of full-size checkpoints in the GOLD fixtures (#7904), with a new tiny SmolLM3 (#7905, #7906), a real trainer in the IW-OPD tests (#7976), real `DistillationLogits` in the SDFT tests (#7977), and real objects in the OpenEnv tests (#7985, #7986, #7989, #7990)\r\n\r\n**MoE LoRA coverage**, all by @albertvillanova: test that LoRA on an MoE model trains its routed experts (#7636) and its dense and shared expert layers (#7768), and the error and warning for PEFT versions that skip expert layers (#7860)\r\n\r\n**Other**, all by @albertvillanova: run the test workflows on push to main only (#7848), use the latest Python patch release in pip workflows (#7881), test the math-verify rewards in CI (#7928), check the LFM2.5-2.6B chat template against the Hub (#7907), skip the async GRPO tests on transformers < 5.2.0 (#7750), skip the Liger fixture and tests without an accelerator (#7868), import trl first in the tests so peft 0.17 imports with transformers v5 (#7873), cap the KV cache in the continuous batching tests (#7825), fix a port race in the OpenReward fixture (#7886), pass `enable_thinking` only to templates that use it (#7898), UTF-8 in tests (#7764), plus GOLD SmolVLM test handling against transformers 5.20.1 (#7728, #7791, #7810, #7929)\r\n\r\n## New Contributors\r\n\r\n* @sfc-gh-truwase made their first contribution in https://github.com/huggingface/trl/pull/7131\r\n* @clementtisseau made their first contribution in https://github.com/huggingface/trl/pull/7793\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.16.0...v1.17.0","publishedAt":"2026-10-29T15:51:15.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.17.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":15307,"contentTokens":4419,"composition":{"bugs":4,"features":7,"enhancements":5}},{"id":"rel_sYn3OMOJV8sislSKeJ_20","version":"v1.16.0","type":"feature","title":"v1.16.0","summary":"TRL now trains on AWS Trainium with a new guide and pad_to_multiple_of defaulting to \"auto\" (128 on Neuron, None elsewhere), and AsyncGRPO packs micro-batch rows as a prefix tree, forwarding shared-prompt tokens once (up to 3.1x faster with peak memory dropping from 35.2 to 13.5 GiB). Adds Muse Spark Open support, AMD GPUs in async trainers, and JSON-schema-typed tool-call arguments.","titleGenerated":"TRL v1.16.0 brings AWS Trainium training and tree attention for AsyncGRPO","titleShort":"AWS Trainium support; AsyncGRPO gets tree attention","breaking":"major","importance":4,"content":"## Features\r\n\r\n### 🛠️ TRL now trains on AWS Trainium\r\n\r\n<img width=\"1672\" height=\"941\" alt=\"Neon Warehouse_ AWS and Trainium Support\" src=\"https://github.com/user-attachments/assets/dcb6e2a1-0f7c-4635-819b-a1ec6f5e143b\" />\r\n\r\nThe fused LM head from v1.15 is what made Neuron tractable; the remaining gaps are closed here. There is a [guide](https://huggingface.co/docs/trl/neuron) to go with it.\r\n\r\nNeuron recompiles once per input shape, so an unpadded run stalls about a minute on every new sequence length. `pad_to_multiple_of` now defaults to `\"auto\"`: 128 on Neuron, `None` elsewhere, so nothing changes on GPU. Trainium2, Qwen3-0.6B SFT, 20 steps: **130 s with the default against 260 s without**.\r\n\r\n* Add a guide for training on AWS Trainium by @qgallouedec in https://github.com/huggingface/trl/pull/7729\r\n* Default `pad_to_multiple_of` to `\"auto\"` by @qgallouedec in https://github.com/huggingface/trl/pull/7724\r\n* Run the fused LM head and distillation kernels on devices without Triton by @qgallouedec in https://github.com/huggingface/trl/pull/7669\r\n* Report the Neuron device in `trl env` and telemetry by @qgallouedec in https://github.com/huggingface/trl/pull/7672\r\n* Support any accelerator in activation offloading by @qgallouedec in https://github.com/huggingface/trl/pull/7670\r\n\r\n### 🌳 Tree attention for AsyncGRPO\r\n\r\nAsyncGRPO now packs every micro-batch row as a **prefix tree** instead of a flat sequence. Multi-turn and multi-generation rollouts repeat a lot of prefix: on a shared prompt, those tokens used to be forwarded once per row. Tree packing forwards them once.\r\n\r\n```text\r\nrollout 0:  [ prompt ] [ completion A ]              prompt ──┬── completion A ── later turn\r\nrollout 1:  [ prompt ] [ completion B ]        =>             └── completion B\r\nrollout 2:  [ prompt ] [ completion A ] [ … ]\r\n```\r\n\r\nChains without a fork collapse into segments, and each segment's queries attend to the K/V gathered along its root-to-segment path in a single FA3 varlen call. No mask tensor: every layer except attention runs once per unique token, and only K/V is duplicated along shared prefixes.\r\n\r\nQwen3-1.7B, 32K tokens per rollout, forward + backward:\r\n\r\n| layout | packing ratio | sequence packing | tree packing | peak GiB |\r\n| --- | --- | --- | --- | --- |\r\n| shared prompt | 1.00 | 8677 ms | 8737 ms (0.99×) | 35.2 → 35.2 |\r\n| shared prompt | 2.29 | 8266 ms | **4341 ms (1.90×)** | 35.2 → 18.0 |\r\n| shared prompt | 3.37 | 8179 ms | **2755 ms (2.97×)** | 35.2 → 13.6 |\r\n| nested fork | 1.60 | 8557 ms | 6217 ms (1.38×) | 35.2 → 23.8 |\r\n| nested fork | 3.56 | 8172 ms | **2635 ms (3.10×)** | 35.2 → 13.5 |\r\n\r\nThere is no `packing` option to choose: with nothing shared a tree row is exactly the old sequence packing, at a ~1% cost.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/7328\r\n\r\n### Muse Spark Open\r\n\r\nSupport for [`meta-models/Muse-Spark-Open-816B-A42B`](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B): the tiny-model script, coverage in the VLM tests (SFT, DPO, KTO, chat-template utils, fused LM head), and a LoRA SFT example on 8 nodes of 8×H100 with `DistributedConfig(fsdp_size=64, ep_size=8)`, token dispatch and packing. Measured at **14.2 s/step, 131k tokens/step, 40.2 GB per GPU**.\r\n\r\nNeeds `transformers>=5.20.0`; the adapter save additionally needs [transformers#49500](https://github.com/huggingface/transformers/pull/49500) with peft 0.21.2.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/7536\r\n\r\n### AMD GPUs in the async trainers\r\n\r\n* Support AMD GPUs in the async GRPO and async distillation trainers by @Abdennacer-Badaoui in https://github.com/huggingface/trl/pull/7689\r\n* Hide the GPU from the async rollout worker on ROCm by @Abdennacer-Badaoui in https://github.com/huggingface/trl/pull/7685\r\n\r\n### Tool-call arguments typed by the tool's JSON schema\r\n\r\n`parse_response` now receives the tools, so tool-call arguments come back typed from the tool's own schema instead of whatever the raw parse produced.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/7650\r\n\r\n### Other\r\n\r\n* Async GRPO tool metrics by @AmineDiro in https://github.com/huggingface/trl/pull/7624\r\n* Store async GRPO and distillation rollout sample tokens as numpy arrays by @AmineDiro in https://github.com/huggingface/trl/pull/7626\r\n* Save the final step when async trainers stop at `num_train_epochs` by @lewtun in https://github.com/huggingface/trl/pull/7628\r\n* Pass sequence boundaries from the padding-free collator by @qgallouedec in https://github.com/huggingface/trl/pull/7599\r\n* Build the reference model with the policy's dtype and attention implementation by @qgallouedec in https://github.com/huggingface/trl/pull/7597, and the SDFT/SDPO teacher with the student's by @qgallouedec in https://github.com/huggingface/trl/pull/7616\r\n* Find unused parameters under DDP for vision-language models by @qgallouedec in https://github.com/huggingface/trl/pull/7598\r\n* Pad the generation prompts to `pad_to_multiple_of` by @qgallouedec in https://github.com/huggingface/trl/pull/7671\r\n* Drop the client-side observe loop from the CARLA examples by @sergiopaniego in https://github.com/huggingface/trl/pull/7721\r\n* Use `None` for unscored `env.reward` in OpenReward by @albertvillanova in https://github.com/huggingface/trl/pull/7541\r\n\r\n### Removals and requirement bumps\r\n\r\n* **Remove the experimental SSD trainer** by @qgallouedec in https://github.com/huggingface/trl/pull/7706\r\n* **Refuse `nn.DataParallel` in `RewardTrainer`** by @albertvillanova in https://github.com/huggingface/trl/pull/7571\r\n* Drop vLLM 0.21.0 support by @qgallouedec in https://github.com/huggingface/trl/pull/7700\r\n* Require `accelerate>=1.10.1` and drop the `ParallelismConfig` shim by @albertvillanova in https://github.com/huggingface/trl/pull/7643\r\n* Declare a minimum bitsandbytes version by @albertvillanova in https://github.com/huggingface/trl/pull/7641\r\n* Declare the direct dependencies TRL gets transitively by @albertvillanova in https://github.com/huggingface/trl/pull/7713\r\n\r\n## Fixes\r\n\r\n* **Fix silently wrong `torch.compile` results in the Triton loss kernels on torch < 2.11.** `torch_compile=True` trained with `loss=0.0` and `grad_norm=nan` at every step. Inductor passes scalar kernel args as `constexpr` before torch 2.11, the `.to(tl.float32)` casts raised on those, and Inductor then assumed every input was mutated and miscompiled. By @albertvillanova in https://github.com/huggingface/trl/pull/7688\r\n* **Fix FSDP2 vLLM colocate weight sync for tied-embedding models** by @Abdennacer-Badaoui in https://github.com/huggingface/trl/pull/7647\r\n* **Prevent VLM policies from generating the image placeholder token** in GRPO, RLOO and Distillation by @albertvillanova in https://github.com/huggingface/trl/pull/7695\r\n* **Fix Online DPO VLM training** when the prompt exceeds `max_length` (https://github.com/huggingface/trl/pull/7703) and for processors with 5D pixel values (https://github.com/huggingface/trl/pull/7704), both by @albertvillanova\r\n* **Fix activation offloading on torch < 2.4** by @albertvillanova in https://github.com/huggingface/trl/pull/7748\r\n* Fix `torch.compile(fullgraph=True)` of `selective_log_softmax_and_entropy` on torch 2.6 by @albertvillanova in https://github.com/huggingface/trl/pull/7719\r\n* Keep the Triton loss kernels in fp32 under `torch.compile` by @qgallouedec in https://github.com/huggingface/trl/pull/7600\r\n* Read `is_triton_available` once in the fused LM head kernels so Dynamo does not warn by @qgallouedec in https://github.com/huggingface/trl/pull/7731\r\n* Fix Distillation for models split across devices by @albertvillanova in https://github.com/huggingface/trl/pull/7573\r\n* Restore the gradient checkpointing kwargs after generation by @qgallouedec in https://github.com/huggingface/trl/pull/7668\r\n* Drop the FlashAttention warnings for padding-free and packing in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/7620\r\n* Restore the distributed env vars that vLLM colocate tests leave behind by @Abdennacer-Badaoui in https://github.com/huggingface/trl/pull/7723\r\n\r\n### Clearer errors\r\n\r\nA sweep making failure modes explicit instead of crashing deep in the stack, all by @albertvillanova:\r\n\r\n* Raise a clear error when the fused LM head is used without Triton (#7639)\r\n* Raise a clear error for prefix tuning on peft versions that cannot support it (#7646)\r\n* Raise a clear error when the Distillation student and teacher heads are on different devices (#7605)\r\n* Reject a text-only teacher in GOLD VLM distillation instead of crashing (#7581)\r\n* Name the missing vLLM server setting when the server cannot serve TRL (#7611)\r\n* Raise early for `use_kernels` with transformers 5.2.0 and 5.3.0 (#7577), and if kernels is installed with transformers 5.3.0 in the tiny NemotronH scripts (#7575)\r\n\r\n## Documentation\r\n\r\n* Add a guide for training on AWS Trainium by @qgallouedec in https://github.com/huggingface/trl/pull/7729\r\n* Add guidance on testing through the public API to the contributing guide and agent instructions by @albertvillanova in https://github.com/huggingface/trl/pull/7606\r\n* Document the criterion for raising a dependency version floor by @albertvillanova in https://github.com/huggingface/trl/pull/7638\r\n\r\n## CI\r\n\r\n**Real objects, not mocks.** The bulk of this cycle, applying the repo's testing rule across the suite, all by @albertvillanova: run the GRPO entropy mask tests on a real trainer instead of a Mock (#7566), the teacher logprob tests on a real trainer instead of a MagicMock (#7579), build real GOLD trainers in the init (#7582), input preparation (#7614), on-policy generation (#7613, #7615), output decoding (#7633), ULD helper (#7634) and no-backend tokenizer (#7635) tests, check GOLD dataset preparation through the trainer (#7632), and build real async trainers in the rollout state checkpoint tests (#7583)\r\n\r\n**Coverage**, all by @albertvillanova: test SFT (#7552) and DPO/KTO/Reward/GRPO/RLOO (#7601) on a model split across devices, test fp16 training across six trainers (#7572)\r\n\r\n**Experimental suite on main**, all by @albertvillanova: run the experimental tests on pushes to main (#7675), nightly (#7681), on demand (#7678) and when their Makefile target or dependencies change (#7680), and report failures to Slack (#7676)\r\n\r\n**Other**: use Python 3.13 by default in CI by @qgallouedec in https://github.com/huggingface/trl/pull/7648, remove the slow test infrastructure (#7604), rename the distributed smoke tests to distributed tests (#7631), skip the kernel tests when Triton is not installed (#7563), guard the aiohttp import in the experimental async rollout workers (#7578), drop `use_cpu` and `bf16` settings the CPU fixture already covers (#7612), re-record the `sft_fa2` invariant reference after the fused LM head by @qgallouedec (#7591), plus warning filters in #7586, #7588, #7590, #7645, #7727\r\n\r\n## New Contributors\r\n* @Abdennacer-Badaoui made their first contribution in https://github.com/huggingface/trl/pull/7647\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.15.0...v1.16.0","publishedAt":"2026-10-15T18:00:33.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.16.0","media":[{"type":"image","url":"https://github.com/user-attachments/assets/dcb6e2a1-0f7c-4635-819b-a1ec6f5e143b","r2Key":"releases/b75e058f04b9babf6c8d3078cdc6abc2dfb74dd416ad00799b005cd93202701b.png","r2Url":"https://media.releases.sh/releases/b75e058f04b9babf6c8d3078cdc6abc2dfb74dd416ad00799b005cd93202701b.png"}],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":11108,"contentTokens":3167,"composition":{"bugs":18,"features":10,"enhancements":6}},{"id":"rel_-QlXNUAbicPpBnhrHxbmg","version":"v1.15.0","type":"feature","title":"v1.15.0","summary":"A new fused LM head, on by default in SFT, DPO, KTO, GRPO, RLOO and Distillation, projects hidden states through the LM head in Triton-kernel tiles to avoid building the full logits tensor, raising max trainable sequence length up to 6.9x and cutting peak memory 52% to 82%. The full-logits scoring path, use_liger_kernel chunked path and _forward_redirection are removed, so a PEFT adapter on lm_head now raises and use_liger_kernel=True is deprecated for v2.0.0 removal.","titleGenerated":"TRL v1.15.0 fused LM head boosts max trainable sequence length up to 6.9x","titleShort":"Fused LM head: up to 6.9x longer sequences, 82% less memory","breaking":"major","importance":4,"content":"<img width=\"1672\" height=\"941\" alt=\"Neon-Gripped Industrial Valve\" src=\"https://github.com/user-attachments/assets/329793d4-cef1-49ae-9ab6-5b1387d13071\" />\r\n\r\n## Features\r\n\r\n### ⚡ Fused LM head: up to 6.9× longer sequences on the same GPU\r\n\r\nSFT, DPO, KTO, GRPO, RLOO and Distillation now score tokens with a **fused LM head**: a Triton kernel projects the hidden states through the LM head in tiles and reduces to per-token log-probs and entropy directly, so the `[batch, seq, vocab]` logits tensor is never built. It is **on by default**, there is nothing to turn on.\r\n\r\n**Max trainable sequence length** (gemma-3-1b, 262k vocabulary, 79 GiB of GPU memory, bf16, batch size 1, gradient checkpointing, sdpa):\r\n\r\n| Trainer | v1.14.2 | v1.15.0 | |\r\n| --- | --- | --- | --- |\r\n| DPO | 10,240 | **59,392** | **5.80×** |\r\n| KTO (batch size 2) | 9,216 | **63,488** | **6.89×** |\r\n| GRPO (scoring only) | 28,672 | **114,688** | **4.00×** |\r\n| RLOO (scoring only) | 23,552 | **100,352** | **4.26×** |\r\n| SFT (`loss_type=\"nll\"`) | 20,480 | **107,520** | **5.25×** |\r\n| SFT (default `chunked_nll`) | 107,520 | 107,520 | 1.00× |\r\n\r\n**At 8,192 tokens**, peak memory drops 52% to 82% and steps are 2.3% to 10.9% faster:\r\n\r\n| Trainer | Peak GiB v1.14.2 → v1.15.0 | Tokens/s |\r\n| --- | --- | --- |\r\n| DPO | 48.97 → **12.00** (-75.5%) | 1.035× |\r\n| KTO | 64.91 → **11.98** (-81.5%) | 1.045× |\r\n| GRPO | 24.15 → **10.30** (-57.4%) | 1.023× |\r\n| RLOO | 29.28 → **14.02** (-52.1%) | 1.029× |\r\n| SFT (`nll`) | 30.33 → **8.87** (-70.7%) | 1.029× |\r\n\r\nThe gain is largest for small models with large vocabularies. SFT's previous default (`chunked_nll`) already avoided full logits, so it is unchanged. Scoring needs Triton on a GPU (Linux with CUDA, ROCm or XPU).\r\n\r\n**💥 What this changes for you.** The full-logits scoring path, the `use_liger_kernel` chunked path and `_forward_redirection` are gone from GRPO, RLOO, DPO and KTO:\r\n\r\n* A PEFT adapter **on `lm_head`** now raises. Use `modules_to_save=[\"lm_head\"]` instead.\r\n* `use_liger_kernel=True` is deprecated in these four trainers and will be removed in v2.0.0. Liger's layer kernels still apply, but `fused_linear_cross_entropy` is forced to `False` (it would replace the fused head) and setting it explicitly raises. Use `model_init_kwargs={\"use_kernels\": True}`.\r\n* In DPO and KTO, `compute_metrics` and `compute_loss(..., return_outputs=True)` still receive the full logits, from an extra forward pass taken only when they are used. Both warn that this is deprecated and will be removed in v2.0.0.\r\n\r\nNothing else the trainers logged is lost: the kernel gained `log_sum_sq_probs` (WPO weighting), `mean_logits` (`logits/chosen`, `logits/rejected`) and `is_top1` (DPO's `mean_token_accuracy`) as opt-in outputs, each checked against the full logits in tests.\r\n\r\n<details>\r\n<summary>Benchmark details</summary>\r\n\r\n1× B300 with the PyTorch allocator capped at 79 GiB (about an H100 80GB's usable memory), `google/gemma-3-1b-pt` built from its config with random bf16 weights, lr 1e-6, AdamW, synthetic token ids at exact length. Max length: one fresh process per attempt, 2 training steps, doubling from 8k then bisection to 1024 tokens, arms interleaved round-robin in the same job. Throughput: 12 steps, first 2 discarded, median of 3 A/B/A/B repeats in the same job. Noise floor (A vs A) is 0.1% to 0.8%, below every gap reported. GRPO/RLOO replace generation with fixed random completions, so only scoring is measured.\r\n\r\nNot measured: H100 hardware itself, flash-attention (sdpa only), real generation, multi-GPU.\r\n\r\n</details>\r\n\r\n* Compute the chunked log-probabilities with a Triton kernel by @qgallouedec in https://github.com/huggingface/trl/pull/7386\r\n* Add an opt-in fused LM head to the patched forward by @qgallouedec in https://github.com/huggingface/trl/pull/7387\r\n* 💥 Score GRPO and RLOO tokens with the fused LM head by @qgallouedec in https://github.com/huggingface/trl/pull/7389\r\n* 💥 Score DPO and KTO tokens with the fused LM head by @qgallouedec in https://github.com/huggingface/trl/pull/7390\r\n* Score SFT tokens with the fused LM head by @qgallouedec in https://github.com/huggingface/trl/pull/7466\r\n* Compute the distillation divergence with a Triton kernel by @qgallouedec in https://github.com/huggingface/trl/pull/7465\r\n* Compute only the requested fused LM head outputs by @qgallouedec in https://github.com/huggingface/trl/pull/7406\r\n* Rename `patch_fused_lm_head` to `add_fused_lm_head` by @qgallouedec in https://github.com/huggingface/trl/pull/7415\r\n* Honor `logits_scaling` and `lm_head_multiplier` in the fused LM head by @YaseenBashaT in https://github.com/huggingface/trl/pull/7439\r\n* Fix the fused LM head for models split across devices by @albertvillanova in https://github.com/huggingface/trl/pull/7479\r\n\r\n### Selective activation checkpointing in SFT\r\n\r\nWith gradient checkpointing on, the attention output is now saved during the forward instead of being recomputed in the backward, recovering most of the checkpointing slowdown at long context for one extra hidden-state-sized tensor per layer. Same eager SAC approach torchtitan uses under FSDP2, no `torch.compile` needed.\r\n\r\n```python\r\nSFTConfig(\r\n    gradient_checkpointing=True,\r\n    gradient_checkpointing_kwargs={\"selective\": True},\r\n)\r\n```\r\n\r\nPEFT + DeepSpeed ZeRO-3 is rejected: that combination needs reentrant checkpointing, which SAC cannot use.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/7194\r\n\r\n### Assistant-only loss on vision datasets in SFT\r\n\r\n`assistant_only_loss=True` now works on vision datasets. `SFTTrainer` used to refuse it because processors did not return usable assistant masks once image placeholders were expanded; [transformers#48793](https://github.com/huggingface/transformers/pull/48793) fixed that, and it ships in `transformers` 5.18.\r\n\r\n```python\r\nSFTConfig(assistant_only_loss=True)  # now fine with an image dataset\r\n```\r\n\r\nThe vision collator tokenizes conversational examples through the processor's `apply_chat_template` and requests the assistant masks to mask non-assistant tokens in the labels. Older `transformers` still raises, now with the required version in the message.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/7377\r\n\r\n### Conversations in the completions table\r\n\r\nPrompts and completions are logged as raw conversation lists instead of `batch_decode`'d flat strings, so the table renders turn by turn. Much more readable for multi-turn and tool-calling setups, and for VLMs.\r\n\r\n<img width=\"1627\" alt=\"Completions table rendered turn by turn\" src=\"https://github.com/user-attachments/assets/649cf786-d4e5-4955-b486-ff7cd8d7439b\" />\r\n\r\nThe rendered system prompt (including tool definitions injected by the chat template) is no longer visible, since the original messages are shown rather than the fully-templated string.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5309\r\n\r\n### AsyncGRPO and AsyncDistillation\r\n\r\n* AsyncGRPO: train OpenEnv harnesses from validated token captures by @adithya-s-k in https://github.com/huggingface/trl/pull/6947\r\n* Add PEFT support to `AsyncDistillationTrainer` by @kashif in https://github.com/huggingface/trl/pull/7476\r\n* Fix duplicate completions in AsyncGRPO groups with data-parallel vLLM by @lewtun in https://github.com/huggingface/trl/pull/7549\r\n\r\n### Other\r\n\r\n* Add SmolVLM original and training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5868\r\n* Declare gradient accumulation loss scaling with `loss_is_scaled_for_ga` by @qgallouedec in https://github.com/huggingface/trl/pull/7508\r\n* Keep SFT's MoE aux loss independent of world size and gradient accumulation by @qgallouedec in https://github.com/huggingface/trl/pull/7570\r\n* Discover environment tools on the class, not the instance by @Rome-1 in https://github.com/huggingface/trl/pull/7360\r\n* Accept an IPv6 `base_url` in `VLLMClient` by @qgallouedec in https://github.com/huggingface/trl/pull/7462\r\n* Pass processor kwargs via `processor_kwargs` in the VLM SFT collator by @qgallouedec in https://github.com/huggingface/trl/pull/7547\r\n* Call `trainer.end()` at the end of the scripts and examples by @qgallouedec in https://github.com/huggingface/trl/pull/7384\r\n* Install vLLM in the TRL Docker images by @qgallouedec in https://github.com/huggingface/trl/pull/7464\r\n* Train harnesses through Harbor: drop the standalone opencode example by @sergiopaniego in https://github.com/huggingface/trl/pull/7457\r\n* Add support for vLLM 0.31.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7567\r\n\r\n## Breaking changes and removals\r\n\r\n* **Drop Python 3.10 support** by @qgallouedec in https://github.com/huggingface/trl/pull/7497\r\n* **Refuse `nn.DataParallel`** (and drop the multi-GPU slow test job) by @qgallouedec in https://github.com/huggingface/trl/pull/7407\r\n* **Remove the experimental MiniLLM trainer** by @qgallouedec in https://github.com/huggingface/trl/pull/7500\r\n* **`use_liger_kernel` is deprecated** in `DPOTrainer`, `KTOTrainer`, `GRPOTrainer` and `RLOOTrainer` and will be removed in v2.0.0. These trainers now use the fused LM head, so only Liger's layer kernels apply; use `model_init_kwargs={\"use_kernels\": True}` instead. Docs updated by @qgallouedec in https://github.com/huggingface/trl/pull/7535\r\n* Drop vLLM 0.20.0 / 0.20.1 / 0.20.2 support by @qgallouedec in https://github.com/huggingface/trl/pull/7435, https://github.com/huggingface/trl/pull/7499 and https://github.com/huggingface/trl/pull/7568\r\n* Remove the unreachable `prepare_multimodal_messages_vllm` by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6946\r\n\r\n## Fixes\r\n\r\n* **Fix `SFTTrainer` not appending EOS when `dataset_text_field` is not `\"text\"`** by @JoeyTan21 in https://github.com/huggingface/trl/pull/7446\r\n* **End completions on every eos id the model declares** in GRPO, RLOO and Distillation by @albertvillanova in https://github.com/huggingface/trl/pull/7505, and in the experimental trainers in https://github.com/huggingface/trl/pull/7506\r\n* **Start the completion where the tokenized prompt and prompt+completion diverge** by @qgallouedec in https://github.com/huggingface/trl/pull/7463\r\n* **Fix the server-mode crash at the first weight sync on vLLM 0.20 to 0.25** by @albertvillanova in https://github.com/huggingface/trl/pull/7410\r\n* **Generate one completion per sample after tool calls in vLLM server mode** by @qgallouedec in https://github.com/huggingface/trl/pull/7418\r\n* **Fix `precompute_ref_log_probs` under FSDP with no reference model** by @qgallouedec in https://github.com/huggingface/trl/pull/7507\r\n* **Keep the DFT loss finite on a batch with no trainable tokens** by @albertvillanova in https://github.com/huggingface/trl/pull/7471\r\n* **Guard the `requests` and `urllib3` imports in the vLLM client** by @qgallouedec in https://github.com/huggingface/trl/pull/7545, and in the experimental async vLLM clients by @albertvillanova in https://github.com/huggingface/trl/pull/7561\r\n* `fix(cpo, orpo)`: truncate responses independently to prevent empty completions by @RohanMali2003 in https://github.com/huggingface/trl/pull/6588\r\n* Fix FLOPs overcount for models with untied word embeddings by @jayzuccarelli in https://github.com/huggingface/trl/pull/7450\r\n* `[MFU]` Use device peak flops instead of hardcoded H100's by @dwarez in https://github.com/huggingface/trl/pull/7197\r\n* Fix segfault on Apple Silicon when loading bf16 checkpoints from a model ID by @albertvillanova in https://github.com/huggingface/trl/pull/7455\r\n* Keep the DoRA magnitude vector in float32 under QLoRA by @qgallouedec in https://github.com/huggingface/trl/pull/7342\r\n* Fix missing OpenReward outcome rewards by @adithya-s-k in https://github.com/huggingface/trl/pull/7467\r\n* Keep the next turn's first token out of the Llava-Next assistant mask by @albertvillanova in https://github.com/huggingface/trl/pull/7375\r\n* Tie weights after loading remote-code models with transformers 5.3.0 by @albertvillanova in https://github.com/huggingface/trl/pull/7559\r\n* Make SDFT importable without peft and align SSD by @albertvillanova in https://github.com/huggingface/trl/pull/7562\r\n* Preserve explicit `model_init_kwargs` in CLI scripts by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/7394\r\n* Honor `resume_from_checkpoint` in CLI scripts by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/7391\r\n* Seed before creating the model so the PEFT adapter init is reproducible by @qgallouedec in https://github.com/huggingface/trl/pull/7385, and in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7401\r\n* Drop the redundant spawn start method from `trl vllm-serve` by @albertvillanova in https://github.com/huggingface/trl/pull/7379\r\n* `FIX`: `test_offloading_with_peft_models` on L40, register model params in activation offloading tests by @AmineDiro in https://github.com/huggingface/trl/pull/7380\r\n* Drop the dead `gate_proj` LoRA target from the Nemotron 3 SFT example by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6866\r\n* Target the Mamba `in_proj` in the Nemotron 3 LoRA example by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6870\r\n* Replace the introspection guard in `sft_qwen3_8b_1m_context.py` with a version check by @kumarrah2002 in https://github.com/huggingface/trl/pull/7537\r\n\r\n## Documentation\r\n\r\n* Expand the Jobs training guide by @davanstrien in https://github.com/huggingface/trl/pull/7441\r\n* Document `expandable_segments` allocator config for memory tuning by @akshansh47 in https://github.com/huggingface/trl/pull/5794\r\n* Update \"What's New\" for the fused LM head by @qgallouedec in https://github.com/huggingface/trl/pull/7593\r\n* Document every stored chat template and check it in CI by @albertvillanova in https://github.com/huggingface/trl/pull/7502\r\n* Document what goes into a patch release by @albertvillanova in https://github.com/huggingface/trl/pull/7440\r\n* Document input validation guidelines by @albertvillanova in https://github.com/huggingface/trl/pull/7472\r\n* Unwrap hard-wrapped paragraphs in the docs by @qgallouedec in https://github.com/huggingface/trl/pull/7405\r\n* Discourage `SimpleNamespace`, `object.__new__` and mocks in tests in the agent instructions by @qgallouedec in https://github.com/huggingface/trl/pull/7459\r\n* Require first-time contributors to link an issue assigned to them by @qgallouedec in https://github.com/huggingface/trl/pull/7313\r\n* Extend the AI usage policy to issues by @albertvillanova in https://github.com/huggingface/trl/pull/7473\r\n* Require bug reports to state how the problem was hit by @albertvillanova in https://github.com/huggingface/trl/pull/7474\r\n* Label CONTRIBUTING.md changes as documentation by @albertvillanova in https://github.com/huggingface/trl/pull/7475\r\n\r\n## CI\r\n\r\n**Chat template coverage** — the stored templates are now documented, checked in CI, and watched weekly against the reference Hub repos, all by @albertvillanova:\r\n\r\n* Document every stored chat template and check it in CI (#7502), add a script checking that reference Hub repos still ship a stored chat template (#7523), check it weekly (#7521)\r\n* Recognize the current LFM2 (#7494) and LFM2.5 (#7493) chat templates, the other Gemma 4 revisions (#7495), parse the current LFM2 tool calls (#7522), keep the earlier LFM2 revision under test (#7527), read templates shipped as named variants by their default variant (#7530)\r\n\r\n**Tiny-model config alignment**, continuing the sweep, all by @albertvillanova: DeepSeek-V3 (#7398) and V3-0528 (#7399), GPT-OSS (#7424), Qwen2-VL (#7425) and Qwen2.5-VL (#7426) mrope sections, Nemotron-3-Nano (#7443), Nemotron-3.5-Lightning (#7444), Nemotron-3-Super (#7447), Nemotron-3-Ultra (#7448), LFM2 (#7400), LFM2.5 (#7532)\r\n\r\n**Test quality**, following the \"real objects, not mocks\" rule, all by @albertvillanova: build real trainers instead of `object.__new__` in the GRPO and SDFT tests (#7531), build real configs instead of `SimpleNamespace` in the GOLD and GKD tests (#7564), build the async rollout loops through `__init__` (#7555), run the async GRPO rollout loop tests on the real tokenizer and response parser (#7556), align the DFT loss tests on a real model output (#7481)\r\n\r\n**Suite hygiene**, all by @albertvillanova unless noted: move the vLLM training (#7488), continuous batching (#7487) and SFT activation offloading (#7477) tests to the regular suite, unmark the vLLM client/server tests as slow (#7490), run them on a single xdist worker (#7489), replace the SFT slow tests with an fp16 test (#7550), remove the always-skipped Gemma 3n tests (#7485), drop the no-op slow and low_priority markers (#7486), stop piping the vLLM test server output (#7414), run the VLM test server on the last visible accelerator (#7412), test the vLLM client against the documented `vllm serve` command (#7378), inject `MODEL_REVISIONS` into the Auto loaders (#7528), mirror the pad token onto the model config in the callback tests (#7383), remove the DataParallel gather warning filter (#7554), tie the mamba-ssm autocast warning filter removal to transformers 5.6 (#7431), remove the nightly cache cleanup workflow (#7395), scope `GITHUB_TOKEN` permissions per job by @hf-security-analysis (#7381), bump the doc-builder workflow pin by @paulinebm (#7542), Dependabot group bump (#7411)\r\n\r\n**Hotfixes**: xfail the GRPO continuous batching test under DataParallel (#7422), xfail Qwen3.5-MoE VLM tests against a broken transformers dev build (#7437) and revert once fixed (#7460)\r\n\r\n## New Contributors\r\n* @dwarez made their first contribution in https://github.com/huggingface/trl/pull/7197\r\n* @Rome-1 made their first contribution in https://github.com/huggingface/trl/pull/7360\r\n* @JoeyTan21 made their first contribution in https://github.com/huggingface/trl/pull/7446\r\n* @jayzuccarelli made their first contribution in https://github.com/huggingface/trl/pull/7450\r\n* @RohanMali2003 made their first contribution in https://github.com/huggingface/trl/pull/6588\r\n* @kumarrah2002 made their first contribution in https://github.com/huggingface/trl/pull/7537\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.14.0...v1.15.0\r\n","publishedAt":"2026-10-08T19:26:48.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.15.0","media":[{"type":"image","url":"https://github.com/user-attachments/assets/329793d4-cef1-49ae-9ab6-5b1387d13071","r2Key":"releases/5018e7c1f394cb3e3ea542b77c448f9f4f8f88e3ae80f66965526184ef7ae059.png","r2Url":"https://media.releases.sh/releases/5018e7c1f394cb3e3ea542b77c448f9f4f8f88e3ae80f66965526184ef7ae059.png"},{"type":"image","url":"https://github.com/user-attachments/assets/649cf786-d4e5-4955-b486-ff7cd8d7439b","r2Key":"releases/e88323bb69b7998dd7e0ed6ebc74d276302e05db4874891056c8dda536564eaf.png","r2Url":"https://media.releases.sh/releases/e88323bb69b7998dd7e0ed6ebc74d276302e05db4874891056c8dda536564eaf.png"}],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":18231,"contentTokens":5289,"composition":{"bugs":4,"features":8,"enhancements":3}},{"id":"rel_H7vNfWdQk6TovPDaKnm17","version":"v1.14.2","type":"feature","title":"v1.14.2","summary":"GRPO, RLOO, and Distillation on the default non-vLLM generation path now stop completions at every eos id the model declares, fixing silent training on tokens generated past the end of the turn that showed up as completions/clipped_ratio near 1. Also fixes DFT loss on batches with no trainable tokens, precompute_ref_log_probs under FSDP with no reference model, completion offsets where the tokenized prompt and prompt+completion diverge, and guards the requests and urllib3 imports in the vLLM client.","titleGenerated":"TRL v1.14.2 fixes GRPO/RLOO training on tokens past end-of-turn","titleShort":"GRPO/RLOO stop at end-of-turn ids; DFS loss stays finite","breaking":"none","importance":3,"content":"Patch release fixing two cases of silently wrong training and three crashes.\r\n\r\nWorth calling out: **#7505** affects GRPO, RLOO and Distillation on the default generation path (`use_vllm=False`). Models that end a turn with an id only their generation config declares (Gemma 3/4, Phi-3.5, ...) did not stop at the end of the turn, and everything generated after it was trained on, up to `max_completion_length`. The only visible sign was `completions/clipped_ratio` close to 1. With vLLM, generation stopped correctly but the metrics were still wrong, and `mask_truncated_completions=True` dropped every finished completion. Runs on models whose tokenizer eos is their only end-of-turn id are unchanged.\r\n\r\n## What's Changed\r\n\r\n* Keep the DFT loss finite on a batch with no trainable tokens by @albertvillanova in https://github.com/huggingface/trl/pull/7471\r\n* Fix `precompute_ref_log_probs` under FSDP with no reference model by @qgallouedec in https://github.com/huggingface/trl/pull/7507\r\n* Fix: Start the completion where the tokenized prompt and prompt+completion diverge by @qgallouedec in https://github.com/huggingface/trl/pull/7463\r\n* End completions on every eos id the model declares in GRPO, RLOO and Distillation by @albertvillanova in https://github.com/huggingface/trl/pull/7505\r\n* Guard the requests and urllib3 imports in the vLLM client by @qgallouedec in https://github.com/huggingface/trl/pull/7545\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.14.1...v1.14.2\r\n","publishedAt":"2026-10-06T20:30:35.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.14.2","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":1505,"contentTokens":399,"composition":{"bugs":5,"features":0,"enhancements":0}},{"id":"rel_BVxRC-qWaAqKKUl9adg9r","version":"v0.21.2","type":"feature","title":"v0.21.2","summary":"Fixes an issue that prevented encoder-decoder models from working when using Transformers 5.18.0 or later.","titleGenerated":"PEFT v0.21.2 fixes encoder-decoder models broken on Transformers 5.18.0+","titleShort":"Encoder-decoder models work again on Transformers 5.18+","breaking":"none","importance":2,"content":"This is a PEFT release fixes an issue that prevented encoder-decoder models to work when using Transformers ≥ 5.18.0.\r\n\r\nChanges:\r\n\r\n- #3845","publishedAt":"2026-10-01T10:29:50.000Z","url":"https://github.com/huggingface/peft/releases/tag/v0.21.2","media":[],"prerelease":false,"source":{"slug":"peft","name":"PEFT","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":140,"contentTokens":34,"composition":{"bugs":1,"features":0,"enhancements":0}},{"id":"rel_vPHZ9rpgU-8rpBiQcc-Kj","version":"v1.14.1","type":"feature","title":"v1.14.1","summary":"Fixed a server-mode crash at the first weight sync on vLLM 0.20 through 0.25, and vLLM server mode now generates one completion per sample after tool calls. CLI scripts again honor resume_from_checkpoint and preserve explicit model_init_kwargs.","titleGenerated":"TRL v1.14.1 fixes server-mode crash on first weight sync in vLLM","titleShort":"vLLM server-mode weight sync crash fixed","breaking":"none","importance":2,"content":"## What's Changed\r\n\r\n* Fix the server-mode crash at the first weight sync on vLLM 0.20 to 0.25 by @albertvillanova in https://github.com/huggingface/trl/pull/7410\r\n* [Fix] Generate one completion per sample after tool calls in vLLM server mode by @qgallouedec in https://github.com/huggingface/trl/pull/7418\r\n* Preserve explicit model_init_kwargs in CLI scripts by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/7394\r\n* Honor resume_from_checkpoint in CLI scripts by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/7391\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.14.0...v1.14.1","publishedAt":"2026-09-29T13:41:37.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.14.1","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":624,"contentTokens":188,"composition":{"bugs":4,"features":0,"enhancements":0}},{"id":"rel_LAKJasTgBfDpiQOVEd5tZ","version":"v0.21.1","type":"feature","title":"v0.21.1","summary":"PEFT now works properly with Tensor Parallel (TP), and requires Transformers 5.17.0 or later.","titleGenerated":"PEFT v0.21.1 enables Tensor Parallel support and requires Transformers 5.17+","titleShort":"Tensor Parallel now works with PEFT; Transformers 5.17+ required","breaking":"minor","importance":2,"content":"This is a small PEFT release to enable Tensor Parallel (TP) to work properly with PEFT. It requires Transformers ≥ 5.17.0 to work.\r\n\r\nChanges:\r\n\r\n* #3614\r\n* #3790","publishedAt":"2026-09-29T09:28:26.000Z","url":"https://github.com/huggingface/peft/releases/tag/v0.21.1","media":[],"prerelease":false,"source":{"slug":"peft","name":"PEFT","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":162,"contentTokens":45,"composition":{"bugs":0,"features":1,"enhancements":0}},{"id":"rel_oGhz1Hr_wFgUSzwDM751v","version":"v1.14.0","type":"feature","title":"v1.14.0","summary":"The trl.losses module introduced in v1.13 is removed, with DPO, KTO, and GRPO now streaming their own per-token log-probs through _ChunkedLogProbFunction, which also unlocks DPO with precomputed reference log-probs that did not exist before. A fused Triton logprob and entropy kernel now ships in-tree at trl.kernels and is on by default on CUDA, ROCm, and XPU, and six experimental trainers were removed from trl.experimental.","titleGenerated":"TRL v1.14.0 removes trl.losses, ships fused Triton logprob kernel","titleShort":"trl.losses gone; fused Triton logprob kernel on by default","breaking":"major","importance":4,"content":"## Features\r\n\r\n### `trl.losses` is gone: DPO, KTO and GRPO stream their own log-probs\r\n\r\n> [!WARNING]\r\n> `from trl.losses import FusedLinearDPOLoss` (or `FusedLinearKTOLoss`, `FusedLinearGRPOLoss`, `FusedLinearJSDLoss`) no longer works. The module introduced in v1.13 has been removed.\r\n\r\nv1.13 vendored Liger's `chunked_loss` into `trl.losses` as a holding action, not a destination (#7063). The problem it was holding: those classes reimplement each trainer's loss math, so TRL carried two copies of every formula, and copies drift. That drift is where the bugs were. GRPO's `grad_norm` differed from the default path, KTO ignored the reference model under PEFT, KTO class weights and DPO `label_smoothing` were silently dropped. Worst of all, under plain DDP the fused path ran the loss on the unwrapped model, so `DistributedDataParallel`'s reducer was never armed and gradients were never all-reduced: every rank silently kept its own.\r\n\r\nThe fix is to stop reimplementing. All DPO, KTO and GRPO need from the fused path is per-token log-probs of the selected tokens, and TRL already had `_ChunkedLogProbFunction` for exactly that: it streams the vocabulary with an online logsumexp and recomputes in the backward pass. Each trainer now runs `backbone -> _ChunkedLogProbFunction -> its own existing loss code`, unchanged. One implementation of each loss, and the memory win is kept, because full logits are still never materialized.\r\n\r\nThirteen restrictions existed only because the loss had been rewritten in a form that could not express those options. Most are gone: DPO with `use_liger_kernel=True` now accepts mixed loss types, f-divergences and precomputed reference log-probs, and GRPO gains entropy and off-policy masking. Still refused: `use_weighting`, `compute_metrics`, a PEFT adapter on `lm_head`, prompt-learning PEFT, and the MoE auxiliary loss.\r\n\r\nSingle H100, Qwen3-0.6B, `bs=4`, seq 512:\r\n\r\n| config | median step | peak |\r\n| --- | --- | --- |\r\n| v1.13 fused, inline ref | 0.2243 s | 7.05 GB |\r\n| v1.14 chunked, inline ref | 0.2457 s (+9.5%) | 7.05 GB |\r\n| v1.14 chunked, `precompute_ref_log_probs=True` | **0.1971 s (-12.1%)** | **4.83 GB (-31%)** |\r\n\r\nThe third row is the point: that configuration did not exist before, because the fused path rejected precomputed reference log-probs outright.\r\n\r\n`use_liger_kernel=True` still works and still enables Liger's model kernels through transformers. In DPO, GRPO and KTO it now selects TRL's chunked log-probability path instead of Liger's fused loss.\r\n\r\n* Stream KTO log probabilities by @kashif in https://github.com/huggingface/trl/pull/7075\r\n* Stream GRPO log probabilities by @kashif in https://github.com/huggingface/trl/pull/7077\r\n* Move DPO to chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7243\r\n* Drop the fused Liger JSD path from the experimental distillation trainers by @kashif in https://github.com/huggingface/trl/pull/7301\r\n* Skip frozen chunked log probability gradients by @kashif in https://github.com/huggingface/trl/pull/7076\r\n* Redirect the KTO liger loss through the distributed wrapper under DDP by @qgallouedec in https://github.com/huggingface/trl/pull/7247\r\n* Document what use_liger_kernel actually does in each trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7361\r\n\r\n### A fused Triton logprob + entropy kernel now ships in TRL\r\n\r\nThe loss head is the only hot path TRL owns: transformers already kernelizes the model internals, and nothing on the Hub covers what happens after the decoder. On one H100 (bf16, `V=151936`, `H=4096`, 8192 tokens), `selective_log_softmax` plus `entropy_from_logits` cost 12.8 ms and 2.32 GiB over `[8, 1024, 151936]`. A fused Triton kernel does the same in 0.89 ms, roughly 14x, and lands closer to the fp32 reference than the bf16 path it replaces. GRPO runs this two to four times per step.\r\n\r\nIt now lives in-tree at `trl.kernels` and is on by default on CUDA, ROCm and XPU, with the PyTorch implementation as fallback. It was briefly loaded from the Hub; that was reversed because this kernel is pure Triton and compiles at runtime, so the Hub bought nothing while costing a version gate unrelated to whether the kernel works, a download that fails offline, `trust_remote_code=True` for anyone with `kernels` installed, a silent `except Exception: pass` wrapped around all of it, and 13 of 13 tests skipped in CI.\r\n\r\n* Use fused kernels for logprobs and entropy by @kashif in https://github.com/huggingface/trl/pull/7253\r\n* Move TRL loss kernels into TRL by @kashif in https://github.com/huggingface/trl/pull/7266\r\n* Ship the logprob and entropy kernel in trl instead of the Hub by @qgallouedec in https://github.com/huggingface/trl/pull/7363\r\n\r\n### Other\r\n\r\n* Ensure device agnostic for examples/docs/templates by @kaixuanliu in https://github.com/huggingface/trl/pull/6770\r\n* AsyncGRPO: PEFT/LoRA support with adapter-only vLLM sync by @AmineDiro in https://github.com/huggingface/trl/pull/7017\r\n* Add support for vLLM 0.29.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7178\r\n* [GOLD] Recommend LFM2.5 and Gemma 4 students by @kashif in https://github.com/huggingface/trl/pull/6388\r\n* Add TailSFT example by @kashif in https://github.com/huggingface/trl/pull/7212\r\n* [AsyncGRPO] Run sync tools on a thread pool and support async tools by @AmineDiro in https://github.com/huggingface/trl/pull/7175\r\n\r\n## Breaking\r\n\r\n### Six experimental trainers removed\r\n\r\n`GRPOWithReplayBufferTrainer`, `BCOTrainer`, `PRMTrainer`, `XPOTrainer`, `NashMDTrainer` and the GSPO-token trainer are gone from `trl.experimental`.\r\n\r\nWhat changed is not the guarantee, which was always that `trl.experimental` has none, but the fact that the decision is now written down. Removals used to be argued one pull request at a time, with every thread relitigating the policy before it got to the feature. #7182 settled what the judgment weighs, and #7254 put it in the docs: usage, external issues and pull requests that come from people actually running the feature, whether a stable trainer already covers it, maintenance cost, downstream consumers, owner, and age. Deliberately inputs to a judgment rather than a test: no threshold, no notice cycle, and the numbers stay in the removal pull request where you can check them. A paper index entry outlives the code it described.\r\n\r\nThe numbers are what make the case. `BCOTrainer` was 1,600 lines carrying its own reference-model handling, DeepSpeed preparation, tokenization and eval loop, for 11 genuine trainings in two months; it was also the only reason TRL depended on scikit-learn and joblib at all, and both are now gone from `pyproject.toml`. `GRPOWithReplayBufferTrainer` had hand-copied `_generate_and_score_completions` from GRPO and drifted to 385 of 616 lines different, for 8 trainings.\r\n\r\n* Remove the experimental BCO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7136\r\n* Remove the experimental GRPO-with-replay-buffer trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7132\r\n* Remove the experimental PRM trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7133\r\n* Remove the experimental Nash-MD trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7255\r\n* Remove the experimental XPO trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7256\r\n* Remove the experimental GSPO-token trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7331\r\n\r\nAlso breaking, and covered above: `trl.losses` was removed along the way.\r\n\r\n* Move DPO to chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7243\r\n\r\n## Fixes\r\n\r\n* **Gradients were never all-reduced for KTO under plain DDP.** `compute_loss` redirected through the distributed wrapper only for ZeRO-3 and FSDP, so on plain DDP the loss ran on the unwrapped model, `DistributedDataParallel.forward()` never ran, its reducer was never armed, and every rank silently kept its own gradients. By @qgallouedec in https://github.com/huggingface/trl/pull/7247\r\n* **Wrapped packing duplicated data when the input table was sliced.** `_pack_wrapped` sized the output from the sliced view but read from the raw child buffer, so it re-read from the start of the buffer instead of from the slice, and `datasets.map(batched=True)` passes sliced tables. By @qgallouedec in https://github.com/huggingface/trl/pull/6670\r\n* **A reward function logging on only some ranks hung or crashed the run.** GRPO and RLOO looped over the keys each rank held and called one collective per key, so ranks called different collectives: `log_metric` on rank 0 alone hung on NCCL, and `log_extra` crashed inside `gather_object` with a 1EB allocation. By @behroozazarkhalili in https://github.com/huggingface/trl/pull/7382\r\n* **Special token ids now reach the config the `Trainer` actually reads**, so it no longer realigns them at train time and logs a change the user did not make (part of #7093): eos in the trainers (#7127), written to the text config for composite models (#7315), the reward model pad token (#7318), and the requested pad token applied to vision datasets (#7316, #7352)\r\n* Do not quantize the teacher by default in the distillation and GOLD entry points by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6769\r\n* Prepare the dataset once per node when the datasets cache is node-local by @qgallouedec in https://github.com/huggingface/trl/pull/7241\r\n* Read the MoE auxiliary loss coefficient from the model config by @qgallouedec in https://github.com/huggingface/trl/pull/7248\r\n* Await async reward functions written as callable classes by @albertvillanova in https://github.com/huggingface/trl/pull/7213\r\n* Fix two stream races in OffloadActivations by @kashif in https://github.com/huggingface/trl/pull/7195\r\n* Only dedupe offloaded activation storages in streams mode by @kashif in https://github.com/huggingface/trl/pull/7193\r\n* Pass the FSDP2 ignored_params kwarg only on torch 2.7 or later by @albertvillanova in https://github.com/huggingface/trl/pull/7259\r\n* Guard the torch 2.6 APIs used by FSDP2 and online DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7260\r\n* Reduce connection-pool warnings in the vLLM client by @vladbataev in https://github.com/huggingface/trl/pull/7199\r\n\r\n## Documentation\r\n\r\n* **Document how a feature leaves `trl.experimental`** by @albertvillanova in https://github.com/huggingface/trl/pull/7254, the rule the six removals above were made under\r\n* Document what use_liger_kernel actually does in each trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7361\r\n* Document that AsyncGRPO tools must be module-level functions by @albertvillanova in https://github.com/huggingface/trl/pull/7226\r\n* Remove outdated \"under construction\" banners from docs by @qgallouedec in https://github.com/huggingface/trl/pull/7177\r\n* docs: fix 'allows to scale' grammar in DeepSpeed guide by @mrchatam in https://github.com/huggingface/trl/pull/7196\r\n\r\n## CI and maintenance\r\n\r\n**Tiny models**, a sweep bringing every tiny test model in line with its reference, all by @albertvillanova:\r\n\r\n* Aligned with their references: Qwen2.5-Coder (#7134), DeepSeek-R1-Distill (#7135), FalconMamba (#7139), Mistral v0.1 (#7203) and v0.2 (#7204), Gemma (#7217), Gemma2 (#7228), OPT (#7233), Phi-3 (#7257), Phi-3.5 (#7276), plus `tie_word_embeddings` across the board (#7130)\r\n* Fields mirrored in a second pass: `use_mrope` (#7332), `use_mambapy` (#7333), `rope_scaling` (#7334), `cache_implementation` and `hidden_act` (#7335), `activation_dropout` and `prefix` (#7336), `attention_bias` (#7337, #7338)\r\n* Generators: seeded so regeneration is idempotent and pushes nothing (#7258), pointed at a FalconMamba reference (#7138), `head_dim` reduced for Gemma (#7216) and Gemma2 (#7227), `ffn_dim` passed for OPT (#7232), the Phi3 leftover replaced by the 3 and 3.5 variants (#7239), three with no test consumer retired (#7186)\r\n* Coverage: the canonical GPTNeoX sequence-classification id (#7185), the tiny DeepSeek-R1-Distill trained in the SFT test (#7187), the dtype check skipped for non-safetensors references (#7238), the chat-template tests off tiny Bloom (#7262)\r\n\r\n**Tests run only when they can be affected**, all by @albertvillanova:\r\n\r\n* Skipped when nothing they cover changed (#7171, #7281), and run when test data or shipped package data does (#7283), or when the experimental workflow itself changes (#7278); code quality moved to its own workflow (#7277)\r\n* Superseded pull request runs cancelled (#7170), the doc build keyed on the pull request number (#7174). The same grouping was applied to `main` (#7172) and then reverted (#7201), because GitHub buckets `cancelled` with failure and `main` looked broken.\r\n\r\n**Release automation.** Publishing now happens on a tag push, and refuses unless `VERSION` in the tagged commit matches the tag (#7165). Release notes come out sectioned, from a label-to-section map (#7166) with documentation and maintenance labels applied automatically from the changed paths (#7205, #7210). Plus the dev version bump (#7160).\r\n\r\n**Test reliability**, all by @albertvillanova: the rerun filter matches the out-of-memory message rather than the exception type (#7292) and needs `pytest-rerunfailures>=16.7` to see wrapped exceptions (#7325); `ACCELERATE_MIXED_PRECISION` is restored between tests (#7321); the Llava assistant-mask tests are un-xfailed now that transformers fixed them (#7373); non-identity adapters in the `use_adapter` tests (#7275); `MODEL_REVISIONS` injected into the PEFT loaders (#7288); the reward model pad token covered (#7350).\r\n\r\n**Workflow hardening.** Token permissions narrowed in the test workflows (#7287), the workflows hardened (#7300), and the upstream issue behind the trufflehog digest pin linked (#7214).\r\n\r\n**Other.** Dependabot bumps (#7192, #7299), the latest-release tests moved from T4 to L40S (#7184) with the Slack action checkout workaround dropped (#7183), the kernel publish job run under a personal namespace (#7343), the token sync comments reworded (#7167) and all three generation-config eos shapes covered (#7168), LM-head gradient buffers and GEMMs skipped when its parameters are frozen (#7076), and the end-of-turn loss mask warning made actionable (#7230).\r\n\r\n## Dependencies\r\n\r\nThe supported vLLM window moves from `>=0.19.1,<=0.28.0` to `>=0.20.0,<=0.30.0`.\r\n\r\n* Add support for vLLM 0.30.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7365\r\n* Drop vLLM 0.19.1 support by @qgallouedec in https://github.com/huggingface/trl/pull/7366\r\n\r\n## New Contributors\r\n\r\n* @mrchatam made their first contribution in https://github.com/huggingface/trl/pull/7196\r\n* @itwangwang made their first contribution in https://github.com/huggingface/trl/pull/7230\r\n* @SuryanshSS1011 made their first contribution in https://github.com/huggingface/trl/pull/6670\r\n* @vladbataev made their first contribution in https://github.com/huggingface/trl/pull/7199\r\n\r\n## What's Changed\r\n\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/7160\r\n* Sync the model config eos token id with the tokenizer in the trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7127\r\n* Align tie_word_embeddings with the reference models by @albertvillanova in https://github.com/huggingface/trl/pull/7130\r\n* Ensure device agnostic for examples/docs/templates by @kaixuanliu in https://github.com/huggingface/trl/pull/6770\r\n* Remove the experimental GRPO-with-replay-buffer trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7132\r\n* Align tiny Qwen2.5-Coder config with Qwen/Qwen2.5-Coder-0.5B by @albertvillanova in https://github.com/huggingface/trl/pull/7134\r\n* Reword the token sync comments to say what the line does by @albertvillanova in https://github.com/huggingface/trl/pull/7167\r\n* Cover the three generation config eos token shapes in the sync test by @albertvillanova in https://github.com/huggingface/trl/pull/7168\r\n* Align tiny DeepSeek-R1-Distill config with deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B by @albertvillanova in https://github.com/huggingface/trl/pull/7135\r\n* Point the tiny FalconMamba generator at a FalconMamba reference by @albertvillanova in https://github.com/huggingface/trl/pull/7138\r\n* Align tiny FalconMamba config with tiiuae/falcon-mamba-7b-instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7139\r\n* AsyncGRPO: PEFT/LoRA support with adapter-only vLLM sync by @AmineDiro in https://github.com/huggingface/trl/pull/7017\r\n* Stream KTO log probabilities by @kashif in https://github.com/huggingface/trl/pull/7075\r\n* Skip frozen chunked log probability gradients by @kashif in https://github.com/huggingface/trl/pull/7076\r\n* Cancel superseded PR runs in the test workflows by @albertvillanova in https://github.com/huggingface/trl/pull/7170\r\n* Add support for vLLM 0.29.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7178\r\n* Skip the tests on pushes that change no tested file by @albertvillanova in https://github.com/huggingface/trl/pull/7171\r\n* Cancel superseded runs on main by @albertvillanova in https://github.com/huggingface/trl/pull/7172\r\n* docs: fix 'allows to scale' grammar in DeepSpeed guide by @mrchatam in https://github.com/huggingface/trl/pull/7196\r\n* Remove outdated \"under construction\" banners from docs by @qgallouedec in https://github.com/huggingface/trl/pull/7177\r\n* Move the latest release tests from T4 to L40S by @albertvillanova in https://github.com/huggingface/trl/pull/7184\r\n* Use the canonical id for the tiny GPTNeoX sequence classification model by @albertvillanova in https://github.com/huggingface/trl/pull/7185\r\n* Train the tiny DeepSeek-R1-Distill model in the SFT test by @albertvillanova in https://github.com/huggingface/trl/pull/7187\r\n* Fix two stream races in OffloadActivations by @kashif in https://github.com/huggingface/trl/pull/7195\r\n* Only dedupe offloaded activation storages in streams mode by @kashif in https://github.com/huggingface/trl/pull/7193\r\n* Stop cancelling superseded runs on main by @albertvillanova in https://github.com/huggingface/trl/pull/7201\r\n* Key the doc build concurrency group on the pull request number by @albertvillanova in https://github.com/huggingface/trl/pull/7174\r\n* Drop the Slack action checkout workaround from the latest release tests by @albertvillanova in https://github.com/huggingface/trl/pull/7183\r\n* Retire three tiny model generators with no test consumer by @albertvillanova in https://github.com/huggingface/trl/pull/7186\r\n* Publish to PyPI on tag push and guard VERSION against the tag by @albertvillanova in https://github.com/huggingface/trl/pull/7165\r\n* Section the auto-generated release notes by label by @albertvillanova in https://github.com/huggingface/trl/pull/7166\r\n* Automatically label documentation and maintenance PRs by @albertvillanova in https://github.com/huggingface/trl/pull/7205\r\n* chore: update labeler.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/7210\r\n* Align tiny Mistral v0.1 config with mistralai/Mistral-7B-Instruct-v0.1 by @albertvillanova in https://github.com/huggingface/trl/pull/7203\r\n* Align tiny Mistral v0.2 config with mistralai/Mistral-7B-Instruct-v0.2 by @albertvillanova in https://github.com/huggingface/trl/pull/7204\r\n* [GOLD] Recommend LFM2.5 and Gemma 4 students by @kashif in https://github.com/huggingface/trl/pull/6388\r\n* Bump the \"github-actions-and-pre-commit\" group with 2 updates across multiple ecosystems by @dependabot[bot] in https://github.com/huggingface/trl/pull/7192\r\n* Link the upstream issue behind the trufflehog digest pin by @albertvillanova in https://github.com/huggingface/trl/pull/7214\r\n* Await async reward functions written as callable classes by @albertvillanova in https://github.com/huggingface/trl/pull/7213\r\n* Reduce head_dim in the tiny Gemma generator by @albertvillanova in https://github.com/huggingface/trl/pull/7216\r\n* Do not quantize the teacher by default in the distillation and GOLD entry points by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6769\r\n* Align tiny Gemma config with google/gemma-7b-it by @albertvillanova in https://github.com/huggingface/trl/pull/7217\r\n* Reduce head_dim in the tiny Gemma2 generator by @albertvillanova in https://github.com/huggingface/trl/pull/7227\r\n* Replace the tiny-Phi3ForCausalLM leftover with the 3 and 3.5 variants by @albertvillanova in https://github.com/huggingface/trl/pull/7239\r\n* Make the end-of-turn loss mask warning actionable by @itwangwang in https://github.com/huggingface/trl/pull/7230\r\n* Fix wrapped packing duplicating data when the input table is sliced by @SuryanshSS1011 in https://github.com/huggingface/trl/pull/6670\r\n* Skip the dtype check when the reference is not a safetensors repo by @albertvillanova in https://github.com/huggingface/trl/pull/7238\r\n* Document that AsyncGRPO tools must be module-level functions by @albertvillanova in https://github.com/huggingface/trl/pull/7226\r\n* Pass ffn_dim instead of intermediate_size in the tiny OPT generator by @albertvillanova in https://github.com/huggingface/trl/pull/7232\r\n* Align tiny OPT config with facebook/opt-1.3b by @albertvillanova in https://github.com/huggingface/trl/pull/7233\r\n* Add TailSFT example by @kashif in https://github.com/huggingface/trl/pull/7212\r\n* [AsyncGRPO] Run sync tools on a thread pool and support async tools by @AmineDiro in https://github.com/huggingface/trl/pull/7175\r\n* Prepare the dataset once per node when the datasets cache is node-local by @qgallouedec in https://github.com/huggingface/trl/pull/7241\r\n* Remove the experimental BCO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7136\r\n* Remove the experimental PRM trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7133\r\n* Document how a feature leaves trl.experimental by @albertvillanova in https://github.com/huggingface/trl/pull/7254\r\n* Remove the experimental XPO trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7256\r\n* Remove the experimental Nash-MD trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7255\r\n* Seed the tiny model generation scripts by @albertvillanova in https://github.com/huggingface/trl/pull/7258\r\n* Align tiny Gemma2 config with google/gemma-2-2b-it by @albertvillanova in https://github.com/huggingface/trl/pull/7228\r\n* Align tiny Phi-3 config with microsoft/Phi-3-mini-4k-instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7257\r\n* Move TRL loss kernels into TRL by @kashif in https://github.com/huggingface/trl/pull/7266\r\n* Narrow the token permissions in the test workflows by @albertvillanova in https://github.com/huggingface/trl/pull/7287\r\n* Redirect the KTO liger loss through the distributed wrapper under DDP by @qgallouedec in https://github.com/huggingface/trl/pull/7247\r\n* Inject MODEL_REVISIONS into the PEFT loaders in tests by @albertvillanova in https://github.com/huggingface/trl/pull/7288\r\n* Run the experimental tests when their workflow changes by @albertvillanova in https://github.com/huggingface/trl/pull/7278\r\n* Move the code quality check to its own workflow by @albertvillanova in https://github.com/huggingface/trl/pull/7277\r\n* Stop running the tests for changes that cannot affect them by @albertvillanova in https://github.com/huggingface/trl/pull/7281\r\n* Stream GRPO log probabilities by @kashif in https://github.com/huggingface/trl/pull/7077\r\n* Restore ACCELERATE_MIXED_PRECISION between tests by @albertvillanova in https://github.com/huggingface/trl/pull/7321\r\n* Bump the \"github-actions-and-pre-commit\" group with 2 updates across multiple ecosystems by @dependabot[bot] in https://github.com/huggingface/trl/pull/7299\r\n* fix(ci): harden GitHub Actions workflows (#7299) by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/7300\r\n* Stop depending on the tiny Bloom model in the chat template tests by @albertvillanova in https://github.com/huggingface/trl/pull/7262\r\n* Use non identity adapters in the use_adapter tests by @albertvillanova in https://github.com/huggingface/trl/pull/7275\r\n* Run the tests when test data or shipped package data changes by @albertvillanova in https://github.com/huggingface/trl/pull/7283\r\n* Match the out-of-memory message instead of the exception type in the rerun filter by @albertvillanova in https://github.com/huggingface/trl/pull/7292\r\n* Raise the pytest-rerunfailures floor to 16.7 so the rerun filter matches wrapped exceptions by @albertvillanova in https://github.com/huggingface/trl/pull/7325\r\n* Pass the FSDP2 ignored_params kwarg only on torch 2.7 or later by @albertvillanova in https://github.com/huggingface/trl/pull/7259\r\n* Guard the torch 2.6 APIs used by FSDP2 and online DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7260\r\n* Align tiny Phi-3.5 config with microsoft/Phi-3.5-mini-instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7276\r\n* Mirror use_mrope in the tiny DeepSeek-R1-Distill config by @albertvillanova in https://github.com/huggingface/trl/pull/7332\r\n* Mirror use_mambapy in the tiny FalconMamba config by @albertvillanova in https://github.com/huggingface/trl/pull/7333\r\n* Use fused kernels for logprobs and entropy by @kashif in https://github.com/huggingface/trl/pull/7253\r\n* Run the kernel publish job under a personal namespace by @qgallouedec in https://github.com/huggingface/trl/pull/7343\r\n* Mirror rope_scaling in the tiny Gemma config by @albertvillanova in https://github.com/huggingface/trl/pull/7334\r\n* Mirror cache_implementation and hidden_act in the tiny Gemma2 config by @albertvillanova in https://github.com/huggingface/trl/pull/7335\r\n* Mirror activation_dropout and prefix in the tiny OPT config by @albertvillanova in https://github.com/huggingface/trl/pull/7336\r\n* Mirror attention_bias in the tiny Phi-3 config by @albertvillanova in https://github.com/huggingface/trl/pull/7337\r\n* Mirror attention_bias in the tiny Phi-3.5 config by @albertvillanova in https://github.com/huggingface/trl/pull/7338\r\n* Set the reward model pad token on the text config by @albertvillanova in https://github.com/huggingface/trl/pull/7318\r\n* Write the aligned token ids to the text config by @albertvillanova in https://github.com/huggingface/trl/pull/7315\r\n* Drop the fused Liger JSD path from the experimental distillation trainers by @kashif in https://github.com/huggingface/trl/pull/7301\r\n* Remove the experimental GSPO-token trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7331\r\n* Move DPO to chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7243\r\n* Apply the requested pad token to vision datasets in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7352\r\n* Reduce connection-pool warnings in the vLLM client by @vladbataev in https://github.com/huggingface/trl/pull/7199\r\n* Read the MoE auxiliary loss coefficient from the model config by @qgallouedec in https://github.com/huggingface/trl/pull/7248\r\n* Hotfix CI: Expect the Llava assistant masks tests to pass now that transformers fixed them by @albertvillanova in https://github.com/huggingface/trl/pull/7373\r\n* Test that the reward model pad token is set on the text config by @albertvillanova in https://github.com/huggingface/trl/pull/7350\r\n* Set the pad token for vision datasets too by @albertvillanova in https://github.com/huggingface/trl/pull/7316\r\n* Drop vLLM 0.19.1 support by @qgallouedec in https://github.com/huggingface/trl/pull/7366\r\n* Document what use_liger_kernel actually does in each trainer by @qgallouedec in https://github.com/huggingface/trl/pull/7361\r\n* Add support for vLLM 0.30.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7365\r\n* Ship the logprob and entropy kernel in trl instead of the Hub by @qgallouedec in https://github.com/huggingface/trl/pull/7363\r\n* Agree on logged keys across ranks before flushing GRPO and RLOO metrics by @behroozazarkhalili in https://github.com/huggingface/trl/pull/7382\r\n* Release: v1.14 by @albertvillanova in https://github.com/huggingface/trl/pull/7392\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.13.0...v1.14.0\r\n","publishedAt":"2026-09-25T06:40:07.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.14.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":28250,"contentTokens":7873,"composition":{"bugs":3,"features":8,"enhancements":4}},{"id":"rel_RkH5ThUioDq7NBUVmU8of","version":"v0.21.0","type":"feature","title":"v0.21.0","summary":"Four new fine-tuning methods ship: a Riemannian-preconditioned LoRA optimizer, the KaSA LoRA variant, Super-Tuning sparsity allocation, and ShadowPEFT. Also fixed a LoRA scaling applied twice in the SVD path of add_weighted_adapter, sped up MoE target parameter computation, and reduced Orthogonal Subspace Fine-tuning memory 22% and training time 46%.","titleGenerated":"PEFT v0.21.0 adds Riemannian, KaSA, Super-Tuning, and ShadowPEFT methods","titleShort":"Riemannian LoRA, KaSA, Super-Tuning, ShadowPEFT added","breaking":"minor","importance":4,"content":"# Highlights\r\n\r\n<img width=\"1248\" height=\"560\" alt=\"peft-v0 21 0\" src=\"https://github.com/user-attachments/assets/c2b8b162-6564-4c7f-b5b6-130b8808ea18\" />\r\n\r\n## New methods\r\n\r\n### Riemannian-preconditional LoRA optimizer\r\n\r\nRiemannian-preconditioned LoRA optimizer (#3382), contributed by @smellslikeml, is a new LoRA-specific optimizer similar to [LoRA-FA](https://huggingface.co/docs/peft/main/en/package_reference/lora#lora-fa-optimizer) or [LoRA+](https://huggingface.co/docs/peft/main/en/package_reference/lora#lora-optimized-lora) that applies a Riemannian pre-conditioning on the gradients to improve feature learning and avoiding separate tuning of learning rates for LoRA's A and B matrices.\r\n\r\n### LoRA variant: Knowledge-Aware Singular-Value Adaption (KaSA)\r\n\r\n[KaSA](https://huggingface.co/docs/peft/main/en/package_reference/lora_variant_kasa) aims at identifying knowledge aware singular values from the base weight. It then removes noisy knowledge from those weights and specifically trains the singular values most relevant to the task. Implemented in #3446 by @robbiebusinessacc and @iambogeumkim.\r\n\r\n### Super-Tuning (#3518)\r\n\r\n[Super-Tuning](https://huggingface.co/docs/peft/main/en/package_reference/supertuning) uses calibration-free magnitude scoring to allocate a sparse tuning budget. With `BottomK` selection, Super adapts low-magnitude weights, motivated by the idea that they provide degrees of freedom for learning a new task while leaving larger-magnitude pretrained weights untouched. Supra adds LoRA on top of this sparse update. It is a good fit for exploring sparsity alongside higher-rank adaptation, particularly on reasoning-heavy fine-tuning tasks. Thanks to @smellslikeml.\r\n\r\n### ShadowPEFT (#3354)\r\n\r\n[ShadowPEFT](https://huggingface.co/docs/peft/main/en/package_reference/shadow) runs a small shadow network alongside the frozen base model.  That shadow state gets updated at every decoder layer, so adaptation is shared across the stack instead of a separate low-rank update on each targeted linear layer (like LoRA). It’s a good choice when you care about downstream or OOD performance and want to stay in the same trainable-parameter range as LoRA, or when you want the small shadow on the edge and the frozen base in the cloud. This works best if the shadow model starts from a small, pretrained model. Contributed by @SeanLee97.\r\n\r\n## Enhancements\r\n\r\n* Speed up MoE target parameter computation (#3577): For the common use case of one active adapter, weight and MoE LoRA delta computation is folded into a single `baddbmm` operation. Expect slight numerical differences in this case.\r\n\r\n* Support torchao v0.18.0 by supporting the newer Int8Tensor, Float8Tensor, etc. tensor subclasses.\r\n\r\n* [Orthogonal Subspace Fine-tuning](https://huggingface.co/docs/peft/main/en/package_reference/osf) received a significant update in #3524 reducing the memory footprint by 22% and the training time by 46% according to the MetaMathQA benchmark while keeping task accuracy and forgetting stable. Expect slight numerical differences, especially in fp16, bf16.\r\n\r\n* https://github.com/huggingface/peft/pull/3639: Thanks to @amogh-nagri-11, prompt learning methods now allow to delete an adapter. For adaption prompt, adding or loading a second adapter no longer automatically activates it, in line with how other PEFT methods handle this.\r\n\r\n\r\n## Changes\r\n\r\n### Refactored state dict handling (#3490)\r\n\r\nAn internal change moved state dict loading/saving method-specific code paths to the respective tuners. This should be transparent to the end-user and was extensively tested and should work exactly the same as previously, but if you suspect that the model behaves differently, let us know.\r\n\r\n### Preserved adapter trainablity during injection (#3495)\r\n\r\nPreviously, when loading or adding a second, third etc. adapter, this could affect the `requires_grad` of the existing adapter (or base model), depending on the value of `is_trainable`. This should no longer be the case thanks to @DaoyuanLi2816.\r\n\r\n### Bug fixes for LoRA adapter combining (#3111)\r\n\r\nCombining LoRA adapters using `add_weighted_adapter` had a bug in the `SVD` path which led to the LoRA scaling being applied twice. This is now fixed.\r\n\r\n### OFT uses generic quantization backend (#3355)\r\n\r\nWe moved OFT from a method-specific quantization backend to the generic quantization backend. This affects bitsandbytes, GPT-Q, etc. This should not affect existing checkpoints, but please let us know if you run into issues.\r\n\r\n### Stricter checks on adapter handling\r\n\r\n* #3667: `inject_adapter_in_model` now rejects PEFT methods that have shared state that would live on the parent model (like VeRA), as no such parent model exists. Contributed by @CoralGarden52.\r\n\r\n* #3534: Thanks to @LK-maker-007, the PEFT now throws an error when adding a second adapter to the PEFT model using `add_adapter` that targets non-existing layers. Before, the adapter would just not adapt anything but be silent about it.  `peft_model.targeted_module_names` and `peft_model.targeted_parameter_names` no longer contain duplicates. They are still sorted by insertion order (not sets). To suppress potential errors, catch `peft.PeftError`.\r\n\r\n* #3504: When deleting an adapter, there is now a check if the adapter is merged. If it's merged, an error is raised, since deleting it would not remove its effect. This was contributed by @DaoyuanLi2816.\r\n\r\n## All Changes\r\n* Fix sequential merge invariants for FRoD, HiRA, and PEANuT by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3482\r\n* docs: improve VeRA documentation by @Ashmitharaja in https://github.com/huggingface/peft/pull/3476\r\n* Bump version to 0.20.1.dev0 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3484\r\n* Fix DeepSpeed guide example script path by @latent-9 in https://github.com/huggingface/peft/pull/3494\r\n* CI: Bump the ci-actions group with 2 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3496\r\n* chore: update torch_compile_tests.yml by @hf-security-analysis[bot] in https://github.com/huggingface/peft/pull/3501\r\n* Add Riemannian-preconditioned LoRA optimizer by @smellslikeml in https://github.com/huggingface/peft/pull/3382\r\n* docs: API coverage tooling + fixes by @githubnemo in https://github.com/huggingface/peft/pull/3465\r\n* ENH Add KaSA (Knowledge-aware Singular-value Adaptation) as a LoRA variant by @robbiebusinessacc in https://github.com/huggingface/peft/pull/3446\r\n* FIX Crash when unloading a model whose active adapter does not use modules_to_save by @javierdejesusda in https://github.com/huggingface/peft/pull/3455\r\n* TST Add regression tests for PEFT checkpoint loading by @BenjaminBossan in https://github.com/huggingface/peft/pull/3489\r\n* Reject deletion of merged adapters by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3504\r\n* TST Add RandLoRA to feature extraction, decoder, encoder-decoder and seq classifier tests by @ishan-1010 in https://github.com/huggingface/peft/pull/3513\r\n* Check merged adapters when merging trainable tokens one at a time by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3514\r\n* CHORE Improve type annotations of PeftModel and get_peft_model by @BenjaminBossan in https://github.com/huggingface/peft/pull/3493\r\n* FIX Route embedding LoRA params to the LoRA+ embedding group by @Kaif10 in https://github.com/huggingface/peft/pull/3503\r\n* MetaMathQA: consistent tokenizer padding behavior by @githubnemo in https://github.com/huggingface/peft/pull/3526\r\n* docs: add installation instructions for NVIDIA Spark (ARM64) devices by @mfuntowicz in https://github.com/huggingface/peft/pull/3536\r\n* Add type hints to helpers.py, hotswap.py, constants.py, integrations.py (consolidates #3448, #3452) by @RudrenduPaul in https://github.com/huggingface/peft/pull/3529\r\n* Refactor state dict handling by @BenjaminBossan in https://github.com/huggingface/peft/pull/3490\r\n* TST Fix some multi GPU tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/3458\r\n* Method comparison: Small optimizations to image-gen by @BenjaminBossan in https://github.com/huggingface/peft/pull/3523\r\n* FIX TST Use distributed_config kwarg for TP tests with transformers >= 5.13 by @peft-jambot in https://github.com/huggingface/peft/pull/3502\r\n* FIX Issue with key matching when dealing with disk offloading by @BenjaminBossan in https://github.com/huggingface/peft/pull/3468\r\n* FEAT Add git dirty flag to method_comparison benchmark results by @peft-jambot in https://github.com/huggingface/peft/pull/3385\r\n* Method comparison: Reduce peak memory usage of image generation task by @BenjaminBossan in https://github.com/huggingface/peft/pull/3395\r\n* CHORE Upgrade ruff to 0.16.2 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3545\r\n* FIX Accept layers_to_transform=0 together with layers_pattern by @winklemad in https://github.com/huggingface/peft/pull/3425\r\n* Docs: cuda windows arm64 update installation guidelines by @mfuntowicz in https://github.com/huggingface/peft/pull/3550\r\n* MetaMathQA: Add bank-1024 VBLoRA configuration by @szzhoujiarui in https://github.com/huggingface/peft/pull/3546\r\n* make device agnostic in docs and examples README by @kaixuanliu in https://github.com/huggingface/peft/pull/3551\r\n* method_comparison/LoKr: new hyper-params by @githubnemo in https://github.com/huggingface/peft/pull/3535\r\n* FIX DoRA unmerge on fan_in_fan_out layers by @LK-maker-007 in https://github.com/huggingface/peft/pull/3532\r\n* Warn when from_pretrained config-field kwargs are ignored by @akshansh47 in https://github.com/huggingface/peft/pull/3516\r\n* FIX inject_adapter raises when a non-first adapter matches nothing by @LK-maker-007 in https://github.com/huggingface/peft/pull/3534\r\n* DOC Fix wrong paper links for LoHa and LoKr by @BenjaminBossan in https://github.com/huggingface/peft/pull/3547\r\n* CI: Bump the third-party-actions group across 1 directory with 7 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3497\r\n* Fix IA3 Conv3d layer error message to reference Conv3d by @uttam12331 in https://github.com/huggingface/peft/pull/3505\r\n* FIX Transformers warnings use return dict, missing inputs_embeds argument, fix causal_mask_creation_function call by @BenjaminBossan in https://github.com/huggingface/peft/pull/3558\r\n* CHORE Dockerfile installs mslk differently by @BenjaminBossan in https://github.com/huggingface/peft/pull/3557\r\n* remove ipex config since it has been removed from accelerate by @sywangyi in https://github.com/huggingface/peft/pull/3564\r\n* STY Remove unused local variables in src/peft by @rajarshidattapy in https://github.com/huggingface/peft/pull/3567\r\n* FIX PVeRA per-module sample_at_inference and generator device by @rajarshidattapy in https://github.com/huggingface/peft/pull/3563\r\n* DOC Fix leftover quotes in GraLoRA target_modules docs by @rajarshidattapy in https://github.com/huggingface/peft/pull/3568\r\n* FIX Reject duplicate adapter names by @tandede in https://github.com/huggingface/peft/pull/3559\r\n* Add Super-Tuning as a new PEFT method by @smellslikeml in https://github.com/huggingface/peft/pull/3518\r\n* Resolve ZeRO-3 partitioned shapes in lora.ParamWrapper by @qgallouedec in https://github.com/huggingface/peft/pull/3576\r\n* fix(peft): make disable_adapter context manager reentrant (#3555) by @lllakshit in https://github.com/huggingface/peft/pull/3565\r\n* FIX Preserve input state dict during adapter loading by @aswanth-07 in https://github.com/huggingface/peft/pull/3596\r\n* Fold the MoE LoRA delta into a single `baddbmm` by @qgallouedec in https://github.com/huggingface/peft/pull/3577\r\n* FIX IA3 grouped convolution feedforward merge by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3575\r\n* Fix LoRA target_parameters raising a dtype error under autocast by @albertvillanova in https://github.com/huggingface/peft/pull/3602\r\n* FIX AdaMSS, PSOFT: fork the RNG of the actual device by @jiqing-feng in https://github.com/huggingface/peft/pull/3598\r\n* DOC Add troubleshooting section for hybrid architecture target_modules by @davidnichols-ops in https://github.com/huggingface/peft/pull/3603\r\n* Add BEFT experiment and results to image-gen benchmark by @Sergasgr in https://github.com/huggingface/peft/pull/3607\r\n* Fix LoRA on FSDP2-sharded models: shape inference and module hooks by @qgallouedec in https://github.com/huggingface/peft/pull/3578\r\n* Feat: support ShadowPEFT by @SeanLee97 in https://github.com/huggingface/peft/pull/3354\r\n* CI Fix ruff version to 0.16.4 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3613\r\n* Preserve adapter trainability during injection by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3495\r\n* TST Ignore floating point noise in gradient checkpointing test by @jiqing-feng in https://github.com/huggingface/peft/pull/3599\r\n* Preserve LN Tuning adapter devices when adding adapters by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3616\r\n* CI: Bump the third-party-actions group with 6 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3634\r\n* fix: support torchao v0.18.0 which removed AffineQuantizedTensor by @peft-jambot in https://github.com/huggingface/peft/pull/3571\r\n* FIX BOFT and HRA crash on grouped Conv2d layers by @AshNicolus in https://github.com/huggingface/peft/pull/3527\r\n* Change layers_pattern logic by @BenjaminBossan in https://github.com/huggingface/peft/pull/2158\r\n* FIX DoRA merge with multiple adapters on the same layer by @LK-maker-007 in https://github.com/huggingface/peft/pull/3609\r\n* Refuse to hot-swap adapters while they are merged into the base weights by @Sravanjangam in https://github.com/huggingface/peft/pull/3590\r\n* Add HiRA experiment and results to image-gen benchmark by @Sergasgr in https://github.com/huggingface/peft/pull/3640\r\n* Add AdaMSS experiment and results to image-gen benchmark; wire up ASA training callback by @Sergasgr in https://github.com/huggingface/peft/pull/3641\r\n* Extend the `prepare_model_for_kbit_training` memory leak regression t… by @sywangyi in https://github.com/huggingface/peft/pull/3627\r\n* Fix nested LoRA sub-configs silently staying dicts after save/load round-trip by @Sravanjangam in https://github.com/huggingface/peft/pull/3587\r\n* Make LoHa/LoKr merge and unmerge deterministic while in train mode by @Sravanjangam in https://github.com/huggingface/peft/pull/3589\r\n* CI Don't install EETQ in Dockerfile as it conflicts with torch 2.14 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3652\r\n* TST Fix nightly CI regarding kernels version, torchao error by @BenjaminBossan in https://github.com/huggingface/peft/pull/3653\r\n* AI Add skill for making changes to PEFT methods by @BenjaminBossan in https://github.com/huggingface/peft/pull/3544\r\n* Treat empty base_model_name_or_path as missing when saving adapters by @victorzhong0110 in https://github.com/huggingface/peft/pull/3647\r\n* Update HiRA experiment with tuned learning rate by @Sergasgr in https://github.com/huggingface/peft/pull/3661\r\n* Update BEFT experiment with expanded target_modules by @Sergasgr in https://github.com/huggingface/peft/pull/3663\r\n* TST Update adaption prompt tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/3573\r\n* refactor: move OFT to generic quantization framework by @peft-jambot in https://github.com/huggingface/peft/pull/3355\r\n* Keep failed LoRA safe merges from changing base weights by @Robertboy18 in https://github.com/huggingface/peft/pull/3611\r\n* DOC Fix broken docstring examples in peft_model and tuner models by @blipbyte in https://github.com/huggingface/peft/pull/3657\r\n* FIX ShadowPEFT last-token pooling with left padding by @aswanth-07 in https://github.com/huggingface/peft/pull/3629\r\n* Fix Trainable Tokens random-init unmerge by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3656\r\n* Validate Super-Tuning index persistence across adapters by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3665\r\n* FIX X-LoRA adapter lifecycle: disable_adapter, training mode and frozen classifier by @amogh-nagri-11 in https://github.com/huggingface/peft/pull/3636\r\n* Reject direct injection for tuners with shared adapter state by @CoralGarden52 in https://github.com/huggingface/peft/pull/3667\r\n* docs: Fix duplicate TOC entries by @githubnemo in https://github.com/huggingface/peft/pull/3696\r\n* CHORE Add triage bot for GitHub PRs by @BenjaminBossan in https://github.com/huggingface/peft/pull/3680\r\n* feat: delta-based forward pass for OSF to reduce memory and compute by @peft-jambot in https://github.com/huggingface/peft/pull/3524\r\n* FIX Adapter management for prompt learning and adaption prompt by @amogh-nagri-11 in https://github.com/huggingface/peft/pull/3639\r\n* Validate IA3 bias before committing safe merges by @Robertboy18 in https://github.com/huggingface/peft/pull/3694\r\n* TST Accelerate mixed adapter batches tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/3630\r\n* TST Add PEFT testing guide by @BenjaminBossan in https://github.com/huggingface/peft/pull/3648\r\n* FIX Several bugs when adding merged LoRA weights by @BenjaminBossan in https://github.com/huggingface/peft/pull/3111\r\n* Fail loud on save_pretrained() for unsharded LoRA tensors by @akshansh47 in https://github.com/huggingface/peft/pull/3251\r\n* Update UniLoRA MetaMathQA benchmark configs with tuned hyperparameters by @peft-jambot in https://github.com/huggingface/peft/pull/3552\r\n* FIX Reject MiSS and PSOFT ranks the layer cannot support by @behroozazarkhalili in https://github.com/huggingface/peft/pull/3708\r\n* FIX Task-type add_adapter: forward autocast_adapter_dtype, stop mutating modules_to_save by @Vedant-Agarwal in https://github.com/huggingface/peft/pull/3668\r\n* Delete every auxiliary adapter and resolve the remaining active ones across layers by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3515\r\n* Add UniLoRA experiment by @Sergasgr in https://github.com/huggingface/peft/pull/3740\r\n* docs: pool four small doc fixes into one PR by @simpleqt in https://github.com/huggingface/peft/pull/3687\r\n* Warn when rank_pattern / alpha_pattern keys match no targeted module by @Sravanjangam in https://github.com/huggingface/peft/pull/3588\r\n* fix: handle scalar adapter state entries by @aswanth-07 in https://github.com/huggingface/peft/pull/3699\r\n* Release: 0.21.0 by @githubnemo in https://github.com/huggingface/peft/pull/3748\r\n\r\n## New Contributors\r\n* @Ashmitharaja made their first contribution in https://github.com/huggingface/peft/pull/3476\r\n* @latent-9 made their first contribution in https://github.com/huggingface/peft/pull/3494\r\n* @smellslikeml made their first contribution in https://github.com/huggingface/peft/pull/3382\r\n* @robbiebusinessacc made their first contribution in https://github.com/huggingface/peft/pull/3446\r\n* @ishan-1010 made their first contribution in https://github.com/huggingface/peft/pull/3513\r\n* @Kaif10 made their first contribution in https://github.com/huggingface/peft/pull/3503\r\n* @mfuntowicz made their first contribution in https://github.com/huggingface/peft/pull/3536\r\n* @winklemad made their first contribution in https://github.com/huggingface/peft/pull/3425\r\n* @szzhoujiarui made their first contribution in https://github.com/huggingface/peft/pull/3546\r\n* @LK-maker-007 made their first contribution in https://github.com/huggingface/peft/pull/3532\r\n* @akshansh47 made their first contribution in https://github.com/huggingface/peft/pull/3516\r\n* @uttam12331 made their first contribution in https://github.com/huggingface/peft/pull/3505\r\n* @rajarshidattapy made their first contribution in https://github.com/huggingface/peft/pull/3567\r\n* @tandede made their first contribution in https://github.com/huggingface/peft/pull/3559\r\n* @lllakshit made their first contribution in https://github.com/huggingface/peft/pull/3565\r\n* @aswanth-07 made their first contribution in https://github.com/huggingface/peft/pull/3596\r\n* @albertvillanova made their first contribution in https://github.com/huggingface/peft/pull/3602\r\n* @davidnichols-ops made their first contribution in https://github.com/huggingface/peft/pull/3603\r\n* @Sergasgr made their first contribution in https://github.com/huggingface/peft/pull/3607\r\n* @SeanLee97 made their first contribution in https://github.com/huggingface/peft/pull/3354\r\n* @Sravanjangam made their first contribution in https://github.com/huggingface/peft/pull/3590\r\n* @victorzhong0110 made their first contribution in https://github.com/huggingface/peft/pull/3647\r\n* @Robertboy18 made their first contribution in https://github.com/huggingface/peft/pull/3611\r\n* @amogh-nagri-11 made their first contribution in https://github.com/huggingface/peft/pull/3636\r\n* @CoralGarden52 made their first contribution in https://github.com/huggingface/peft/pull/3667\r\n* @behroozazarkhalili made their first contribution in https://github.com/huggingface/peft/pull/3708\r\n* @Vedant-Agarwal made their first contribution in https://github.com/huggingface/peft/pull/3668\r\n* @simpleqt made their first contribution in https://github.com/huggingface/peft/pull/3687\r\n\r\n**Full Changelog**: https://github.com/huggingface/peft/compare/v0.20.0...v0.21.0","publishedAt":"2026-09-15T15:31:52.000Z","url":"https://github.com/huggingface/peft/releases/tag/v0.21.0","media":[{"type":"image","url":"https://github.com/user-attachments/assets/c2b8b162-6564-4c7f-b5b6-130b8808ea18","r2Key":"releases/289fd06533e59fa4f07ca81b35c27870fece6f5942f46ff1c266c38a9622921a.png","r2Url":"https://media.releases.sh/releases/289fd06533e59fa4f07ca81b35c27870fece6f5942f46ff1c266c38a9622921a.png"}],"prerelease":false,"source":{"slug":"peft","name":"PEFT","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":21245,"contentTokens":5848,"composition":{"bugs":4,"features":4,"enhancements":4}},{"id":"rel_uJYz2MFBk1vhv7Ecg7aa0","version":"v1.13.0","type":"feature","title":"v1.13.0","summary":"PPOTrainer, PPOConfig, and the value-head modeling modules are removed. The default chunked_nll lm_head projection now runs on bf16 tensor cores instead of fp32 SIMT, cutting a 23.37 ms fwd+bwd chunk to 3.86 ms (−6×) and peak memory from 5.99 GB to 3.03 GB, with end-to-end tokens/s gains of 1.2–1.7× across models. Vendored fused linear losses (DPO, KTO, GRPO, JSD) into trl.losses, added a long-context guide for 1M-token training, raised dependency floors (peft, deepspeed, vLLM), and added QLoRA test coverage.","titleGenerated":"TRL v1.13.0 drops PPO trainer and accelerates chunked-NLL lm_head projection 6x","titleShort":"PPO trainer removed; chunked-CE lm_head 6× faster","breaking":"major","importance":4,"content":"## Features\r\n\r\n### Training beyond 1M tokens\r\n\r\n<img width=\"1672\" height=\"941\" alt=\"image\" src=\"https://github.com/user-attachments/assets/806f9153-d098-4a45-bef4-5d4119b7ef25\" />\r\n\r\n\r\nA new [long-context guide](https://huggingface.co/docs/trl/long_context_training) plus a runnable example that trains a book-length sequence per step on a single 8×H100 node.\r\n\r\nMeasured: **Qwen3-8B at 1,048,576 tokens, 380 s/step, 56.2 GB per GPU** (bf16, `per_device_train_batch_size=1`, `loss_type=\"chunked_nll\"`). Runs end to end and saves. The guide documents the levers in the order you hit them (`chunked_nll`, gradient-checkpointing `offload`, YaRN RoPE) and the constraints (full attention only, no packing). Needs `transformers >= 5.16` for checkpointing `offload`.\r\n\r\n* Document 1M-token training and add a context-parallelism example by @qgallouedec in https://github.com/huggingface/trl/pull/6846\r\n* Rewrite the long context guide by @qgallouedec in https://github.com/huggingface/trl/pull/7003\r\n\r\n\r\n### Chunked-CE `lm_head` projection now runs on tensor cores\r\n\r\nThe inner loop of the default `loss_type=\"chunked_nll\"` was doing `h.float() @ w.float().t()`. Both operands are already bf16, so the upcast bought no information: it moved the GEMM off the tensor cores onto the fp32 SIMT path and materialized an fp32 copy of the whole `lm_head` weight (2.03 GB for a 248k vocabulary), rebuilt for every chunk and again on every gradient-checkpoint recompute. In an 8×H100 profile of `trl sft` on Qwen3.6-35B-A3B those two fp32 SIMT GEMMs were **21.6% of all GPU kernel time**.\r\n\r\nOne chunk, 256 tokens × vocab 248,320 × hidden 2048, 1×H100, bf16, fwd+bwd:\r\n\r\n| | fwd+bwd | peak mem |\r\n| --- | --- | --- |\r\n| before | 23.37 ms | 5.99 GB |\r\n| after | **3.86 ms (6.0×)** | **3.03 GB** |\r\n\r\nEnd to end, 16,384 tokens per step, tokens/s/GPU:\r\n\r\n| model | mode | before | after | |\r\n| --- | --- | --- | --- | --- |\r\n| gemma-3-270m (vocab 262k) | full FT, 1×H100 | 24036 | **31609** | 1.32× |\r\n| Qwen3-0.6B (vocab 152k) | full FT, 1×H100 | 21276 | **26641** | 1.25× |\r\n| Qwen3-8B | full FT, 2×H100 FSDP2 | 3554 | **6009** | **1.69×** |\r\n| Qwen3-8B | LoRA r16, 2×H100 FSDP2 | 4531 | **7125** | **1.57×** |\r\n| Qwen3-30B-A3B (MoE) | LoRA attn, 2×H100 FSDP2 | 4251 | **5101** | 1.20× |\r\n\r\nSame fix applied to the distillation trainers (which paid it twice per chunk, student and teacher). Under accelerate mixed precision the numerics are bit-identical (autocast was already casting the operands back down); without autocast over the projection the GEMM moves to bf16 tensor cores, which is exactly what `loss_type=\"nll\"` already does.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6863\r\n\r\n### Fused linear losses vendored into `trl.losses`\r\n\r\nTRL's only import from `liger-kernel` was `liger_kernel.chunked_loss` (the fused linear DPO / KTO / GRPO / JSD losses). That module is pure PyTorch, and most of the GRPO and DPO variants in it were written for TRL. Upstream reviews had stalled, so the code comes home as `trl.losses`: `FusedLinearDPOLoss`, `FusedLinearKTOLoss`, `FusedLinearGRPOLoss`, `FusedLinearJSDLoss` (copied from Liger-Kernel v0.8.2, BSD-2 notice kept, bitwise identical to the installed Liger on random inputs).\r\n\r\nNo new config flag: `use_liger_kernel=True` keeps selecting the fused loss exactly as before, and still needs `liger-kernel` installed because transformers' `Trainer` patches the model kernels with it.\r\n\r\n* Vendor the fused linear losses from Liger-Kernel into `trl.losses` by @qgallouedec in https://github.com/huggingface/trl/pull/7059\r\n* Fix DPO and KTO Liger loss gaps by @kashif in https://github.com/huggingface/trl/pull/7062\r\n* Harden chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7065\r\n* Drop fused JSD from `DistillationTrainer` by @kashif in https://github.com/huggingface/trl/pull/7064\r\n* Test liger vs non-liger GRPO loss parity by @kashif in https://github.com/huggingface/trl/pull/6814\r\n* Run the vendored loss parity suites as slow tests only by @qgallouedec in https://github.com/huggingface/trl/pull/7067\r\n\r\n### Contributor skills at `.agents/skills`\r\n\r\nAgent skills for contributing to TRL now ship in the repo at `.agents/skills`.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6901\r\n\r\n### Docs: stable vs experimental taxonomy, and how to change the objective\r\n\r\n* **Split the docs taxonomy into stable and experimental trainers** by @qgallouedec in https://github.com/huggingface/trl/pull/7022\r\n* **Document how to change the training objective by subclassing a trainer** by @sergiopaniego in https://github.com/huggingface/trl/pull/6918\r\n* **Document how to choose and pin a Hub kernel version** by @albertvillanova in https://github.com/huggingface/trl/pull/6998\r\n* Add missing TRL blog posts to the docs landing page by @qgallouedec in https://github.com/huggingface/trl/pull/7023\r\n\r\n### QLoRA on the `quantization_config` argument\r\n\r\nThe QLoRA test suites move onto the `quantization_config` trainer argument added in v1.8, and gain coverage on the VLM paths.\r\n\r\n* Add a QLoRA test to the GRPO and RLOO VLM test suites by @albertvillanova in https://github.com/huggingface/trl/pull/6909\r\n* Add a QLoRA test to the trainers that support `quantization_config` by @albertvillanova in https://github.com/huggingface/trl/pull/6910\r\n* Convert the QLoRA tests to the `quantization_config` trainer argument by @albertvillanova in https://github.com/huggingface/trl/pull/6927\r\n\r\n### Dependency floors raised\r\n\r\n* Require `peft>=0.13.0` and drop the 0.12 version guards by @qgallouedec in https://github.com/huggingface/trl/pull/7102\r\n* Require `deepspeed>=0.18.6` and drop the 0.16.4 guard by @albertvillanova in https://github.com/huggingface/trl/pull/7118\r\n* Guard `target_parameters` use for `peft<0.17.0` by @albertvillanova in https://github.com/huggingface/trl/pull/7119\r\n* Add support for vLLM 0.28.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7001\r\n* Drop vLLM 0.19.0 support by @qgallouedec in https://github.com/huggingface/trl/pull/7002\r\n\r\n### Other\r\n\r\n* **Count each token once in `num_tokens` under tensor parallelism** by @qgallouedec in https://github.com/huggingface/trl/pull/7101\r\n* **XPO / NashMD perf**: avoid `output_hidden_states` when only `last_hidden_state` is used by @ciaoyizhen in https://github.com/huggingface/trl/pull/4755\r\n* Sync the model config pad token id with the tokenizer in the trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7097 and (experimental) https://github.com/huggingface/trl/pull/7128\r\n* Forward model revision to the auto-loaded `processing_class` by @kashif in https://github.com/huggingface/trl/pull/6978\r\n* Inject the model revision into concrete config loads too by @albertvillanova in https://github.com/huggingface/trl/pull/7099\r\n* Match hub kernels by repo id when checking the attention implementation by @albertvillanova in https://github.com/huggingface/trl/pull/6985\r\n* Catch `AsyncDistillationTrainer` up with recent AsyncGRPO changes by @kashif in https://github.com/huggingface/trl/pull/6823 and https://github.com/huggingface/trl/pull/6987\r\n* Remove the leftover `dataset_formatting` module by @albertvillanova in https://github.com/huggingface/trl/pull/7117\r\n* Remove the unused `FLASH_ATTENTION_VARIANTS` constant from the DPO trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7013\r\n* Document that padding-free is temporarily unavailable in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7014\r\n* Realign duplicated comments across the trainers by @qgallouedec in https://github.com/huggingface/trl/pull/7145\r\n\r\n## Breaking\r\n\r\n### ⚠️ PPO is removed\r\n\r\n`PPOTrainer`, `PPOConfig`, and `modeling_value_head.py` (`PreTrainedModelWrapper`, `AutoModelForCausalLMWithValueHead`, `AutoModelForSeq2SeqLMWithValueHead`) are gone, along with their tests, doc page, and examples.\r\n\r\nBit of a moment for TRL: `PPOTrainer` predates every PR in this repo. It landed in `dfb6a580` on 2020-03-28, the commit that first added the library, back when the package was called `lm_ppo`. It was the oldest thing in TRL and the last piece of the original codebase.\r\n\r\nWhy now: unmaintained (no feature work in over a year, every PPO commit since #4482 a drive-by fix), the only trainer never aligned on the input format (it still took tokenized `input_ids`), near-zero recorded usage, and a magnet for automated bug hunters filing real reports against code nobody runs. Nobody loses anything: `from trl import PPOTrainer` already stopped working in v1.10, so anyone actually running PPO is pinned to an older TRL and those installs keep resolving exactly as they do today.\r\n\r\n`create_reference_model` stays (BCO, A2PO and Online DPO use it).\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/7020\r\n\r\n## Fixes\r\n\r\n* **Fix missing entropy gradient in `_ChunkedLogProbFunction` backward** — `backward` accepted `grad_entropy` but never used it, so any loss backpropagating through the `entropy` output silently got zero gradient from that term, with no error or warning. Only `AsyncGRPOTrainer` read `entropy` (under `no_grad`, for logging), so no user-facing regression today, but this is exported non-experimental code and an entropy-bonus loss term would have hit it silently. By @verma8076 in https://github.com/huggingface/trl/pull/6625\r\n* **vLLM weight sync now fails instead of hanging** by @AmineDiro in https://github.com/huggingface/trl/pull/6913\r\n* **Fail fast on incompatible vLLM for default Async GRPO weight transfer** by @DimensionSTP in https://github.com/huggingface/trl/pull/6830\r\n* **Fix VLM support in AsyncGRPO** by @AmineDiro in https://github.com/huggingface/trl/pull/6839\r\n* Fix variable reference in BCO docs by @albertvillanova in https://github.com/huggingface/trl/pull/7046\r\n* Fix the IW-OPD self-distillation loss assertion in tests by @albertvillanova in https://github.com/huggingface/trl/pull/7123\r\n* Fix typos in help strings and error messages by @qgallouedec in https://github.com/huggingface/trl/pull/7146\r\n* Fix two messages that describe the wrong behaviour by @qgallouedec in https://github.com/huggingface/trl/pull/7147\r\n* Fix three broken doc links by @qgallouedec in https://github.com/huggingface/trl/pull/7155\r\n* Fix the example commands in the script docstrings by @qgallouedec in https://github.com/huggingface/trl/pull/7156\r\n* Correct what the TRL Docker image provides in the Jobs guide by @albertvillanova in https://github.com/huggingface/trl/pull/7054\r\n* Fix `paper_index` paths and add GOLD, IW-OPD, and AsyncDistillation telemetry by @YeonwooSung in https://github.com/huggingface/trl/pull/7051\r\n* Qualify doc references to experimental classes by @qgallouedec in https://github.com/huggingface/trl/pull/7158\r\n\r\n## Documentation\r\n\r\nA docstring and docs consistency pass across the repo, all by @qgallouedec:\r\n\r\n* Fix docstring type specs (#7148), reformat five docstrings to the repo style (#7149), fix docstring content drift (#7150), match config field order to the docstring order (#7151), reflow two docstrings to the doc-builder width (#7153), fix the `DistillationTrainer` signature annotations (#7152), docs formatting pass (#7154), tidy pyproject / Makefile / .gitignore (#7157)\r\n* Document the TRL Docker image tags by @albertvillanova in https://github.com/huggingface/trl/pull/7055\r\n\r\n## CI\r\n\r\n**Docker images** — released and dev builds are now separate workflows, the image version comes from the release tag, and the installed TRL version is pinned:\r\n\r\n* Split the released and dev Docker image builds into separate workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6990\r\n* Read the Docker image version from the release tag by @albertvillanova in https://github.com/huggingface/trl/pull/6991\r\n* Pin the TRL version installed in the Docker image by @albertvillanova in https://github.com/huggingface/trl/pull/6992\r\n* Add the missing trailing newline to the dev Dockerfile by @albertvillanova in https://github.com/huggingface/trl/pull/7030\r\n* chore: update `docker-build.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6997\r\n\r\n**Tiny-model config alignment** — a sweep bringing every tiny test model's config in line with its reference model, all by @albertvillanova:\r\n\r\n* Qwen2.5 (#7098), Qwen3 tiny+small (#7106), Qwen3 Instruct-2507 (#7107), GptOss (#7108), GPTNeoX / pythia-14m (#7109), Llama 3.2 + Remote (#7110), Llama 3 (#7112), Llama 3.1 (#7113)\r\n* Pin `layer_types` so tiny Gemma3 / Olmo3 cover both attention types (#6962) and tiny Cohere2 (#6963)\r\n* Show list lengths and mark truncation in the tiny-model config diff (#7105)\r\n\r\n**Other CI**, mostly by @albertvillanova:\r\n\r\n* Use the doc-builder pre-commit hook (#6799), bump its revision (#6969, #7078)\r\n* Retry tests failing on cuBLAS allocation errors the rerun filter missed (#6916)\r\n* Use ruff `select` instead of `extend-select` to keep the rule set explicit (#6964), stop dependabot from opening ruff pre-commit bumps (#6965), reformat lambda functions for code clarity (#7092)\r\n* Remove the flash-attn2 xfail markers now that the padding-free tests pass (#7057), fix the invariant suite recording instructions (#7058)\r\n* Pin the invariant suite's FA2 hub kernel to the v2 branch by @qgallouedec in https://github.com/huggingface/trl/pull/6979\r\n* Filter the benign torch DataParallel scalar-gather warning (#7019)\r\n* Fix the input metadata of the post-slack composite action (#7029)\r\n* Group Dependabot GitHub Actions and pre-commit updates in a single PR (#7088), merge the two Dependabot github-actions entries into one (#7091)\r\n* Dependabot bumps in #6958, #7000, #7079, #7089\r\n\r\n## New Contributors\r\n* @ciaoyizhen made their first contribution in https://github.com/huggingface/trl/pull/4755\r\n* @DimensionSTP made their first contribution in https://github.com/huggingface/trl/pull/6830\r\n* @YeonwooSung made their first contribution in https://github.com/huggingface/trl/pull/7051\r\n\r\n## What's Changed\r\n* ⬆️ Fix dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6939\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6940\r\n* Add a QLoRA test to the GRPO and RLOO VLM test suites by @albertvillanova in https://github.com/huggingface/trl/pull/6909\r\n* Add a QLoRA test to the trainers that support quantization_config by @albertvillanova in https://github.com/huggingface/trl/pull/6910\r\n* Convert the QLoRA tests to the quantization_config trainer argument by @albertvillanova in https://github.com/huggingface/trl/pull/6927\r\n* Fix missing entropy gradient in _ChunkedLogProbFunction backward by @verma8076 in https://github.com/huggingface/trl/pull/6625\r\n* Document how to change the training objective by subclassing a trainer by @sergiopaniego in https://github.com/huggingface/trl/pull/6918\r\n* Use the doc-builder pre-commit hook by @albertvillanova in https://github.com/huggingface/trl/pull/6799\r\n* Retry tests failing on cuBLAS allocation errors the rerun filter missed by @albertvillanova in https://github.com/huggingface/trl/pull/6916\r\n* Fix:  vLLM weight sync fails instead of hanging by @AmineDiro in https://github.com/huggingface/trl/pull/6913\r\n* Bump the actions group across 1 directory with 5 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6958\r\n* Bump https://github.com/huggingface/doc-builder from 0ab9ea03baf111ed8dd83e88233430b663127368 to 1b16dac5e33043af565fdf4c1b5b0fe81d0891c8 by @dependabot[bot] in https://github.com/huggingface/trl/pull/6969\r\n* Pin layer_types so the tiny Gemma3 and Olmo3 models cover both attention types by @albertvillanova in https://github.com/huggingface/trl/pull/6962\r\n* Pin layer_types so the tiny Cohere2 model covers both attention types by @albertvillanova in https://github.com/huggingface/trl/pull/6963\r\n* Use ruff `select` instead of `extend-select` to keep the rule set explicit by @albertvillanova in https://github.com/huggingface/trl/pull/6964\r\n* fix vlm support asyncgrpo by @AmineDiro in https://github.com/huggingface/trl/pull/6839\r\n* Split the released and dev Docker image builds into separate workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6990\r\n* chore: update docker-build.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6997\r\n* Stop dependabot from opening ruff pre-commit bumps by @albertvillanova in https://github.com/huggingface/trl/pull/6965\r\n* Read the Docker image version from the release tag by @albertvillanova in https://github.com/huggingface/trl/pull/6991\r\n* Pin the TRL version installed in the Docker image by @albertvillanova in https://github.com/huggingface/trl/pull/6992\r\n* Bump trufflesecurity/trufflehog from 3.97.0 to 3.97.1 in the actions group by @dependabot[bot] in https://github.com/huggingface/trl/pull/7000\r\n* Match hub kernels by repo id when checking the attention implementation by @albertvillanova in https://github.com/huggingface/trl/pull/6985\r\n* Remove the unused FLASH_ATTENTION_VARIANTS constant from the DPO trainer by @albertvillanova in https://github.com/huggingface/trl/pull/7013\r\n* Document that padding-free is temporarily unavailable in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/7014\r\n* Drop vLLM 0.19.0 support by @qgallouedec in https://github.com/huggingface/trl/pull/7002\r\n* Pin the invariant suite's FA2 hub kernel to the v2 branch by @qgallouedec in https://github.com/huggingface/trl/pull/6979\r\n* Filter the benign torch DataParallel scalar-gather warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/7019\r\n* Test liger vs non-liger GRPO loss parity by @kashif in https://github.com/huggingface/trl/pull/6814\r\n* Add the missing trailing newline to the dev Dockerfile by @albertvillanova in https://github.com/huggingface/trl/pull/7030\r\n* Fix the input metadata of the post-slack composite action by @albertvillanova in https://github.com/huggingface/trl/pull/7029\r\n* Catch AsyncDistillationTrainer up with recent AsyncGRPO changes by @kashif in https://github.com/huggingface/trl/pull/6823\r\n* Document 1M-token training and add a context-parallelism example by @qgallouedec in https://github.com/huggingface/trl/pull/6846\r\n* Rewrite the long context guide by @qgallouedec in https://github.com/huggingface/trl/pull/7003\r\n* Document how to choose and pin a Hub kernel version by @albertvillanova in https://github.com/huggingface/trl/pull/6998\r\n* Fix variable reference in BCO docs by @albertvillanova in https://github.com/huggingface/trl/pull/7046\r\n* Remove PPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/7020\r\n* Correct what the TRL Docker image provides in the Jobs guide by @albertvillanova in https://github.com/huggingface/trl/pull/7054\r\n* Document the TRL Docker image tags by @albertvillanova in https://github.com/huggingface/trl/pull/7055\r\n* Do the chunked-CE `lm_head` projection on tensor cores instead of in fp32 by @qgallouedec in https://github.com/huggingface/trl/pull/6863\r\n* Add contributor skills at `.agents/skills` by @qgallouedec in https://github.com/huggingface/trl/pull/6901\r\n* Split the docs taxonomy into stable and experimental trainers by @qgallouedec in https://github.com/huggingface/trl/pull/7022\r\n* Vendor the fused linear losses from Liger-Kernel into `trl.losses` by @qgallouedec in https://github.com/huggingface/trl/pull/7059\r\n* Drop fused JSD from DistillationTrainer by @kashif in https://github.com/huggingface/trl/pull/7064\r\n* Fix DPO and KTO Liger loss gaps by @kashif in https://github.com/huggingface/trl/pull/7062\r\n* Harden chunked log probabilities by @kashif in https://github.com/huggingface/trl/pull/7065\r\n* Run the vendored loss parity suites as slow tests only by @qgallouedec in https://github.com/huggingface/trl/pull/7067\r\n* XPO/NashMD perf: avoid `output_hidden_states` when only `last_hidden_state` is used by @ciaoyizhen in https://github.com/huggingface/trl/pull/4755\r\n* Bump https://github.com/huggingface/doc-builder from 1b16dac5e33043af565fdf4c1b5b0fe81d0891c8 to c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27 by @dependabot[bot] in https://github.com/huggingface/trl/pull/7078\r\n* Bump the actions group with 5 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/7079\r\n* Group Dependabot GitHub Actions and pre-commit updates in a single PR by @albertvillanova in https://github.com/huggingface/trl/pull/7088\r\n* Fix the invariant suite recording instructions by @albertvillanova in https://github.com/huggingface/trl/pull/7058\r\n* Remove the flash-attn2 xfail markers now that the padding-free tests pass by @albertvillanova in https://github.com/huggingface/trl/pull/7057\r\n* Bump the \"github-actions-and-pre-commit\" group with 2 updates across multiple ecosystems by @dependabot[bot] in https://github.com/huggingface/trl/pull/7089\r\n* Merge the two Dependabot github-actions entries into one by @albertvillanova in https://github.com/huggingface/trl/pull/7091\r\n* Reformat lambda functions for code clarity by @albertvillanova in https://github.com/huggingface/trl/pull/7092\r\n* Fail fast on incompatible vLLM for default Async GRPO weight transfer by @DimensionSTP in https://github.com/huggingface/trl/pull/6830\r\n* Inject the model revision into concrete config loads too by @albertvillanova in https://github.com/huggingface/trl/pull/7099\r\n* Align tiny/small Qwen2.5 config with Qwen/Qwen2.5-32B-Instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7098\r\n* Sync the async dist. trainers with recent AsyncGRPO changes by @kashif in https://github.com/huggingface/trl/pull/6987\r\n* Align tiny and small Qwen3 configs with their reference models by @albertvillanova in https://github.com/huggingface/trl/pull/7106\r\n* Align tiny Qwen3 Instruct-2507 config with Qwen/Qwen3-4B-Instruct-2507 by @albertvillanova in https://github.com/huggingface/trl/pull/7107\r\n* Require peft>=0.13.0 and drop the 0.12 version guards by @qgallouedec in https://github.com/huggingface/trl/pull/7102\r\n* Align tiny GptOss config with openai/gpt-oss-20b by @albertvillanova in https://github.com/huggingface/trl/pull/7108\r\n* Require deepspeed>=0.18.6 and drop the 0.16.4 guard by @albertvillanova in https://github.com/huggingface/trl/pull/7118\r\n* Remove the leftover dataset_formatting module by @albertvillanova in https://github.com/huggingface/trl/pull/7117\r\n* Align tiny GPTNeoX config with EleutherAI/pythia-14m by @albertvillanova in https://github.com/huggingface/trl/pull/7109\r\n* Align tiny Llama 3.2 and Remote configs with meta-llama/Llama-3.2-1B-Instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7110\r\n* Align tiny Llama 3 config with meta-llama/Meta-Llama-3-8B-Instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7112\r\n* Align tiny Llama 3.1 config with meta-llama/Llama-3.1-8B-Instruct by @albertvillanova in https://github.com/huggingface/trl/pull/7113\r\n* Show list lengths and mark truncation in the tiny-model config diff by @albertvillanova in https://github.com/huggingface/trl/pull/7105\r\n* Fix the IW-OPD self-distillation loss assertion in tests by @albertvillanova in https://github.com/huggingface/trl/pull/7123\r\n* Sync the model config pad token id with the tokenizer in the trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7097\r\n* Sync the model config pad token id with the tokenizer in the experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/7128\r\n* Guard target_parameters use for peft<0.17.0 by @albertvillanova in https://github.com/huggingface/trl/pull/7119\r\n* Forward model revision to the auto-loaded processing_class by @kashif in https://github.com/huggingface/trl/pull/6978\r\n* Add missing TRL blog posts to the docs landing page by @qgallouedec in https://github.com/huggingface/trl/pull/7023\r\n* Add support for vLLM 0.28.0 by @qgallouedec in https://github.com/huggingface/trl/pull/7001\r\n* Count each token once in num_tokens under tensor parallelism by @qgallouedec in https://github.com/huggingface/trl/pull/7101\r\n* Realign duplicated comments across the trainers by @qgallouedec in https://github.com/huggingface/trl/pull/7145\r\n* Fix typos in help strings and error messages by @qgallouedec in https://github.com/huggingface/trl/pull/7146\r\n* Fix two messages that describe the wrong behaviour by @qgallouedec in https://github.com/huggingface/trl/pull/7147\r\n* Fix docstring type specs by @qgallouedec in https://github.com/huggingface/trl/pull/7148\r\n* Reformat five docstrings to the repo style by @qgallouedec in https://github.com/huggingface/trl/pull/7149\r\n* Fix docstring content drift by @qgallouedec in https://github.com/huggingface/trl/pull/7150\r\n* Match config field order to the docstring order by @qgallouedec in https://github.com/huggingface/trl/pull/7151\r\n* Reflow two docstrings to the doc-builder width by @qgallouedec in https://github.com/huggingface/trl/pull/7153\r\n* Fix the DistillationTrainer signature annotations by @qgallouedec in https://github.com/huggingface/trl/pull/7152\r\n* Docs formatting pass by @qgallouedec in https://github.com/huggingface/trl/pull/7154\r\n* Tidy pyproject, Makefile and .gitignore by @qgallouedec in https://github.com/huggingface/trl/pull/7157\r\n* Fix the example commands in the script docstrings by @qgallouedec in https://github.com/huggingface/trl/pull/7156\r\n* Fix three broken doc links by @qgallouedec in https://github.com/huggingface/trl/pull/7155\r\n* Fix paper_index paths and add GOLD, IW-OPD, and AsyncDistillation telemetry by @YeonwooSung in https://github.com/huggingface/trl/pull/7051\r\n* Qualify doc references to experimental classes by @qgallouedec in https://github.com/huggingface/trl/pull/7158\r\n* Release: v1.13 by @qgallouedec in https://github.com/huggingface/trl/pull/7159\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.12.0...v1.13.0\r\n","publishedAt":"2026-09-10T00:50:26.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.13.0","media":[{"type":"image","url":"https://github.com/user-attachments/assets/806f9153-d098-4a45-bef4-5d4119b7ef25","r2Key":"releases/efcaa3f83f5f796e611964a3a89ee3bddb5f86eb99f34afaa4e68bd02f52732d.png","r2Url":"https://media.releases.sh/releases/efcaa3f83f5f796e611964a3a89ee3bddb5f86eb99f34afaa4e68bd02f52732d.png"}],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":25887,"contentTokens":7615,"composition":{"bugs":6,"features":7,"enhancements":4}},{"id":"rel_GJEoPJemdKuwPynV7CzX1","version":"v1.12.0","type":"feature","title":"v1.12.0","summary":"v1.12.0 was accidentally auto-published to PyPI with identical code to v1.11.0, so skip it — the next feature release is v1.13.0.","titleGenerated":"TRL v1.12.0 is accidental duplicate of v1.11.0; next feature release is v1.13.0","titleShort":"v1.12.0 duplicates v1.11.0; skip to v1.13.0","breaking":"none","importance":3,"content":"## ⚠️ v1.12.0 is an accidental duplicate of v1.11.0\r\n\r\nDuring the v1.11.0 release, `VERSION` was bumped to `1.12.0` instead of `1.12.0.dev0`. Our publish workflow fires on any push to `main` that touches `VERSION` and only skips the upload when the version contains `\"dev\"`, so **`trl 1.12.0` was auto-published to PyPI at 19:46 on 2026-08-26, 90 seconds after `1.11.0`, with identical code**.\r\n\r\n**`trl==1.12.0` on PyPI is bit-identical to `trl==1.11.0`.** There are no features, fixes, or behavior changes in this version.\r\n\r\nThe version number is burned (PyPI does not allow yanking + re-uploading the same version), so we skip `1.12` entirely for real development. The next feature release is **`v1.13.0`**.\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.11.0...v1.12.0 (empty)\r\n","publishedAt":"2026-08-26T20:05:50.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.12.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":805,"contentTokens":259,"composition":null},{"id":"rel_lK_Rd3S5nhpIYWvV4peDt","version":"v1.11.0","type":"feature","title":"v1.11.0","summary":"trl vllm-serve now delegates to vLLM's own server, cutting code from 1218 to ~130 lines and delivering 1.4–1.6× faster GRPO server-mode steps with bit-identical completions. A new experimental AsyncDistillationTrainer supports multi-teacher on-policy distillation, plus support for several new models and templates. Old examples/scripts and examples/notebooks links now 404 after the folder reorganization.","titleGenerated":"TRL v1.11.0 replaces custom vLLM server and adds AsyncDistillationTrainer","titleShort":"vLLM's own server replaces custom; async distillation trainer new","breaking":"major","importance":4,"content":"## Features\r\n\r\n### Use vLLM's own server instead of TRL's\r\n\r\n`trl vllm-serve` used to run a custom FastAPI app around vLLM's `LLM`, with its own data-parallel fan-out and a worker extension for weight sync. vLLM's own server covers all of that. The custom server is gone: `trl/scripts/vllm_serve.py` drops from **1218 to ~130 lines** and now translates its flags into `vllm serve`, prints the equivalent command, forwards any extra argument, and sets the three settings TRL needs (`--weight-transfer-config`, `--logprobs-mode processed_logprobs`, `--max-logprobs -1`). `fastapi`, `uvicorn` and `pydantic` leave the `vllm` extra.\r\n\r\nWeight sync now uses vLLM's NCCL weight-transfer engine — tensors are announced once and streamed in a single packed broadcast, instead of one HTTP request + broadcast per tensor. Distillation teacher logprobs come from `prompt_logprobs`.\r\n\r\n**Benchmark** (GRPO server-mode on H100, Qwen2.5-1.5B / Qwen2.5-VL-3B, same seed, same data):\r\n\r\n| Setup | before | after | speedup |\r\n| --- | --- | --- | --- |\r\n| Text, TP=1 | 1.08, 0.91 s/step | **0.63, 0.62 s/step** | **1.59×** |\r\n| Text, TP=2 | 0.96, 0.96 s/step | **0.62, 0.60 s/step** | **1.57×** |\r\n| VLM, TP=1 | 2.14, 2.13 s/step | **1.48, 1.48 s/step** | **1.44×** |\r\n\r\nBit-identical completions vs the old server (16/16 over 4 prompts × n=4, logprobs identical to 0.0 over 499 tokens). `trl vllm-serve` still accepts the same flags — it just forwards to `vllm serve` now.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6765\r\n\r\n### New experimental `AsyncDistillationTrainer` (+ multi-teacher MOPD)\r\n\r\nAsync on-policy distillation, architected like `AsyncGRPOTrainer`: a background rollout worker generates the student's own completions and scores them against a teacher served over HTTP, so generation and training overlap instead of alternating. The teacher is never loaded locally — just a vLLM server URL.\r\n\r\nAlso supports **MOPD (multi-teacher on-policy distillation)**: pass more than one entry in `teacher_server_urls` and route each sample to a teacher via a `teacher_id` column (e.g. a math teacher and a code teacher, each served independently).\r\n\r\n```python\r\nfrom trl.experimental.async_distillation import AsyncDistillationConfig, AsyncDistillationTrainer\r\n\r\ntrainer = AsyncDistillationTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=AsyncDistillationConfig(\r\n        teacher_server_urls=[\"http://teacher-math:8000\", \"http://teacher-code:8000\"],\r\n        teacher_top_k=16,\r\n    ),\r\n    train_dataset=dataset,  # rows carry a `teacher_id` column for MOPD\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/6705\r\n\r\n### New model & template support\r\n\r\n* **Muse Glimmer** (+ chat template) by @qgallouedec in https://github.com/huggingface/trl/pull/6738 and https://github.com/huggingface/trl/pull/6739\r\n* **Nemotron 3.5 Lightning** by @qgallouedec in https://github.com/huggingface/trl/pull/6736\r\n* **LFM2.5-VL** by @qgallouedec in https://github.com/huggingface/trl/pull/6737\r\n* **Qwen3.8** by @qgallouedec in https://github.com/huggingface/trl/pull/6764\r\n* **DeepSeek-R1-Distill chat template** by @qgallouedec in https://github.com/huggingface/trl/pull/6766\r\n\r\n### Examples reorganized into per-example folders\r\n\r\n`examples/` was split by format (`scripts/` vs `notebooks/`), which scattered related files (e.g. `sft_nemotron_3` existed as both a script *and* a notebook, in different folders; OpenEnv notebooks lived apart from OpenEnv scripts). Everything is now **one folder per story** — the name says the method and the task (`grpo_wordle`, `sft_gpt_oss`, `ppo_tldr`), and the folder holds every file the example needs (scripts, notebooks, prompts, chat templates, eval code). Thin single-trainer demo scripts are removed (they live in `trl/scripts/` and each trainer doc page has a runnable snippet). Old `examples/scripts/…` and `examples/notebooks/…` GitHub / Colab links will 404 — worth flagging in downstream posts.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6820\r\n\r\n### AsyncGRPO: metrics rework, dtype control, and cleanup\r\n\r\n* **Rework AsyncGRPO metrics + docs** by @AmineDiro in https://github.com/huggingface/trl/pull/6715\r\n* **Training dtype in `AsyncGRPOConfig`** by @AmineDiro in https://github.com/huggingface/trl/pull/6774\r\n* **Simplify `AsyncGRPOTrainer`** — drop redundant metric guards, merge train-begin callbacks by @qgallouedec in https://github.com/huggingface/trl/pull/6378\r\n* **Fix checkpoint resume triggering unnecessary vLLM inference** by @AmmarHassona in https://github.com/huggingface/trl/pull/5911\r\n\r\n### Rewritten TRL skill (Python-API first) + drop `trl skills install`\r\n\r\nThe TRL skill is now written around the Python API — trainer selection, dataset format, load-bearing config fields, GRPO reward-function contract — with the CLI as a footnote. `trl skills install` is dropped and the skill source moves out of the package.\r\n\r\n* Rewrite the TRL skill around the Python API by @qgallouedec in https://github.com/huggingface/trl/pull/6794\r\n* Drop `trl skills install` and move the skill source out of the package by @qgallouedec in https://github.com/huggingface/trl/pull/6793\r\n\r\n### `DistillationTrainer` supports tool calling\r\n\r\nContinuing the graduation cycle from v1.10.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6723\r\n\r\n### Suggest migrating to `DistillationTrainer` when the GKD config is fully covered\r\n\r\n`GKDTrainer` now emits a hint when the config is fully covered by the stable `DistillationTrainer` — no behavior change, just a nudge for users still on the older API.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6847\r\n\r\n### Other\r\n\r\n* Allow dict config fields to be set from the command line by @qgallouedec in https://github.com/huggingface/trl/pull/6792\r\n* Raise when packing is combined with context parallelism by @qgallouedec in https://github.com/huggingface/trl/pull/6843\r\n* Emit `position_ids` from the SFT collator when sequence parallelism is enabled by @qgallouedec in https://github.com/huggingface/trl/pull/6812\r\n* Remove one host sync from SFT `chunked_nll` loss by @qgallouedec in https://github.com/huggingface/trl/pull/6842\r\n* Align the chunked-loss config access between SFT and Distillation by @qgallouedec in https://github.com/huggingface/trl/pull/6744\r\n* Add `dataset_source` to telemetry by @qgallouedec in https://github.com/huggingface/trl/pull/6599\r\n* Raise a clear error when generation returns no completions by @albertvillanova in https://github.com/huggingface/trl/pull/6714\r\n* Move GPU CI from T4 to L40S by @qgallouedec in https://github.com/huggingface/trl/pull/6741\r\n* Add on-policy multimodal GKD regression test by @sergiopaniego in https://github.com/huggingface/trl/pull/6342\r\n\r\n### vLLM version sweep\r\n\r\n* Add support for vLLM 0.27.0 and 0.27.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6742\r\n* Drop vLLM 0.17 by @qgallouedec in https://github.com/huggingface/trl/pull/6722\r\n* Drop vLLM 0.18 by @qgallouedec in https://github.com/huggingface/trl/pull/6844\r\n* Require `liger-kernel` 0.8.2 and drop the SAPO warning filter by @albertvillanova in https://github.com/huggingface/trl/pull/6768\r\n\r\n## Fixes\r\n\r\n* **Fix GRPO/RLOO training with multiple images for VLMs without grid metadata** by @qgallouedec in https://github.com/huggingface/trl/pull/6570\r\n* **Fix GRPO/RLOO with dynamic-tiling VLMs (InternVL)** by @qgallouedec in https://github.com/huggingface/trl/pull/6619\r\n* **Handle `None` token logprobs from vLLM in GRPO importance sampling** by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6693\r\n* **`[GRPO]` Match the Liger `dapo` / `cispo` / `vespo` normalizer to the non-Liger path** — completes the fix from v1.10 (#6024) on the Liger path. By @kashif in https://github.com/huggingface/trl/pull/5890\r\n* **`[SFT]` Treat `past_key_values` and `attentions` as optional on the model output** by @Hakureirm in https://github.com/huggingface/trl/pull/6686\r\n* **Fix `AsyncGRPOTrainer` crashing on environment-owned datasets** by @albertvillanova in https://github.com/huggingface/trl/pull/6824\r\n* **Fix SDPO `scale_rewards` CLI parsing** and warn on ignored feedback by @sergiopaniego in https://github.com/huggingface/trl/pull/6341\r\n* Fix `opencode.py` proxy trace / log paths after OpenEnv moved them to functions by @sergiopaniego in https://github.com/huggingface/trl/pull/6672\r\n\r\n## Documentation and Examples\r\n\r\n* Rework AsyncGRPO docs alongside the metrics rework (@AmineDiro, #6715)\r\n* Reorganize examples into self-contained per-example folders (see above)\r\n* Suggest migrating to `DistillationTrainer` when GKD config is covered (see above)\r\n\r\n## CI\r\n\r\nBig CI-hygiene cycle by @albertvillanova. Highlights:\r\n\r\n* Test one Python version on PRs, matrix nightly by @qgallouedec in https://github.com/huggingface/trl/pull/6763\r\n* Notify Slack when the tests of the latest release fail by @albertvillanova in https://github.com/huggingface/trl/pull/6900\r\n* Post the Docker image build results to the shared CI Slack channel by @albertvillanova in https://github.com/huggingface/trl/pull/6903\r\n* Factor the Slack notification into a local composite action by @albertvillanova in https://github.com/huggingface/trl/pull/6895 (+ updates in #6896, #6897, #6908, #6934, #6936)\r\n* Move slow-test job status handling, dependencies, and reporting to Slack by @albertvillanova in https://github.com/huggingface/trl/pull/6752, https://github.com/huggingface/trl/pull/6754, https://github.com/huggingface/trl/pull/6756\r\n* Pin the TruffleHog scanner image by digest + tighten secret scanning by @albertvillanova in https://github.com/huggingface/trl/pull/6677, https://github.com/huggingface/trl/pull/6757, https://github.com/huggingface/trl/pull/6758, https://github.com/huggingface/trl/pull/6759\r\n* Various hotfix pins and reverts across the cycle (bitsandbytes / liger-kernel / peft dev builds / transformers dev) by @albertvillanova in https://github.com/huggingface/trl/pull/6517, https://github.com/huggingface/trl/pull/6835, https://github.com/huggingface/trl/pull/6840, https://github.com/huggingface/trl/pull/6841, https://github.com/huggingface/trl/pull/6838, https://github.com/huggingface/trl/pull/6915, https://github.com/huggingface/trl/pull/6919\r\n* Bump the actions group across 1 directory with 8 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6727 and https://github.com/huggingface/trl/pull/6902\r\n* Restore Liger Kernel's global patches between tests by @albertvillanova in https://github.com/huggingface/trl/pull/6783\r\n* Stub the weight transfer in the AsyncGRPO epoch-stop test by @albertvillanova in https://github.com/huggingface/trl/pull/6826\r\n* Align AsyncDistillationTrainer with AsyncGRPOTrainer on pre-init args by @albertvillanova in https://github.com/huggingface/trl/pull/6825\r\n* Use Flash Attention compatible models in the continuous batching test / async trainer tests by @albertvillanova in https://github.com/huggingface/trl/pull/6853 and https://github.com/huggingface/trl/pull/6854\r\n* Derive `head_dim` when the model config does not declare it by @albertvillanova in https://github.com/huggingface/trl/pull/6855\r\n* Build the GRPO VLM test images as PIL images to avoid a datasets downcast warning by @albertvillanova in https://github.com/huggingface/trl/pull/6889\r\n* Remove the slow GRPO VLM training test by @albertvillanova in https://github.com/huggingface/trl/pull/6912\r\n* Ignore benign NumPy `__array_wrap__` deprecation and torch inductor TF32 advisory warnings in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6850 and https://github.com/huggingface/trl/pull/6851\r\n* Stop excluding the TruffleHog `lob` / `postgres` detectors by @albertvillanova in https://github.com/huggingface/trl/pull/6920, https://github.com/huggingface/trl/pull/6921 (+ #6926)\r\n* Fix misleading SFT step names, misaligned failed-test report headers, use built-in `GITHUB_ACTIONS` by @albertvillanova in https://github.com/huggingface/trl/pull/6726, https://github.com/huggingface/trl/pull/6753, https://github.com/huggingface/trl/pull/6749\r\n* Pin Python version in the publish workflow / align Python version in the experimental tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6883, https://github.com/huggingface/trl/pull/6881, https://github.com/huggingface/trl/pull/6882, https://github.com/huggingface/trl/pull/6884\r\n* Bump doc-builder pre-commit revision + drop ruff workaround by @albertvillanova in https://github.com/huggingface/trl/pull/6784, https://github.com/huggingface/trl/pull/6790\r\n* Remove the redundant `VERSION` env from the Docker build step / fix the Slack title of the TRL Docker image build job by @albertvillanova in https://github.com/huggingface/trl/pull/6931, https://github.com/huggingface/trl/pull/6932\r\n* Remove unused `RUN_SLOW` env variable from the slow tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6748\r\n* Remove unnecessary `docstyle-ignore` tag from docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6785\r\n* Inline the pre-commit action in the code-quality jobs by @albertvillanova in https://github.com/huggingface/trl/pull/6886\r\n\r\n## New Contributors\r\n* @Hakureirm made their first contribution in https://github.com/huggingface/trl/pull/6686\r\n* @AmmarHassona made their first contribution in https://github.com/huggingface/trl/pull/5911\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6725\r\n* Hotfix CI: Skip transformers continuous batching test on pre-Ampere GPUs by @albertvillanova in https://github.com/huggingface/trl/pull/6712\r\n* Raise a clear error when generation returns no completions by @albertvillanova in https://github.com/huggingface/trl/pull/6714\r\n* Pin the TruffleHog scanner image by digest by @albertvillanova in https://github.com/huggingface/trl/pull/6677\r\n* Handle None token logprobs from vLLM in GRPO importance sampling by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6693\r\n* Drop vLLM 0.17 support by @qgallouedec in https://github.com/huggingface/trl/pull/6722\r\n* Fix misleading SFT step names in slow tests CI workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6726\r\n* Bump the actions group across 1 directory with 8 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6727\r\n* [SFT] Treat past_key_values and attentions as optional on the model output by @Hakureirm in https://github.com/huggingface/trl/pull/6686\r\n* Add support for vLLM 0.27.0 and 0.27.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6742\r\n* Remove unused RUN_SLOW env variable from slow tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6748\r\n* Do not run slow tests when the setup steps failed by @albertvillanova in https://github.com/huggingface/trl/pull/6752\r\n* Fix misaligned headers in the failed tests report table by @albertvillanova in https://github.com/huggingface/trl/pull/6753\r\n* Use built-in GITHUB_ACTIONS variable instead of custom IS_GITHUB_CI by @albertvillanova in https://github.com/huggingface/trl/pull/6749\r\n* Fix opencode.py proxy trace/log paths after OpenEnv moved them to functions by @sergiopaniego in https://github.com/huggingface/trl/pull/6672\r\n* Add DeepSeek-R1-Distill chat template by @qgallouedec in https://github.com/huggingface/trl/pull/6766\r\n* Report slow tests results to Slack based on the job status by @albertvillanova in https://github.com/huggingface/trl/pull/6754\r\n* Declare the test report dependencies in the test extra by @albertvillanova in https://github.com/huggingface/trl/pull/6756\r\n* Pass the pytest report log flag from the workflow instead of the Makefile by @albertvillanova in https://github.com/huggingface/trl/pull/6750\r\n* Add Muse Glimmer support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6738\r\n* Add Nemotron 3.5 Lightning support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6736\r\n* Add LFM2.5-VL support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6737\r\n* Add Qwen3.8 support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6764\r\n* Add Muse Glimmer chat template by @qgallouedec in https://github.com/huggingface/trl/pull/6739\r\n* Restore Liger Kernel's global patches between tests by @albertvillanova in https://github.com/huggingface/trl/pull/6783\r\n* Bump doc-builder pre-commit revision by @albertvillanova in https://github.com/huggingface/trl/pull/6784\r\n* Restrict token access in the secret scanning workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6757\r\n* Fail the secret scanning job when the scan errors out by @albertvillanova in https://github.com/huggingface/trl/pull/6758\r\n* Remove unnecessary docstyle-ignore tag from docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6785\r\n* Rework AsyncGRPO metrics metrics and docs by @AmineDiro in https://github.com/huggingface/trl/pull/6715\r\n* Scan the pushed commits in the secret scanning workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6759\r\n* Drop ruff workaround from doc-builder pre-commit hook by @albertvillanova in https://github.com/huggingface/trl/pull/6790\r\n* Require liger-kernel 0.8.2 and drop the SAPO warning filter by @albertvillanova in https://github.com/huggingface/trl/pull/6768\r\n* AsyncGRPOConfig : dtype asyncgrpo training by @AmineDiro in https://github.com/huggingface/trl/pull/6774\r\n* Add AsyncDistillationTrainer by @kashif in https://github.com/huggingface/trl/pull/6705\r\n* Simplify AsyncGRPOTrainer: drop redundant metric guards, merge train-begin callbacks by @qgallouedec in https://github.com/huggingface/trl/pull/6378\r\n* [GRPO] Match the Liger dapo/cispo/vespo normalizer to the non-Liger path by @kashif in https://github.com/huggingface/trl/pull/5890\r\n* Fix AsyncGRPO checkpoint resume triggering unnecessary vLLM inference by @AmmarHassona in https://github.com/huggingface/trl/pull/5911\r\n* Fix AsyncGRPOTrainer crashing on environment-owned datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6824\r\n* Move GPU CI from T4 to L40S by @qgallouedec in https://github.com/huggingface/trl/pull/6741\r\n* Align the chunked-loss config access between SFT and Distillation by @qgallouedec in https://github.com/huggingface/trl/pull/6744\r\n* Allow dict config fields to be set from the command line by @qgallouedec in https://github.com/huggingface/trl/pull/6792\r\n* Hotfix CI: xfail the transformers continuous batching test by @albertvillanova in https://github.com/huggingface/trl/pull/6835\r\n* Hotfix CI: skip the offloading test when several accelerators are visible by @albertvillanova in https://github.com/huggingface/trl/pull/6840\r\n* Hotfix CI: xfail the tests that DataParallel breaks on multiple GPUs by @albertvillanova in https://github.com/huggingface/trl/pull/6841\r\n* Add `dataset_source` to telemetry by @qgallouedec in https://github.com/huggingface/trl/pull/6599\r\n* Test one Python version on PRs, matrix nightly by @qgallouedec in https://github.com/huggingface/trl/pull/6763\r\n* Fix GRPO/RLOO training with multiple images for VLMs without grid metadata by @qgallouedec in https://github.com/huggingface/trl/pull/6570\r\n* Hotfix CI: xfail the experimental async trainer training tests by @albertvillanova in https://github.com/huggingface/trl/pull/6838\r\n* Fix GRPO/RLOO with dynamic-tiling VLMs (InternVL) by @qgallouedec in https://github.com/huggingface/trl/pull/6619\r\n* Ignore benign NumPy `__array_wrap__` deprecation warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6850\r\n* Ignore benign torch inductor TF32 advisory warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6851\r\n* Use Flash Attention compatible models in the continuous batching test by @albertvillanova in https://github.com/huggingface/trl/pull/6853\r\n* Stub the weight transfer in the AsyncGRPO epoch-stop test by @albertvillanova in https://github.com/huggingface/trl/pull/6826\r\n* Derive head_dim when the model config does not declare it by @albertvillanova in https://github.com/huggingface/trl/pull/6855\r\n* Use a Flash Attention compatible model in the async trainer tests by @albertvillanova in https://github.com/huggingface/trl/pull/6854\r\n* Fix SDPO scale_rewards CLI parsing and warn on ignored feedback by @sergiopaniego in https://github.com/huggingface/trl/pull/6341\r\n* Add on-policy multimodal GKD regression test by @sergiopaniego in https://github.com/huggingface/trl/pull/6342\r\n* Align AsyncDistillationTrainer with AsyncGRPOTrainer on pre-init args by @albertvillanova in https://github.com/huggingface/trl/pull/6825\r\n* Align Python version in the experimental tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6881\r\n* Quote python-version value in the tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6882\r\n* Set up Python explicitly in the slow tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6884\r\n* Pin the mutable post-slack action reference to a commit SHA by @albertvillanova in https://github.com/huggingface/trl/pull/6891\r\n* Bump the pinned post-slack action SHA to the current main by @albertvillanova in https://github.com/huggingface/trl/pull/6892\r\n* Emit position_ids from the SFT collator when sequence parallelism is enabled by @qgallouedec in https://github.com/huggingface/trl/pull/6812\r\n* Inline the pre-commit action in the code quality jobs by @albertvillanova in https://github.com/huggingface/trl/pull/6886\r\n* Pin Python version in the publish workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6883\r\n* Notify Slack when the tests of the latest release fail by @albertvillanova in https://github.com/huggingface/trl/pull/6900\r\n* Reorganize examples into self-contained per-example folders by @qgallouedec in https://github.com/huggingface/trl/pull/6820\r\n* [DistillationTrainer refactor] Support tool calling by @qgallouedec in https://github.com/huggingface/trl/pull/6723\r\n* Factor the Slack notification into a local composite action by @albertvillanova in https://github.com/huggingface/trl/pull/6895\r\n* Update slack-github-action from v1 to v4 in the local Slack action by @albertvillanova in https://github.com/huggingface/trl/pull/6896\r\n* Let Dependabot update actions pinned in local composite actions by @albertvillanova in https://github.com/huggingface/trl/pull/6897\r\n* Build the GRPO VLM test images as PIL images to avoid a datasets downcast warning by @albertvillanova in https://github.com/huggingface/trl/pull/6889\r\n* Hotfix CI: Xfail MoE LoRA tests against a broken peft dev build by @albertvillanova in https://github.com/huggingface/trl/pull/6915\r\n* Bump the actions group across 1 directory with 8 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6902\r\n* Revert xfail for MoE LoRA tests now that peft#3602 fixed the autocast dtype bug by @albertvillanova in https://github.com/huggingface/trl/pull/6919\r\n* Stop excluding the trufflehog lob detector by @albertvillanova in https://github.com/huggingface/trl/pull/6920\r\n* Stop excluding the trufflehog postgres detector by @albertvillanova in https://github.com/huggingface/trl/pull/6921\r\n* Remove the stale detector exclusion comment by @albertvillanova in https://github.com/huggingface/trl/pull/6926\r\n* Remove the slow GRPO VLM training test by @albertvillanova in https://github.com/huggingface/trl/pull/6912\r\n* Post the Docker image build results to the shared CI Slack channel by @albertvillanova in https://github.com/huggingface/trl/pull/6903\r\n* Fix the Slack title of the TRL Docker image build job by @albertvillanova in https://github.com/huggingface/trl/pull/6931\r\n* Remove the redundant VERSION env from the Docker build step by @albertvillanova in https://github.com/huggingface/trl/pull/6932\r\n* Read the job status from the job context in the local Slack action by @albertvillanova in https://github.com/huggingface/trl/pull/6908\r\n* Drop vLLM 0.18 support by @qgallouedec in https://github.com/huggingface/trl/pull/6844\r\n* Use vLLM's own server instead of TRL's by @qgallouedec in https://github.com/huggingface/trl/pull/6765\r\n* Drop `trl skills install` and move the skill source out of the package by @qgallouedec in https://github.com/huggingface/trl/pull/6793\r\n* Make the Slack CI report readable at a glance by @qgallouedec in https://github.com/huggingface/trl/pull/6934\r\n* Fix the Slack report posting twice by @qgallouedec in https://github.com/huggingface/trl/pull/6936\r\n* Rewrite the TRL skill around the Python API by @qgallouedec in https://github.com/huggingface/trl/pull/6794\r\n* Raise when packing is combined with context parallelism by @qgallouedec in https://github.com/huggingface/trl/pull/6843\r\n* Suggest migrating to DistillationTrainer when the GKD config is fully covered by @qgallouedec in https://github.com/huggingface/trl/pull/6847\r\n* Remove one host sync from SFT chunked_nll loss by @qgallouedec in https://github.com/huggingface/trl/pull/6842\r\n* Release: v1.11 by @qgallouedec in https://github.com/huggingface/trl/pull/6937\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.10.0...v1.11.0\r\n","publishedAt":"2026-08-26T20:02:08.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.11.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":25322,"contentTokens":7341,"composition":{"bugs":8,"features":8,"enhancements":12}},{"id":"rel_XU019-yMF4AdzT2wUZEAp","version":"v1.10.0","type":"feature","title":"v1.10.0","summary":"DistillationTrainer and DistillationConfig move from trl.experimental.distillation to the top-level trl package, with the old path emitting a FutureWarning until removal in v2.0.0. AsyncGRPOTrainer gains an experimental loop-owning path for training external agents like opencode, plus VLM support in DistillationTrainer and new DiffusionGemma SFT example.","titleGenerated":"TRL v1.10.0 graduates DistillationTrainer and adds loop-owning AsyncGRPO training","titleShort":"DistillationTrainer graduates; AsyncGRPO trains loop-owning agents","breaking":"minor","importance":4,"content":"## Features\r\n\r\n### 🎓 `DistillationTrainer` is now a stable trainer\r\n\r\nAfter a ~30-PR refactor that reshaped its data contract, loss path, generation stack, config surface, and tests, `DistillationTrainer` and `DistillationConfig` graduate from `trl.experimental.distillation` to the top-level `trl` package. Same import surface as SFT / DPO / GRPO / KTO. The old experimental path still works and emits a `FutureWarning` (removal in v2.0.0).\r\n\r\n```python\r\n# Before\r\nfrom trl.experimental.distillation import DistillationConfig, DistillationTrainer\r\n\r\n# Now\r\nfrom trl import DistillationConfig, DistillationTrainer\r\n```\r\n\r\nAlso lands a `trl distillation` CLI and moves the tests to `tests/`. The full refactor arc: switched signature columns to `prompt` (deprecated `messages`-format), pinned the generation stack to GRPO's, wired the chunked JSD loss and deleted the full-logit path, cleaned the Liger path to share extraction with the chunked path, adopted GRPO's `log()` / `training_step` timing / `_save_checkpoint`, rebuilt the test suite to GRPO shape, and much more.\r\n\r\nby @qgallouedec across ~30 PRs (#6479, #6480, #6481, #6482, #6484, #6487, #6497, #6508, #6509, #6510, #6511, #6512, #6513, #6521, #6522, #6523, #6524, #6525, #6526, #6530, #6537, #6604, #6605, #6606, #6607, #6609, #6610, #6611, #6612, #6613, #6614, #6629, #6630, #6632, #6633, #6634, #6639, #6640, #6641, #6642, #6643, #6644, #6645, #6647, #6653).\r\n\r\n### `DistillationTrainer` supports Vision Language Models\r\n\r\nAlongside the graduation, VLMs work end-to-end in `DistillationTrainer`.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6650\r\n\r\n### AsyncGRPO: loop-owning agent training (OpenEnv harness)\r\n\r\nNew experimental **loop-owning (black-box)** path for `AsyncGRPOTrainer` — for training external agents like [`opencode`](https://opencode.ai) that run their own tool loop, rather than TRL driving each turn as in the `environment_factory` white-box path.\r\n\r\nThe agent runs in an OpenEnv session in `transparent_proxy` mode; an in-sandbox proxy captures each turn's token ids and logprobs. On completion, TRL reads the trace, rebuilds per-turn training rows, and scores the workspace with the session's `verify()`. Ships with three caller-supplied policy hooks (`rollout_reward_fn`, `train_turn_fn`, `agent_turn_fn`) so you can drop framework aux calls (title generator, context summarizer) and reinforce only the turns you want.\r\n\r\nIncludes a self-contained `examples/scripts/openenv/opencode.py` (local subprocess sandbox + DeepCoder held-out stdin/stdout verifier), validated end-to-end on Qwen3-4B.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6420, plus HF-sandbox variant by @sergiopaniego in https://github.com/huggingface/trl/pull/6565\r\n\r\n### AsyncGRPO: observability + sampling controls\r\n\r\n* **Record rollout traces to trackio** by @AmineDiro in https://github.com/huggingface/trl/pull/6491\r\n* **Step-time metric** by @AmineDiro in https://github.com/huggingface/trl/pull/6490\r\n* **Fix queue wait time metric** by @AmineDiro in https://github.com/huggingface/trl/pull/6489\r\n* **`top_p` / `top_k` / `min_p` / `repetition_penalty` in `AsyncGRPOConfig`** by @AmineDiro in https://github.com/huggingface/trl/pull/6608\r\n* **Fork-independent epoch counting** — landed in v1.9 window\r\n* Close in-flight harness sessions concurrently on shutdown by @sergiopaniego in https://github.com/huggingface/trl/pull/6564\r\n\r\n### DiffusionGemma block-diffusion SFT example\r\n\r\nA new SFT example for [`google/diffusiongemma-26B-A4B-it`](https://huggingface.co/google/diffusiongemma-26B-A4B-it) implementing the reference recipe: one response block per step, uniform random token corruption with `t ~ U(eps, 1)`, two-pass self-conditioning, flat cross-entropy over the whole canvas plus an autoregressive co-loss on the encoder. LoRA targets attention + dense MLP linears. Requires `transformers >= 5.12.0`.\r\n\r\nShips with `diffusion_gemma.jinja` / `diffusion_gemma_training.jinja` chat templates (with `{% generation %}` markers) so `assistant_only_loss=True` works out of the box.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/6003\r\n\r\n### More model & template support\r\n\r\n* **LFM2 and LFM2.5** — day-zero training support by @qgallouedec in https://github.com/huggingface/trl/pull/6428\r\n* **Gemma 4 response template** by @qgallouedec in https://github.com/huggingface/trl/pull/6535\r\n\r\n### Default changes\r\n\r\nTwo default flips this cycle — pinning them explicitly is recommended if you want the old behavior.\r\n\r\n* **`SFTConfig.max_completion_length` / `GRPOConfig.max_completion_length`**: default bumped from `256` to `512`. By @dhruvnigam93 in https://github.com/huggingface/trl/pull/6264\r\n* **`GRPOConfig.use_bias_correction_kl`** now defaults to `True`. By @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6503\r\n\r\n### Packing and padding-free on VLMs with text-only data\r\n\r\nVLM configurations no longer reject packing / padding-free when the batch happens to be text-only.\r\n\r\nby @cris96spa in https://github.com/huggingface/trl/pull/6547\r\n\r\n### Other\r\n\r\n* Use a reference adapter with `target_parameters` on `peft>=0.20.0` by @albertvillanova in https://github.com/huggingface/trl/pull/6591\r\n* Reject unsupported `train_dataset` types in core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6493 and https://github.com/huggingface/trl/pull/6494\r\n* Reject unsupported `f_divergence_type` / loss type combinations in DPO by @qgallouedec in https://github.com/huggingface/trl/pull/6559\r\n* Validate that reward functions return one reward per completion by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6534\r\n* Validate YAML config root type by @RerankerGuo in https://github.com/huggingface/trl/pull/6656\r\n* Respect `TQDM_DISABLE` in DPO/KTO/BCO reference log-prob loops by @qgallouedec in https://github.com/huggingface/trl/pull/6507\r\n* Respect `use_cpu` in `create_model_from_path` `device_map` default by @Strongich in https://github.com/huggingface/trl/pull/6295\r\n* Use `dtype` instead of `torch_dtype` (deprecated) by @qgallouedec in https://github.com/huggingface/trl/pull/6560\r\n* Add missing logged metrics to the GRPO and RLOO docs by @PoojanTa in https://github.com/huggingface/trl/pull/6624\r\n* Add `test_train_moe_peft_model` for KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6589\r\n\r\n### vLLM version sweep\r\n\r\n* Add support for vLLM 0.26.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6569\r\n\r\n## Fixes\r\n\r\n* **Fix DAPO / CISPO / VESPO loss normalization** when `steps_per_generation != gradient_accumulation_steps` — gradients were silently mis-scaled by `gradient_accumulation_steps / steps_per_generation` (e.g. **0.5×** or **4×** the intended token-mean in common configs). Default-equivalent configs are unaffected. By @0xadvait in https://github.com/huggingface/trl/pull/6024\r\n* **Fix Liger kernel crash on pre-Ampere GPUs** by @albertvillanova in https://github.com/huggingface/trl/pull/6518\r\n* **Fix vLLM weight synchronization** when `vllm_enable_sleep_mode=True` by @muupan in https://github.com/huggingface/trl/pull/5313\r\n* **Fix vLLM server-mode communicator initialization** to use the current accelerator device by @sywangyi in https://github.com/huggingface/trl/pull/6417\r\n* **Use accelerator device instead of hardcoded `cuda`** in FSDP2 vLLM weight sync by @verma8076 in https://github.com/huggingface/trl/pull/6592\r\n* **Fix `prepare_deepspeed` crash with CPU offload optimizer** by @roycho96 in https://github.com/huggingface/trl/pull/5916\r\n* **`fix(ppo)`: exclude padding tokens from entropy calculation** by @mukund1985 in https://github.com/huggingface/trl/pull/6121\r\n* **`fix`: avoid `CopySlices` when scaling policy logits** by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6554\r\n* **Fix multimodal field handling in GRPO tool-call loop** by @HariomJangra in https://github.com/huggingface/trl/pull/6286\r\n* **`[GRPO]` Apply the completion mask elementwise in the LuSPO loss aggregation** by @YaseenBashaT in https://github.com/huggingface/trl/pull/6654\r\n* **`[GRPO]` Fix entropy bonus normalization inconsistency across loss types** by @YaseenBashaT in https://github.com/huggingface/trl/pull/6648\r\n* Fix incorrect example script path in Online DPO docs by @latent-9 in https://github.com/huggingface/trl/pull/6598\r\n* Point the opencode example at `huggingface/OpenEnv` by @sergiopaniego in https://github.com/huggingface/trl/pull/6529\r\n* Remove symlink support from skills installation by @albertvillanova in https://github.com/huggingface/trl/pull/6665\r\n\r\n## Documentation and Examples\r\n\r\n* docs: complete `GRPOConfig` `loss_type` help by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6477\r\n* Fix inaccuracies in the release instructions by @albertvillanova in https://github.com/huggingface/trl/pull/6566\r\n* Fix patch release instructions: pull release branch instead of main by @albertvillanova in https://github.com/huggingface/trl/pull/6563\r\n* Add `opencode_hf_sandbox.py`: loop-owning training in remote HF sandboxes by @sergiopaniego in https://github.com/huggingface/trl/pull/6565\r\n\r\n## CI\r\n\r\n* `test`: don't hard-code `bf16=True` on devices that lack bf16 support by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6036\r\n* Cleanup xpu cache memory after each test by @kaixuanliu in https://github.com/huggingface/trl/pull/5771\r\n* Force `use_cpu` in tests when no CUDA device is available by @albertvillanova in https://github.com/huggingface/trl/pull/6504\r\n* Silence transformers `LOAD REPORT` tables in test output by @albertvillanova in https://github.com/huggingface/trl/pull/6502\r\n* Require vision for VLM response parsing tests by @albertvillanova in https://github.com/huggingface/trl/pull/6602\r\n* Require `jmespath` only on transformers below 5.13.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6597\r\n* Add regression tests for evaluating init-time eval datasets after training with precomputed reference log-probs by @albertvillanova in https://github.com/huggingface/trl/pull/6488\r\n* Test `train_dataset=None` raises for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6492\r\n* Remove unnecessary `loss_type` in KTO precompute ref log-probs test by @albertvillanova in https://github.com/huggingface/trl/pull/6501\r\n* Fix `test_peft_with_quantization` tests for bitsandbytes 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6546\r\n* Ignore benign bitsandbytes 4-bit blocksize alignment warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6582\r\n* Suppress deprecated `torch.get_autocast_gpu_dtype()` warning from mamba-ssm kernel by @albertvillanova in https://github.com/huggingface/trl/pull/6556\r\n* Document removal criteria for pytest warning filters by @albertvillanova in https://github.com/huggingface/trl/pull/6595\r\n* Fix test: don't call `add_response_schema` when the model already ships a response template by @qgallouedec in https://github.com/huggingface/trl/pull/6574\r\n* Fix CI quality: fix copyright year in `openenv_harness.py` by @albertvillanova in https://github.com/huggingface/trl/pull/6531\r\n* Hotfix CI: temporarily pin `liger-kernel < 0.8.1` by @albertvillanova in https://github.com/huggingface/trl/pull/6517\r\n* Hotfix CI: temporarily pin `bitsandbytes < 0.50.0` by @albertvillanova in https://github.com/huggingface/trl/pull/6544\r\n* Hotfix CI: xfail NemotronH GRPO/RLOO tests against a broken transformers dev build by @albertvillanova in https://github.com/huggingface/trl/pull/6542 (reverted in https://github.com/huggingface/trl/pull/6558)\r\n* `ci`: exclude the buggy TruffleHog LOB detector by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6674\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6540\r\n* Revert transformers rotary device deprecation warning filter by @albertvillanova in https://github.com/huggingface/trl/pull/6596\r\n\r\n## New Contributors\r\n* @gowtham-sai-yadav made their first contribution in https://github.com/huggingface/trl/pull/6477\r\n* @dhruvnigam93 made their first contribution in https://github.com/huggingface/trl/pull/6264\r\n* @cris96spa made their first contribution in https://github.com/huggingface/trl/pull/6547\r\n* @mukund1985 made their first contribution in https://github.com/huggingface/trl/pull/6121\r\n* @latent-9 made their first contribution in https://github.com/huggingface/trl/pull/6598\r\n* @verma8076 made their first contribution in https://github.com/huggingface/trl/pull/6592\r\n* @PoojanTa made their first contribution in https://github.com/huggingface/trl/pull/6624\r\n* @YaseenBashaT made their first contribution in https://github.com/huggingface/trl/pull/6654\r\n* @RerankerGuo made their first contribution in https://github.com/huggingface/trl/pull/6656\r\n* @HariomJangra made their first contribution in https://github.com/huggingface/trl/pull/6286\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6496\r\n* Test train_dataset=None raises for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6492\r\n* Add regression tests for evaluating init-time eval datasets after training with precomputed reference log-probs by @albertvillanova in https://github.com/huggingface/trl/pull/6488\r\n* docs: complete GRPOConfig loss_type help by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6477\r\n* test: don't hard-code bf16=True on devices that lack bf16 support by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6036\r\n* [DistillationTrainer refactor]  Switch tests, docs, and example to prompt-only datasets by @qgallouedec in https://github.com/huggingface/trl/pull/6479\r\n* cleanup xpu cache memory after each test by @kaixuanliu in https://github.com/huggingface/trl/pull/5771\r\n* Fix vLLM server-mode communicator initialization to use the current A… by @sywangyi in https://github.com/huggingface/trl/pull/6417\r\n* [DistillationTrainer refactor] Remove messages-format support and prompt-length config by @qgallouedec in https://github.com/huggingface/trl/pull/6480\r\n* [DistillationTrainer refactor] Pin signature columns to `[\"prompt\", \"image\", \"images\"]` by @qgallouedec in https://github.com/huggingface/trl/pull/6481\r\n* [DistillationTrainer refactor] Emit `completion_mask` alongside `labels` by @qgallouedec in https://github.com/huggingface/trl/pull/6482\r\n* [DistillationTrainer refactor] Loss consumes `completion_mask` by @qgallouedec in https://github.com/huggingface/trl/pull/6484\r\n* Fix : queue wait time metric AsyncGRPOTrainer by @AmineDiro in https://github.com/huggingface/trl/pull/6489\r\n* [DistillationTrainer refactor] Emit prompt_ids / prompt_mask / completion_ids by @qgallouedec in https://github.com/huggingface/trl/pull/6487\r\n* Remove unnecessary loss_type in KTO precompute ref log-probs test by @albertvillanova in https://github.com/huggingface/trl/pull/6501\r\n* Force use_cpu in tests when no CUDA device is available by @albertvillanova in https://github.com/huggingface/trl/pull/6504\r\n* Hotfix CI: Temporarily pin liger-kernel < 0.8.1 by @albertvillanova in https://github.com/huggingface/trl/pull/6517\r\n* Silence transformers LOAD REPORT tables in test output by @albertvillanova in https://github.com/huggingface/trl/pull/6502\r\n* Change default `max_completion_length` from 256 to 512 by @dhruvnigam93 in https://github.com/huggingface/trl/pull/6264\r\n* Add DiffusionGemma block-diffusion SFT example by @kashif in https://github.com/huggingface/trl/pull/6003\r\n* Record rollout traces to trackio  by @AmineDiro in https://github.com/huggingface/trl/pull/6491\r\n* [DistillationTrainer refactor] Switch the loss to GRPO's keys by @qgallouedec in https://github.com/huggingface/trl/pull/6497\r\n* [DistillationTrainer refactor] Freeze ServerDistillationTrainer self-contained by @qgallouedec in https://github.com/huggingface/trl/pull/6508\r\n* Add LFM2 and LFM2.5 support and testing by @qgallouedec in https://github.com/huggingface/trl/pull/6428\r\n* [DistillationTrainer refactor] Delete `_compute_prompt_length` by @qgallouedec in https://github.com/huggingface/trl/pull/6509\r\n* Async grpo OpenEnv harness rollout by @AmineDiro in https://github.com/huggingface/trl/pull/6420\r\n* [DistillationTrainer refactor] Add `_tokenize_prompts` (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6510\r\n* [DistillationTrainer refactor] Add the shared generation config fields (add-only) by @qgallouedec in https://github.com/huggingface/trl/pull/6511\r\n* Fix crash when training with Liger kernel on pre-Ampere GPUs by @albertvillanova in https://github.com/huggingface/trl/pull/6518\r\n* [DistillationTrainer refactor] Add the ctor generation-supporting state (add-only) by @qgallouedec in https://github.com/huggingface/trl/pull/6512\r\n* Fix prepare_deepspeed crash with cpu offload optimizer by @roycho96 in https://github.com/huggingface/trl/pull/5916\r\n* Fix CI quality: Fix copyright year in trl/experimental/async_grpo/openenv_harness.py by @albertvillanova in https://github.com/huggingface/trl/pull/6531\r\n* Reject unsupported train_dataset types in core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6493\r\n* [DistillationTrainer refactor] Add `_generate_single_turn` (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6513\r\n* [DistillationTrainer refactor] Add `_generate` + `_generate_and_score_completions` (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6521\r\n* Test train_dataset types at init for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6494\r\n* Add Gemma4 response template by @qgallouedec in https://github.com/huggingface/trl/pull/6535\r\n* Validate that reward functions return one reward per completion by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6534\r\n* fix(grpo): correct DAPO/CISPO/VESPO loss normalization when steps_per_generation != gradient_accumulation_steps by @0xadvait in https://github.com/huggingface/trl/pull/6024\r\n* Hotfix CI: Temporarily pin bitsandbytes < 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6544\r\n* Hotfix CI: Xfail NemotronH GRPO/RLOO tests against a broken transformers dev build by @albertvillanova in https://github.com/huggingface/trl/pull/6542\r\n* added step time metric to AsyncGRPOTrainer by @AmineDiro in https://github.com/huggingface/trl/pull/6490\r\n* Allow packing and padding-free on VLMs with text-only data by @cris96spa in https://github.com/huggingface/trl/pull/6547\r\n* fix(ppo): exclude padding tokens from entropy calculation by @mukund1985 in https://github.com/huggingface/trl/pull/6121\r\n* Respect `TQDM_DISABLE` in DPO/KTO/BCO reference log-prob loops by @qgallouedec in https://github.com/huggingface/trl/pull/6507\r\n* Revert xfail for NemotronH GRPO/RLOO tests now that transformers#47569 fixed the kernels bug by @albertvillanova in https://github.com/huggingface/trl/pull/6558\r\n* fix: avoid CopySlices when scaling policy logits by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6554\r\n* Fix a bug in vLLM weight synchronization when `vllm_enable_sleep_mode=True` by @muupan in https://github.com/huggingface/trl/pull/5313\r\n* Point the opencode example at huggingface/OpenEnv by @sergiopaniego in https://github.com/huggingface/trl/pull/6529\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6540\r\n* Suppress deprecated torch.get_autocast_gpu_dtype() warning from mamba-ssm kernel by @albertvillanova in https://github.com/huggingface/trl/pull/6556\r\n* Fix test_peft_with_quantization tests for bitsandbytes 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6546\r\n* Fix patch release instructions: Pull release branch instead of main by @albertvillanova in https://github.com/huggingface/trl/pull/6563\r\n* Fix inaccuracies in the release instructions by @albertvillanova in https://github.com/huggingface/trl/pull/6566\r\n* Use `dtype` instead of `torch_dtype` (deprecated) by @qgallouedec in https://github.com/huggingface/trl/pull/6560\r\n* Fix test: don't call `add_response_schema` when the model already ships a response template by @qgallouedec in https://github.com/huggingface/trl/pull/6574\r\n* [DistillationTrainer refactor] Switch generation to GRPO's stack; delete the buffer by @qgallouedec in https://github.com/huggingface/trl/pull/6522\r\n* [DistillationTrainer refactor] Align the generation_kwargs dict with GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/6523\r\n* [DistillationTrainer refactor] Align the vLLM ctor block with GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/6524\r\n* [DistillationTrainer refactor] Add `_get_last_hidden_state` (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6525\r\n* Default `use_bias_correction_kl=True` in GRPOConfig by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6503\r\n* Ignore benign bitsandbytes 4-bit blocksize alignment warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6582\r\n* Respect use_cpu in create_model_from_path device_map default by @Strongich in https://github.com/huggingface/trl/pull/6295\r\n* Ignore transformers rotary `device` deprecation warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6585\r\n* Add `test_train_moe_peft_model` for KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6589\r\n* Revert transformers rotary device deprecation warning filter by @albertvillanova in https://github.com/huggingface/trl/pull/6596\r\n* Document removal criteria for pytest warning filters by @albertvillanova in https://github.com/huggingface/trl/pull/6595\r\n* Reject unsupported `f_divergence_type` / loss type combinations in DPO by @qgallouedec in https://github.com/huggingface/trl/pull/6559\r\n* Add support for vLLM 0.26.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6569\r\n* Use a reference adapter with `target_parameters` on `peft>=0.20.0` by @albertvillanova in https://github.com/huggingface/trl/pull/6591\r\n* Fix incorrect example script path in Online DPO docs by @latent-9 in https://github.com/huggingface/trl/pull/6598\r\n* Add top_p/top_k/min_p/repetition_penalty to AsyncGRPOConfig by @AmineDiro in https://github.com/huggingface/trl/pull/6608\r\n* Use accelerator device instead of hardcoded cuda in FSDP2 vLLM weight sync by @verma8076 in https://github.com/huggingface/trl/pull/6592\r\n* [DistillationTrainer refactor] Add the chunked JSD loss (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6526\r\n* [DistillationTrainer refactor] Wire compute_loss to the chunked JSD path; delete the full-logit loss by @qgallouedec in https://github.com/huggingface/trl/pull/6530\r\n* [DistillationTrainer refactor] Heavy-clean the Liger path: share extraction with the chunked path by @qgallouedec in https://github.com/huggingface/trl/pull/6537\r\n* [DistillationTrainer refactor] Default `disable_dropout` to `False` by @qgallouedec in https://github.com/huggingface/trl/pull/6604\r\n* [DistillationTrainer refactor] Config finish by @qgallouedec in https://github.com/huggingface/trl/pull/6605\r\n* [DistillationTrainer refactor] Constructor cleanup: safe internal cleanups by @qgallouedec in https://github.com/huggingface/trl/pull/6606\r\n* [DistillationTrainer refactor] Config cosmetic parity by @qgallouedec in https://github.com/huggingface/trl/pull/6607\r\n* [DistillationTrainer refactor] Constructor cleanup (40b): drop eval-metric params, require `model` by @qgallouedec in https://github.com/huggingface/trl/pull/6609\r\n* [DistillationTrainer refactor] Constructor cleanup (40c): quantization + PEFT/QLoRA parity by @qgallouedec in https://github.com/huggingface/trl/pull/6610\r\n* [DistillationTrainer refactor] Make `advantages` optional in `print_prompt_completions_sample` by @qgallouedec in https://github.com/huggingface/trl/pull/6611\r\n* [DistillationTrainer refactor] Use the shared `print_prompt_completions_sample` by @qgallouedec in https://github.com/huggingface/trl/pull/6612\r\n* [DistillationTrainer refactor] Adopt GRPO's `log()` + bounded `_logs` deques by @qgallouedec in https://github.com/huggingface/trl/pull/6613\r\n* [DistillationTrainer refactor] Adopt GRPO's `training_step` timing + `_save_checkpoint` by @qgallouedec in https://github.com/huggingface/trl/pull/6614\r\n* Require vision for VLM response parsing tests by @albertvillanova in https://github.com/huggingface/trl/pull/6602\r\n* Require jmespath only on transformers below 5.13.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6597\r\n* Add missing logged metrics to the GRPO and RLOO docs by @PoojanTa in https://github.com/huggingface/trl/pull/6624\r\n* Close in-flight harness sessions concurrently on shutdown by @sergiopaniego in https://github.com/huggingface/trl/pull/6564\r\n* Add opencode_hf_sandbox.py: loop-owning training in remote HF sandboxes by @sergiopaniego in https://github.com/huggingface/trl/pull/6565\r\n* Remove symlink support from skills installation by @albertvillanova in https://github.com/huggingface/trl/pull/6665\r\n* ci: exclude the buggy TruffleHog lob detector by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6674\r\n* [GRPO] Apply the completion mask elementwise in the luspo loss aggregation by @YaseenBashaT in https://github.com/huggingface/trl/pull/6654\r\n* [GRPO] Fix entropy bonus normalization inconsistency across loss types by @YaseenBashaT in https://github.com/huggingface/trl/pull/6648\r\n* Validate YAML config root type by @RerankerGuo in https://github.com/huggingface/trl/pull/6656\r\n* Fix multimodal field handling in GRPO tool call loop by @HariomJangra in https://github.com/huggingface/trl/pull/6286\r\n* [DistillationTrainer refactor] Reject `use_liger_kernel` with `logit_scale` / `final_logit_softcapping` by @qgallouedec in https://github.com/huggingface/trl/pull/6629\r\n* [DistillationTrainer refactor] Convention/cosmetic sweep by @qgallouedec in https://github.com/huggingface/trl/pull/6630\r\n* [DistillationTrainer refactor] Config: GRPO parity drift fixes by @qgallouedec in https://github.com/huggingface/trl/pull/6632\r\n* [DistillationTrainer refactor] Log the student entropy metric by @qgallouedec in https://github.com/huggingface/trl/pull/6633\r\n* [DistillationTrainer refactor] Rewrite the test suite to GRPO shape by @qgallouedec in https://github.com/huggingface/trl/pull/6634\r\n* [DistillationTrainer refactor] Docs: align distillation_trainer.md with stable-trainer conventions by @qgallouedec in https://github.com/huggingface/trl/pull/6639\r\n* [DistillationTrainer refactor] Promote DistillationTrainer to the stable API by @qgallouedec in https://github.com/huggingface/trl/pull/6640\r\n* [DistillationTrainer refactor] Move DistillationTrainer to the Stable telemetry group by @qgallouedec in https://github.com/huggingface/trl/pull/6641\r\n* [DistillationTrainer refactor] Add the `trl distillation` CLI by @qgallouedec in https://github.com/huggingface/trl/pull/6642\r\n* [DistillationTrainer refactor] Move the distillation tests to `tests/` by @qgallouedec in https://github.com/huggingface/trl/pull/6647\r\n* [DistillationTrainer refactor] Docs: de-namespace DistillationTrainer for the stable API by @qgallouedec in https://github.com/huggingface/trl/pull/6643\r\n* [DistillationTrainer refactor] Update the example script for the stable API by @qgallouedec in https://github.com/huggingface/trl/pull/6644\r\n* [DistillationTrainer refactor] Re-point ServerDistillationTrainer at the stable base by @qgallouedec in https://github.com/huggingface/trl/pull/6645\r\n* [DistillationTrainer refactor] Support Vision Language Models by @qgallouedec in https://github.com/huggingface/trl/pull/6650\r\n* [DistillationTrainer refactor] Docs: announce the DistillationTrainer promotion in What's New by @qgallouedec in https://github.com/huggingface/trl/pull/6653\r\n* Release: v1.10 by @qgallouedec in https://github.com/huggingface/trl/pull/6724\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.9.0...v1.10.0\r\n","publishedAt":"2026-08-13T01:36:04.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.10.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":28138,"contentTokens":8183,"composition":{"bugs":12,"features":8,"enhancements":6}},{"id":"rel_2ohdKI7sZ5xBot4yVnyxr","version":"v0.20.0","type":"feature","title":"v0.20.0","summary":"PEFT v0.20.0 adds nine new methods including HiRA, GLoRA, BEFT, MonteCLoRA, VeLoRA, Uni-LoRA, FRoD, MiCA, and DEFT, along with automatic LoRA target selection, broader quantization support for BOFT, MiSS, VeRA, and SHiRA, and a new image generation benchmark. AutoGPTQ is formally deprecated following the switch to GPT-QModel.","titleGenerated":"PEFT v0.20.0 adds nine new fine-tuning methods and broader quantization support","titleShort":"Nine new PEFT methods ship; AutoGPTQ deprecated","breaking":"minor","importance":4,"content":"# Highlights\r\n\r\n<img width=\"1250\" height=\"560\" alt=\"peft-v0 20 0\" src=\"https://github.com/user-attachments/assets/6194fae9-f3c3-48ca-b43b-40c774bfe6eb\" />\r\n\r\nThis release adds no less than nine new PEFT methods and puts a lot of work into the surrounding infrastructure, for example adding a new image generation benchmark for the method comparison suite and greatly improving the documentation structure.\r\n\r\n## New Methods\r\n\r\n### HiRA\r\n\r\n@hqsiswiliam added [\"HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models\"](https://openreview.net/forum?id=TwJrTz9cRS) to PEFT (#2668). Instead of adding the low-rank product `BA` to the base weight, HiRA multiplies it elementwise (Hadamard product) with the frozen base weight. Because the base weight itself is full rank, the resulting update is no longer constrained to be low rank, while the trainable parameter count stays the same as LoRA's.\r\n\r\n### GLoRA\r\n\r\n@not-lain contributed GLoRA: [\"One-for-All: Generalized LoRA for Parameter-Efficient Fine-Tuning\"](https://arxiv.org/abs/2306.07967) in #3098. It is a flexible PEFT method that extends LoRA with configurable weight, activation, and bias adaptation, delivering richer fine-tuning with no extra inference cost. Use it when you need per-layer flexibility or stronger adaptation than vanilla LoRA. Skip it for non-Linear layers (e.g. Conv/Embedding) or when standard LoRA is already sufficient and simplicity matters.\r\n\r\n### BEFT\r\n\r\n@whubaichuan added [\"BEFT: Bias-Efficient Fine-Tuning of Language Models\"](https://arxiv.org/abs/2509.15974v2) in #3195. BEFT builds on the observation that fine-tuning bias terms alone can be competitive in low-data regimes, but goes further: rather than training *all* biases, it targets the value projection by default, as the authors found this to be most efficient. This brings the trainable parameter count down to roughly 0.01% of the total parameters.\r\n\r\n### MonteCLoRA\r\n\r\n@victor7246 integrated MonteCLoRA, [\"Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation\"](https://openreview.net/forum?id=2HFmicB8kh) in #2943. LoRA is known to be sensitive to hyperparameters like learning rate and batch size. This new LoRA variant addresses this by treating the low-rank parameters as a distribution rather than a point estimate, using Monte Carlo estimation to obtain a low-variance posterior estimate. This should make training noticeably more robust, at the cost of only _O(rank)_ additional parameters.\r\n\r\n### VeLoRA\r\n\r\n@roymiles added VeLoRA: [\"Memory Efficient Training using Rank-1 Sub-Token Projections\"](https://arxiv.org/abs/2405.17991) in #3159. Unlike most PEFT methods, this LoRA variant targets activation memory rather than parameter count: intermediate activations are split into sub-tokens and compressed with a fixed rank-1 projection before being cached for the backward pass, then reconstructed during backpropagation. Conceptually, it is similar to gradient checkpointing, with a lower memory saving but running faster.\r\n\r\n### Uni-LoRA\r\n\r\n@KaiyangLi1992 contributed Uni-LoRA: [\"One Vector is All You Need\"](https://arxiv.org/abs/2506.00799) in #3257. The paper shows that parameter-efficient LoRA variants like VeRA and VB-LoRA can all be described as projecting the full LoRA parameter space down to a much smaller subspace, differing only in the choice of projection. Uni-LoRA uses a single global projection across the whole model instead of layer-wise ones, which allows cross-layer parameter sharing and thus very low parameter counts.\r\n\r\n### FRoD\r\n\r\n@Bane-Elvin added FRoD, [\"Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence\"](https://arxiv.org/abs/2512.23485), in #3270. Instead of adding low-rank deltas like LoRA, it reconstructs selected weights with shared rotational subspaces and sparse trainable coefficients. It is especially useful when fast convergence and a higher full-rank capacity ceiling are important, and its large sparse rotational subspace may also be promising for model merging. The main tradeoffs are the costly joint-decomposition initialization and slightly slower forward/backward passes than LoRA, so it may be less attractive for a one-off single-task fine-tune.\r\n\r\n### MiCA\r\n\r\n@sr-networks contributed MiCA, [\"MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning\"](https://arxiv.org/abs/2604.01694), in #3260. MiCA is a LoRA variant that initializes `B` from the SVD of the base weight, taking the left singular vectors belonging to the *smallest* singular values, and then trains only `A`. The idea is that adapting these underused directions integrates new knowledge more effectively than adapting the dominant subspace, with less interference with existing capabilities.\r\n\r\n### DEFT\r\n\r\n@MAXNORM8650 added DEFT, [\"Decompositional Efficient Fine-Tuning for Text-to-Image Models\"](https://arxiv.org/abs/2509.22793), in #3342. This PEFT method splits a weight update into two learned low-rank parts: a projection that removes a sub-space of the frozen weight, and a low-rank update that injects new content in its place. Use DEFT for personalizing a text-to-image model from a few images while retaining the base model's instruction-following/editability with minimal forgetting. It is less of a fit if you don't need to preserve the base model's other capabilities (a plain additive adapter is simpler) or for layers beyond Linear/Conv1D. The [PaRa method](https://arxiv.org/abs/2406.05641) is also supported as a special case of DEFT.\r\n\r\n## Enhancements\r\n\r\n### Automatic LoRA target selection\r\n\r\n@oswaldoludwig added `KappaTuneSelector` in #3106 based on [\"The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units\"](https://arxiv.org/abs/2506.16289). The PR adds a new function, `find_kappa_target_modules`, which automatically identifies which modules best to target based on the condition number of the base weight matrices. This removes some of the guesswork from choosing `target_modules` when configuring LoRA or other PEFT methods.\r\n\r\n### Broader quantization support\r\n\r\n#3117 adds generic quantization support to BOFT, MiSS, and VeRA, and #3321 does the same for SHiRA. Previously, quantization support had to be implemented per method and per backend; more methods now work with the common quantization backends out of the box. Note that as a consequence, VeRA now uses the generic quantization backend instead of its previous bitsandbytes quantization backend. If you encounter issues because of that, please let us know.\r\n\r\n### Multiple adapters with `target_parameters`\r\n\r\n`target_parameters`, which allows targeting `nn.Parameter`s directly (useful for MoE models that pack experts into a single parameter), now supports adding and loading multiple adapters (#3350).\r\n\r\n### AdaLoRA for Conv2d\r\n\r\n@Anai-Guo added `SVDConv2d` in #3196, so AdaLoRA can now target `torch.nn.Conv2d` layers.\r\n\r\n### Documentation restructure\r\n\r\nThe [PEFT documentation](https://huggingface.co/docs/peft/index) was reorganized in #3300 and #3325, which should make it considerably easier to navigate now that PEFT supports so many methods. Each PEFT method also provides an overview of how well it fares in the PEFT benchmarks. There is also a long tail of documentation improvements from many contributors this release, thanks to everyone who worked on those.\r\n\r\n## Changes\r\n\r\n### AutoGPTQ deprecation\r\n\r\nFollowing the switch to [GPT-QModel](https://github.com/ModelCloud/GPTQModel) as the backend in the previous release, AutoGPTQ is now formally deprecated (#3190) thanks to @Qubitium.\r\n\r\n## Benchmarks\r\n\r\n### Image generation benchmark\r\n\r\nThe method comparison suite now has an image generation benchmark (#3082). This complements the existing MetaMathQA benchmark and allows to compare PEFT methods on a completely different task. This gives us a more complete picture of the capabilities of different PEFT methods. To check the results, visit the [Gradio Space](https://huggingface.co/spaces/peft-internal-testing/PEFT-method-comparison) and on the top left, select the task \"image-gen\".\r\n\r\n<img width=\"925\" height=\"631\" alt=\"Screenshot from 2026-07-28 15-48-53\" src=\"https://github.com/user-attachments/assets/e6260b44-9b83-45b7-bd6c-d15bd429e006\" />\r\n\r\n_A look at the results from the image generation benchmark_\r\n\r\n### MetaMathQA\r\n\r\nSince the last release, we also made several improvements to the [MetaMathQA benchmark](https://github.com/huggingface/peft/method_comparison/MetaMathQA). You should check the updated results in the [Gradio Space](https://huggingface.co/spaces/peft-internal-testing/PEFT-method-comparison). With the help of the community, we now cover a much broader range of experiments. Moreover, we discovered a bug in the forgetting metric -- the new numbers now accurately reflect how much fine-tuning retains of the original knowledge of the base model.\r\n\r\n### PEFT Shop\r\n\r\n#3317 adds a new [Gradio Space](https://huggingface.co/spaces/peft-internal-testing/PEFT-shop) with the goal of making it very easy to compare different PEFT methods not only with regard to performance but also capabilities (e.g. if quantization is supported). Give this Space a visit to find out which PEFT method is the best fit for your use case.\r\n\r\n<img width=\"1013\" height=\"697\" alt=\"Screenshot from 2026-07-28 15-30-21\" src=\"https://github.com/user-attachments/assets/bd9ad132-a44f-4a2f-b28d-e499c9a5eaea\" />\r\n\r\n_Browse the PEFT shop for the best PEFT method for your use case_\r\n\r\n## All Changes\r\n\r\n* chore: bump doc-builder SHA for PR upload workflow by @rtrompier in https://github.com/huggingface/peft/pull/3162\r\n* Benchmark results for new methods by @githubnemo in https://github.com/huggingface/peft/pull/3160\r\n* fix: guard UPCAST_DTYPES lookup with hasattr to support older PyTorch by @Anai-Guo in https://github.com/huggingface/peft/pull/3161\r\n* Bump version to 0.19.1.dev0 after release by @BenjaminBossan in https://github.com/huggingface/peft/pull/3156\r\n* ENH: Option for gradient checkpointing in MetaMath by @BenjaminBossan in https://github.com/huggingface/peft/pull/3157\r\n* MetaMathQA: upload checkpoint to bucket after training by @githubnemo in https://github.com/huggingface/peft/pull/3163\r\n* FIX How to swapped in/out features for transposed by @BenjaminBossan in https://github.com/huggingface/peft/pull/3165\r\n* Implement import allowlist in AutoPeftModel, limit access in megatron by @githubnemo in https://github.com/huggingface/peft/pull/3090\r\n* Refactor layer initialization: PR 2960 continued by @BenjaminBossan in https://github.com/huggingface/peft/pull/3047\r\n* Fix SVDQuantLinear __repr__ nested inside forward instead of on the class by @Chessing234 in https://github.com/huggingface/peft/pull/3174\r\n* Resolve #2944: ensure_weight_tying respects config + improved tests by @Adya-Prasad in https://github.com/huggingface/peft/pull/3171\r\n* FIX Clip module structure since transformers > 5.5 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3179\r\n* Bump version to 0.19.2.dev0 after release by @BenjaminBossan in https://github.com/huggingface/peft/pull/3178\r\n* FIX TinyLoRA non-determinism by @BenjaminBossan in https://github.com/huggingface/peft/pull/3180\r\n* FIX VeRA and OFT error after layer init refactor by @BenjaminBossan in https://github.com/huggingface/peft/pull/3187\r\n* fix the test_hotswapping_compiled_model_xxx case of diffusers failure… by @sywangyi in https://github.com/huggingface/peft/pull/3183\r\n* CI Fix GPTQmodel install in Docker build by @BenjaminBossan in https://github.com/huggingface/peft/pull/3188\r\n* fix aux4 kwargs-only modules in modules_to_save by @ved1beta in https://github.com/huggingface/peft/pull/3199\r\n* chore: bump doc-builder SHA for main doc build workflow by @rtrompier in https://github.com/huggingface/peft/pull/3198\r\n* FIX PVeRA forward implementation for bitsandbytes by @BenjaminBossan in https://github.com/huggingface/peft/pull/3189\r\n* Update GPT-QModel references and deprecate AutoGPTQ by @Qubitium in https://github.com/huggingface/peft/pull/3190\r\n* FIX Transformers weight conversion regression by @BenjaminBossan in https://github.com/huggingface/peft/pull/3197\r\n* Add a generator to the PVeRA reparameterization function. by @leofillioux in https://github.com/huggingface/peft/pull/3200\r\n* Add BEFT as a new PEFT method by @whubaichuan in https://github.com/huggingface/peft/pull/3195\r\n* feat(adalora): add SVDConv2d to support torch.nn.Conv2d target modules by @Anai-Guo in https://github.com/huggingface/peft/pull/3196\r\n* MiSS update by @Joluck in https://github.com/huggingface/peft/pull/3194\r\n* Bump actions/cache from 5.0.4 to 5.0.5 in the ci-actions group by @dependabot[bot] in https://github.com/huggingface/peft/pull/3206\r\n* Bump the third-party-actions group with 7 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3207\r\n* FIX Error when prefix tuning Gemma 4 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3205\r\n* FIX Use tiny Gemma 4 model for faster tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/3215\r\n* FIX State dict injection with weight conversion by @BenjaminBossan in https://github.com/huggingface/peft/pull/3212\r\n* fix: use torch.div in _ConvNd.unmerge for bias (was torch.mul) by @Chessing234 in https://github.com/huggingface/peft/pull/3216\r\n* Pull Request: Adding HiRA integration into PEFT library by @hqsiswiliam in https://github.com/huggingface/peft/pull/2668\r\n* Add KappaTuneSelector: condition-number-based automatic LoRA target selection by @oswaldoludwig in https://github.com/huggingface/peft/pull/3106\r\n* Implement VeLoRA and add tests. by @roymiles in https://github.com/huggingface/peft/pull/3159\r\n* Add type hints to utility functions in merge_utils and other modules by @RudrenduPaul in https://github.com/huggingface/peft/pull/3144\r\n* fix compatible issue with datasets version for beft example by @kaixuanliu in https://github.com/huggingface/peft/pull/3218\r\n* refine test case for `test_lora_seq2seq_lm_multi_gpu_inference` by @kaixuanliu in https://github.com/huggingface/peft/pull/3226\r\n* fix CI bug for multi-device scene by @kaixuanliu in https://github.com/huggingface/peft/pull/3228\r\n* refine test_causal_lm_training_multi_accelerator test case by @kaixuanliu in https://github.com/huggingface/peft/pull/3225\r\n* CHORE: Upgrade ruff 0.15.12 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3202\r\n* DOC Improve LoHa docs with overview, when-to-use, and quick start exa… by @Dhruv-1710 in https://github.com/huggingface/peft/pull/3224\r\n* fix(xlora): duplicated \"for\" in adapters warning message by @vip892766gma in https://github.com/huggingface/peft/pull/3235\r\n* docs: improve MiSS documentation with practical guidance and LLM example by @vedantnavle13 in https://github.com/huggingface/peft/pull/3231\r\n* CI Remove Transformer Engine from Dockerfile by @BenjaminBossan in https://github.com/huggingface/peft/pull/3248\r\n* Intergrate MonteCLoRA (TMLR 2025 accepted) into PEFT by @victor7246 in https://github.com/huggingface/peft/pull/2943\r\n* Bump diffusers from 0.34.0 to 0.38.0 in /examples/hra_dreambooth by @dependabot[bot] in https://github.com/huggingface/peft/pull/3250\r\n* chore: add missing return type annotations in import_utils and other by @lphuc2250gma in https://github.com/huggingface/peft/pull/3245\r\n* TST Remove unnecessary use of trust_remote_code by @BenjaminBossan in https://github.com/huggingface/peft/pull/3242\r\n* Fix prepare model for compiled hotswap requires grad by @BenjaminBossan in https://github.com/huggingface/peft/pull/3241\r\n* DOC Fix incorrect imports in helpers.py docstring examples by @blipbyte in https://github.com/huggingface/peft/pull/3249\r\n* FIX Transformers weight conversion undo incorrect key renaming by @BenjaminBossan in https://github.com/huggingface/peft/pull/3262\r\n* [BUG FIX] tuners/lora/eva: fix label mask guard using 'in' instead of hasattr for dict by @Dev-X25874 in https://github.com/huggingface/peft/pull/3247\r\n* chore: add missing return type annotations by @lphuc2250gma in https://github.com/huggingface/peft/pull/3272\r\n* TST Correctly resolve GLUE dataset by @BenjaminBossan in https://github.com/huggingface/peft/pull/3263\r\n* SEC Add 7 day cooldown to Dependabot by @BenjaminBossan in https://github.com/huggingface/peft/pull/3273\r\n* DOC Fix broken docstring examples in tuner model.py files by @blipbyte in https://github.com/huggingface/peft/pull/3254\r\n* Bump the third-party-actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3276\r\n* fix(tuners/lora/torchao): make get_apply_tensor_subclass optional (#3223) by @Anai-Guo in https://github.com/huggingface/peft/pull/3234\r\n* FIX Transformers weight conversion PEFT model prefix by @BenjaminBossan in https://github.com/huggingface/peft/pull/3230\r\n* FIX Transformers argument name in rename_source_key by @BenjaminBossan in https://github.com/huggingface/peft/pull/3284\r\n* DOC Fix incorrect class references in tuner config docstrings by @javierdejesusda in https://github.com/huggingface/peft/pull/3286\r\n* Avoid TP helper imports for non-TP distributed LoRA loads by @jiqing-feng in https://github.com/huggingface/peft/pull/3261\r\n* Add BEFT and HiRA results by @githubnemo in https://github.com/huggingface/peft/pull/3277\r\n* Add PVeRA rank ablation benchmark experiments by @anrohanro in https://github.com/huggingface/peft/pull/3256\r\n* experiment method comparison #2310 : adaptation_prompt by @Adya-Prasad in https://github.com/huggingface/peft/pull/3301\r\n* Add decoupled weight-decay one-step update test by @eSVeeF in https://github.com/huggingface/peft/pull/3296\r\n* Image generation benchmark by @BenjaminBossan in https://github.com/huggingface/peft/pull/3082\r\n* Generic quantization support for BOFT, MiSS, VeRA by @BenjaminBossan in https://github.com/huggingface/peft/pull/3117\r\n* CI GPU run: Remove gptmodel from Dockerfile because of pcre by @BenjaminBossan in https://github.com/huggingface/peft/pull/3253\r\n* DOC Fix wrong function name in docstring by @BenjaminBossan in https://github.com/huggingface/peft/pull/3281\r\n* FIX Saving adapter when other adapters partially match name by @BenjaminBossan in https://github.com/huggingface/peft/pull/3282\r\n* TST Regression tests download regression artifacts lazily by @BenjaminBossan in https://github.com/huggingface/peft/pull/3285\r\n* CHORE Add instructions for coding agents by @BenjaminBossan in https://github.com/huggingface/peft/pull/3288\r\n* FIX TST Failing HiRA and EETQ GPU tests. by @BenjaminBossan in https://github.com/huggingface/peft/pull/3287\r\n* Separate, embeddable method comparison space by @githubnemo in https://github.com/huggingface/peft/pull/3303\r\n* FIX Image benchmark dataset wrong prompt order by @BenjaminBossan in https://github.com/huggingface/peft/pull/3308\r\n* Fix app deployment CI issue by @githubnemo in https://github.com/huggingface/peft/pull/3312\r\n* Method comparison: Image generation results by @BenjaminBossan in https://github.com/huggingface/peft/pull/3313\r\n* Method comparison: Evaluation script for image gen benchmark by @BenjaminBossan in https://github.com/huggingface/peft/pull/3314\r\n* Fix app deployment CI issue by @githubnemo in https://github.com/huggingface/peft/pull/3316\r\n* Method comparison: Image benchmark shows sample images by @BenjaminBossan in https://github.com/huggingface/peft/pull/3309\r\n* FIX layers_to_transform matching wrong index in MoE models by @MichalMraz in https://github.com/huggingface/peft/pull/3318\r\n* Documentation re-structure by @githubnemo in https://github.com/huggingface/peft/pull/3300\r\n* docs: fix typos in OFTConfig docstring (paramters, speficy) by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3326\r\n* FIX AdaMSS save/load reproduction by making slice_pca SVD deterministic by @AshNicolus in https://github.com/huggingface/peft/pull/3310\r\n* Docs: Make API reference sorted and collapsed by @githubnemo in https://github.com/huggingface/peft/pull/3325\r\n* FEAT Add PEFT shop Gradio space by @BenjaminBossan in https://github.com/huggingface/peft/pull/3317\r\n* FIX PEFT Shop deployment workflow error by @BenjaminBossan in https://github.com/huggingface/peft/pull/3332\r\n* FIX PEFT Shop deployment workflow error 2 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3334\r\n* FIX PEFT shop app description extraction by @BenjaminBossan in https://github.com/huggingface/peft/pull/3335\r\n* docs: sync PEFT config docstrings with their fields by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3319\r\n* MetaMathQA: Optimized configs for IA3 by @githubnemo in https://github.com/huggingface/peft/pull/3337\r\n* Fix typos in docstrings, comments, and docs by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3328\r\n* Feature/add frod by @Bane-Elvin in https://github.com/huggingface/peft/pull/3270\r\n* refactor: dedupe prompt-learning kwarg adjustments in peft_model by @eSVeeF in https://github.com/huggingface/peft/pull/3336\r\n* docs: clarify subtract_mutated_init by @githubnemo in https://github.com/huggingface/peft/pull/3343\r\n* DOC Fix incorrect LoRA references in hira and ln_tuning model docstrings by @javierdejesusda in https://github.com/huggingface/peft/pull/3291\r\n* FIX Bug when targeting an nn.Parameter on the root module by @BenjaminBossan in https://github.com/huggingface/peft/pull/3302\r\n* DOC: fix adapter_names parameter name in set_requires_grad docstrings by @kratos0718 in https://github.com/huggingface/peft/pull/3271\r\n* TST Add tests for hotswapping targeted LoRA parameters by @BenjaminBossan in https://github.com/huggingface/peft/pull/3304\r\n* Mica by @sr-networks in https://github.com/huggingface/peft/pull/3260\r\n* TST Extend multiple active adapter tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/3361\r\n* fix: increase FRoD init_weights=False std to prevent flaky test by @peft-jambot in https://github.com/huggingface/peft/pull/3353\r\n* CI Tests: Extend ignored paths by @BenjaminBossan in https://github.com/huggingface/peft/pull/3339\r\n* CI Skip EETQ test if kernel not found by @BenjaminBossan in https://github.com/huggingface/peft/pull/3351\r\n* Add DoRA experiment for image-gen benchmark by @spideyashith in https://github.com/huggingface/peft/pull/3341\r\n* FIX Upcast bf16 adapters to fp32 on CPU in MultiheadAttention and VBLoRA delta weight by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3364\r\n* FIX Mixed: honor bias='<prefix>_only' for all compatible tuners (not just lora_only) by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3366\r\n* Sync function docstrings with their signatures by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3329\r\n* TST Add dedicated BD-LoRA math-correctness test suite (follow-up to #2895) by @eSVeeF in https://github.com/huggingface/peft/pull/3362\r\n* FIX Call super().__post_init__() in CPTConfig and RandLoraConfig by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3365\r\n* add glora by @not-lain in https://github.com/huggingface/peft/pull/3098\r\n* Add UniLora tuner to PEFT by @KaiyangLi1992 in https://github.com/huggingface/peft/pull/3257\r\n* FIX Remove duplicate VBLoRAConfig entry from peft.__all__ by @SHAI-Akshay-Tripathi in https://github.com/huggingface/peft/pull/3279\r\n* Bump the ci-actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3374\r\n* DOC Fix LoRA-GA 'Usage Tips' subsection by @BenjaminBossan in https://github.com/huggingface/peft/pull/3331\r\n* FEAT Add DEFT (Decompositional Efficient Fine-Tuning (NeurIPS'25) by @MAXNORM8650 in https://github.com/huggingface/peft/pull/3342\r\n* ENH Allow multiple adapters when using target_parameters by @BenjaminBossan in https://github.com/huggingface/peft/pull/3350\r\n* Docs: Improve VeRA conceptual explanation and parameter scaling by @SanjayMuthuswamy in https://github.com/huggingface/peft/pull/3386\r\n* docs: fix missing torch import and variable name in README examples by @lxcxjxhx in https://github.com/huggingface/peft/pull/3392\r\n* CI Pin kernels version by @BenjaminBossan in https://github.com/huggingface/peft/pull/3377\r\n* FIX Resolve regex target_modules in MoE config conversion by @1fanwang in https://github.com/huggingface/peft/pull/3232\r\n* Make doc builds faster by @mishig25 in https://github.com/huggingface/peft/pull/3393\r\n* TST Skip failing Diffusers tests until v0.40.0 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3394\r\n* fix(utils): call empty_cache() after fp16→fp32 casts in prepare_model_for_kbit_training by @umutonuryasar in https://github.com/huggingface/peft/pull/3293\r\n* Add default target_modules for nemotron_h hybrid Mamba-MoE models by @A1c0r-Z in https://github.com/huggingface/peft/pull/3289\r\n* CI Fix error with Windows loading stable diffusion models by @BenjaminBossan in https://github.com/huggingface/peft/pull/3399\r\n* feat: registry-based dispatch for resolve_lora_variant in Linear layer by @isha822 in https://github.com/huggingface/peft/pull/3219\r\n* fix: remove dead code in LoRA get_delta_weight and fix unsafe merge dtype inconsistency by @lxcxjxhx in https://github.com/huggingface/peft/pull/3396\r\n* Fix typos in examples, tests, and docs by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3398\r\n* DOC Expand contributing installation guide with fork and setup steps by @Strongich in https://github.com/huggingface/peft/pull/3406\r\n* FIX Register per-adapter containers so delete_adapter removes them (unilora, fourierft, psoft) by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3376\r\n* FIX Crash in reduce_intruder_dimension on fp16/bf16 adapters with autocast_adapter_dtype=False by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3410\r\n* FIX GLoRA safe_merge mutating the base layer before the isfinite check by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3411\r\n* fix device mismatch for bnb dequantization by @kaixuanliu in https://github.com/huggingface/peft/pull/3418\r\n* Minor app embed improvements by @githubnemo in https://github.com/huggingface/peft/pull/3322\r\n* Improve IA3 documentation: add usage guidance, module selection tips, and LoRA comparison by @DresdenGman in https://github.com/huggingface/peft/pull/3417\r\n* FIX Bug in forgetting metric in MetaMathQA by @BenjaminBossan in https://github.com/huggingface/peft/pull/3419\r\n* Create SECURITY.md by @githubnemo in https://github.com/huggingface/peft/pull/3432\r\n* Update MetaMAthQA results by @githubnemo in https://github.com/huggingface/peft/pull/3445\r\n* fix: upgrade http:// to https:// in README by @hanu-14 in https://github.com/huggingface/peft/pull/3454\r\n* CI: Bump the third-party-actions group across 1 directory with 7 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3375\r\n* Return False from is_gptqmodel_available when gptqmodel isn't installed by @c-tonneslan in https://github.com/huggingface/peft/pull/3255\r\n* Method comparison: Fix adaption prompt experiment by @BenjaminBossan in https://github.com/huggingface/peft/pull/3463\r\n* fix: replace bare except with except Exception by @lxcxjxhx in https://github.com/huggingface/peft/pull/3462\r\n* Fix BdLoraConfig.nblocks default being a tuple instead of an int by @Osamaali313 in https://github.com/huggingface/peft/pull/3456\r\n* ENH: Warn in get_peft_model when base model parameters are on the meta device by @immu4989 in https://github.com/huggingface/peft/pull/3415\r\n* Track git dirty state in method_comparison benchmark results (#3384) by @eduanmoldeep in https://github.com/huggingface/peft/pull/3464\r\n* FIX DeLoRA crash with UnboundLocalError on unsupported layers by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3434\r\n* FIX HiRA ConvNd layers with groups > 1 crash on forward, not just merge by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3430\r\n* FIX TinyLoRA weight_tying corruption when adding overlapping adapters by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3422\r\n* FIX DEFT's cached merge factor: precision loss and device staleness by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3412\r\n* FIX MiSS with bat init error with some quantization backends by @BenjaminBossan in https://github.com/huggingface/peft/pull/3381\r\n* Release 0.20.0 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3388\r\n* Update AdaptionPrompt results by @githubnemo in https://github.com/huggingface/peft/pull/3466\r\n* Add quantization support for SHiRA by @githubnemo in https://github.com/huggingface/peft/pull/3321\r\n* Fix grouped convolutions in LoHa and LoKr by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3440\r\n* fix: respect layer-specific LoRA-FA scaling in optimizer by @eSVeeF in https://github.com/huggingface/peft/pull/3315\r\n* FIX grouped Conv2d LoRA rank padding for hotswapping (#3416) by @eSVeeF in https://github.com/huggingface/peft/pull/3427\r\n* Fix Gemma4 prefix tuning with per_layer_config by @peft-jambot in https://github.com/huggingface/peft/pull/3471\r\n* AI: Update contribution guide based on recent experience by @BenjaminBossan in https://github.com/huggingface/peft/pull/3459\r\n* CI Enable test coverage only on one of the runs by @BenjaminBossan in https://github.com/huggingface/peft/pull/3469\r\n* Validate GraLoRA, AdaLoRA, WaveFT, and AdaMSS configs by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3438\r\n* Forward revision and token in AutoPeftModel loading by @DaoyuanLi2816 in https://github.com/huggingface/peft/pull/3442\r\n* Add get base model state dict by @Isalia20 in https://github.com/huggingface/peft/pull/3000\r\n* Imagegen results for new PEFT methods by @githubnemo in https://github.com/huggingface/peft/pull/3483\r\n\r\n## New Contributors\r\n\r\n* @rtrompier made their first contribution in https://github.com/huggingface/peft/pull/3162\r\n* @Anai-Guo made their first contribution in https://github.com/huggingface/peft/pull/3161\r\n* @Adya-Prasad made their first contribution in https://github.com/huggingface/peft/pull/3171\r\n* @whubaichuan made their first contribution in https://github.com/huggingface/peft/pull/3195\r\n* @hqsiswiliam made their first contribution in https://github.com/huggingface/peft/pull/2668\r\n* @oswaldoludwig made their first contribution in https://github.com/huggingface/peft/pull/3106\r\n* @roymiles made their first contribution in https://github.com/huggingface/peft/pull/3159\r\n* @RudrenduPaul made their first contribution in https://github.com/huggingface/peft/pull/3144\r\n* @Dhruv-1710 made their first contribution in https://github.com/huggingface/peft/pull/3224\r\n* @vip892766gma made their first contribution in https://github.com/huggingface/peft/pull/3235\r\n* @vedantnavle13 made their first contribution in https://github.com/huggingface/peft/pull/3231\r\n* @victor7246 made their first contribution in https://github.com/huggingface/peft/pull/2943\r\n* @lphuc2250gma made their first contribution in https://github.com/huggingface/peft/pull/3245\r\n* @blipbyte made their first contribution in https://github.com/huggingface/peft/pull/3249\r\n* @Dev-X25874 made their first contribution in https://github.com/huggingface/peft/pull/3247\r\n* @javierdejesusda made their first contribution in https://github.com/huggingface/peft/pull/3286\r\n* @anrohanro made their first contribution in https://github.com/huggingface/peft/pull/3256\r\n* @eSVeeF made their first contribution in https://github.com/huggingface/peft/pull/3296\r\n* @MichalMraz made their first contribution in https://github.com/huggingface/peft/pull/3318\r\n* @DaoyuanLi2816 made their first contribution in https://github.com/huggingface/peft/pull/3326\r\n* @AshNicolus made their first contribution in https://github.com/huggingface/peft/pull/3310\r\n* @Bane-Elvin made their first contribution in https://github.com/huggingface/peft/pull/3270\r\n* @kratos0718 made their first contribution in https://github.com/huggingface/peft/pull/3271\r\n* @sr-networks made their first contribution in https://github.com/huggingface/peft/pull/3260\r\n* @peft-jambot made their first contribution in https://github.com/huggingface/peft/pull/3353\r\n* @spideyashith made their first contribution in https://github.com/huggingface/peft/pull/3341\r\n* @not-lain made their first contribution in https://github.com/huggingface/peft/pull/3098\r\n* @KaiyangLi1992 made their first contribution in https://github.com/huggingface/peft/pull/3257\r\n* @SHAI-Akshay-Tripathi made their first contribution in https://github.com/huggingface/peft/pull/3279\r\n* @MAXNORM8650 made their first contribution in https://github.com/huggingface/peft/pull/3342\r\n* @SanjayMuthuswamy made their first contribution in https://github.com/huggingface/peft/pull/3386\r\n* @lxcxjxhx made their first contribution in https://github.com/huggingface/peft/pull/3392\r\n* @1fanwang made their first contribution in https://github.com/huggingface/peft/pull/3232\r\n* @umutonuryasar made their first contribution in https://github.com/huggingface/peft/pull/3293\r\n* @A1c0r-Z made their first contribution in https://github.com/huggingface/peft/pull/3289\r\n* @isha822 made their first contribution in https://github.com/huggingface/peft/pull/3219\r\n* @Strongich made their first contribution in https://github.com/huggingface/peft/pull/3406\r\n* @DresdenGman made their first contribution in https://github.com/huggingface/peft/pull/3417\r\n* @hanu-14 made their first contribution in https://github.com/huggingface/peft/pull/3454\r\n* @c-tonneslan made their first contribution in https://github.com/huggingface/peft/pull/3255\r\n* @Osamaali313 made their first contribution in https://github.com/huggingface/peft/pull/3456\r\n* @immu4989 made their first contribution in https://github.com/huggingface/peft/pull/3415\r\n* @eduanmoldeep made their first contribution in https://github.com/huggingface/peft/pull/3464\r\n\r\n**Full Changelog**: https://github.com/huggingface/peft/compare/v0.19.0...v0.20.0","publishedAt":"2026-07-28T13:49:42.000Z","url":"https://github.com/huggingface/peft/releases/tag/v0.20.0","media":[{"type":"image","url":"https://github.com/user-attachments/assets/6194fae9-f3c3-48ca-b43b-40c774bfe6eb","r2Key":"releases/eeddab5e31d27ff75de508eb932ad3c18e3b3ce620e7bc2150dabcefd33473cc.png","r2Url":"https://media.releases.sh/releases/eeddab5e31d27ff75de508eb932ad3c18e3b3ce620e7bc2150dabcefd33473cc.png"},{"type":"image","url":"https://github.com/user-attachments/assets/e6260b44-9b83-45b7-bd6c-d15bd429e006","r2Key":"releases/de19d41e99ca915a4c1ea237553d81bc44019c1105ff80914332d10b61798c56.png","r2Url":"https://media.releases.sh/releases/de19d41e99ca915a4c1ea237553d81bc44019c1105ff80914332d10b61798c56.png"},{"type":"image","url":"https://github.com/user-attachments/assets/bd9ad132-a44f-4a2f-b28d-e499c9a5eaea","r2Key":"releases/48126b4b7dd522f2b9c53e25e081a123b9194b422a9b94ecd1f6c0fda96a30d2.png","r2Url":"https://media.releases.sh/releases/48126b4b7dd522f2b9c53e25e081a123b9194b422a9b94ecd1f6c0fda96a30d2.png"}],"prerelease":false,"source":{"slug":"peft","name":"PEFT","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":33897,"contentTokens":9441,"composition":{"bugs":0,"features":9,"enhancements":5}},{"id":"rel_PupF6nL1tk3deaqBRAazI","version":"v1.9.2","type":"feature","title":"v1.9.2","summary":"Fixed quantization tests for bitsandbytes 0.50.0 and reverted an xfail skip for Nemotron H GRPO/RLOO tests after a transformers kernel bug fix.","titleGenerated":"TRL v1.9.2 fixes quantization test and unblocks Nemotron kernels","titleShort":"Quantization tests fixed; Nemotron kernels unblocked","breaking":"none","importance":2,"content":"## What's Changed\r\n\r\n* Revert xfail for NemotronH GRPO/RLOO tests now that transformers#47569 fixed the kernels bug by @albertvillanova in https://github.com/huggingface/trl/pull/6558\r\n* Fix test_peft_with_quantization tests for bitsandbytes 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6546\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.9.1...v1.9.2","publishedAt":"2026-07-28T10:39:01.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.9.2","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":398,"contentTokens":124,"composition":{"bugs":2,"features":0,"enhancements":0}},{"id":"rel_WJkG0IXM71QVvkGnKuG6s","version":"v1.9.1","type":"feature","title":"v1.9.1","summary":"Fixes a crash when training with Liger kernel on pre-Ampere (Fermi, Kepler, Maxwell, Pascal) GPUs and corrects DAPO, CISPO, and VESPO loss normalization when steps_per_generation differs from gradient_accumulation_steps. Also fixes vLLM server-mode communicator initialization, queue wait time metric in AsyncGRPOTrainer, and prepare_deepspeed crash with CPU offload optimizer.","titleGenerated":"TRL v1.9.1 fixes Liger kernel crash on pre-Ampere GPUs and DAPO/VESPO loss normalization","titleShort":"Liger kernel crash on pre-Ampere GPUs fixed; DAPO/VESPO loss normalization corrected","breaking":"none","importance":2,"content":"## What's Changed\r\n\r\n* Fix vLLM server-mode communicator initialization to use the current A… by @sywangyi in https://github.com/huggingface/trl/pull/6417\r\n* Fix : queue wait time metric AsyncGRPOTrainer by @AmineDiro in https://github.com/huggingface/trl/pull/6489\r\n* Hotfix CI: Temporarily pin liger-kernel < 0.8.1 by @albertvillanova in https://github.com/huggingface/trl/pull/6517\r\n* Fix crash when training with Liger kernel on pre-Ampere GPUs by @albertvillanova in https://github.com/huggingface/trl/pull/6518\r\n* Fix prepare_deepspeed crash with cpu offload optimizer by @roycho96 in https://github.com/huggingface/trl/pull/5916\r\n* Add Gemma4 response template by @qgallouedec in https://github.com/huggingface/trl/pull/6535\r\n* fix(grpo): correct DAPO/CISPO/VESPO loss normalization when steps_per_generation != gradient_accumulation_steps by @0xadvait in https://github.com/huggingface/trl/pull/6024\r\n* Hotfix CI: Temporarily pin bitsandbytes < 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6544\r\n* Hotfix CI: Xfail NemotronH GRPO/RLOO tests against a broken transformers dev build by @albertvillanova in https://github.com/huggingface/trl/pull/6542\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.9.0...v1.9.1\r\n","publishedAt":"2026-07-26T16:49:29.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.9.1","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":1265,"contentTokens":382,"composition":{"bugs":6,"features":1,"enhancements":0}},{"id":"rel_1Rmxks9DRP1V0tp4Z4gjz","version":"v1.9.0","type":"feature","title":"v1.9.0","summary":"GRPO and RLOO trainers now support iterable datasets via a new repeat_iterable_dataset generator, and environments can own the data (train_dataset becomes optional). Message-level rollouts land in AsyncGRPO, and the DistillationTrainer received a major refactor including a fix for num_items_in_batch that counted the wrong tokens and NaN'd on prompt-only datasets.","titleGenerated":"TRL v1.9.0 supports streaming datasets in GRPO/RLOO and environment-owned data","titleShort":"Streaming datasets in GRPO/RLOO; environments own data without dummy dataset","breaking":"minor","importance":3,"content":"## Features\r\n\r\n### Iterable / streaming datasets in GRPO and RLOO\r\n\r\nLong-standing request finally landed. GRPO and RLOO rely on `RepeatSampler` to repeat each prompt `num_generations` times and group them across processes — but samplers can't attach to iterable datasets (no length, no indexing), and the old behavior silently bypassed the sampler, quietly corrupting advantage computation.\r\n\r\nA new `repeat_iterable_dataset` generator reproduces `RepeatSampler`'s exact ordering by transforming the stream itself instead of reordering indices. Trainers wrap iterable datasets via `IterableDataset.from_generator`, force `dispatch_batches=False`, and shard the stream per-process to preserve prompt groups after cross-process gathering.\r\n\r\n```python\r\nfrom datasets import load_dataset\r\nfrom trl import GRPOConfig, GRPOTrainer\r\n\r\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\", streaming=True)\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=GRPOConfig(max_steps=1000),  # required for iterable datasets\r\n    train_dataset=dataset,\r\n    reward_funcs=accuracy_reward,\r\n)\r\n```\r\n\r\n`max_steps` is required for iterable datasets (no length). A clear error also fires if `dispatch_batches=True` is combined with an iterable dataset.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6351\r\n\r\n### Environment-owned datasets\r\n\r\nWhen an `environment_factory` is provided, the environment can now own the data. `train_dataset` becomes optional: `reset()` returns the prompt. No more fabricating a dummy dataset just to drive the loop.\r\n\r\n```python\r\nclass WordleEnv:\r\n    def reset(self, **kwargs) -> str:      # returns the prompt\r\n        self._target = sample(words)\r\n        return f\"Guess a {len(self._target)}-letter word.\"\r\n    def get_reward(self) -> float:\r\n        return 1.0 if self._solved else 0.0\r\n    def guess(self, word: str) -> str:     # exposed tool\r\n        self._solved = word == self._target\r\n        return _feedback(word, self._target)\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=GRPOConfig(max_steps=1000),\r\n    environment_factory=WordleEnv,          # no train_dataset needed\r\n)\r\n```\r\n\r\nApplies to both `GRPOTrainer` and `AsyncGRPOTrainer`. A provided `train_dataset` still works exactly as before; multi-environment routing datasets can now be **`environment`-only** (no `prompt` column required).\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6349\r\n\r\n### Message-level rollouts in AsyncGRPO\r\n\r\nAn opt-in way to build training rows from a multi-turn conversation. Message mode keeps the conversation as messages and **re-tokenizes the whole thing each turn**, then checks whether the fresh tokens still start with the tokens held so far. If yes → append the new part (same as token mode). If no → a rewrite happened → close the row and open a new one that matches what the model actually read.\r\n\r\n```python\r\nAsyncGRPOConfig(\r\n    rollout_protocol=\"message\",   # \"token\" (default) | \"message\"\r\n    fork_threshold_tokens=1024,   # message-mode only\r\n)\r\n```\r\n\r\nPer-turn drift is classified as **CLEAN** (append), **REALIGN** (last-answer tail wobble → overwrite same row), or **FORK** (real rewrite → new row). One advantage per conversation, stamped on every row it produced; under token-mean loss a fork is invisible. Sets up the plumbing for future tree-trajectory support — scoring is already branch-agnostic (groups by `rollout_id`).\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6250\r\n\r\n### AsyncGRPO: `VLLMClient` and decoupled weight sync\r\n\r\nA follow-up to v1.8's rollout worker split. A small `VLLMClient` becomes the single place that speaks HTTP to the vLLM server (named methods for `wait_for_server_ready`, `get_max_model_len`, `pause`/`resume`, weight-update endpoints — stateless, so picklable). Generation deliberately stays on its own async `aiohttp` session in the rollout child.\r\n\r\nWeight sync is now built from config independent of `rollout_worker` (previously a custom rollout worker force-set it to `None`), and is injectable via a new `WeightTransferProtocol`. `token_budget` now waits for `/health` before defaulting to `get_max_model_len()`, so a slow-starting vLLM waits instead of failing the run.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6269\r\n\r\n### `DistillationTrainer` refactor\r\n\r\nA ~13-PR sweep that reshapes the experimental `DistillationTrainer` for future stability:\r\n\r\n- **Extract `ServerDistillationTrainer`** for the teacher-server path (#6454)\r\n- **Move IW-OPD** to a dedicated experimental trainer (#6453)\r\n- **Remove the local sparse top-1 loss path** from the base trainer (#6455)\r\n- **Remove top-k support** from the base generalized JSD loss (#6456)\r\n- **Validate the teacher by vocab size** instead of tokenizer identity (#6457)\r\n- **Deprecate & remove `lmbda`** on/off-policy mixing and the off-policy training branch (#6458, #6460); GKD paper reproduction now points at `GKDTrainer` (#6459)\r\n- **Accept a `prompt` column** alongside `messages` (#6461); **deprecate messages-format datasets** (#6474)\r\n- ⚠️ **Fix `num_items_in_batch`** to count generated completion tokens — the loss denominator was wrong for on-policy training (counted dataset completions, not generated ones) and **NaN'd on prompt-only datasets**. This changes loss values for anyone on the old path (#6478)\r\n- Preparatory pins on the objective and on/off-policy behavior end to end (#6450, #6451, #6452)\r\n\r\nAll by @qgallouedec.\r\n\r\n### New IW-OPD distillation objective\r\n\r\nImportance-Weighted On-Policy Distillation lands as an optional objective on the experimental `DistillationTrainer` — detached IW-OPD advantages built from sampled-token teacher and rollout logprobs (with cached vLLM rollout logprobs when `use_vllm=True`).\r\n\r\n```python\r\nDistillationConfig(\r\n    distillation_objective=\"iw_opd\",\r\n    iw_opd_gamma=1.0,\r\n    iw_opd_epsilon=0.2,\r\n    lmbda=1.0, reverse_kl_top_1_mode=\"sampled\",  # required\r\n)\r\n```\r\n\r\nNow split into its own experimental trainer as part of the refactor above.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/6191\r\n\r\n### `log_multimodal` for GRPO / RLOO\r\n\r\nNew config toggle to control whether images from multimodal completions are logged to trackers. Also applied to experimental DPPO and GRPO-with-replay-buffer.\r\n\r\nby @apardyl in https://github.com/huggingface/trl/pull/5408 and by @albertvillanova in https://github.com/huggingface/trl/pull/6352\r\n\r\n### vLLM version sweep\r\n\r\n* Support vLLM 0.24.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6355\r\n* Support vLLM 0.25.0 and 0.25.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6406\r\n* Drop vLLM 0.16 by @qgallouedec in https://github.com/huggingface/trl/pull/6404\r\n\r\n### Other\r\n\r\n* Simplify KTO KL completion construction into a single batched map by @albertvillanova in https://github.com/huggingface/trl/pull/6335\r\n* Add security policy to the repo by @albertvillanova in https://github.com/huggingface/trl/pull/6362\r\n* Removed `generate_rollout_completions` by @sergiopaniego in https://github.com/huggingface/trl/pull/5870\r\n* Set explicit neutral `top_p` in GKD on-policy generation by @sergiopaniego in https://github.com/huggingface/trl/pull/6337\r\n* Support `DatasetDict` and `IterableDatasetDict` as `eval_dataset` in `evaluate` by @albertvillanova in https://github.com/huggingface/trl/pull/6326\r\n* AsyncGRPO fork-independent epoch counting by @AmineDiro in https://github.com/huggingface/trl/pull/6385\r\n* Align `AsyncGRPOTrainer` args handling with stable trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6377\r\n* Reject `auto_find_batch_size` in GRPO and RLOO by @qgallouedec in https://github.com/huggingface/trl/pull/6411\r\n* Drop fully-masked examples after truncation in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/6320\r\n* Drop examples with no completion left after truncation in DPO and KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6321\r\n* Add `GMPOTrainer` to the telemetry allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/6410\r\n* `[GRPO]` Treat explicit `max_tool_calling_iterations=0` as zero, not unlimited by @Strongich in https://github.com/huggingface/trl/pull/6423\r\n* Add a tip to the Paper Index on reading papers with the `hf` CLI by @davanstrien in https://github.com/huggingface/trl/pull/6422\r\n\r\n## Deprecations and removals\r\n\r\n* **Remove the experimental DPPO trainer** by @qgallouedec in https://github.com/huggingface/trl/pull/6402\r\n* **Deprecate `messages`-format datasets in `DistillationTrainer`** — use `prompt` column instead — by @qgallouedec in https://github.com/huggingface/trl/pull/6474\r\n* **Deprecate `lmbda` on/off-policy mixing** in `DistillationTrainer` — off-policy path fully removed by @qgallouedec in https://github.com/huggingface/trl/pull/6458 and https://github.com/huggingface/trl/pull/6460\r\n\r\n## Fixes\r\n\r\n* **Fix biased cross-rank aggregation of token-weighted metrics** in GRPO / RLOO — mean-of-means was biased when per-rank token counts differ; now reduces numerator + count jointly for a true global mean. Also fixes `clip_ratio/low_min` / `high_max` to be per-completion instead of per-rank-batch-mean (DP-layout-independent). Logging-only, no loss change. By @qgallouedec in https://github.com/huggingface/trl/pull/6380\r\n* **Fix DPO/KTO `use_liger_kernel` under DeepSpeed ZeRO-3** by @qgallouedec in https://github.com/huggingface/trl/pull/6372\r\n* **Force `use_reentrant=True` for PEFT + ZeRO-3 + gradient checkpointing** across all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6356\r\n* **Fix `precompute_ref_log_probs=True` with DeepSpeed in DPO and KTO** by @qgallouedec in https://github.com/huggingface/trl/pull/6403\r\n* **Fix `cast_lm_head_to_fp32` under FSDP2 / DeepSpeed ZeRO-3** by @qgallouedec in https://github.com/huggingface/trl/pull/6330\r\n* **Fix corrupted reasoning blocks in Qwen3-family training chat templates** — the both-tags guard broke replay of self-generated thinking traces (SFT / distillation on model outputs); reverted to the original `</think>`-only check. By @qgallouedec in https://github.com/huggingface/trl/pull/6363\r\n* **Fix EOS being masked when `pad_token_id == eos_token_id` in GKD and GOLD** by @roycho96 in https://github.com/huggingface/trl/pull/6357\r\n* **Fix GRPO truncated-completion loss normalization** by @Functionhx in https://github.com/huggingface/trl/pull/6439\r\n* **Fix `apply_chat_template` dropping leading tokens from `chosen`** when the prompt common prefix shrinks against `rejected` by @sohumt123 in https://github.com/huggingface/trl/pull/6433\r\n* Fix crash when reward function is a `functools.partial` or callable instance by @qgallouedec in https://github.com/huggingface/trl/pull/6313\r\n* Fix QLoRA + FSDP1 mixed-dtype crash in `prepare_peft_model` by @sergiopaniego in https://github.com/huggingface/trl/pull/6343\r\n* Truncate GOLD on-policy prompts before generation, keeping the prompt end by @roycho96 in https://github.com/huggingface/trl/pull/6359\r\n* Fix `FileNotFoundError` in DPO/KTO `precompute_ref_log_probs` by @Functionhx in https://github.com/huggingface/trl/pull/6348\r\n* Fix `GeometricMixtureWrapper.generate()` crash after transformers' MoE decode optimization change by @albertvillanova in https://github.com/huggingface/trl/pull/6413\r\n* Fix GKD `vocab_size` check for multimodal model configs by @sergiopaniego in https://github.com/huggingface/trl/pull/6340\r\n* Fix OpenEnv docker-image example bootstrap by @sergiopaniego in https://github.com/huggingface/trl/pull/6329\r\n* Fix error type for `IterableDataset` in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6324\r\n* Fix KTO evaluate crash on streaming datasets with unpaired-preference data by @albertvillanova in https://github.com/huggingface/trl/pull/6325\r\n* Fix DPO/KTO evaluate crash with a dict `eval_dataset` and `precompute_ref_log_probs` by @albertvillanova in https://github.com/huggingface/trl/pull/6370\r\n* Fix DPO/KTO double-preprocessing `eval_dataset` dict passed to `evaluate()` by @albertvillanova in https://github.com/huggingface/trl/pull/6434\r\n* Raise when DPO/KTO cannot precompute correct reference log-probs at eval time by @albertvillanova in https://github.com/huggingface/trl/pull/6443\r\n* Avoid in-place mutation of input prompts in GRPO and DPPO environment loops by @albertvillanova in https://github.com/huggingface/trl/pull/6364 and https://github.com/huggingface/trl/pull/6366\r\n* Align KTO with DPO: drop `hasattr` guards on `lm_head.bias` in the Liger path by @qgallouedec in https://github.com/huggingface/trl/pull/6425\r\n* Align KTO with DPO: fix `logits/chosen` and `logits/rejected` metrics by @qgallouedec in https://github.com/huggingface/trl/pull/6424\r\n* Fix missing `eos_token_id` in non-vLLM on-policy GOLD test stubs by @kashif in https://github.com/huggingface/trl/pull/6394\r\n* Fix `BEMACallback` `bias_power` docstring by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6390\r\n\r\n## Documentation and Examples\r\n\r\n* Document `max_steps` requirement for iterable datasets in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6432\r\n* Align `AsyncGRPOConfig.max_completion_length` description with `GRPOConfig` by @qgallouedec in https://github.com/huggingface/trl/pull/6374\r\n* Align `AsyncGRPOTrainer.processing_class` docstring with `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6375\r\n* Align `AsyncGRPOTrainer` clip-ratio comment with `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6376\r\n* Fix formatting of error message for `loss_type` by @qgallouedec in https://github.com/huggingface/trl/pull/6373\r\n* Fix misleading name of `BEMACallback` `bias_power=0.0` test by @albertvillanova in https://github.com/huggingface/trl/pull/6464\r\n\r\n## CI\r\n\r\n* Destroy vLLM colocate mode process group after each CI test by @albertvillanova in https://github.com/huggingface/trl/pull/6353\r\n* Test KTO training on iterable datasets with the KL loss by @albertvillanova in https://github.com/huggingface/trl/pull/6338\r\n* Test all supported `eval_dataset` input types in core trainer `init` / `evaluate()` tests by @albertvillanova in https://github.com/huggingface/trl/pull/6336 and https://github.com/huggingface/trl/pull/6371\r\n* Add `eval_dataset` init tests for experimental GRPO / Online-DPO / SFT & distillation / preference / BCO+CPO+ORPO+PRM trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6395, https://github.com/huggingface/trl/pull/6396, https://github.com/huggingface/trl/pull/6397, https://github.com/huggingface/trl/pull/6398 and https://github.com/huggingface/trl/pull/6399\r\n* Silence experimental `environment_factory` warning in GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6445\r\n* Silence PEFT `ensure_weight_tying` warning on newer PEFT in GRPO tests / DPO+KTO liger tests by @albertvillanova in https://github.com/huggingface/trl/pull/6446 and https://github.com/huggingface/trl/pull/6463\r\n* Ignore benign `bitsandbytes` `_check_is_size` `FutureWarning` in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6448\r\n* Ignore benign `IndexPutBackward0` warning from Liger SAPO loss by @albertvillanova in https://github.com/huggingface/trl/pull/6436\r\n* Filter reworded `torch.jit.script_method` warning on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6465\r\n* Filter triton `AnnAssign` `DeprecationWarning` on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6467\r\n* Fix `create_reference_model` deprecation warning by @albertvillanova in https://github.com/huggingface/trl/pull/6418\r\n* Fix uncaptured cross-architecture VLM warning in GOLD trainer CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/6419\r\n* Re-record the `dpo` invariant reference after #6321 by @qgallouedec in https://github.com/huggingface/trl/pull/6421\r\n* Remove redundant `torch.tensor` re-wrapping of batch labels in KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6431\r\n* Bump the actions group across 1 directory with 10 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6442\r\n\r\n## New Contributors\r\n* @Functionhx made their first contribution in https://github.com/huggingface/trl/pull/6348\r\n* @sohumt123 made their first contribution in https://github.com/huggingface/trl/pull/6433\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6347\r\n* Fix FileNotFoundError in DPO/KTO precompute_ref_log_probs by @Functionhx in https://github.com/huggingface/trl/pull/6348\r\n* Add `log_multimodal` param to GRPOConfig and RLOOConfig to control image logging by @apardyl in https://github.com/huggingface/trl/pull/5408\r\n* Set explicit neutral top_p in GKD on-policy generation by @sergiopaniego in https://github.com/huggingface/trl/pull/6337\r\n* Fix GKD vocab_size check for multimodal model configs by @sergiopaniego in https://github.com/huggingface/trl/pull/6340\r\n* Fix OpenEnv docker-image example bootstrap by @sergiopaniego in https://github.com/huggingface/trl/pull/6329\r\n* Align log_multimodal in the DPPO and GRPO-with-replay-buffer experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6352\r\n* Fix QLoRA + FSDP1 mixed-dtype crash in prepare_peft_model by @sergiopaniego in https://github.com/huggingface/trl/pull/6343\r\n* Destroy vLLM colocate mode process group after each CI test by @albertvillanova in https://github.com/huggingface/trl/pull/6353\r\n* Fix error type for IterableDataset in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6324\r\n* Fix KTO evaluate crash on streaming datasets with unpaired-preference data by @albertvillanova in https://github.com/huggingface/trl/pull/6325\r\n* Support DatasetDict and IterableDatasetDict as eval_dataset in evaluate by @albertvillanova in https://github.com/huggingface/trl/pull/6326\r\n* Simplify KTO KL completion construction into a single batched map by @albertvillanova in https://github.com/huggingface/trl/pull/6335\r\n* implement message level rollout with linear trajectories by @AmineDiro in https://github.com/huggingface/trl/pull/6250\r\n* Add security policy to the repo by @albertvillanova in https://github.com/huggingface/trl/pull/6362\r\n* Removed `generate_rollout_completions` by @sergiopaniego in https://github.com/huggingface/trl/pull/5870\r\n* Fix crash when reward function is a `functools.partial` or callable instance by @qgallouedec in https://github.com/huggingface/trl/pull/6313\r\n* Drop fully-masked examples after truncation in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/6320\r\n* Fix `cast_lm_head_to_fp32` under FSDP2 / DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/6330\r\n* Add support for vLLM 0.24.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6355\r\n* Fix DPO/KTO `use_liger_kernel` under DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/6372\r\n* Test KTO training on iterable datasets with the KL loss by @albertvillanova in https://github.com/huggingface/trl/pull/6338\r\n* Avoid in-place mutation of input prompts in GRPO environments loop by @albertvillanova in https://github.com/huggingface/trl/pull/6364\r\n* Avoid in-place mutation of input prompts in DPPO environments loop by @albertvillanova in https://github.com/huggingface/trl/pull/6366\r\n* Fix DPO/KTO evaluate crash with a dict eval_dataset and precompute_ref_log_probs by @albertvillanova in https://github.com/huggingface/trl/pull/6370\r\n* Test all supported eval_dataset input types in core trainer init tests by @albertvillanova in https://github.com/huggingface/trl/pull/6336\r\n* Test all supported eval_dataset input types in core trainer evaluate() tests by @albertvillanova in https://github.com/huggingface/trl/pull/6371\r\n* Drop examples with no completion left after truncation in DPO and KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6321\r\n* Fix EOS being masked when pad_token_id equals eos_token_id in GKD and GOLD by @roycho96 in https://github.com/huggingface/trl/pull/6357\r\n* Truncate GOLD on-policy prompts before generation, keeping the prompt end by @roycho96 in https://github.com/huggingface/trl/pull/6359\r\n* Fix missing eos_token_id in non-vLLM on-policy GOLD test stubs by @kashif in https://github.com/huggingface/trl/pull/6394\r\n* add iw-opd distillation by @kashif in https://github.com/huggingface/trl/pull/6191\r\n* Force `use_reentrant=True` for PEFT + ZeRO-3 + gradient checkpointing in all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6356\r\n* Fix GeometricMixtureWrapper.generate() crash after transformers' MoE decode optimization change by @albertvillanova in https://github.com/huggingface/trl/pull/6413\r\n* Add eval_dataset init tests for experimental GRPO-family trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6395\r\n* Add eval_dataset init tests for experimental Online-DPO-family trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6396\r\n* Add eval_dataset init tests for experimental SFT and distillation trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6397\r\n* Add eval_dataset init tests for experimental preference trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6398\r\n* Add eval_dataset init tests for experimental BCO, CPO, ORPO and PRM trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6399\r\n* Fix corrupted reasoning blocks in Qwen3-family training chat templates by @qgallouedec in https://github.com/huggingface/trl/pull/6363\r\n* Fix uncaptured cross-architecture VLM warning in GOLD trainer CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/6419\r\n* Fix create_reference_model deprecation warning by @albertvillanova in https://github.com/huggingface/trl/pull/6418\r\n* Add `GMPOTrainer` to the telemetry allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/6410\r\n* Async grpo vllm client by @AmineDiro in https://github.com/huggingface/trl/pull/6269\r\n* Re-record the `dpo` invariant reference after #6321 by @qgallouedec in https://github.com/huggingface/trl/pull/6421\r\n* Add a tip to the Paper Index on reading papers with the hf CLI by @davanstrien in https://github.com/huggingface/trl/pull/6422\r\n* [GRPO] Treat explicit max_tool_calling_iterations=0 as zero, not unlimited  by @Strongich in https://github.com/huggingface/trl/pull/6423\r\n* Align KTO with DPO: Drop `hasattr` guards on `lm_head.bias` in the Liger path by @qgallouedec in https://github.com/huggingface/trl/pull/6425\r\n* Align KTO with DPO: Fix `logits/chosen` and `logits/rejected` metrics by @qgallouedec in https://github.com/huggingface/trl/pull/6424\r\n* Support iterable datasets in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6351\r\n* Align `AsyncGRPOTrainer` clip-ratio comment with `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6376\r\n* Ignore benign IndexPutBackward0 warning from Liger SAPO loss by @albertvillanova in https://github.com/huggingface/trl/pull/6436\r\n* Align `AsyncGRPOConfig.max_completion_length` description with `GRPOConfig` by @qgallouedec in https://github.com/huggingface/trl/pull/6374\r\n* Fix formatting of error message for loss_type by @qgallouedec in https://github.com/huggingface/trl/pull/6373\r\n* Align `AsyncGRPOTrainer.processing_class` docstring with `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6375\r\n* Remove redundant torch.tensor re-wrapping of batch labels in KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6431\r\n* Document max_steps requirement for iterable datasets in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6432\r\n* Fix DPO/KTO double-preprocessing eval_dataset dict passed to evaluate() by @albertvillanova in https://github.com/huggingface/trl/pull/6434\r\n* Bump the actions group across 1 directory with 10 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6442\r\n* Fix GRPO truncated-completion loss normalization by @Functionhx in https://github.com/huggingface/trl/pull/6439\r\n* Environment-owned dataset: make `train_dataset` optional by @qgallouedec in https://github.com/huggingface/trl/pull/6349\r\n* Align `AsyncGRPOTrainer` args handling with stable trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6377\r\n* Fix biased cross-rank aggregation of token-weighted metrics (GRPO, RLOO) by @qgallouedec in https://github.com/huggingface/trl/pull/6380\r\n* Silence experimental environment_factory warning in GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6445\r\n* Ignore benign bitsandbytes _check_is_size FutureWarning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6448\r\n* Silence PEFT ensure_weight_tying warning on newer PEFT in GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6446\r\n* Fix BEMACallback bias_power docstring, which described the opposite of the actual behavior by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6390\r\n* AsyncGRPO fork-independent epoch counting by @AmineDiro in https://github.com/huggingface/trl/pull/6385\r\n* Add support for vLLM 0.25.0 and 0.25.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6406\r\n* Drop vLLM 0.16 support by @qgallouedec in https://github.com/huggingface/trl/pull/6404\r\n* Remove the experimental DPPO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6402\r\n* Reject `auto_find_batch_size` in GRPO and RLOO by @qgallouedec in https://github.com/huggingface/trl/pull/6411\r\n* Fix `precompute_ref_log_probs=True` with DeepSpeed in DPO and KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6403\r\n* [DistillationTrainer refactor] Pin the distillation objective before refactoring by @qgallouedec in https://github.com/huggingface/trl/pull/6450\r\n* [DistillationTrainer refactor] Pin on-policy and off-policy training end to end by @qgallouedec in https://github.com/huggingface/trl/pull/6451\r\n* [DistillationTrainer refactor] Document the num_items_in_batch denominator bug by @qgallouedec in https://github.com/huggingface/trl/pull/6452\r\n* Fix `apply_chat_template` dropping leading tokens from `chosen` when the prompt common prefix shrinks against `rejected` by @sohumt123 in https://github.com/huggingface/trl/pull/6433\r\n* [DistillationTrainer refactor] Move IW-OPD to a dedicated experimental trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6453\r\n* [DistillationTrainer refactor] Extract the teacher-server path into ServerDistillationTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6454\r\n* [DistillationTrainer refactor] Remove the local sparse top-1 loss path from the base trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6455\r\n* [DistillationTrainer refactor] Remove top-k support from the base generalized JSD loss by @qgallouedec in https://github.com/huggingface/trl/pull/6456\r\n* [DistillationTrainer refactor] Validate the teacher by vocab size instead of tokenizer identity by @qgallouedec in https://github.com/huggingface/trl/pull/6457\r\n* [DistillationTrainer refactor] Deprecate `lmbda` on/off-policy mixing by @qgallouedec in https://github.com/huggingface/trl/pull/6458\r\n* [DistillationTrainer refactor] Point the GKD paper reproduction at `GKDTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6459\r\n* [DistillationTrainer refactor] Remove `lmbda` and the off-policy training branch by @qgallouedec in https://github.com/huggingface/trl/pull/6460\r\n* Raise when DPO/KTO cannot precompute correct reference log-probs at eval time by @albertvillanova in https://github.com/huggingface/trl/pull/6443\r\n* Silence PEFT ensure_weight_tying warning in DPO and KTO liger tests by @albertvillanova in https://github.com/huggingface/trl/pull/6463\r\n* Fix misleading name of BEMACallback bias_power=0.0 test by @albertvillanova in https://github.com/huggingface/trl/pull/6464\r\n* Filter reworded torch.jit.script_method warning on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6465\r\n* Filter triton AnnAssign DeprecationWarning on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6467\r\n* [DistillationTrainer refactor] Accept a `prompt` column alongside `messages` by @qgallouedec in https://github.com/huggingface/trl/pull/6461\r\n* [DistillationTrainer refactor] Deprecate `messages`-format datasets by @qgallouedec in https://github.com/huggingface/trl/pull/6474\r\n* [DistillationTrainer refactor] Fix `num_items_in_batch` to count generated completion tokens ⚠️ changes loss values by @qgallouedec in https://github.com/huggingface/trl/pull/6478\r\n* Release: v1.9 by @qgallouedec in https://github.com/huggingface/trl/pull/6495\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.8.0...v1.9.0\r\n","publishedAt":"2026-07-21T20:36:12.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.9.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":29053,"contentTokens":8230,"composition":{"bugs":1,"features":4,"enhancements":6}},{"id":"rel_YYu6ID45bA0Ojq05mBYK8","version":"v1.8.0","type":"feature","title":"v1.8.0","summary":"KTO trainer graduates from experimental to the top-level trl package with the same API as DPO/GRPO/SFT, and the experimental import path still works with a FutureWarning. Environment-owned rewards let agentic RL environments define their own reward via a reserved get_reward() method, and multi-environment support allows a single training run to handle multiple environments with environment-specific tool schemas. GRPO now supports both static and adaptive entropy regularization to encourage exploration and prevent policy collapse.","titleGenerated":"TRL v1.8.0 graduates KTO trainer to stable, adds environment-owned rewards and entropy regularization","titleShort":"KTO now stable; environment rewards and GRPO entropy regularization added","breaking":"none","importance":4,"content":"## Features\r\n\r\n### 🎓 KTO is now a stable trainer\r\n\r\nAfter many cycles of `KTOTrainer` ↔ `DPOTrainer` alignment work, KTO graduates from `trl.experimental.kto` to the top-level `trl` package. Same API as DPO/GRPO/SFT — imports move from experimental, tests move to the main test tree, docs no longer flag it as experimental. The experimental path still works and emits a `FutureWarning` (removal in v2.0.0).\r\n\r\n```python\r\n# Before\r\nfrom trl.experimental.kto import KTOConfig, KTOTrainer\r\n\r\n# Now\r\nfrom trl import KTOConfig, KTOTrainer\r\n```\r\n\r\nPer our telemetry, KTO is the 4th most used trainer in TRL — this graduation was overdue.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6175, https://github.com/huggingface/trl/pull/6287 and https://github.com/huggingface/trl/pull/6345\r\n\r\n### Environment-owned rewards & multi-environment support\r\n\r\nThree interrelated changes make agentic RL training with environments substantially more ergonomic.\r\n\r\n**Environment-owned reward.** If your `environment_factory` env defines a reserved `get_reward()` method (no args → `float`), it's called once per completed rollout and treated as a reward source. `reward_funcs` becomes optional — no more leaking env state back out to trainer-owned reward funcs.\r\n\r\n```python\r\nclass WordleEnv:\r\n    def reset(self, **kwargs):\r\n        self._target = sample(words); self._solved = False\r\n\r\n    def get_reward(self) -> float:       # optional, reserved (not a tool)\r\n        return 1.0 if self._solved else 0.0\r\n\r\n    def guess(self, word: str) -> str:   # exposed as a tool\r\n        self._solved = word == self._target; ...\r\n\r\ntrainer = GRPOTrainer(\r\n    model=model,\r\n    train_dataset=dataset,\r\n    environment_factory=WordleEnv,       # no reward_funcs needed\r\n)\r\n```\r\n\r\n**Multi-environment support.** `environment_factory` now accepts `dict[str, factory]` in addition to a single callable. Each dataset row selects its environment via an `environment` column, and **only that env's tools are exposed in that row's prompt** — so a coding task and a game can train together in one run without leaking each other's tool schemas. Single-callable usage is unchanged.\r\n\r\nSame wiring lands in GRPO, AsyncGRPO, DPPO, and GRPO-with-replay-buffer.\r\n\r\nEnv-owned reward by @qgallouedec in https://github.com/huggingface/trl/pull/6238; multi-env in https://github.com/huggingface/trl/pull/6001 and https://github.com/huggingface/trl/pull/6002\r\n\r\n### Entropy regularization for GRPO\r\n\r\n`GRPOConfig` now supports both static and adaptive entropy regularization ([Skywork-OR1](https://huggingface.co/papers/2505.22617)). The bonus encourages exploration and helps prevent premature policy collapse.\r\n\r\n```python\r\nGRPOConfig(\r\n    entropy_coef=0.01,          # static\r\n    # or:\r\n    use_adaptive_entropy=True,  # adjust to target entropy\r\n    entropy_target=1.5,\r\n    entropy_coef_delta=0.01,\r\n    entropy_coef_min=0.0,\r\n    entropy_coef_max=0.1,\r\n)\r\n```\r\n\r\nAdaptive mode adjusts the coefficient once per optimizer step from window-aggregated entropy (gradient accumulation-aware) and persists `entropy_coef` in the checkpoint for resume. Not compatible with the Liger kernel.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6140\r\n\r\n### `quantization_config` trainer argument (streamlined QLoRA)\r\n\r\nQLoRA no longer requires reaching into `model_init_kwargs` or pre-loading the model manually.\r\n\r\n```python\r\nSFTTrainer(\r\n    model=\"meta-llama/Llama-2-7b-hf\",\r\n    quantization_config=BitsAndBytesConfig(load_in_4bit=True),\r\n    peft_config=LoraConfig(),\r\n    train_dataset=dataset,\r\n)\r\n```\r\n\r\nAdded to `SFTTrainer`, `DPOTrainer`, `GRPOTrainer`, `RLOOTrainer`, `RewardTrainer`, and `KTOTrainer`. Sits next to `peft_config` (the other non-serializable QLoRA ingredient), flows into `from_pretrained`, and raises if also set in `args.model_init_kwargs`. Also drops the redundant `get_kbit_device_map()` line — QLoRA trains identically without it across all tested configurations.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6157 and https://github.com/huggingface/trl/pull/6276\r\n\r\n### MoE aux loss extends to DPO and KTO\r\n\r\nv1.7 added the router load-balancing auxiliary loss to `GRPOTrainer` / `RLOOTrainer` / `AsyncGRPOTrainer`. It's now on `DPOTrainer` and `KTOTrainer` too, so post-training MoE models with preference data keeps experts balanced.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6208 and https://github.com/huggingface/trl/pull/6275\r\n\r\n### Neuron-friendly `chunked_nll` via static-shape token packing\r\n\r\nThe chunked NLL path had data-dependent indexing that broke XLA/Neuron compilation. This PR reworks it to pack valid tokens to the front via a stable `argsort` on the validity mask, iterate over `ceil(n_valid / chunk_size) * chunk_size` whole chunks (a tensor, no Python-int sync), and use `ignore_index=-100` inside each chunk. GPU behavior is unchanged; Neuron now works too — same tiny memory footprint (~1.33 GiB vs the naive 5.94 GiB alternative) with no per-mask recompilation.\r\n\r\nby @michaelbenayoun in https://github.com/huggingface/trl/pull/6314\r\n\r\n### Packing-aware dynamic batching in AsyncGRPO\r\n\r\nAsyncGRPO micro-batching becomes packing-aware and token-bounded on top of the padding-free path from v1.7:\r\n\r\n- **Σ Lᵢ² row balancing** — a greedy longest-first assignment gives every DP row the same Σ Lᵢ² instead of the same token count (attention is O(L²), FFN is O(L), so equal token counts don't equalize wall-clock). **Cross-rank stragglers vanish; +19% MFU at 4B in the benchmark.**\r\n- **Token-budget packing** (opt-in) — cap each row at `token_budget` tokens with a variable sample count, decoupling peak memory from `per_device_train_batch_size`. Useful in memory-bound regimes (no gradient checkpointing, long context, very large models).\r\n\r\nBoth ride HF Trainer's existing gradient accumulation — no training-loop surgery, FSDP/EP collectives stay in lockstep.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6092\r\n\r\n### VLM support in `GOLDTrainer`\r\n\r\n`GOLDTrainer` now supports vision-language models end-to-end.\r\n\r\nby @Strongich in https://github.com/huggingface/trl/pull/5969\r\n\r\n### Support tool calling in KTO\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6259\r\n\r\n### Support PEFT + Liger in DPO and KTO\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6159 and https://github.com/huggingface/trl/pull/6277\r\n\r\n### Per-dataset `fraction` in dataset mixtures\r\n\r\nYou can now weight a `DatasetMixtureConfig` by fraction instead of only by explicit counts.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6199\r\n\r\n### SFT: truncate during dataset preparation\r\n\r\nContinuing the label-refactor from v1.7 (#6037), truncation moves out of the collator and into dataset preparation. `SFTTrainer` and `DPOTrainer` now truncate consistently at the same phase.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6155\r\n\r\n### Simplify tokenization [1-5/N]\r\n\r\nA big refactor pass merging DPO / SFT / Reward / KTO tokenization into one shared implementation:\r\n\r\n* Remove redundant `is_vlm` parameter — https://github.com/huggingface/trl/pull/6298\r\n* Make `_tokenize` a module-level function — https://github.com/huggingface/trl/pull/6301\r\n* Factor `_tokenize` into a single shared function — https://github.com/huggingface/trl/pull/6302\r\n* Bundle `apply_chat_template` kwargs — https://github.com/huggingface/trl/pull/6305\r\n* Pass `chat_template` as `apply_chat_template_kwargs` — https://github.com/huggingface/trl/pull/6315\r\n\r\nAll by @albertvillanova. Related: align collators across DPO / SFT / Reward / KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6178.\r\n\r\n### Other\r\n\r\n* Support `DatasetDict` and `IterableDatasetDict` as `eval_dataset` in trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6322\r\n* Auto-load `processing_class` in CPO/ORPO trainers when omitted by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6087\r\n* Add `trust_remote_code` to `GRPOConfig` by @muupan in https://github.com/huggingface/trl/pull/4186\r\n* Integrate the new response parsing API by @qgallouedec in https://github.com/huggingface/trl/pull/5791\r\n* Raise a clear error when the ChatML collator truncates the whole prompt by @qgallouedec in https://github.com/huggingface/trl/pull/6310\r\n* Raise a clear error when GKD student and teacher vocab sizes differ by @sergiopaniego in https://github.com/huggingface/trl/pull/6252\r\n* Add prompt-learning guard for PEFT with Liger in GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6186\r\n* `[async GRPO]` Per-generation `reset()` observation by @qgallouedec in https://github.com/huggingface/trl/pull/6072\r\n* Default AsyncGRPO `token_budget` to vLLM `max_model_len` by @AmineDiro in https://github.com/huggingface/trl/pull/6218\r\n* Drop vLLM 0.14 support by @qgallouedec in https://github.com/huggingface/trl/pull/6209\r\n* Drop vLLM 0.15 support by @qgallouedec in https://github.com/huggingface/trl/pull/6239\r\n* Log `entropy` and `num_tokens` metrics in KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6257 and https://github.com/huggingface/trl/pull/6256\r\n* Standardize `TrainerCallback` import to the public top-level path by @qgallouedec in https://github.com/huggingface/trl/pull/6249\r\n* Remove redundant `get_kbit_device_map()` by @qgallouedec in https://github.com/huggingface/trl/pull/6158\r\n* Factorize `get_dataset_column_names` from core and experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6272 and https://github.com/huggingface/trl/pull/6273\r\n\r\n## Deprecations and removals\r\n\r\n* **Remove `GFPOTrainer`** by @qgallouedec in https://github.com/huggingface/trl/pull/6309 — no known usage, and its behavior can be reproduced by adjusting `reward_funcs`.\r\n* **Remove `PAPOTrainer`** by @qgallouedec in https://github.com/huggingface/trl/pull/6235\r\n* **Remove post-training-toolkit integration** by @qgallouedec in https://github.com/huggingface/trl/pull/6308\r\n* **Remove `sft_video_llm.py` script** by @qgallouedec in https://github.com/huggingface/trl/pull/6193\r\n\r\n## Fixes\r\n\r\n* **Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware** by @albertvillanova in https://github.com/huggingface/trl/pull/6139 and https://github.com/huggingface/trl/pull/6187\r\n* **Align CPO / ORPO / BCO / Distillation / TPO with DPO: fix ZeRO-3 + PEFT mixed-dtype error** by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6192\r\n* **Fix `chunked_nll` patch hiding VLM kwargs from `generate`** by @Strongich in https://github.com/huggingface/trl/pull/6156\r\n* **Fix vLLM server-mode generation in `OnlineDPOTrainer`** by @qgallouedec in https://github.com/huggingface/trl/pull/6228\r\n* **Fix activation offload storage dedupe reuse** by @winglian in https://github.com/huggingface/trl/pull/6241\r\n* **Update merged probability computation to follow Bayes' rule** by @vman049 in https://github.com/huggingface/trl/pull/5905\r\n* Fix dataset fingerprinting in DPO/SFT tokenization by @qgallouedec in https://github.com/huggingface/trl/pull/6206\r\n* Fix data types for PPO models by @albertvillanova in https://github.com/huggingface/trl/pull/6202\r\n* Fix `precompute_ref_log_probs` guard for iterable datasets nested in a dict by @albertvillanova in https://github.com/huggingface/trl/pull/6307\r\n* Fix teacher quantization kwargs and guard eval callback in GKD example by @sergiopaniego in https://github.com/huggingface/trl/pull/6251\r\n* Raise on `quantization_config` + already-instantiated model in `DPOTrainer` / `KTOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6312\r\n* Fix incorrect examples in distillation docs by @sergiopaniego in https://github.com/huggingface/trl/pull/6334\r\n* Fix `KTOTrainer._prepare_dataset` type hint by @qgallouedec in https://github.com/huggingface/trl/pull/6207\r\n* Fix PEFT `ensure_weight_tying` warning in liger + PEFT GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6188\r\n* Fix spurious liger token_accuracy CI warnings in SFT tests by @albertvillanova in https://github.com/huggingface/trl/pull/6189\r\n* Fix `add_response_schema` tests for the new `parse_response` prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236\r\n* Fix tiny DeepSeek-V3 configs for MLA: `num_key_value_heads = num_attention_heads` by @albertvillanova in https://github.com/huggingface/trl/pull/6268\r\n* Raise `ValueError` instead of `NotImplementedError` for Liger DPO with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6225\r\n\r\n## Documentation and Examples\r\n\r\n* Docs: fix Jobs requirement — positive credit balance, not a Pro/Team/Enterprise plan by @davanstrien in https://github.com/huggingface/trl/pull/6306\r\n* Docs: clarify dtype defaults between transformers v5 and TRL by @casinca in https://github.com/huggingface/trl/pull/5457\r\n* Document that `max_steps` is required for iterable train datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6333\r\n* Align KTO doc with DPO and fix Logged-metrics wording by @qgallouedec in https://github.com/huggingface/trl/pull/6258\r\n* Align `epsilon` help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014 (v1.7 window)\r\n* Fix KTO `max_length` docstring (truncation direction) by @qgallouedec in https://github.com/huggingface/trl/pull/6254\r\n* Fix KTO `compute_metrics` type hint (`EvalLoopOutput` → `EvalPrediction`) by @qgallouedec in https://github.com/huggingface/trl/pull/6253\r\n\r\n## CI\r\n\r\n* Liger teardown in experimental tests by @cmpatino in https://github.com/huggingface/trl/pull/6203\r\n* Sort conditional imports by @qgallouedec in https://github.com/huggingface/trl/pull/6180\r\n* Align format of versions in CI workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6223\r\n* Update CI `huggingface/doc-builder` by @albertvillanova in https://github.com/huggingface/trl/pull/6304\r\n* Fix Upload PR Documentation: bump `doc-builder` pin by @mishig25 in https://github.com/huggingface/trl/pull/6317\r\n* Fix / set explicit owner in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6200 and https://github.com/huggingface/trl/pull/6201\r\n* Fix CI `test_chatml_collator_truncates_keeping_completion_end` by @albertvillanova in https://github.com/huggingface/trl/pull/6318\r\n* Test DPO / KTO init fails with `compute_metrics` and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6231 and https://github.com/huggingface/trl/pull/6232\r\n* Add `TestKTOTrainerSlow` with `test_train_vlm_gemma_3n` by @albertvillanova in https://github.com/huggingface/trl/pull/6162\r\n* Test `DataCollatorForChatML` raises when completion fills window by @albertvillanova in https://github.com/huggingface/trl/pull/6319\r\n* Improve consistency of Liger Kernel initialization in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6242\r\n* Align Liger loss naming across core and experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6243, https://github.com/huggingface/trl/pull/6245 and https://github.com/huggingface/trl/pull/6244\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6214, https://github.com/huggingface/trl/pull/6285\r\n\r\n## New Contributors\r\n* @vman049 made their first contribution in https://github.com/huggingface/trl/pull/5905\r\n* @Strongich made their first contribution in https://github.com/huggingface/trl/pull/5969\r\n* @michaelbenayoun made their first contribution in https://github.com/huggingface/trl/pull/6314\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6185\r\n* Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware by @albertvillanova in https://github.com/huggingface/trl/pull/6139\r\n* Remove RUNNING_NAME from KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6179\r\n* Align CPO/ORPO/BCO/Distillation/TPO with DPO: fix ZeRO-3 + PEFT mixed-dtype error by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6192\r\n* Align KTO collator keys with the DPO convention by @qgallouedec in https://github.com/huggingface/trl/pull/6182\r\n* Move `_get_kl_completion_ids` into `_get_kl_dataset` by @qgallouedec in https://github.com/huggingface/trl/pull/6181\r\n* Align experimental KTOTrainer docstring and signature with DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6183\r\n* Add packing-aware dynamic batching to AsyncGRPO by @AmineDiro in https://github.com/huggingface/trl/pull/6092\r\n* Liger Teardown in Experimental Tests by @cmpatino in https://github.com/huggingface/trl/pull/6203\r\n* Sort conditional imports by @qgallouedec in https://github.com/huggingface/trl/pull/6180\r\n* Align TPO with DPO: Add evaluate() override by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6195\r\n* Auto-load `processing_class` in CPO/ORPO trainers when omitted by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6087\r\n* Fix dataset fingerprinting in DPO/SFT tokenization by @qgallouedec in https://github.com/huggingface/trl/pull/6206\r\n* Pass GPU device_ids to barrier fix in GRPO + vLLM colocate + PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6187\r\n* Fix PEFT ensure_weight_tying warning in liger + PEFT GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6188\r\n* Fix spurious liger token_accuracy CI warnings in SFT tests by @albertvillanova in https://github.com/huggingface/trl/pull/6189\r\n* Fix deprecation in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6200\r\n* Set explicit owner in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6201\r\n* Fix data types for PPO models by @albertvillanova in https://github.com/huggingface/trl/pull/6202\r\n* Fix: update merged probability computation to follow Bayes' rule by @vman049 in https://github.com/huggingface/trl/pull/5905\r\n* Bump the actions group across 1 directory with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6214\r\n* Remove sft_video_llm.py script by @qgallouedec in https://github.com/huggingface/trl/pull/6193\r\n* Drop vLLM 0.14 support by @qgallouedec in https://github.com/huggingface/trl/pull/6209\r\n* Integrate the new response parsing API by @qgallouedec in https://github.com/huggingface/trl/pull/5791\r\n* [docs] Clarify dtype defaults between trf v5 and TRL by @casinca in https://github.com/huggingface/trl/pull/5457\r\n* Align format of versions in CI workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6223\r\n* Align KTO with DPO: Align validation of liger and compute_metrics by @albertvillanova in https://github.com/huggingface/trl/pull/6224\r\n* Raise ValueError instead of NotImplementedError for Liger DPO with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6225\r\n* Support multiple environments [1/2]: Pool and build environment tool dicts at batch time by @qgallouedec in https://github.com/huggingface/trl/pull/6001\r\n* Test DPO init fails with compute_metrics and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6231\r\n* Align KTO with DPO: Test init fails with compute_metrics and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6232\r\n* [async GRPO] Per-generation reset() observation by @qgallouedec in https://github.com/huggingface/trl/pull/6072\r\n* Fix `add_response_schema` tests for the new `parse_response` prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236\r\n* Align KTO with DPO: Add TestKTOTrainerSlow with test_train_vlm_gemma_3n by @albertvillanova in https://github.com/huggingface/trl/pull/6162\r\n* Support PEFT with Liger in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6159\r\n* Drop `policy_` prefix from KTO loss-internal logps/logits by @qgallouedec in https://github.com/huggingface/trl/pull/6204\r\n* Fix KTOTrainer._prepare_dataset type hint by @qgallouedec in https://github.com/huggingface/trl/pull/6207\r\n* Improve consistency of Liger Kernel initialization in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6242\r\n* Align Liger loss naming across core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6243\r\n* Align Liger loss naming across experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6245\r\n* Align KTO with DPO: Align _tokenize by @albertvillanova in https://github.com/huggingface/trl/pull/6212\r\n* Align KTO with DPO: Align error for Liger with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6226\r\n* Remove redundant `get_kbit_device_map()` by @qgallouedec in https://github.com/huggingface/trl/pull/6158\r\n* Fix vLLM server-mode generation in `OnlineDPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6228\r\n* Use trl's guarded `is_liger_kernel_available` in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6247\r\n* Default AsyncGRPO token_budget to vLLM max_model_len by @AmineDiro in https://github.com/huggingface/trl/pull/6218\r\n* Fix tiny DeepSeek-V3 configs for MLA: num_key_value_heads =num_attention_heads by @albertvillanova in https://github.com/huggingface/trl/pull/6268\r\n* Align KTO with DPO: Align Liger loss naming by @albertvillanova in https://github.com/huggingface/trl/pull/6244\r\n* Raise a clear error when GKD student and teacher vocab sizes differ by @sergiopaniego in https://github.com/huggingface/trl/pull/6252\r\n* SFT: Truncate during dataset preparation, not collation by @qgallouedec in https://github.com/huggingface/trl/pull/6155\r\n* Add prompt-learning guard for PEFT with Liger in GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6186\r\n* Add MoE load-balancing auxiliary loss to `DPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6208\r\n* Add `quantization_config` trainer argument (streamline QLoRA) by @qgallouedec in https://github.com/huggingface/trl/pull/6157\r\n* Align KTO with DPO: Support tool calling by @qgallouedec in https://github.com/huggingface/trl/pull/6259\r\n* Align KTO with DPO: Align error for Liger with precompute_ref_logps by @albertvillanova in https://github.com/huggingface/trl/pull/6270\r\n* Remove unused get_dataset_column_names from DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6271\r\n* Factorize get_dataset_column_names from core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6272\r\n* Factorize get_dataset_column_names from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6273\r\n* Align KTO with DPO: Log `entropy` metric by @qgallouedec in https://github.com/huggingface/trl/pull/6257\r\n* Align KTO doc with DPO and fix Logged metrics wording by @qgallouedec in https://github.com/huggingface/trl/pull/6258\r\n* Align KTO with DPO: Validate Liger kernel configuration before trainer initialization by @albertvillanova in https://github.com/huggingface/trl/pull/6227\r\n* Align KTO with DPO: Remove stray misplaced comment in KTO `_get_train_sampler` by @qgallouedec in https://github.com/huggingface/trl/pull/6255\r\n* Align KTO with DPO: Fix KTO `max_length` docstring (truncation direction) by @qgallouedec in https://github.com/huggingface/trl/pull/6254\r\n* Align KTO with DPO: Fix KTO `compute_metrics` type hint (`EvalLoopOutput` → `EvalPrediction`) by @qgallouedec in https://github.com/huggingface/trl/pull/6253\r\n* Align KTO with DPO: Align models import by @qgallouedec in https://github.com/huggingface/trl/pull/6248\r\n* Align KTO with DPO: Align `F` import with the rest of the repo by @qgallouedec in https://github.com/huggingface/trl/pull/6246\r\n* Fix teacher quantization kwargs and guard eval callback in GKD example by @sergiopaniego in https://github.com/huggingface/trl/pull/6251\r\n* Align experimental TPO _tokenize with core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6213\r\n* Align KTO with DPO: Support PEFT with Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6277\r\n* Align KTO with DPO: clarify dtype default in `model` docstring by @qgallouedec in https://github.com/huggingface/trl/pull/6278\r\n* Align KTO with DPO: Log `num_tokens` metric by @qgallouedec in https://github.com/huggingface/trl/pull/6256\r\n* Standardize `TrainerCallback` import to the public top-level path by @qgallouedec in https://github.com/huggingface/trl/pull/6249\r\n* Fix activation offload storage dedupe reuse by @winglian in https://github.com/huggingface/trl/pull/6241\r\n* Align KTO with DPO: MoE load-balancing auxiliary loss by @qgallouedec in https://github.com/huggingface/trl/pull/6275\r\n* Align data collators across DPO / SFT / Reward / KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6178\r\n* Drop vLLM 0.15 support by @qgallouedec in https://github.com/huggingface/trl/pull/6239\r\n* Add entropy regularization to GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6140\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6285\r\n* Remove KTO shims from stable by @albertvillanova in https://github.com/huggingface/trl/pull/6287\r\n* Promote KTO to stable API by @albertvillanova in https://github.com/huggingface/trl/pull/6175\r\n* [GOLD] VLM support for GOLDTrainer by @Strongich in https://github.com/huggingface/trl/pull/5969\r\n* Docs: fix Jobs requirement — positive credit balance, not a Pro/Team/Enterprise plan by @davanstrien in https://github.com/huggingface/trl/pull/6306\r\n* Simplify tokenization [1/N]: Remove redundant is_vlm parameter by @albertvillanova in https://github.com/huggingface/trl/pull/6298\r\n* Update CI huggingface/doc-builder by @albertvillanova in https://github.com/huggingface/trl/pull/6304\r\n* Simplify tokenization [2/N]: Make _tokenize a module-level function by @albertvillanova in https://github.com/huggingface/trl/pull/6301\r\n* Raise clear error when ChatML collator truncates the whole prompt by @qgallouedec in https://github.com/huggingface/trl/pull/6310\r\n* Remove the PAPO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6235\r\n* Add trust_remote_code to GRPOConfig by @muupan in https://github.com/huggingface/trl/pull/4186\r\n* Simplify tokenization [3/N]: Factor _tokenize into a single shared function by @albertvillanova in https://github.com/huggingface/trl/pull/6302\r\n* Simplify tokenization [4/N]: Bundle apply_chat_template kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/6305\r\n* Fix precompute_ref_log_probs guard for iterable datasets nested in a dict by @albertvillanova in https://github.com/huggingface/trl/pull/6307\r\n* Fix Upload PR Documentation: bump doc-builder pin by @mishig25 in https://github.com/huggingface/trl/pull/6317\r\n* Removes GFPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6309\r\n* Fix CI test_chatml_collator_truncates_keeping_completion_end: no prompt tokens left after truncation by @albertvillanova in https://github.com/huggingface/trl/pull/6318\r\n* Add per-dataset `fraction` to dataset mixtures by @qgallouedec in https://github.com/huggingface/trl/pull/6199\r\n* Test DataCollatorForChatML raises when completion fills window by @albertvillanova in https://github.com/huggingface/trl/pull/6319\r\n* Simplify tokenization [5/N]: Pass chat_template as apply_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/6315\r\n* Support multiple environments [2/2]: Per-example environment selection by @qgallouedec in https://github.com/huggingface/trl/pull/6002\r\n* Remove post-training-toolkit integration by @qgallouedec in https://github.com/huggingface/trl/pull/6308\r\n* Environment-owned reward by @qgallouedec in https://github.com/huggingface/trl/pull/6238\r\n* Align KTO with DPO: `quantization_config` trainer argument by @qgallouedec in https://github.com/huggingface/trl/pull/6276\r\n* Support DatasetDict and IterableDatasetDict as eval_dataset in trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6322\r\n* Document that max_steps is required for iterable train datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6333\r\n* Raise on `quantization_config` + already-instantiated model in `DPOTrainer`/ `KTOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6312\r\n* Fix chunked_nll patch hiding VLM kwargs from generate by @Strongich in https://github.com/huggingface/trl/pull/6156\r\n* Neuron-friendly `chunked_nll` via static-shape token packing by @michaelbenayoun in https://github.com/huggingface/trl/pull/6314\r\n* Fix incorrect examples in distillation docs by @sergiopaniego in https://github.com/huggingface/trl/pull/6334\r\n* Docs: treat KTO as a stable trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6345\r\n* Release: v1.8 by @qgallouedec in https://github.com/huggingface/trl/pull/6346\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.7.0...v1.8.0\r\n","publishedAt":"2026-07-09T18:41:28.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.8.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":29065,"contentTokens":8443,"composition":{"bugs":0,"features":11,"enhancements":7}},{"id":"rel_m-2DKXJ2DPtKCbQ7eiKwY","version":"v1.7.1","type":"feature","title":"v1.7.1","summary":"Fixed a hang in GRPO + vLLM colocate + PEFT on non-NVLink hardware and corrected dataset fingerprinting in DPO/SFT tokenization. Also integrated the new response parsing API, added a prompt-learning guard for PEFT with Liger in GRPO, and fixed activation offload storage deduplication.","titleGenerated":"TRL v1.7.1 fixes GRPO + vLLM hang and dataset fingerprinting","titleShort":"GRPO + vLLM hang fixed on non-NVLink; dataset fingerprinting corrected","breaking":"none","importance":2,"content":"## What's Changed\r\n\r\n* Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware by @albertvillanova in https://github.com/huggingface/trl/pull/6139\r\n* Fix dataset fingerprinting in DPO/SFT tokenization by @qgallouedec in https://github.com/huggingface/trl/pull/6206\r\n* Pass GPU device_ids to barrier fix in GRPO + vLLM colocate + PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6187\r\n* Integrate the new response parsing API by @qgallouedec in https://github.com/huggingface/trl/pull/5791\r\n* Fix `add_response_schema` tests for the new `parse_response` prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236\r\n* Add prompt-learning guard for PEFT with Liger in GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6186\r\n* Fix activation offload storage dedupe reuse by @winglian in https://github.com/huggingface/trl/pull/6241\r\n\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.7.0...v1.7.1","publishedAt":"2026-07-04T04:05:54.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.7.1","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":975,"contentTokens":295,"composition":{"bugs":5,"features":1,"enhancements":1}}],"pagination":{"nextCursor":"2026-07-04T04:05:54.000Z|2026-07-04T05:03:01.742Z|rel_m-2DKXJ2DPtKCbQ7eiKwY","limit":20}}