{"id":"prod_pCEM29no4janpSW6GK3TC","name":"Fine-tuning","slug":"fine-tuning","orgId":"org_GDdYeYynEgCEBNBwy-m6s","url":null,"description":"Libraries for efficient model fine-tuning and alignment","category":"ai","kind":"sdk","avatarUrl":null,"createdAt":"2026-04-10T16:06:59.610Z","embeddedAt":"2026-04-15T16:19:40.805Z","deletedAt":null,"sources":[{"id":"src_p8uVHjEHY1lSK3poHvlWI","slug":"peft","name":"PEFT","type":"github","url":"https://github.com/huggingface/peft","metadata":"{\"evaluatedMethod\":\"github\",\"evaluatedAt\":\"2026-04-07T17:19:21.796Z\",\"changelogDetectedAt\":\"2026-04-07T17:29:08.155Z\",\"wellKnownSweptAt\":\"2026-07-08T06:00:52.047Z\",\"sourceActor\":{\"nextAlarmAt\":\"2026-07-12T02:16:43.090Z\",\"lastAlarmAt\":\"2026-07-11T02:16:43.937Z\",\"managed\":true}}","kind":"sdk"},{"id":"src_g_3xbu2oRX52tM8mwXJzq","slug":"trl","name":"TRL","type":"github","url":"https://github.com/huggingface/trl","metadata":"{\"evaluatedMethod\":\"github\",\"evaluatedAt\":\"2026-04-07T17:19:15.726Z\",\"changelogDetectedAt\":\"2026-04-07T17:28:23.856Z\",\"wellKnownSweptAt\":\"2026-07-08T06:00:52.047Z\",\"sourceActor\":{\"nextAlarmAt\":\"2026-07-12T01:05:13.831Z\",\"lastAlarmAt\":\"2026-07-11T21:05:15.251Z\",\"managed\":true}}","kind":"sdk"}],"tags":["fine-tuning","python","rlhf"],"aliases":[],"notice":null,"releases":[{"id":"rel_YYu6ID45bA0Ojq05mBYK8","version":"v1.8.0","type":"feature","title":"v1.8.0","summary":"KTO trainer graduates from experimental to the top-level trl package with the same API as DPO/GRPO/SFT, and the experimental import path still works with a FutureWarning. Environment-owned rewards let agentic RL environments define their own reward via a reserved get_reward() method, and multi-environment support allows a single training run to handle multiple environments with environment-specific tool schemas. GRPO now supports both static and adaptive entropy regularization to encourage exploration and prevent policy collapse.","titleGenerated":"TRL v1.8.0 graduates KTO trainer to stable, adds environment-owned rewards and entropy regularization","titleShort":"KTO now stable; environment rewards and GRPO entropy regularization added","breaking":"none","importance":4,"content":"## Features\r\n\r\n### 🎓 KTO is now a stable trainer\r\n\r\nAfter many cycles of `KTOTrainer` ↔ `DPOTrainer` alignment work, KTO graduates from `trl.experimental.kto` to the top-level `trl` package. Same API as DPO/GRPO/SFT — imports move from experimental, tests move to the main test tree, docs no longer flag it as experimental. The experimental path still works and emits a `FutureWarning` (removal in v2.0.0).\r\n\r\n```python\r\n# Before\r\nfrom trl.experimental.kto import KTOConfig, KTOTrainer\r\n\r\n# Now\r\nfrom trl import KTOConfig, KTOTrainer\r\n```\r\n\r\nPer our telemetry, KTO is the 4th most used trainer in TRL — this graduation was overdue.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6175, https://github.com/huggingface/trl/pull/6287 and https://github.com/huggingface/trl/pull/6345\r\n\r\n### Environment-owned rewards & multi-environment support\r\n\r\nThree interrelated changes make agentic RL training with environments substantially more ergonomic.\r\n\r\n**Environment-owned reward.** If your `environment_factory` env defines a reserved `get_reward()` method (no args → `float`), it's called once per completed rollout and treated as a reward source. `reward_funcs` becomes optional — no more leaking env state back out to trainer-owned reward funcs.\r\n\r\n```python\r\nclass WordleEnv:\r\n    def reset(self, **kwargs):\r\n        self._target = sample(words); self._solved = False\r\n\r\n    def get_reward(self) -> float:       # optional, reserved (not a tool)\r\n        return 1.0 if self._solved else 0.0\r\n\r\n    def guess(self, word: str) -> str:   # exposed as a tool\r\n        self._solved = word == self._target; ...\r\n\r\ntrainer = GRPOTrainer(\r\n    model=model,\r\n    train_dataset=dataset,\r\n    environment_factory=WordleEnv,       # no reward_funcs needed\r\n)\r\n```\r\n\r\n**Multi-environment support.** `environment_factory` now accepts `dict[str, factory]` in addition to a single callable. Each dataset row selects its environment via an `environment` column, and **only that env's tools are exposed in that row's prompt** — so a coding task and a game can train together in one run without leaking each other's tool schemas. Single-callable usage is unchanged.\r\n\r\nSame wiring lands in GRPO, AsyncGRPO, DPPO, and GRPO-with-replay-buffer.\r\n\r\nEnv-owned reward by @qgallouedec in https://github.com/huggingface/trl/pull/6238; multi-env in https://github.com/huggingface/trl/pull/6001 and https://github.com/huggingface/trl/pull/6002\r\n\r\n### Entropy regularization for GRPO\r\n\r\n`GRPOConfig` now supports both static and adaptive entropy regularization ([Skywork-OR1](https://huggingface.co/papers/2505.22617)). The bonus encourages exploration and helps prevent premature policy collapse.\r\n\r\n```python\r\nGRPOConfig(\r\n    entropy_coef=0.01,          # static\r\n    # or:\r\n    use_adaptive_entropy=True,  # adjust to target entropy\r\n    entropy_target=1.5,\r\n    entropy_coef_delta=0.01,\r\n    entropy_coef_min=0.0,\r\n    entropy_coef_max=0.1,\r\n)\r\n```\r\n\r\nAdaptive mode adjusts the coefficient once per optimizer step from window-aggregated entropy (gradient accumulation-aware) and persists `entropy_coef` in the checkpoint for resume. Not compatible with the Liger kernel.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6140\r\n\r\n### `quantization_config` trainer argument (streamlined QLoRA)\r\n\r\nQLoRA no longer requires reaching into `model_init_kwargs` or pre-loading the model manually.\r\n\r\n```python\r\nSFTTrainer(\r\n    model=\"meta-llama/Llama-2-7b-hf\",\r\n    quantization_config=BitsAndBytesConfig(load_in_4bit=True),\r\n    peft_config=LoraConfig(),\r\n    train_dataset=dataset,\r\n)\r\n```\r\n\r\nAdded to `SFTTrainer`, `DPOTrainer`, `GRPOTrainer`, `RLOOTrainer`, `RewardTrainer`, and `KTOTrainer`. Sits next to `peft_config` (the other non-serializable QLoRA ingredient), flows into `from_pretrained`, and raises if also set in `args.model_init_kwargs`. Also drops the redundant `get_kbit_device_map()` line — QLoRA trains identically without it across all tested configurations.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6157 and https://github.com/huggingface/trl/pull/6276\r\n\r\n### MoE aux loss extends to DPO and KTO\r\n\r\nv1.7 added the router load-balancing auxiliary loss to `GRPOTrainer` / `RLOOTrainer` / `AsyncGRPOTrainer`. It's now on `DPOTrainer` and `KTOTrainer` too, so post-training MoE models with preference data keeps experts balanced.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6208 and https://github.com/huggingface/trl/pull/6275\r\n\r\n### Neuron-friendly `chunked_nll` via static-shape token packing\r\n\r\nThe chunked NLL path had data-dependent indexing that broke XLA/Neuron compilation. This PR reworks it to pack valid tokens to the front via a stable `argsort` on the validity mask, iterate over `ceil(n_valid / chunk_size) * chunk_size` whole chunks (a tensor, no Python-int sync), and use `ignore_index=-100` inside each chunk. GPU behavior is unchanged; Neuron now works too — same tiny memory footprint (~1.33 GiB vs the naive 5.94 GiB alternative) with no per-mask recompilation.\r\n\r\nby @michaelbenayoun in https://github.com/huggingface/trl/pull/6314\r\n\r\n### Packing-aware dynamic batching in AsyncGRPO\r\n\r\nAsyncGRPO micro-batching becomes packing-aware and token-bounded on top of the padding-free path from v1.7:\r\n\r\n- **Σ Lᵢ² row balancing** — a greedy longest-first assignment gives every DP row the same Σ Lᵢ² instead of the same token count (attention is O(L²), FFN is O(L), so equal token counts don't equalize wall-clock). **Cross-rank stragglers vanish; +19% MFU at 4B in the benchmark.**\r\n- **Token-budget packing** (opt-in) — cap each row at `token_budget` tokens with a variable sample count, decoupling peak memory from `per_device_train_batch_size`. Useful in memory-bound regimes (no gradient checkpointing, long context, very large models).\r\n\r\nBoth ride HF Trainer's existing gradient accumulation — no training-loop surgery, FSDP/EP collectives stay in lockstep.\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6092\r\n\r\n### VLM support in `GOLDTrainer`\r\n\r\n`GOLDTrainer` now supports vision-language models end-to-end.\r\n\r\nby @Strongich in https://github.com/huggingface/trl/pull/5969\r\n\r\n### Support tool calling in KTO\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6259\r\n\r\n### Support PEFT + Liger in DPO and KTO\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6159 and https://github.com/huggingface/trl/pull/6277\r\n\r\n### Per-dataset `fraction` in dataset mixtures\r\n\r\nYou can now weight a `DatasetMixtureConfig` by fraction instead of only by explicit counts.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6199\r\n\r\n### SFT: truncate during dataset preparation\r\n\r\nContinuing the label-refactor from v1.7 (#6037), truncation moves out of the collator and into dataset preparation. `SFTTrainer` and `DPOTrainer` now truncate consistently at the same phase.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/6155\r\n\r\n### Simplify tokenization [1-5/N]\r\n\r\nA big refactor pass merging DPO / SFT / Reward / KTO tokenization into one shared implementation:\r\n\r\n* Remove redundant `is_vlm` parameter — https://github.com/huggingface/trl/pull/6298\r\n* Make `_tokenize` a module-level function — https://github.com/huggingface/trl/pull/6301\r\n* Factor `_tokenize` into a single shared function — https://github.com/huggingface/trl/pull/6302\r\n* Bundle `apply_chat_template` kwargs — https://github.com/huggingface/trl/pull/6305\r\n* Pass `chat_template` as `apply_chat_template_kwargs` — https://github.com/huggingface/trl/pull/6315\r\n\r\nAll by @albertvillanova. Related: align collators across DPO / SFT / Reward / KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6178.\r\n\r\n### Other\r\n\r\n* Support `DatasetDict` and `IterableDatasetDict` as `eval_dataset` in trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6322\r\n* Auto-load `processing_class` in CPO/ORPO trainers when omitted by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6087\r\n* Add `trust_remote_code` to `GRPOConfig` by @muupan in https://github.com/huggingface/trl/pull/4186\r\n* Integrate the new response parsing API by @qgallouedec in https://github.com/huggingface/trl/pull/5791\r\n* Raise a clear error when the ChatML collator truncates the whole prompt by @qgallouedec in https://github.com/huggingface/trl/pull/6310\r\n* Raise a clear error when GKD student and teacher vocab sizes differ by @sergiopaniego in https://github.com/huggingface/trl/pull/6252\r\n* Add prompt-learning guard for PEFT with Liger in GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6186\r\n* `[async GRPO]` Per-generation `reset()` observation by @qgallouedec in https://github.com/huggingface/trl/pull/6072\r\n* Default AsyncGRPO `token_budget` to vLLM `max_model_len` by @AmineDiro in https://github.com/huggingface/trl/pull/6218\r\n* Drop vLLM 0.14 support by @qgallouedec in https://github.com/huggingface/trl/pull/6209\r\n* Drop vLLM 0.15 support by @qgallouedec in https://github.com/huggingface/trl/pull/6239\r\n* Log `entropy` and `num_tokens` metrics in KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6257 and https://github.com/huggingface/trl/pull/6256\r\n* Standardize `TrainerCallback` import to the public top-level path by @qgallouedec in https://github.com/huggingface/trl/pull/6249\r\n* Remove redundant `get_kbit_device_map()` by @qgallouedec in https://github.com/huggingface/trl/pull/6158\r\n* Factorize `get_dataset_column_names` from core and experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6272 and https://github.com/huggingface/trl/pull/6273\r\n\r\n## Deprecations and removals\r\n\r\n* **Remove `GFPOTrainer`** by @qgallouedec in https://github.com/huggingface/trl/pull/6309 — no known usage, and its behavior can be reproduced by adjusting `reward_funcs`.\r\n* **Remove `PAPOTrainer`** by @qgallouedec in https://github.com/huggingface/trl/pull/6235\r\n* **Remove post-training-toolkit integration** by @qgallouedec in https://github.com/huggingface/trl/pull/6308\r\n* **Remove `sft_video_llm.py` script** by @qgallouedec in https://github.com/huggingface/trl/pull/6193\r\n\r\n## Fixes\r\n\r\n* **Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware** by @albertvillanova in https://github.com/huggingface/trl/pull/6139 and https://github.com/huggingface/trl/pull/6187\r\n* **Align CPO / ORPO / BCO / Distillation / TPO with DPO: fix ZeRO-3 + PEFT mixed-dtype error** by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6192\r\n* **Fix `chunked_nll` patch hiding VLM kwargs from `generate`** by @Strongich in https://github.com/huggingface/trl/pull/6156\r\n* **Fix vLLM server-mode generation in `OnlineDPOTrainer`** by @qgallouedec in https://github.com/huggingface/trl/pull/6228\r\n* **Fix activation offload storage dedupe reuse** by @winglian in https://github.com/huggingface/trl/pull/6241\r\n* **Update merged probability computation to follow Bayes' rule** by @vman049 in https://github.com/huggingface/trl/pull/5905\r\n* Fix dataset fingerprinting in DPO/SFT tokenization by @qgallouedec in https://github.com/huggingface/trl/pull/6206\r\n* Fix data types for PPO models by @albertvillanova in https://github.com/huggingface/trl/pull/6202\r\n* Fix `precompute_ref_log_probs` guard for iterable datasets nested in a dict by @albertvillanova in https://github.com/huggingface/trl/pull/6307\r\n* Fix teacher quantization kwargs and guard eval callback in GKD example by @sergiopaniego in https://github.com/huggingface/trl/pull/6251\r\n* Raise on `quantization_config` + already-instantiated model in `DPOTrainer` / `KTOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6312\r\n* Fix incorrect examples in distillation docs by @sergiopaniego in https://github.com/huggingface/trl/pull/6334\r\n* Fix `KTOTrainer._prepare_dataset` type hint by @qgallouedec in https://github.com/huggingface/trl/pull/6207\r\n* Fix PEFT `ensure_weight_tying` warning in liger + PEFT GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6188\r\n* Fix spurious liger token_accuracy CI warnings in SFT tests by @albertvillanova in https://github.com/huggingface/trl/pull/6189\r\n* Fix `add_response_schema` tests for the new `parse_response` prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236\r\n* Fix tiny DeepSeek-V3 configs for MLA: `num_key_value_heads = num_attention_heads` by @albertvillanova in https://github.com/huggingface/trl/pull/6268\r\n* Raise `ValueError` instead of `NotImplementedError` for Liger DPO with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6225\r\n\r\n## Documentation and Examples\r\n\r\n* Docs: fix Jobs requirement — positive credit balance, not a Pro/Team/Enterprise plan by @davanstrien in https://github.com/huggingface/trl/pull/6306\r\n* Docs: clarify dtype defaults between transformers v5 and TRL by @casinca in https://github.com/huggingface/trl/pull/5457\r\n* Document that `max_steps` is required for iterable train datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6333\r\n* Align KTO doc with DPO and fix Logged-metrics wording by @qgallouedec in https://github.com/huggingface/trl/pull/6258\r\n* Align `epsilon` help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014 (v1.7 window)\r\n* Fix KTO `max_length` docstring (truncation direction) by @qgallouedec in https://github.com/huggingface/trl/pull/6254\r\n* Fix KTO `compute_metrics` type hint (`EvalLoopOutput` → `EvalPrediction`) by @qgallouedec in https://github.com/huggingface/trl/pull/6253\r\n\r\n## CI\r\n\r\n* Liger teardown in experimental tests by @cmpatino in https://github.com/huggingface/trl/pull/6203\r\n* Sort conditional imports by @qgallouedec in https://github.com/huggingface/trl/pull/6180\r\n* Align format of versions in CI workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6223\r\n* Update CI `huggingface/doc-builder` by @albertvillanova in https://github.com/huggingface/trl/pull/6304\r\n* Fix Upload PR Documentation: bump `doc-builder` pin by @mishig25 in https://github.com/huggingface/trl/pull/6317\r\n* Fix / set explicit owner in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6200 and https://github.com/huggingface/trl/pull/6201\r\n* Fix CI `test_chatml_collator_truncates_keeping_completion_end` by @albertvillanova in https://github.com/huggingface/trl/pull/6318\r\n* Test DPO / KTO init fails with `compute_metrics` and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6231 and https://github.com/huggingface/trl/pull/6232\r\n* Add `TestKTOTrainerSlow` with `test_train_vlm_gemma_3n` by @albertvillanova in https://github.com/huggingface/trl/pull/6162\r\n* Test `DataCollatorForChatML` raises when completion fills window by @albertvillanova in https://github.com/huggingface/trl/pull/6319\r\n* Improve consistency of Liger Kernel initialization in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6242\r\n* Align Liger loss naming across core and experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6243, https://github.com/huggingface/trl/pull/6245 and https://github.com/huggingface/trl/pull/6244\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6214, https://github.com/huggingface/trl/pull/6285\r\n\r\n## New Contributors\r\n* @vman049 made their first contribution in https://github.com/huggingface/trl/pull/5905\r\n* @Strongich made their first contribution in https://github.com/huggingface/trl/pull/5969\r\n* @michaelbenayoun made their first contribution in https://github.com/huggingface/trl/pull/6314\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6185\r\n* Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware by @albertvillanova in https://github.com/huggingface/trl/pull/6139\r\n* Remove RUNNING_NAME from KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6179\r\n* Align CPO/ORPO/BCO/Distillation/TPO with DPO: fix ZeRO-3 + PEFT mixed-dtype error by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6192\r\n* Align KTO collator keys with the DPO convention by @qgallouedec in https://github.com/huggingface/trl/pull/6182\r\n* Move `_get_kl_completion_ids` into `_get_kl_dataset` by @qgallouedec in https://github.com/huggingface/trl/pull/6181\r\n* Align experimental KTOTrainer docstring and signature with DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6183\r\n* Add packing-aware dynamic batching to AsyncGRPO by @AmineDiro in https://github.com/huggingface/trl/pull/6092\r\n* Liger Teardown in Experimental Tests by @cmpatino in https://github.com/huggingface/trl/pull/6203\r\n* Sort conditional imports by @qgallouedec in https://github.com/huggingface/trl/pull/6180\r\n* Align TPO with DPO: Add evaluate() override by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6195\r\n* Auto-load `processing_class` in CPO/ORPO trainers when omitted by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6087\r\n* Fix dataset fingerprinting in DPO/SFT tokenization by @qgallouedec in https://github.com/huggingface/trl/pull/6206\r\n* Pass GPU device_ids to barrier fix in GRPO + vLLM colocate + PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6187\r\n* Fix PEFT ensure_weight_tying warning in liger + PEFT GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6188\r\n* Fix spurious liger token_accuracy CI warnings in SFT tests by @albertvillanova in https://github.com/huggingface/trl/pull/6189\r\n* Fix deprecation in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6200\r\n* Set explicit owner in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6201\r\n* Fix data types for PPO models by @albertvillanova in https://github.com/huggingface/trl/pull/6202\r\n* Fix: update merged probability computation to follow Bayes' rule by @vman049 in https://github.com/huggingface/trl/pull/5905\r\n* Bump the actions group across 1 directory with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6214\r\n* Remove sft_video_llm.py script by @qgallouedec in https://github.com/huggingface/trl/pull/6193\r\n* Drop vLLM 0.14 support by @qgallouedec in https://github.com/huggingface/trl/pull/6209\r\n* Integrate the new response parsing API by @qgallouedec in https://github.com/huggingface/trl/pull/5791\r\n* [docs] Clarify dtype defaults between trf v5 and TRL by @casinca in https://github.com/huggingface/trl/pull/5457\r\n* Align format of versions in CI workflows by @albertvillanova in https://github.com/huggingface/trl/pull/6223\r\n* Align KTO with DPO: Align validation of liger and compute_metrics by @albertvillanova in https://github.com/huggingface/trl/pull/6224\r\n* Raise ValueError instead of NotImplementedError for Liger DPO with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6225\r\n* Support multiple environments [1/2]: Pool and build environment tool dicts at batch time by @qgallouedec in https://github.com/huggingface/trl/pull/6001\r\n* Test DPO init fails with compute_metrics and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6231\r\n* Align KTO with DPO: Test init fails with compute_metrics and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6232\r\n* [async GRPO] Per-generation reset() observation by @qgallouedec in https://github.com/huggingface/trl/pull/6072\r\n* Fix `add_response_schema` tests for the new `parse_response` prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236\r\n* Align KTO with DPO: Add TestKTOTrainerSlow with test_train_vlm_gemma_3n by @albertvillanova in https://github.com/huggingface/trl/pull/6162\r\n* Support PEFT with Liger in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6159\r\n* Drop `policy_` prefix from KTO loss-internal logps/logits by @qgallouedec in https://github.com/huggingface/trl/pull/6204\r\n* Fix KTOTrainer._prepare_dataset type hint by @qgallouedec in https://github.com/huggingface/trl/pull/6207\r\n* Improve consistency of Liger Kernel initialization in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6242\r\n* Align Liger loss naming across core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6243\r\n* Align Liger loss naming across experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6245\r\n* Align KTO with DPO: Align _tokenize by @albertvillanova in https://github.com/huggingface/trl/pull/6212\r\n* Align KTO with DPO: Align error for Liger with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6226\r\n* Remove redundant `get_kbit_device_map()` by @qgallouedec in https://github.com/huggingface/trl/pull/6158\r\n* Fix vLLM server-mode generation in `OnlineDPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6228\r\n* Use trl's guarded `is_liger_kernel_available` in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6247\r\n* Default AsyncGRPO token_budget to vLLM max_model_len by @AmineDiro in https://github.com/huggingface/trl/pull/6218\r\n* Fix tiny DeepSeek-V3 configs for MLA: num_key_value_heads =num_attention_heads by @albertvillanova in https://github.com/huggingface/trl/pull/6268\r\n* Align KTO with DPO: Align Liger loss naming by @albertvillanova in https://github.com/huggingface/trl/pull/6244\r\n* Raise a clear error when GKD student and teacher vocab sizes differ by @sergiopaniego in https://github.com/huggingface/trl/pull/6252\r\n* SFT: Truncate during dataset preparation, not collation by @qgallouedec in https://github.com/huggingface/trl/pull/6155\r\n* Add prompt-learning guard for PEFT with Liger in GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6186\r\n* Add MoE load-balancing auxiliary loss to `DPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6208\r\n* Add `quantization_config` trainer argument (streamline QLoRA) by @qgallouedec in https://github.com/huggingface/trl/pull/6157\r\n* Align KTO with DPO: Support tool calling by @qgallouedec in https://github.com/huggingface/trl/pull/6259\r\n* Align KTO with DPO: Align error for Liger with precompute_ref_logps by @albertvillanova in https://github.com/huggingface/trl/pull/6270\r\n* Remove unused get_dataset_column_names from DPO by @albertvillanova in https://github.com/huggingface/trl/pull/6271\r\n* Factorize get_dataset_column_names from core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6272\r\n* Factorize get_dataset_column_names from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6273\r\n* Align KTO with DPO: Log `entropy` metric by @qgallouedec in https://github.com/huggingface/trl/pull/6257\r\n* Align KTO doc with DPO and fix Logged metrics wording by @qgallouedec in https://github.com/huggingface/trl/pull/6258\r\n* Align KTO with DPO: Validate Liger kernel configuration before trainer initialization by @albertvillanova in https://github.com/huggingface/trl/pull/6227\r\n* Align KTO with DPO: Remove stray misplaced comment in KTO `_get_train_sampler` by @qgallouedec in https://github.com/huggingface/trl/pull/6255\r\n* Align KTO with DPO: Fix KTO `max_length` docstring (truncation direction) by @qgallouedec in https://github.com/huggingface/trl/pull/6254\r\n* Align KTO with DPO: Fix KTO `compute_metrics` type hint (`EvalLoopOutput` → `EvalPrediction`) by @qgallouedec in https://github.com/huggingface/trl/pull/6253\r\n* Align KTO with DPO: Align models import by @qgallouedec in https://github.com/huggingface/trl/pull/6248\r\n* Align KTO with DPO: Align `F` import with the rest of the repo by @qgallouedec in https://github.com/huggingface/trl/pull/6246\r\n* Fix teacher quantization kwargs and guard eval callback in GKD example by @sergiopaniego in https://github.com/huggingface/trl/pull/6251\r\n* Align experimental TPO _tokenize with core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6213\r\n* Align KTO with DPO: Support PEFT with Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6277\r\n* Align KTO with DPO: clarify dtype default in `model` docstring by @qgallouedec in https://github.com/huggingface/trl/pull/6278\r\n* Align KTO with DPO: Log `num_tokens` metric by @qgallouedec in https://github.com/huggingface/trl/pull/6256\r\n* Standardize `TrainerCallback` import to the public top-level path by @qgallouedec in https://github.com/huggingface/trl/pull/6249\r\n* Fix activation offload storage dedupe reuse by @winglian in https://github.com/huggingface/trl/pull/6241\r\n* Align KTO with DPO: MoE load-balancing auxiliary loss by @qgallouedec in https://github.com/huggingface/trl/pull/6275\r\n* Align data collators across DPO / SFT / Reward / KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6178\r\n* Drop vLLM 0.15 support by @qgallouedec in https://github.com/huggingface/trl/pull/6239\r\n* Add entropy regularization to GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6140\r\n* Bump the actions group with 3 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6285\r\n* Remove KTO shims from stable by @albertvillanova in https://github.com/huggingface/trl/pull/6287\r\n* Promote KTO to stable API by @albertvillanova in https://github.com/huggingface/trl/pull/6175\r\n* [GOLD] VLM support for GOLDTrainer by @Strongich in https://github.com/huggingface/trl/pull/5969\r\n* Docs: fix Jobs requirement — positive credit balance, not a Pro/Team/Enterprise plan by @davanstrien in https://github.com/huggingface/trl/pull/6306\r\n* Simplify tokenization [1/N]: Remove redundant is_vlm parameter by @albertvillanova in https://github.com/huggingface/trl/pull/6298\r\n* Update CI huggingface/doc-builder by @albertvillanova in https://github.com/huggingface/trl/pull/6304\r\n* Simplify tokenization [2/N]: Make _tokenize a module-level function by @albertvillanova in https://github.com/huggingface/trl/pull/6301\r\n* Raise clear error when ChatML collator truncates the whole prompt by @qgallouedec in https://github.com/huggingface/trl/pull/6310\r\n* Remove the PAPO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6235\r\n* Add trust_remote_code to GRPOConfig by @muupan in https://github.com/huggingface/trl/pull/4186\r\n* Simplify tokenization [3/N]: Factor _tokenize into a single shared function by @albertvillanova in https://github.com/huggingface/trl/pull/6302\r\n* Simplify tokenization [4/N]: Bundle apply_chat_template kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/6305\r\n* Fix precompute_ref_log_probs guard for iterable datasets nested in a dict by @albertvillanova in https://github.com/huggingface/trl/pull/6307\r\n* Fix Upload PR Documentation: bump doc-builder pin by @mishig25 in https://github.com/huggingface/trl/pull/6317\r\n* Removes GFPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6309\r\n* Fix CI test_chatml_collator_truncates_keeping_completion_end: no prompt tokens left after truncation by @albertvillanova in https://github.com/huggingface/trl/pull/6318\r\n* Add per-dataset `fraction` to dataset mixtures by @qgallouedec in https://github.com/huggingface/trl/pull/6199\r\n* Test DataCollatorForChatML raises when completion fills window by @albertvillanova in https://github.com/huggingface/trl/pull/6319\r\n* Simplify tokenization [5/N]: Pass chat_template as apply_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/6315\r\n* Support multiple environments [2/2]: Per-example environment selection by @qgallouedec in https://github.com/huggingface/trl/pull/6002\r\n* Remove post-training-toolkit integration by @qgallouedec in https://github.com/huggingface/trl/pull/6308\r\n* Environment-owned reward by @qgallouedec in https://github.com/huggingface/trl/pull/6238\r\n* Align KTO with DPO: `quantization_config` trainer argument by @qgallouedec in https://github.com/huggingface/trl/pull/6276\r\n* Support DatasetDict and IterableDatasetDict as eval_dataset in trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6322\r\n* Document that max_steps is required for iterable train datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6333\r\n* Raise on `quantization_config` + already-instantiated model in `DPOTrainer`/ `KTOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/6312\r\n* Fix chunked_nll patch hiding VLM kwargs from generate by @Strongich in https://github.com/huggingface/trl/pull/6156\r\n* Neuron-friendly `chunked_nll` via static-shape token packing by @michaelbenayoun in https://github.com/huggingface/trl/pull/6314\r\n* Fix incorrect examples in distillation docs by @sergiopaniego in https://github.com/huggingface/trl/pull/6334\r\n* Docs: treat KTO as a stable trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6345\r\n* Release: v1.8 by @qgallouedec in https://github.com/huggingface/trl/pull/6346\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.7.0...v1.8.0\r\n","publishedAt":"2026-07-09T18:41:28.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.8.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":29065,"contentTokens":8443,"composition":{"bugs":0,"features":11,"enhancements":7}},{"id":"rel_m-2DKXJ2DPtKCbQ7eiKwY","version":"v1.7.1","type":"feature","title":"v1.7.1","summary":"Fixed a hang in GRPO + vLLM colocate + PEFT on non-NVLink hardware and corrected dataset fingerprinting in DPO/SFT tokenization. Also integrated the new response parsing API, added a prompt-learning guard for PEFT with Liger in GRPO, and fixed activation offload storage deduplication.","titleGenerated":"TRL v1.7.1 fixes GRPO + vLLM hang and dataset fingerprinting","titleShort":"GRPO + vLLM hang fixed on non-NVLink; dataset fingerprinting corrected","breaking":"none","importance":2,"content":"## What's Changed\r\n\r\n* Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware by @albertvillanova in https://github.com/huggingface/trl/pull/6139\r\n* Fix dataset fingerprinting in DPO/SFT tokenization by @qgallouedec in https://github.com/huggingface/trl/pull/6206\r\n* Pass GPU device_ids to barrier fix in GRPO + vLLM colocate + PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6187\r\n* Integrate the new response parsing API by @qgallouedec in https://github.com/huggingface/trl/pull/5791\r\n* Fix `add_response_schema` tests for the new `parse_response` prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236\r\n* Add prompt-learning guard for PEFT with Liger in GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/6186\r\n* Fix activation offload storage dedupe reuse by @winglian in https://github.com/huggingface/trl/pull/6241\r\n\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.7.0...v1.7.1","publishedAt":"2026-07-04T04:05:54.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.7.1","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":975,"contentTokens":295,"composition":{"bugs":5,"features":1,"enhancements":1}},{"id":"rel_b-tcAO3CgHjd-FDiEqan7","version":"v1.7.0","type":"feature","title":"v1.7.0","summary":"The default SFT loss_type is now \"chunked_nll\", delivering ~30% less peak VRAM on average with neutral or slightly faster wall-clock time. Also introduces experimental GMPO trainer, transformers continuous batching, AsyncGRPO weight sync with vLLM 0.22+, and paddding-free AsyncGRPO.","titleGenerated":"TRL v1.7.0 makes chunked cross-entropy loss default for SFT and adds GMPO trainer","titleShort":"SFT default loss flips to chunked_nll; GMPO trainer arrives","breaking":"major","importance":null,"content":"## Features\r\n\r\n### SFT default `loss_type` is now `\"chunked_nll\"`\r\n\r\nThe flip announced in v1.6 has landed. Setting `loss_type` is optional, and the default now resolves to `\"chunked_nll\"` — giving every `SFTTrainer` run **~30% less peak VRAM on average (up to ~50% on large-vocab models)** with wall-clock time neutral or slightly faster. No action needed.\r\n\r\nThe auto-resolve falls back to `\"nll\"` when `use_liger_kernel=True` (the two paths are incompatible). If you want the old behavior — e.g. for custom heads — pin it explicitly:\r\n\r\n```python\r\nSFTConfig(loss_type=\"nll\")\r\n```\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5846\r\n\r\n### MoE auxiliary loss in GRPO / RLOO / AsyncGRPO\r\n\r\nPost-training MoE models now correctly include the router load-balancing auxiliary loss, matching the model's own reference forward and `SFTTrainer`. Enable via `model_init_kwargs`:\r\n\r\n```python\r\nGRPOConfig(\r\n    ...,\r\n    model_init_kwargs={\"output_router_logits\": True, \"router_aux_loss_coef\": 0.001},\r\n)\r\n```\r\n\r\nPlumbed through `_get_per_token_logps_and_entropies` (now returns a 3-tuple including `aux_loss`), folded into the policy loss with grad-accum scaling matched per trainer, and logged as `aux_loss`. AsyncGRPO recomputes it via `load_balancing_loss_func` in the chunked LM-head path (same as SFT's chunked path).\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/6083, plus `router_aux_loss_coef` config wiring by @qgallouedec in https://github.com/huggingface/trl/pull/6085\r\n\r\n### New experimental GMPO trainer\r\n\r\n[Geometric-Mean Policy Optimization](https://huggingface.co/papers/2507.20673) lands as an experimental trainer. Replaces GRPO's per-token arithmetic mean of importance ratios with a **sequence-level geometric mean** (mean of clipped log-ratios, then `exp`); clipping is one-sided by advantage sign and applied in log space. Default `epsilon=0.4` per the paper.\r\n\r\n```python\r\nfrom trl.experimental.gmpo import GMPOConfig, GMPOTrainer\r\n\r\ntrainer = GMPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=GMPOConfig(epsilon=0.4),\r\n    reward_funcs=accuracy_reward,\r\n    train_dataset=dataset,\r\n)\r\n```\r\n\r\nby @raghulchandramouli in https://github.com/huggingface/trl/pull/6078\r\n\r\n### Transformers continuous batching in GRPO / RLOO\r\n\r\n`use_transformers_paged` was deprecated in v1.4; it's now replaced with proper [transformers continuous batching](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationMixin.generate_batch). The old branch silently bypassed importance-sampling correction (`logprobs = None`); the new path captures logprobs from `output.logprobs` and exposes a `ContinuousBatchingConfig` for KV-cache tuning.\r\n\r\n```python\r\nGRPOConfig(\r\n    ...,\r\n    use_transformers_continuous_batching=True,\r\n    transformers_continuous_batching_config={\r\n        \"use_cuda_graph\": False,\r\n        \"max_memory_percent\": 0.4,  # leave headroom for training\r\n    },\r\n)\r\n```\r\n\r\nBenchmark (Llama-3.2-1B-Instruct, A100 80GB, GSM8K): **1.25× faster at N=64 generations** with **-16 GB peak VRAM** vs default `generate()`. Use when N ≥ 32 with variable completion lengths.\r\n\r\n`use_transformers_paged=True` still works and forwards to the new flag with a `FutureWarning`. Requires `transformers>=5.8.0`.\r\n\r\nby @sergiopaniego in https://github.com/huggingface/trl/pull/5765\r\n\r\n### AsyncGRPO: native weight sync with vLLM ≥ 0.22.0\r\n\r\n`WeightTransferClient` now drives vLLM's native [4-phase RL weight-transfer API](https://docs.vllm.ai/en/latest/training/weight_transfer/) instead of the older 2-call flow: `pause(mode=\"keep\")` → `start_weight_update` → threaded `update_weights` + NCCL broadcast → `finish_weight_update` → `resume`. Validated end-to-end on H100 across single-node, FSDP2×4 + TP=4, and 2-node FSDP2×4 + DP=2×TP=4 (weight-sync time ≈ 0.18-0.8 s).\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/5892\r\n\r\n### Padding-free training in AsyncGRPO\r\n\r\nAsyncGRPO now supports the same padding-free path SFT already had. Flattens the batch and uses `position_ids`-based document boundaries instead of right-padding to the longest sequence — meaningful speedup and memory savings on heterogeneous-length workloads.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5854\r\n\r\n### Experimental Harbor integration\r\n\r\nA new `trl.experimental.harbor` adapter plugs [Harbor](https://www.harborframework.com) agentic task suites into `GRPOTrainer` via `environment_factory`. Same pattern as the OpenReward integration — one spec wires all three trainer slots:\r\n\r\n```python\r\nfrom trl import GRPOConfig, GRPOTrainer\r\nfrom trl.experimental.harbor import HarborSpec\r\n\r\nspec = HarborSpec(\"AdithyaSK/data_agent_rl_environment_train\", agent=\"bash\", num_tasks=64)\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=GRPOConfig(num_generations=8, max_steps=50, max_tool_calling_iterations=25),\r\n    train_dataset=spec.train_dataset,\r\n    environment_factory=spec.environment_factory,\r\n    reward_funcs=spec.reward_funcs,\r\n)\r\n```\r\n\r\nBuilt-in `bash` harness, plus `jupyter` and `terminal_notes` example harnesses. Gated by the new `trl[harbor]` extra.\r\n\r\nby @adithya-s-k in https://github.com/huggingface/trl/pull/6018\r\n\r\n### `trust_remote_code` in trainer configs\r\n\r\nA single `trust_remote_code: bool = False` field on the trainer configs now covers the **whole load surface** — model, processor / tokenizer, reference model, reward model, reward tokenizer, teacher — instead of forcing users to thread it through several independent kwarg dicts.\r\n\r\n```python\r\nSFTConfig(trust_remote_code=True)\r\n```\r\n\r\n`ModelConfig.trust_remote_code` is removed to avoid duplicate `--trust_remote_code` when combining dataclasses; CLI behavior is unchanged.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5802\r\n\r\n### KTO ↔ DPO alignment: tests, evaluate, sync_ref_model\r\n\r\nThe last alignment cycle before graduation: KTO now has parity with DPO on `pad_to_multiple_of`, `sync_ref_model`, `evaluate()`, method order/signature, metric placement (all moved into `_compute_loss`), and a real text + VLM (including multi-image) test suite.\r\n\r\nPRs all by @albertvillanova: #6029, #6030, #6033, #6034, #6035, #6080, #6093, #6148, #6149, #6150, #6152, #6160, #6163.\r\n\r\n### LFM2-VL multimodal inputs in GRPO / RLOO\r\n\r\nGRPO and RLOO now support [LFM2-VL](https://huggingface.co/LiquidAI/LFM2-VL-1.6B) multimodal inputs end-to-end.\r\n\r\nby @zwischenraum in https://github.com/huggingface/trl/pull/6114\r\n\r\n### New built-in reward helpers\r\n\r\n* `get_repetition_penalty_reward` by @qgallouedec in https://github.com/huggingface/trl/pull/6058\r\n* `get_cosine_scaled_reward` by @qgallouedec in https://github.com/huggingface/trl/pull/6066\r\n\r\n### SFT refactor: build labels during dataset preparation\r\n\r\nLabel construction moves out of the collator and into dataset preparation, so \"what's trainable\" is defined in exactly one place. A single batched `map` produces a `labels` column where each token keeps its ID when every applicable mask is 1, else `-100`. Plain LM stays storage-neutral; pre-tokenized datasets with mask columns now go through the same path. Step 1 toward fixing #3927.\r\n\r\nby @0xadvait in https://github.com/huggingface/trl/pull/6037\r\n\r\n### Idefics3 chat template\r\n\r\n`{% generation %}`-marker training template for Idefics3, enabling `assistant_only_loss=True`.\r\n\r\nby @aazizyan in https://github.com/huggingface/trl/pull/5871\r\n\r\n### vLLM version sweep\r\n\r\n* Support vLLM 0.19.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6107\r\n* Support vLLM 0.20.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6108\r\n* Support vLLM 0.22.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6119\r\n* Support vLLM 0.23.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6153\r\n* Drop vLLM 0.12 by @qgallouedec in https://github.com/huggingface/trl/pull/6109\r\n* Drop vLLM 0.13 by @qgallouedec in https://github.com/huggingface/trl/pull/6154\r\n\r\n### Other\r\n\r\n* Normalize JSD distillation loss by `num_items_in_batch` for gradient accumulation by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6006\r\n* `[AsyncGRPO]` Rollout worker: set aiohttp limit to `max(100, max_inflight_tasks)` by @ggcr in https://github.com/huggingface/trl/pull/5861\r\n* Align SDPO with GRPO/RLOO: drop NaN values when averaging logged metrics by @anshulkulhari7 in https://github.com/huggingface/trl/pull/6055\r\n* Make `evaluate()` accept the same dataset types as the trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6116\r\n* Keep extra columns in `unpair_preference_dataset` by @albertvillanova in https://github.com/huggingface/trl/pull/6161\r\n* Warn when sequence-level importance sampling is combined with a token-summed loss type by @discobot in https://github.com/huggingface/trl/pull/6042\r\n* `fix(profiling)`: log `ProfilingContext` metrics to Trackio backend by @Anai-Guo in https://github.com/huggingface/trl/pull/5979\r\n* Move experimental example scripts out of the packaged tree by @sergiopaniego in https://github.com/huggingface/trl/pull/6141\r\n* Remove redundant `.contiguous()` calls by @qgallouedec in https://github.com/huggingface/trl/pull/6045 and https://github.com/huggingface/trl/pull/6046\r\n* Harmonize logger imports by @qgallouedec in https://github.com/huggingface/trl/pull/6142\r\n\r\n## Fixes\r\n\r\n* **Share frozen layers with reference model instead of duplicating in memory** — `create_reference_model` with `num_shared_layers` was double-allocating the \"shared\" frozen layers because the loop never assigned `_ref_param` back. Now it does, so shared layers are held once. By @behroozazarkhalili in https://github.com/huggingface/trl/pull/6053\r\n* **Fix `chunked_nll` mixed Tensor/DTensor error under FSDP2 + PEFT** by @albertvillanova in https://github.com/huggingface/trl/pull/6065\r\n* **Fix per-chunk `lm_head.weight` all-gathers under FSDP2 + `chunked_nll`** by @albertvillanova in https://github.com/huggingface/trl/pull/6077\r\n* **Fix ZeRO-3 + PEFT mixed-dtype error** for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6091 and KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6093\r\n* Fix ref adapter creation when the LoRA config uses `target_parameters` by @discobot in https://github.com/huggingface/trl/pull/6043\r\n* Raise when `use_liger_kernel` is combined with a PEFT adapter on `lm_head` by @akshansh47 in https://github.com/huggingface/trl/pull/5977\r\n* `[fix]` GLM-4-MoE template: turn-terminating token to the turn itself by @qgallouedec in https://github.com/huggingface/trl/pull/6044\r\n* Fix `unpair_preference_dataset` dropping extra columns by @albertvillanova in https://github.com/huggingface/trl/pull/6059\r\n* `fix(gold)`: preserve vllm prompt special tokens by @he-yufeng in https://github.com/huggingface/trl/pull/6063\r\n* `fix(sft)`: reject transformed datasets during preparation by @he-yufeng in https://github.com/huggingface/trl/pull/6054\r\n* Fix broken light-mode banner image in README by @strickvl in https://github.com/huggingface/trl/pull/6112\r\n* Fix BrowserGym OpenEnv example dependency and task wiring by @burtenshaw in https://github.com/huggingface/trl/pull/6117\r\n* Fix signature of `_unpair_row` by @albertvillanova in https://github.com/huggingface/trl/pull/6062\r\n* Remove silently-ignored W&B/Hub fields from GOLD and Distillation configs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6023\r\n\r\n## Documentation and Examples\r\n\r\n* docs: sync experimental trainer docstrings with their `__init__` signatures by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6011\r\n* docs: fix stale default values in config docstrings by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6015\r\n* docs: fix function docstrings that drifted from their signatures by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6022\r\n* Fix broken doc links in `paper_index` (experimental page was split) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6070\r\n* Fix broken import in GOLD trainer docs (GOLD is experimental) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6073\r\n* Fix broken section anchors in docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6071\r\n* Fix broken example link in BCO trainer docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6084\r\n* `docs(rloo)`: add clarifying comment on KL penalty formula divergence from GRPO by @abderahmane-ai in https://github.com/huggingface/trl/pull/6096\r\n* Fix async GRPO docs to require `vllm>=0.22.0` by @sergiopaniego in https://github.com/huggingface/trl/pull/6101\r\n* Fix broken internal doc link to GKD Trainer in MiniLLM docs by @ShamSaleem in https://github.com/huggingface/trl/pull/6131\r\n* Align `epsilon` help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014\r\n* Fix style of optional parameters in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6081\r\n* Align format of code examples in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6147\r\n* Add license to Harbor examples by @qgallouedec in https://github.com/huggingface/trl/pull/6104\r\n\r\n## CI\r\n\r\n* Extract VLM tests into dedicated classes for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6033\r\n* Add raises test with vision dataset and text model for DPO and SFT by @albertvillanova in https://github.com/huggingface/trl/pull/6026\r\n* Extract `_push_param_to_vllm` helper in `VLLMGeneration` by @albertvillanova in https://github.com/huggingface/trl/pull/6004\r\n* Use relative imports in `async_grpo` to match the rest of `trl/experimental` by @qgallouedec in https://github.com/huggingface/trl/pull/6012\r\n* Align AsyncGRPO with GRPO: `num_completions_to_print`, `epsilon_high` fallback, `logging_steps` docstring, loss variable names, clip-ratio metrics by @qgallouedec in https://github.com/huggingface/trl/pull/6020, https://github.com/huggingface/trl/pull/6019, https://github.com/huggingface/trl/pull/6016, https://github.com/huggingface/trl/pull/6013 and https://github.com/huggingface/trl/pull/6021\r\n* Add vision requirement marker to Idefics3 test parameter by @qgallouedec in https://github.com/huggingface/trl/pull/6106\r\n* Update distributed SFT test after default `chunked_nll` loss by @albertvillanova in https://github.com/huggingface/trl/pull/6074\r\n* `test`: bound memory in `test_gkd_trainer_with_liger` to avoid OOM on shared runners by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6103\r\n* Fix `sft_fa2` invariant test by @qgallouedec in https://github.com/huggingface/trl/pull/6069\r\n* Pin SHA instead of version tag for CI `actions/checkout` / `astral-sh/setup-uv` / `actions/setup-python` / `pre-commit/action` by @albertvillanova in https://github.com/huggingface/trl/pull/6097, https://github.com/huggingface/trl/pull/6098, https://github.com/huggingface/trl/pull/6099 and https://github.com/huggingface/trl/pull/6100\r\n* Replace `parse_version` with `Version` by @albertvillanova in https://github.com/huggingface/trl/pull/6164\r\n* Hotfix CI: temporarily pin `deepspeed < 0.19.2` by @albertvillanova in https://github.com/huggingface/trl/pull/6090\r\n* `chore`: update `tests_transformers_branch.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6051\r\n* `chore`: update `clear_cache.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6047\r\n* `chore`: update `docker-build.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6048\r\n* Delete CI `pr_style_bot` workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6082\r\n* Remove issue labeller by @qgallouedec in https://github.com/huggingface/trl/pull/6052\r\n* Bump the actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6041\r\n* Change `_get_train_sampler` comment to mention `num_iterations > 1` by @anidoesdev in https://github.com/huggingface/trl/pull/6125\r\n\r\n## New Contributors\r\n* @Anai-Guo made their first contribution in https://github.com/huggingface/trl/pull/5979\r\n* @ggcr made their first contribution in https://github.com/huggingface/trl/pull/5861\r\n* @he-yufeng made their first contribution in https://github.com/huggingface/trl/pull/6063\r\n* @discobot made their first contribution in https://github.com/huggingface/trl/pull/6043\r\n* @anshulkulhari7 made their first contribution in https://github.com/huggingface/trl/pull/6055\r\n* @abderahmane-ai made their first contribution in https://github.com/huggingface/trl/pull/6096\r\n* @akshansh47 made their first contribution in https://github.com/huggingface/trl/pull/5977\r\n* @strickvl made their first contribution in https://github.com/huggingface/trl/pull/6112\r\n* @ShamSaleem made their first contribution in https://github.com/huggingface/trl/pull/6131\r\n* @0xadvait made their first contribution in https://github.com/huggingface/trl/pull/6037\r\n* @anidoesdev made their first contribution in https://github.com/huggingface/trl/pull/6125\r\n* @zwischenraum made their first contribution in https://github.com/huggingface/trl/pull/6114\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/6010\r\n* docs: sync experimental trainer docstrings with their __init__ signatures by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6011\r\n* docs: fix stale default values in config docstrings by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6015\r\n* fix(profiling): log ProfilingContext metrics to Trackio backend by @Anai-Guo in https://github.com/huggingface/trl/pull/5979\r\n* docs: fix function docstrings that drifted from their signatures by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6022\r\n* Add raises test with vision dataset and text model for DPO and SFT by @albertvillanova in https://github.com/huggingface/trl/pull/6026\r\n* Align KTO with DPO: Unwrap VLM batch dimension for text-only data in _tokenize by @albertvillanova in https://github.com/huggingface/trl/pull/6029\r\n* Extract VLM tests into dedicated classes for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6033\r\n* Extract _push_param_to_vllm helper in VLLMGeneration by @albertvillanova in https://github.com/huggingface/trl/pull/6004\r\n* Align KTO with DPO: Add tests for text data collator by @albertvillanova in https://github.com/huggingface/trl/pull/6034\r\n* Align KTO with DPO: Support pad_to_multiple_of by @albertvillanova in https://github.com/huggingface/trl/pull/6035\r\n* Align KTO with DPO: Add VLM tests by @albertvillanova in https://github.com/huggingface/trl/pull/6030\r\n* Normalize JSD distillation loss by num_items_in_batch for gradient accumulation by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6006\r\n* Bump the actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/6041\r\n* [fix] GLM-4-MoE template: turn-terminating token to the turn itself by @qgallouedec in https://github.com/huggingface/trl/pull/6044\r\n* fix: share frozen layers with reference model instead of duplicating in memory by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6053\r\n* Default SFT loss to chunked_nll by @qgallouedec in https://github.com/huggingface/trl/pull/5846\r\n* Remove redundant `.contiguous()` calls by @qgallouedec in https://github.com/huggingface/trl/pull/6045\r\n* Remove silently-ignored W&B/Hub fields from GOLD and Distillation configs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6023\r\n* [AsyncGRPO] Rollout worker: set aiohttp limit to max(100, max_inflight_tasks) by @ggcr in https://github.com/huggingface/trl/pull/5861\r\n* async grpo native weight sync with vllm>=0.22.0 by @AmineDiro in https://github.com/huggingface/trl/pull/5892\r\n* Fix `unpair_preference_dataset` dropping extra columns by @albertvillanova in https://github.com/huggingface/trl/pull/6059\r\n* fix(gold): preserve vllm prompt special tokens by @he-yufeng in https://github.com/huggingface/trl/pull/6063\r\n* Fix broken doc links in paper_index (experimental page was split) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6070\r\n* chore: update tests_transformers_branch.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6051\r\n* Fix chunked_nll mixed Tensor/DTensor error under FSDP2 + PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6065\r\n* Fix signature of _unpair_row by @albertvillanova in https://github.com/huggingface/trl/pull/6062\r\n* fix(sft): reject transformed datasets during preparation by @he-yufeng in https://github.com/huggingface/trl/pull/6054\r\n* Fix broken import in GOLD trainer docs (GOLD is experimental) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6073\r\n* Fix broken section anchors in docs (slug mismatches, renamed/removed sections) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6071\r\n* Update distributed SFT test after default chunked_nll loss by @albertvillanova in https://github.com/huggingface/trl/pull/6074\r\n* Add GMPO (Geometric-Mean Policy Optimization) experimental trainer by @raghulchandramouli in https://github.com/huggingface/trl/pull/6078\r\n* Fix style of optional parameters in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6081\r\n* Align KTO with DPO: Support config pad_to_multiple_of by @albertvillanova in https://github.com/huggingface/trl/pull/6080\r\n* chore: update clear_cache.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6047\r\n* chore: update docker-build.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6048\r\n* Padding-free training in AsyncGRPO by @qgallouedec in https://github.com/huggingface/trl/pull/5854\r\n* Delete CI pr_style_bot workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6082\r\n* Fix broken example link in BCO trainer docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6084\r\n* Hotfix CI: Temporarily pin deepspeed < 0.19.2 by @albertvillanova in https://github.com/huggingface/trl/pull/6090\r\n* Add MoE auxiliary loss to GRPO, RLOO, and AsyncGRPO trainers by @AmineDiro in https://github.com/huggingface/trl/pull/6083\r\n* Add experimental Harbor integration for GRPO environment training by @adithya-s-k in https://github.com/huggingface/trl/pull/6018\r\n* Fix `sft_fa2` invariant test by @qgallouedec in https://github.com/huggingface/trl/pull/6069\r\n* Use relative imports in async_grpo to match the rest of trl/experimental` by @qgallouedec in https://github.com/huggingface/trl/pull/6012\r\n* Fix ref adapter creation when the LoRA config uses `target_parameters` by @discobot in https://github.com/huggingface/trl/pull/6043\r\n* Remove issue labeller by @qgallouedec in https://github.com/huggingface/trl/pull/6052\r\n* Add Idefics3 original and training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5871\r\n* Align SDPO with GRPO/RLOO: drop NaN values when averaging logged metrics by @anshulkulhari7 in https://github.com/huggingface/trl/pull/6055\r\n* docs(rloo): add clarifying comment on KL penalty formula divergence from GRPO by @abderahmane-ai in https://github.com/huggingface/trl/pull/6096\r\n* feat(grpo): replace deprecated `use_transformers_paged` with transformers continuous batching by @sergiopaniego in https://github.com/huggingface/trl/pull/5765\r\n* test: bound memory in test_gkd_trainer_with_liger to avoid OOM on shared runners by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6103\r\n* Add vision requirement marker to Idefics3 test parameter by @qgallouedec in https://github.com/huggingface/trl/pull/6106\r\n* Pin SHA instead of version tag for CI actions/checkout by @albertvillanova in https://github.com/huggingface/trl/pull/6097\r\n* Pin SHA instead of version tag for CI astral-sh/setup-uv by @albertvillanova in https://github.com/huggingface/trl/pull/6098\r\n* Pin SHA instead of version tag for CI actions/setup-python by @albertvillanova in https://github.com/huggingface/trl/pull/6099\r\n* Pin SHA instead of version tag for CI pre-commit/action by @albertvillanova in https://github.com/huggingface/trl/pull/6100\r\n* Fix ZeRO-3 + PEFT mixed-dtype error for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6091\r\n* Align KTO with DPO: Fix ZeRO-3 + PEFT dtype mismatch for non-quantized models by @albertvillanova in https://github.com/huggingface/trl/pull/6093\r\n* Fix per-chunk lm_head.weight all-gathers under FSDP2 + chunked_nll by @albertvillanova in https://github.com/huggingface/trl/pull/6077\r\n* Raise when use_liger_kernel is combined with a PEFT adapter on lm_head by @akshansh47 in https://github.com/huggingface/trl/pull/5977\r\n* Align `epsilon` help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014\r\n* Fix broken light-mode banner image in README by @strickvl in https://github.com/huggingface/trl/pull/6112\r\n* Drop vLLM 0.12 support by @qgallouedec in https://github.com/huggingface/trl/pull/6109\r\n* Add `router_aux_loss_coef` by @qgallouedec in https://github.com/huggingface/trl/pull/6085\r\n* Add support for vLLM 0.19.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6107\r\n* Add license to Harbor examples by @qgallouedec in https://github.com/huggingface/trl/pull/6104\r\n* Remove redundant `.contiguous()` from the shift logits/labels pattern by @qgallouedec in https://github.com/huggingface/trl/pull/6046\r\n* Fix BrowserGym OpenEnv example dependency and task wiring by @burtenshaw in https://github.com/huggingface/trl/pull/6117\r\n* Fix async GRPO docs to require vllm>=0.22.0 by @sergiopaniego in https://github.com/huggingface/trl/pull/6101\r\n* Add `get_repetition_penalty_reward` by @qgallouedec in https://github.com/huggingface/trl/pull/6058\r\n* Add `trust_remote_code` to trainer configs by @qgallouedec in https://github.com/huggingface/trl/pull/5802\r\n* Align AsyncGRPO num_completions_to_print with GRPO (int | None) by @qgallouedec in https://github.com/huggingface/trl/pull/6020\r\n* Align AsyncGRPO epsilon_high with GRPO (None fallback to epsilon) by @qgallouedec in https://github.com/huggingface/trl/pull/6019\r\n* Fix logging_steps default mentioned in AsyncGRPOConfig docstring by @qgallouedec in https://github.com/huggingface/trl/pull/6016\r\n* Align async GRPO loss variable names with GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6013\r\n* Make `evaluate()` accept the same dataset types as the trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6116\r\n* Add support for vLLM 0.20.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6108\r\n* Add support for vLLM 0.22.1 by @qgallouedec in https://github.com/huggingface/trl/pull/6119\r\n* Fix broken internal doc link to GKD Trainer in MiniLLM docs by @ShamSaleem in https://github.com/huggingface/trl/pull/6131\r\n* Align AsyncGRPO clip-ratio metrics with GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6021\r\n* Add `get_cosine_scaled_reward` by @qgallouedec in https://github.com/huggingface/trl/pull/6066\r\n* Move experimental example scripts out of the packaged tree by @sergiopaniego in https://github.com/huggingface/trl/pull/6141\r\n* Harmonize logger imports by @qgallouedec in https://github.com/huggingface/trl/pull/6142\r\n* refactor(sft): build labels during dataset preparation instead of collation by @0xadvait in https://github.com/huggingface/trl/pull/6037\r\n* Add support for vLLM 0.23.0 by @qgallouedec in https://github.com/huggingface/trl/pull/6153\r\n* Change `_get_train_sampler` comment to mention `num_iterations > 1` by @anidoesdev in https://github.com/huggingface/trl/pull/6125\r\n* Warn when sequence-level importance sampling is combined with a token-summed loss type by @discobot in https://github.com/huggingface/trl/pull/6042\r\n* Drop vLLM 0.13 support by @qgallouedec in https://github.com/huggingface/trl/pull/6154\r\n* Align KTO with DPO: Add evaluate() override by @albertvillanova in https://github.com/huggingface/trl/pull/6148\r\n* Align KTO with DPO: Align order and signature of methods by @albertvillanova in https://github.com/huggingface/trl/pull/6149\r\n* Align KTO with DPO: Move all metrics computation from log to _compute_loss by @albertvillanova in https://github.com/huggingface/trl/pull/6150\r\n* Align KTO with DPO: Support sync_ref_model by @albertvillanova in https://github.com/huggingface/trl/pull/6152\r\n* Replace parse_version with Version by @albertvillanova in https://github.com/huggingface/trl/pull/6164\r\n* Keep extra columns in unpair_preference_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/6161\r\n* Align KTO with DPO: Add tests by @albertvillanova in https://github.com/huggingface/trl/pull/6160\r\n* Align format of code examples in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6147\r\n* Align KTO with DPO: Add VLM multi-image test by @albertvillanova in https://github.com/huggingface/trl/pull/6163\r\n* Support LFM2-VL multimodal inputs in GRPO and RLOO by @zwischenraum in https://github.com/huggingface/trl/pull/6114\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.6.0...v1.7.0\r\n","publishedAt":"2026-06-25T22:52:18.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.7.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":29224,"contentTokens":8495,"composition":{"bugs":0,"features":8,"enhancements":6}},{"id":"rel_4aRgXtA9OXhhfQPQK3xdf","version":"v1.6.0","type":"feature","title":"v1.6.0","summary":"The release introduces a new experimental A2POTrainer for optimal advantage regression and grants KTO trainer support for vision-language models. The AsyncRolloutWorker now runs in a separate process to avoid GIL contention and potential NCCL watchdog timeouts, along with fixes for aiohttp retries and all-NaN reward columns. Gold distillation trainer now aligns tokens via byte offsets, and SDFT/SDPO leverage the vLLM server for live teacher logprobs. Other features include bidirectional masked importance sampling for IcePop, support for NemotronH and Nemotron 3 Ultra, additional training chat templates, and decoupled self-distillation trainers.","titleGenerated":"Fine-tuning TRL v1.6.0 debuts A2PO trainer, supports VLM KTO, and improves async GRPO","titleShort":"A2PO trainer debuts; VLM KTO support; Async GRPO spawns process","breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### AsyncGRPO rollout worker now runs in a separate process\r\n\r\n`AsyncRolloutWorker` is no longer a thread — it's a spawned child process with its own GIL. The trainer's autograd engine no longer competes with `recursive_parse` / `accuracy_reward` for the GIL, which was causing 1-5s stalls in real Qwen3-30B-A3B @ 16k runs and ultimately NCCL watchdog timeouts on other ranks.\r\n\r\nArchitectural changes:\r\n- `AsyncRolloutWorker` (parent) owns the child process + shared `mp.Queue` / `mp.Value` / `mp.Event`.\r\n- `_AsyncRolloutLoop` (child-only) handles tokenization, dataset iteration, reward funcs, and asyncio loops.\r\n- A new `WeightTransferClient` owns the NCCL group with vLLM (`/pause`, `/resume`, `/init_weight_transfer_engine`, `/update_weights`); the rollout child only talks to `/v1/completions`.\r\n\r\nTwo correctness fixes shipped alongside (they would have conflicted otherwise): broader aiohttp retry (now catches `ClientPayloadError`) with bounded exponential backoff, and **all-NaN reward columns** are now preserved — `np.nansum` was silently returning 0, giving unscorable completions a real advantage signal and pushing the policy *away* from correct answers (~30% of DeepMath / OpenR1-Math rows).\r\n\r\n> [!NOTE]\r\n> `reward_funcs` / `tools` / `environment_factory` must now be picklable, and the child runs CPU-only (`CUDA_VISIBLE_DEVICES=\"\"`).\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/5749\r\n\r\n### New experimental A2PO trainer (Optimal Advantage Regression)\r\n\r\nA new `A2POTrainer` implements [A\\*-PO from \"Accelerating RL for LLM Reasoning with Optimal Advantage Regression\"](https://huggingface.co/papers/2505.20686). Two stages: an **offline V\\* estimation pass** from reference policy samples (with optional `filter_all_incorrect` to drop prompts where every reference completion fails), then **on-policy training with one generation per prompt** and a plain least-squares loss on `β₂·log(π/π_ref)` vs `r − V*`. No group, no critic, no clipping, no reward normalization.\r\n\r\n```python\r\nfrom trl.experimental.a2po import A2POConfig, A2POTrainer\r\n\r\ntrainer = A2POTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=A2POConfig(num_value_samples=8, filter_all_incorrect=True),\r\n    train_dataset=dataset,\r\n    reward_funcs=accuracy_reward,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nDesigned for binary verifiable rewards (math/code), not open-ended problems.\r\n\r\nby @raghulchandramouli in https://github.com/huggingface/trl/pull/5940\r\n\r\n### KTO now supports VLMs + big alignment push\r\n\r\nThe biggest KTO ↔ DPO alignment cycle yet — **KTOTrainer now supports vision-language models**, plus a deep restructuring of compute_loss, KL dataset generation, ref-logp precomputation, activation offloading, sampler strategy, metrics, and more. KTO graduation is very close.\r\n\r\n```python\r\nfrom trl.experimental.kto import KTOConfig, KTOTrainer\r\n\r\ntrainer = KTOTrainer(\r\n    model=\"Qwen/Qwen2.5-VL-3B-Instruct\",\r\n    args=KTOConfig(...),\r\n    train_dataset=vision_kto_dataset,\r\n)\r\n```\r\n\r\nVLM support: by @albertvillanova in https://github.com/huggingface/trl/pull/5939. Plus ~20 alignment PRs all by @albertvillanova: #5820, #5849, #5852, #5850, #5866, #5864, #5856, #5872, #5875, #5900, #5901, #5899, #5906, #5909, #5914, #5982, #5936, #5996, #5998, #5999.\r\n\r\n### Cross-tokenizer alignment in GOLD via byte offsets\r\n\r\nThe GOLD distillation trainer used to align student/teacher tokens by extending two decoded strings and flushing on equality. It silently broke on **any byte-level disagreement** — including the common case of one tokenizer prepending BOS while the other doesn't (Llama-3 ↔ Qwen-3). The X-Token paper called this out by name.\r\n\r\nEach side now carries `(start_byte, end_byte)` spans derived once from the fast tokenizer's char offsets, and the walker syncs on cumulative byte boundaries. On the on-policy path, spans come from `piece_byte_len` over the sampled token ids (not from re-encoding the decoded completion — BPE makes that round-trip non-injective).\r\n\r\nTwo related fixes shipped: long rows no longer lose the completion (now keeping the last `max_length` tokens), and the vLLM on-policy `original_prompt_text` is now decoded from the truncated ids the student actually consumed.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/5885\r\n\r\n### SDFT / SDPO: live teacher logprobs from the vLLM server\r\n\r\nWhen `teacher_model_kind=\"live\"` and `vllm_mode=\"server\"`, the vLLM generation server already holds the current student weights (synced every step for rollouts). The new `use_teacher_server=True` flag scores the teacher's log-probs on **that same server** instead of running a separate local teacher forward — removing the teacher from the training step entirely.\r\n\r\nSupported modes: `sampled_token` (reverse KL on the realized token) and `topk_logits`. When buffered batches reuse steps (`num_iterations > 1`), weights are re-synced before scoring so the teacher never scores stale.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/5989\r\n\r\n### Bidirectional masked importance sampling (MIS) for IcePop\r\n\r\nvLLM importance sampling in GRPO now uses a **two-sided band** `[C_min, C_max]` instead of a single upper cap, aligning TIS/MIS with [IcePop's bidirectional handling](https://github.com/huggingface/trl/pull/4732) of train–inference ratio outliers.\r\n\r\n```python\r\nfrom trl import GRPOConfig\r\n\r\nconfig = GRPOConfig(\r\n    vllm_importance_sampling_clip_min=0.5,\r\n    vllm_importance_sampling_clip_max=2.0,\r\n    vllm_importance_sampling_correction=\"mask\",  # or \"truncate\"\r\n)\r\n```\r\n\r\nThe old `vllm_importance_sampling_cap` is deprecated and maps to `clip_max`.\r\n\r\nby @casinca in https://github.com/huggingface/trl/pull/4732\r\n\r\n### NemotronH and Nemotron 3 Ultra support\r\n\r\nDay-zero training support for NVIDIA's new model families.\r\n\r\n* **NemotronH integration** by @qgallouedec in https://github.com/huggingface/trl/pull/5938\r\n* **Nemotron 3 Ultra support** by @qgallouedec in https://github.com/huggingface/trl/pull/5942\r\n* Enable gradient checkpointing in Nemotron 3 SFT example by @sergiopaniego in https://github.com/huggingface/trl/pull/5944\r\n\r\n### Even more training chat templates\r\n\r\nThree more model families with `{% generation %}` markers (assistant-only loss out of the box):\r\n\r\n- **Qwen2.5-VL** by @aazizyan in https://github.com/huggingface/trl/pull/5838\r\n- **Qwen2-VL** by @aazizyan in https://github.com/huggingface/trl/pull/5839\r\n- **Llava-Next** by @aazizyan in https://github.com/huggingface/trl/pull/5959\r\n\r\n### Distributed backend boilerplate, hidden\r\n\r\nA new `trl/distributed.py` introduces a single `DistributedBackend` class that detects ZeRO stage and FSDP version once, then exposes two context managers (`gather_params`, `summon_full_params`) used everywhere. Replaces the scattered `getattr(state, \"fsdp_plugin\", None)` / `gather_if_zero3` / `summon_full_params if ... else nullcontext()` boilerplate spread across `vllm_generation.py`, `models/utils.py`, and the main trainers. Future deprecations land in one place.\r\n\r\nby @albertvillanova in https://github.com/huggingface/trl/pull/6000\r\n\r\n### Decoupled self-distillation trainers\r\n\r\nA two-PR refactor that disentangles SDPO, SDFT, and other self-distillation trainers from their shared base, making each one self-contained and consistent with the rest of the codebase.\r\n\r\nby @LeonEricsson in https://github.com/huggingface/trl/pull/5862 and https://github.com/huggingface/trl/pull/5883\r\n\r\n### Heads-up: SFT default `loss_type` will change in 1.7\r\n\r\nSetting `SFTConfig.loss_type` is now optional, and leaving it unset emits a `FutureWarning`: in TRL **1.7** the default will switch from `\"nll\"` to `\"chunked_nll\"`. No action needed — you'll just get the new default automatically on upgrade — unless you want to pin the current behavior (e.g. for custom models) with `loss_type=\"nll\"`.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5997\r\n\r\n### Other\r\n\r\n* Support `'None'` as CLI value for `Optional[T]` fields by @qgallouedec in https://github.com/huggingface/trl/pull/5843\r\n* Support non-`lm_head` output projections in chunked SFT loss (GPTNeoX) by @qgallouedec in https://github.com/huggingface/trl/pull/5857\r\n* `SFTTrainer`: merge entropy and accuracy computation to eliminate redundant logits copy by @flutist in https://github.com/huggingface/trl/pull/5897\r\n* Remove redundant `.contiguous()` calls in `DPOTrainer` to reduce peak memory by @flutist in https://github.com/huggingface/trl/pull/5926\r\n* Remove unnecessary explicit `.contiguous()` before `entropy_from_logits` by @qgallouedec in https://github.com/huggingface/trl/pull/5930\r\n* Exclude `None` reward completions from GRPO/RLOO advantage baseline by @AmineDiro in https://github.com/huggingface/trl/pull/5902\r\n* Support multimodal config in PPO ValueHead by @albertvillanova in https://github.com/huggingface/trl/pull/5907\r\n* Support vision datasets for Liger in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5943\r\n* Raise if `precompute_ref_log_probs` with vision datasets in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5867\r\n* 🔒 Gate trainer telemetry on an explicit class-name allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/5851\r\n* Update vLLM version support to 0.19.0 by @sergiopaniego in https://github.com/huggingface/trl/pull/5879\r\n* Improve error message when image tokens are truncated by `max_length` by @lxk8998 in https://github.com/huggingface/trl/pull/5927\r\n* Padding-free invariance test by @qgallouedec in https://github.com/huggingface/trl/pull/5842\r\n* Per-field invariance tolerances, calibrated by @qgallouedec in https://github.com/huggingface/trl/pull/5844\r\n\r\n## Fixes\r\n\r\n* **Fix `loss_type=\"chunked_nll\"` under DeepSpeed ZeRO-3** by @qgallouedec in https://github.com/huggingface/trl/pull/5873\r\n* **Fix GRPO `use_liger_kernel` under DeepSpeed ZeRO-3** by @kashif in https://github.com/huggingface/trl/pull/5891\r\n* `async_grpo`: don't return on `queue.Empty` by @AmineDiro in https://github.com/huggingface/trl/pull/5751\r\n* Don't treat ROCm GPUs as Ampere by @kashif in https://github.com/huggingface/trl/pull/5917\r\n* Route liger student forward through DDP wrapper in GKD, GOLD, and Distillation trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5934\r\n* Fix backbone access in GRPO by aligning with SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5949\r\n* Fix priority order in PPO ValueHead and raise ValueError for unsupported config by @albertvillanova in https://github.com/huggingface/trl/pull/5908\r\n* Fix `generate_batch`: inference tensors block inplace ops in background thread by @albertvillanova in https://github.com/huggingface/trl/pull/5818 (cross-listed from v1.5 changelog window)\r\n* Fix SFT padding-free test config by @kashif in https://github.com/huggingface/trl/pull/5923\r\n* Specify `encoding=\"utf-8\"` when reading `.jinja` chat templates on Windows by @ColebyPearson in https://github.com/huggingface/trl/pull/5869\r\n* Fix `ValueError` by pinning `kernels < 0.15.1` by @albertvillanova in https://github.com/huggingface/trl/pull/5880\r\n* Set `kernels` optional dependency via transformers by @albertvillanova in https://github.com/huggingface/trl/pull/5884\r\n* Support `kernels` extra for `transformers < 5.1.0` by @albertvillanova in https://github.com/huggingface/trl/pull/5928\r\n* Add missing `use_liger_kernel` guard to SDPO teacher-server validation by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5994\r\n* Flash Attention capitalization fix by @qgallouedec in https://github.com/huggingface/trl/pull/5855\r\n\r\n## Documentation and Examples\r\n\r\n* Remove NeMo Gym Integration Guide (broken) by @cmunley1 in https://github.com/huggingface/trl/pull/5840\r\n* docs(GRPOTrainer): remove duplicate sentence by @zafstojano in https://github.com/huggingface/trl/pull/5957\r\n* docs(RLOOTrainer): fix blockquote math not rendering by @zafstojano in https://github.com/huggingface/trl/pull/5958\r\n* docs: highlight the role of KL in RLOO compared to GRPO by @zafstojano in https://github.com/huggingface/trl/pull/5966\r\n* docs: clarify PPO entropy metrics in PPO trainer docs by @biefan in https://github.com/huggingface/trl/pull/5289\r\n* docs: update OpenEnv GitHub org references and package name by @sergiopaniego in https://github.com/huggingface/trl/pull/5919\r\n* docs: update OpenEnv doc URLs to `huggingface.co/docs/openenv` by @sergiopaniego in https://github.com/huggingface/trl/pull/5929\r\n* docs: sync SDFT/SDPO config docstrings with their fields by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5992\r\n* docs: sync Distillation/GOLD/OnlineDPO config docstrings with their fields by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5995\r\n* docs: Document `bnb_4bit_quant_storage` and normalize docstring param headers by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5993\r\n* docs: fix rendering typos by @zafstojano in https://github.com/huggingface/trl/pull/5991\r\n* fix(docs): correct broken GKD Trainer link in MiniLLM docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5960 and https://github.com/huggingface/trl/pull/5961\r\n* fix(docs): drop duplicate \"a\" in `online_dpo_vlm` example description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5978\r\n* Fix broken doc links by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5971\r\n* Fix broken code examples in docs (RLOO syntax, `SFTConfig` `max_length`) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5970\r\n* Fix malformed ScaleRL paper link in `GRPOConfig` `epsilon_high` help by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5972\r\n* fix(cli): drop duplicate \"to\" in `trl skills install` description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6008\r\n* Remove invalid `max_prompt_length` argument from GRPO example by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5964\r\n\r\n## CI\r\n\r\n* Refresh `sft.json` / `dpo.json` snapshots after transformers `num_items_in_batch fix` by @qgallouedec in https://github.com/huggingface/trl/pull/5845\r\n* Add testing for Olmo 3 by @qgallouedec in https://github.com/huggingface/trl/pull/5962\r\n* Align trainer train tests by @qgallouedec in https://github.com/huggingface/trl/pull/5963\r\n* Align trainers: Remove redundant else branch by @albertvillanova in https://github.com/huggingface/trl/pull/5983\r\n* [CI] Check that training chat templates keep the stop token in the loss mask by @kashif in https://github.com/huggingface/trl/pull/5988\r\n* Create CI workflow to sync TRL skill with `huggingface/skills` by @albertvillanova in https://github.com/huggingface/trl/pull/5950\r\n* Simplify agent skills target and default to `.agents` by @albertvillanova in https://github.com/huggingface/trl/pull/5987\r\n* chore: enable Dependabot weekly GitHub Actions bumps by @hf-dependantbot-rollout[bot] in https://github.com/huggingface/trl/pull/5910\r\n* Bump the actions group with 9 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/5913\r\n* Bump the actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/5954\r\n* chore: update `docker-build.yml` with version parsing by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5920\r\n* ci: use GitHub App auth for doc preview comment bot by @sergiopaniego in https://github.com/huggingface/trl/pull/5915\r\n\r\n## New Contributors\r\n* @ColebyPearson made their first contribution in https://github.com/huggingface/trl/pull/5869\r\n* @hf-dependantbot-rollout[bot] made their first contribution in https://github.com/huggingface/trl/pull/5910\r\n* @raghulchandramouli made their first contribution in https://github.com/huggingface/trl/pull/5940\r\n* @zafstojano made their first contribution in https://github.com/huggingface/trl/pull/5957\r\n* @DaoyuanLi2816 made their first contribution in https://github.com/huggingface/trl/pull/5964\r\n* @lxk8998 made their first contribution in https://github.com/huggingface/trl/pull/5927\r\n* @biefan made their first contribution in https://github.com/huggingface/trl/pull/5289\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5836\r\n* Add Qwen2.5-VL original and training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5838\r\n* Align KTO with DPO: Simplify metrics from sum/count to direct averages by @albertvillanova in https://github.com/huggingface/trl/pull/5820\r\n* async_grpo don't return on queue.Empty by @AmineDiro in https://github.com/huggingface/trl/pull/5751\r\n* Align KTO with DPO: Refactor forward by @albertvillanova in https://github.com/huggingface/trl/pull/5849\r\n* Per-field invariance tolerances, calibrated by @qgallouedec in https://github.com/huggingface/trl/pull/5844\r\n* Add Qwen2-VL original and training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5839\r\n* Remove NeMo Gym Integration Guide (broken) by @cmunley1 in https://github.com/huggingface/trl/pull/5840\r\n* Align KTO with DPO: Align compute_ref_log_probs by @albertvillanova in https://github.com/huggingface/trl/pull/5852\r\n* Align KTO with DPO: Align precompute_ref_logps by @albertvillanova in https://github.com/huggingface/trl/pull/5850\r\n* Flash Attention capitalization fix by @qgallouedec in https://github.com/huggingface/trl/pull/5855\r\n* 🔒 Gate trainer telemetry on an explicit class-name allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/5851\r\n* Align KTO with DPO: Support remove_unused_columns by @albertvillanova in https://github.com/huggingface/trl/pull/5866\r\n* Raise if precompute_ref_log_probs with vision datasets in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5867\r\n* Support `'None'` as CLI value for `Optional[T]` fields by @qgallouedec in https://github.com/huggingface/trl/pull/5843\r\n* KTO: Replace _get_train_sampler with train_sampling_strategy for transformers >= 5.2.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5864\r\n* Fix: specify encoding=\"utf-8\" when reading .jinja chat templates on Windows by @ColebyPearson in https://github.com/huggingface/trl/pull/5869\r\n* Align KTO with DPO: Align ref log probability names by @albertvillanova in https://github.com/huggingface/trl/pull/5856\r\n* KTO: Support non-sequential train_sampling_strategy for apo_zero_unpaired by @albertvillanova in https://github.com/huggingface/trl/pull/5872\r\n* Align KTO with DPO: Remove null_ref_context by @albertvillanova in https://github.com/huggingface/trl/pull/5875\r\n* Fix ValueError by pinning kernels < 0.15.1 by @albertvillanova in https://github.com/huggingface/trl/pull/5880\r\n* Update vLLM version support to 0.19.0 by @sergiopaniego in https://github.com/huggingface/trl/pull/5879\r\n* Set kernels optional dependency via transformers by @albertvillanova in https://github.com/huggingface/trl/pull/5884\r\n* Support non-lm_head output projections in chunked SFT loss (GPTNeoX) by @qgallouedec in https://github.com/huggingface/trl/pull/5857\r\n* SFTTrainer: merge entropy and accuracy computation to eliminate redundant logits copy by @flutist in https://github.com/huggingface/trl/pull/5897\r\n* Align KTO with DPO: Add disable_gradient_checkpointing to ref model forward passes by @albertvillanova in https://github.com/huggingface/trl/pull/5900\r\n* Align KTO with DPO: Add activation offloading support by @albertvillanova in https://github.com/huggingface/trl/pull/5901\r\n* Align KTO with DPO: Decouple KL dataset generation by @albertvillanova in https://github.com/huggingface/trl/pull/5899\r\n* Fix GRPO use_liger_kernel under DeepSpeed ZeRO-3 by @kashif in https://github.com/huggingface/trl/pull/5891\r\n* Replace custom numpy cache in precompute_ref_logps with native datasets by @albertvillanova in https://github.com/huggingface/trl/pull/5906\r\n* [1/2] refactor: decoupled self distillation trainers (sdpo, sdft, ...) by @LeonEricsson in https://github.com/huggingface/trl/pull/5862\r\n* Align KTO with DPO: Use datasets caching in precompute_ref_logps by @albertvillanova in https://github.com/huggingface/trl/pull/5909\r\n* Support multimodal config in PPO ValueHead by @albertvillanova in https://github.com/huggingface/trl/pull/5907\r\n* Fix priority order in PPO ValueHead and raise ValueError for unsupported config by @albertvillanova in https://github.com/huggingface/trl/pull/5908\r\n* Fix `loss_type=\"chunked_nll\"` under DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/5873\r\n* chore: enable Dependabot weekly GitHub Actions bumps by @hf-dependantbot-rollout[bot] in https://github.com/huggingface/trl/pull/5910\r\n* Exclude None  reward completions  from GRPO/RLOO advantage baseline by @AmineDiro in https://github.com/huggingface/trl/pull/5902\r\n* Don't treat ROCm GPUs as Ampere by @kashif in https://github.com/huggingface/trl/pull/5917\r\n* ci: use GitHub App auth for doc preview comment bot by @sergiopaniego in https://github.com/huggingface/trl/pull/5915\r\n* Bump the actions group with 9 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/5913\r\n* Align KTO with DPO: Replace completion_labels/get_batch_logps with completion_mask by @albertvillanova in https://github.com/huggingface/trl/pull/5914\r\n* chore: update docker-build.yml with version parsing by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5920\r\n* docs: update OpenEnv GitHub org references and package name by @sergiopaniego in https://github.com/huggingface/trl/pull/5919\r\n* Support kernels extra for transformers < 5.1.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5928\r\n* feat: move async rollout worker to separate process by @AmineDiro in https://github.com/huggingface/trl/pull/5749\r\n* Remove redundant .contiguous() calls in DPOTrainer to reduce peak memory by @flutist in https://github.com/huggingface/trl/pull/5926\r\n* docs: update OpenEnv doc URLs from meta-pytorch.org to huggingface.co/docs/openenv by @sergiopaniego in https://github.com/huggingface/trl/pull/5929\r\n* Refresh `sft.json` / `dpo.json` snapshots after transformers `num_items_in_batch fix` by @qgallouedec in https://github.com/huggingface/trl/pull/5845\r\n* NemotronH integration by @qgallouedec in https://github.com/huggingface/trl/pull/5938\r\n* Nemotron 3 Ultra support by @qgallouedec in https://github.com/huggingface/trl/pull/5942\r\n* Enable gradient checkpointing in Nemotron 3 SFT example (transformers>=5.7.0) by @sergiopaniego in https://github.com/huggingface/trl/pull/5944\r\n* Fix SFT padding-free test config by @kashif in https://github.com/huggingface/trl/pull/5923\r\n* Add experimental A2PO trainer (Optimal Advantage Regression) by @raghulchandramouli in https://github.com/huggingface/trl/pull/5940\r\n* Align KTO with DPO: Inline _compute_logps into _compute_loss by @albertvillanova in https://github.com/huggingface/trl/pull/5936\r\n* Fix: Route liger student forward through DDP wrapper in GKD, GOLD, and Distillation trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5934\r\n* Fix backbone access in GRPO by aligning with SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5949\r\n* fix(docs): Remove duplicate sentence in GRPOTrainer docs by @zafstojano in https://github.com/huggingface/trl/pull/5957\r\n* fix(docs): Blockquote math not rendering in RLooTrainer docs by @zafstojano in https://github.com/huggingface/trl/pull/5958\r\n* Remove invalid max_prompt_length argument from GRPO example by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5964\r\n* Bump the actions group with 4 updates by @dependabot[bot] in https://github.com/huggingface/trl/pull/5954\r\n* Add Llava-Next training tempalates support with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5959\r\n* fix(docs): correct broken GKD Trainer link in MiniLLM docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5961\r\n* Remove unnecessary explicit `.contiguous()` before `entropy_from_logits` by @qgallouedec in https://github.com/huggingface/trl/pull/5930\r\n* Improve error message when image tokens are truncated by max_length by @lxk8998 in https://github.com/huggingface/trl/pull/5927\r\n* fix(docs): correct broken GKD Trainer link in MiniLLM docs by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5960\r\n* Cross-tokenizer alignment via byte offsets in GOLD trainer by @kashif in https://github.com/huggingface/trl/pull/5885\r\n* [2/2] refactor: decoupled self distillation trainers; cleanup by @LeonEricsson in https://github.com/huggingface/trl/pull/5883\r\n* Create CI workflow to sync TRL skill with huggingface/skills by @albertvillanova in https://github.com/huggingface/trl/pull/5950\r\n* Support vision datasets for Liger in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5943\r\n* Align trainer train tests by @qgallouedec in https://github.com/huggingface/trl/pull/5963\r\n* Fix malformed ScaleRL paper link in GRPOConfig epsilon_high help by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5972\r\n* Add testing for Olmo3 by @qgallouedec in https://github.com/huggingface/trl/pull/5962\r\n* chore(docs): Highlight the role of KL in RLOO compared to GRPO by @zafstojano in https://github.com/huggingface/trl/pull/5966\r\n* fix(docs): drop duplicate \"a\" in online_dpo_vlm example description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5978\r\n* Fix broken doc links (CONTRIBUTING online DPO paths, async GRPO anchor) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5971\r\n* Align KTO with DPO: Support VLM by @albertvillanova in https://github.com/huggingface/trl/pull/5939\r\n* Fix broken code examples in docs (RLOO syntax, SFTConfig max_length) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5970\r\n* Align KTO with DPO: Improve error message for VLM truncation by @albertvillanova in https://github.com/huggingface/trl/pull/5982\r\n* Align trainers: Remove redundant else branch by @albertvillanova in https://github.com/huggingface/trl/pull/5983\r\n* SDFT/SDPO: live teacher logprobs from the vLLM server by @kashif in https://github.com/huggingface/trl/pull/5989\r\n* docs: sync SDFT/SDPO config docstrings with their fields by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5992\r\n* fix(docs): Fix rendering typos by @zafstojano in https://github.com/huggingface/trl/pull/5991\r\n* Add missing use_liger_kernel guard to SDPO teacher-server validation by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5994\r\n* docs: sync Distillation/GOLD/OnlineDPO config docstrings with their fields by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5995\r\n* feat: Bidirectional masked importance sampling ratio (MIS) for IcePop by @casinca in https://github.com/huggingface/trl/pull/4732\r\n* Simplify agent skills target and default to `.agents` by @albertvillanova in https://github.com/huggingface/trl/pull/5987\r\n* Align KTO with DPO: Remove unused use_dpo_data_collator attribute by @albertvillanova in https://github.com/huggingface/trl/pull/5996\r\n* Align KTO with DPO: Rename kto_loss_fn to liger_loss_fn by @albertvillanova in https://github.com/huggingface/trl/pull/5998\r\n* Align KTO with DPO: Inline kto_loss in _compute_loss by @albertvillanova in https://github.com/huggingface/trl/pull/5999\r\n* Document bnb_4bit_quant_storage and normalize docstring param headers by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5993\r\n* [CI] Check that training chat templates keep the stop token in the loss mask by @kashif in https://github.com/huggingface/trl/pull/5988\r\n* Announce upcoming SFT `loss_type` default change from `'nll'` to `'chunked_nll'` by @qgallouedec in https://github.com/huggingface/trl/pull/5997\r\n* Padding-free invariance test by @qgallouedec in https://github.com/huggingface/trl/pull/5842\r\n* Hide DeepSpeed/FSDP distributed backend boilerplate by @albertvillanova in https://github.com/huggingface/trl/pull/6000\r\n* fix(cli): drop duplicate \"to\" in trl skills install description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6008\r\n* docs: clarify PPO entropy metrics in PPO trainer docs by @biefan in https://github.com/huggingface/trl/pull/5289\r\n* Release: v1.6 by @qgallouedec in https://github.com/huggingface/trl/pull/6009\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.5.0...v1.6.0\r\n","publishedAt":"2026-06-11T22:00:15.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.6.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":28354,"contentTokens":8125,"composition":{"bugs":2,"features":7,"enhancements":5}},{"id":"rel_8_eXU-zwuSdzRk6xMoDTB","version":"v1.5.1","type":"feature","title":"v1.5.1","summary":"Trainer telemetry is now gated on an explicit class-name allowlist, restricting which trainer classes can send telemetry.","titleGenerated":"TRL Fine-tuning v1.5.1 gates trainer telemetry on allowlist","titleShort":"Trainer telemetry now allowlisted","breaking":"unknown","importance":null,"content":"## What's Changed\r\n\r\n* 🔒 Gate trainer telemetry on an explicit class-name allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/5851\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.5.0...v1.5.1","publishedAt":"2026-05-27T15:26:53.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.5.1","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":230,"contentTokens":71,"composition":{"bugs":0,"features":0,"enhancements":1}},{"id":"rel_DGph8QcHlLU1aD0WmSMY3","version":"v1.5.0","type":"feature","title":"v1.5.0","summary":"Fixed an exponential backtracking bug in Qwen3/Qwen3.5/GLM4MoE response parsing that caused GRPOTrainer to hang indefinitely on truncated tool-call blocks, reducing worst-case complexity from O(2ⁿ) to O(n). Also fixed a CUDA memory leak in BNB dequantization buffers and stale state in OffloadActivations. Added training chat templates for Phi-3.5, Qwen3-VL, and Qwen3.5 Think/NoThink, and final logits softcapping support for AsyncGRPOTrainer on models like Gemma 2.","titleGenerated":"TRL v1.5.0 fixes exponential backtracking in response parsing and CUDA memory leak","titleShort":"Response parsing hang fixed; CUDA memory leak patched","breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### Even more training chat templates\r\n\r\nThree more model families gain training-compatible templates with `{% generation %}` markers (so `assistant_only_loss=True` just works):\r\n\r\n- **Phi-3.5** by @DagaBhai in https://github.com/huggingface/trl/pull/5746\r\n- **Qwen3-VL** by @aazizyan in https://github.com/huggingface/trl/pull/5764\r\n- **Qwen3.5 Think / NoThink** by @aazizyan in https://github.com/huggingface/trl/pull/5824\r\n\r\n### Final logits softcapping for async GRPO\r\n\r\nThe chunked LM-head path used by `AsyncGRPOTrainer` now supports models that use `final_logit_softcapping` (notably Gemma 2). `_ChunkedLogProbFunction` applies `logit_scale`, optional tanh-based softcapping, and temperature consistently in both forward and backward — softcapped models are no longer rejected.\r\n\r\nby @mlarnouhet in https://github.com/huggingface/trl/pull/5691\r\n\r\n### KTO ↔ DPO alignment continues\r\n\r\nTwo more cycles closer to KTO graduation:\r\n\r\n* Align `compute_loss` flow by @albertvillanova in https://github.com/huggingface/trl/pull/5810\r\n* Align `_compute_loss_liger` flow by @albertvillanova in https://github.com/huggingface/trl/pull/5816\r\n\r\n### Trainer telemetry (opt-out)\r\n\r\n`_BaseTrainer.__init__` now emits a single anonymous `huggingface_hub.send_telemetry` ping per trainer instantiation, so we can finally see which trainers / model families / distributed backends are actually being used in practice and prioritize accordingly.\r\n\r\nThe payload is intentionally minimal — TRL version, trainer class name, model architecture, PEFT yes/no, distributed backend (`deepspeed`/`fsdp`/`ddp`/`none`), bucketed world size, device type, GPU model when available. No user data, no dataset names, no model paths, no hyperparameter values, never sent in CI / offline / `HF_HUB_DISABLE_TELEMETRY` mode.\r\n\r\nSee [`usage_stats.md`](https://huggingface.co/docs/trl/usage_stats) for what's collected and how to opt out.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5758\r\n\r\n### Other\r\n\r\n* `OpenRewardSpec`: fix omitting task-scoped tools during rollout binding (fixes #5727) by @rycerzes in https://github.com/huggingface/trl/pull/5729\r\n* Add OpenReward example to the list of examples by @sergiopaniego in https://github.com/huggingface/trl/pull/5752\r\n* Add DDP-2 members to invariant test suite by @qgallouedec in https://github.com/huggingface/trl/pull/5736\r\n* Align and simplify the stable training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/5812\r\n* Replace uv installation script with setup action by @qgallouedec in https://github.com/huggingface/trl/pull/5735\r\n\r\n## Fixes\r\n\r\n* **Fix exponential backtracking in qwen3 / qwen3_5 / glm4moe response parsing** — `GRPOTrainer` was hanging indefinitely on truncated `<tool_call>` blocks (a degenerate case that happens naturally when generation hits `max_completion_length` mid-tool-call). Rewrote the regex to be non-backtracking — worst case goes from O(2ⁿ) to O(n). By @xodn348 in https://github.com/huggingface/trl/pull/5798\r\n* **CUDA memory leak: release BNB dequantization buffers & stale state in `OffloadActivations`** — follow-up to v1.4's activation-offloading leak fix. By @butterwecksolutions in https://github.com/huggingface/trl/pull/5730\r\n* Invalidate ZeRO-3 param coordinator trace in `add_hooks` by @roycho96 in https://github.com/huggingface/trl/pull/4693\r\n* Fix nested `vocab_size` for `DistillationTrainer` and `GOLDTrainer` by @Beichen-Ma in https://github.com/huggingface/trl/pull/5592\r\n* Fix MPS support in experimental `empty_cache()` by @jamie-peterson-ml in https://github.com/huggingface/trl/pull/5799\r\n* Fix `metric_for_best_model` for trainer-specific eval metrics by @qgallouedec in https://github.com/huggingface/trl/pull/5811\r\n* Fix `generate_batch`: inference tensors blocking inplace ops in background thread by @albertvillanova in https://github.com/huggingface/trl/pull/5818\r\n* Replace deprecated `torch_dtype` with `dtype` across examples, docs, notebooks, tests, and experimental `distillation` / `gold` trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5717\r\n\r\n## Documentation and Examples\r\n\r\n* docs(grpo): align model to Qwen2.5 and add GRPO OOM tab in quickstart by @xodn348 in https://github.com/huggingface/trl/pull/5740\r\n\r\n## CI\r\n\r\n* Migrate tests to **Qwen3.5 Think/NoThink fixtures** + tiny-model generation scripts by @aazizyan in https://github.com/huggingface/trl/pull/5819 and https://github.com/huggingface/trl/pull/5821\r\n* Align tiny `Glm4MoeForCausalLM` / `Cohere` / `Cohere2` / `Qwen2.5-VL` configs with their reference models by @qgallouedec in https://github.com/huggingface/trl/pull/5638, https://github.com/huggingface/trl/pull/5706, https://github.com/huggingface/trl/pull/5707 and https://github.com/huggingface/trl/pull/5739\r\n* Fix tiny Qwen3-VL `deepstack_visual_indexes` and drop the test skip by @qgallouedec in https://github.com/huggingface/trl/pull/5779\r\n* Fix tiny Qwen2.5-VL `fullatt_block_indexes` out of range for depth=2 by @albertvillanova in https://github.com/huggingface/trl/pull/5805\r\n* Remove non-existent params from tiny Qwen2-VL model by @albertvillanova in https://github.com/huggingface/trl/pull/5795\r\n* Fix vision config `num_heads` key in Qwen VL tiny model scripts by @matdou in https://github.com/huggingface/trl/pull/5792\r\n* Drop unjustified `model.visual.` skip in GRPO/RLOO Qwen2.5-VL tests by @qgallouedec in https://github.com/huggingface/trl/pull/5780\r\n* Make the LLaVA / LLaVA-Next test guard explicit by @qgallouedec in https://github.com/huggingface/trl/pull/5778\r\n* Remove obsolete Gemma 3 vision-head guard from VLM training tests by @qgallouedec in https://github.com/huggingface/trl/pull/5772\r\n* Fix OOM in CI: reduce batch size in VLM SFT / GRPO/RLOO VLM / toolcall tests by @albertvillanova in https://github.com/huggingface/trl/pull/5687, https://github.com/huggingface/trl/pull/5767, https://github.com/huggingface/trl/pull/5801\r\n* Fix OOM in CI by clearing chained exception tracebacks by @albertvillanova in https://github.com/huggingface/trl/pull/5776\r\n* Fix OOM in CI by reducing intermediate_size and image token budget for tiny Gemma 4 by @albertvillanova in https://github.com/huggingface/trl/pull/5760\r\n* Fix CI errors in response parsing for gpt-oss/llama with transformers v5 by @albertvillanova in https://github.com/huggingface/trl/pull/5755\r\n* Fix CI `AttributeError: 'GptOssConfig' object has no attribute 'num_experts'` by @albertvillanova in https://github.com/huggingface/trl/pull/5756\r\n* Fix CI `apply_model_revisions` by removing `_commit_hash` kwarg by @albertvillanova in https://github.com/huggingface/trl/pull/5762\r\n* Fix CI test to avoid skipping `model.visual` params by @albertvillanova in https://github.com/huggingface/trl/pull/5806\r\n* Fix transformers min version for tiny gemma 4 as 5.5.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5763\r\n* Hotfix CI: pin `torch < 2.12.0` (later reverted) by @albertvillanova in https://github.com/huggingface/trl/pull/5769\r\n* Fix catch-all empty string in Makefile `pytest --only-rerun` by @albertvillanova in https://github.com/huggingface/trl/pull/5784\r\n* chore: update `tests_latest.yml` by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5733\r\n\r\n## New Contributors\r\n* @hf-security-analysis[bot] made their first contribution in https://github.com/huggingface/trl/pull/5733\r\n* @Beichen-Ma made their first contribution in https://github.com/huggingface/trl/pull/5592\r\n* @DagaBhai made their first contribution in https://github.com/huggingface/trl/pull/5746\r\n* @xodn348 made their first contribution in https://github.com/huggingface/trl/pull/5740\r\n* @mlarnouhet made their first contribution in https://github.com/huggingface/trl/pull/5691\r\n* @matdou made their first contribution in https://github.com/huggingface/trl/pull/5792\r\n* @jamie-peterson-ml made their first contribution in https://github.com/huggingface/trl/pull/5799\r\n* @rycerzes made their first contribution in https://github.com/huggingface/trl/pull/5729\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5734\r\n* chore: update tests_latest.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5733\r\n* fix: CUDA memory leak / release BNB dequantization buffers & stale state in OffloadActivations by @butterwecksolutions in https://github.com/huggingface/trl/pull/5730\r\n* fix: invalidate ZeRO-3 param coordinator trace in add_hooks by @roycho96 in https://github.com/huggingface/trl/pull/4693\r\n* Fix nested vocab_size for DistillationTrainer and GOLDTrainer by @Beichen-Ma in https://github.com/huggingface/trl/pull/5592\r\n* feat:  add Phi-3.5 training chat templates with generation markers by @DagaBhai in https://github.com/huggingface/trl/pull/5746\r\n* docs(grpo): align model to Qwen2.5 and add GRPO OOM tab in quickstart by @xodn348 in https://github.com/huggingface/trl/pull/5740\r\n* `torch_dtype` -> `dtype` by @qgallouedec in https://github.com/huggingface/trl/pull/5717\r\n* Add OpenReward example to the list of examples by @sergiopaniego in https://github.com/huggingface/trl/pull/5752\r\n* Fix CI errors in response parsing for gptoss/llama with transformers v5 by @albertvillanova in https://github.com/huggingface/trl/pull/5755\r\n* Add DDP-2 members to invariant test suite by @qgallouedec in https://github.com/huggingface/trl/pull/5736\r\n* Hotfix CI param not updated AssertionError: Pin torch < 2.12.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5769\r\n* Align tiny-Glm4MoeForCausalLM with GLM-4.5 reference config by @qgallouedec in https://github.com/huggingface/trl/pull/5638\r\n* Align tiny Cohere config with aya-expanse-8b by @qgallouedec in https://github.com/huggingface/trl/pull/5706\r\n* Align tiny Cohere2 config with tiny-aya-earth by @qgallouedec in https://github.com/huggingface/trl/pull/5707\r\n* Fix OOM in CI by reducing intermediate_size and image token budget for tiny Gemma4 by @albertvillanova in https://github.com/huggingface/trl/pull/5760\r\n* Fix CI AttributeError: 'GptOssConfig' object has no attribute 'num_experts' by @albertvillanova in https://github.com/huggingface/trl/pull/5756\r\n* Fix CI apply_model_revisions by removing _commit_hash kwarg by @albertvillanova in https://github.com/huggingface/trl/pull/5762\r\n* Remove obsolete Gemma3 vision-head guard from VLM training tests by @qgallouedec in https://github.com/huggingface/trl/pull/5772\r\n* Replace uv installation script with setup action by @qgallouedec in https://github.com/huggingface/trl/pull/5735\r\n* Fix OOM in CI by clearing chained exception tracebacks by @albertvillanova in https://github.com/huggingface/trl/pull/5776\r\n* Fix transformers min version for tiny gemma4 as 5.5.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5763\r\n* Final logits softcapping support for async GRPO Trainer by @mlarnouhet in https://github.com/huggingface/trl/pull/5691\r\n* Fix vision config num_heads key in Qwen VL tiny model scripts and revert torch pin by @matdou in https://github.com/huggingface/trl/pull/5792\r\n* Drop unjustified `model.visual.` skip in GRPO / RLOO Qwen2.5-VL tests by @qgallouedec in https://github.com/huggingface/trl/pull/5780\r\n* Fix OOM in CI by reducing batch size and sequence length for toolcall tests by @albertvillanova in https://github.com/huggingface/trl/pull/5801\r\n* Fix exponential backtracking in qwen3 / qwen3_5 / glm4moe response parsing by @xodn348 in https://github.com/huggingface/trl/pull/5798\r\n* Add telemetry to trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5758\r\n* Add Qwen3-VL training chat template with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5764\r\n* Align tiny Qwen2.5-VL with Qwen/Qwen2.5-VL-3B-Instruct by @qgallouedec in https://github.com/huggingface/trl/pull/5739\r\n* Fix tiny Qwen3-VL `deepstack_visual_indexes` and drop the test skip by @qgallouedec in https://github.com/huggingface/trl/pull/5779\r\n* Fix OOM in CI by reducing batch size in GRPO/RLOO VLM tests by @albertvillanova in https://github.com/huggingface/trl/pull/5767\r\n* Fix catch-all empty string in Makefile pytest --only-rerun by @albertvillanova in https://github.com/huggingface/trl/pull/5784\r\n* Remove non-existent params from tiny Qwen2-VL model by @albertvillanova in https://github.com/huggingface/trl/pull/5795\r\n* Fix tiny Qwen2.5-VL fullatt_block_indexes out of range for depth=2 by @albertvillanova in https://github.com/huggingface/trl/pull/5805\r\n* Make the LLaVA / LLaVA-Next test guard explicit by @qgallouedec in https://github.com/huggingface/trl/pull/5778\r\n* Fix MPS support in experimental empty_cache() by @jamie-peterson-ml in https://github.com/huggingface/trl/pull/5799\r\n* Fix CI test to avoid skipping model.visual params by @albertvillanova in https://github.com/huggingface/trl/pull/5806\r\n* Align KTO with DPO: Align compute_loss flow by @albertvillanova in https://github.com/huggingface/trl/pull/5810\r\n* Fix generate_batch: inference tensors block inplace ops in background thread by @albertvillanova in https://github.com/huggingface/trl/pull/5818\r\n* Fix `metric_for_best_model` for trainer-specific eval metrics by @qgallouedec in https://github.com/huggingface/trl/pull/5811\r\n* Align and simplify the stable training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/5812\r\n* Align KTO with DPO: Align _compute_loss_liger flow by @albertvillanova in https://github.com/huggingface/trl/pull/5816\r\n* Add tiny Qwen3.5 Think/NoThink fixture generation scripts by @aazizyan in https://github.com/huggingface/trl/pull/5819\r\n* Migrate tests to Qwen3.5 Think/NoThink fixtures by @aazizyan in https://github.com/huggingface/trl/pull/5821\r\n* Fix `OpenRewardSpec` omitting task‑scoped tools during rollout binding (fixes #5727) by @rycerzes in https://github.com/huggingface/trl/pull/5729\r\n* Add Qwen3.5 Think/NoThink training chat templates with generation markers by @aazizyan in https://github.com/huggingface/trl/pull/5824\r\n* Release: v1.5 by @qgallouedec in https://github.com/huggingface/trl/pull/5835\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.4.0...v1.5.0\r\n","publishedAt":"2026-05-25T15:33:19.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.5.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":14180,"contentTokens":4181,"composition":{"bugs":8,"features":4,"enhancements":5}},{"id":"rel_BrSDFutYy_LBOtbeHH_OH","version":"v1.4.0","type":"feature","title":"v1.4.0","summary":"A new `loss_type=\"chunked_nll\"` option for SFT drastically reduces peak activation memory by computing cross-entropy over tokens in checkpointed chunks instead of materializing the full `[batch × seq × vocab]` logits tensor, unlocking sequence lengths that previously caused out-of-memory errors. Also added OpenReward Standard environment adapter support, length-normalized DPO sigmoid loss, training chat templates for Cohere, Cohere2, Gemma 3, Qwen3, and Qwen2.5, and a training-invariance test suite to catch numerical drift across trainer configurations.","titleGenerated":"Hugging Face Fine-tuning v1.4.0 cuts SFT peak VRAM by 50% with chunked cross-entropy loss","titleShort":"Chunked cross-entropy loss cuts SFT VRAM by 50%","breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### Chunked cross-entropy loss for SFT (up to –50% VRAM)\r\n\r\n<img width=\"2704\" height=\"1455\" alt=\"chunked_loss_idea\" src=\"https://github.com/user-attachments/assets/3957f39b-3e71-4465-949a-22b2cf894d03\" />\r\n\r\nA new `loss_type=\"chunked_nll\"` option drastically reduces peak activation memory in SFT by avoiding the full `[batch × seq × vocab]` logits tensor. Ignored-label tokens are dropped before the `lm_head` matmul, and the cross-entropy is computed over the remaining tokens in checkpointed chunks (default `chunk_size=256`, the sweet spot consistent across model sizes and sequence lengths).\r\n\r\n```python\r\nfrom trl import SFTConfig, SFTTrainer\r\n\r\ntrainer = SFTTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=SFTConfig(loss_type=\"chunked_nll\"),\r\n    train_dataset=dataset,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nPeak GPU memory, AdamW fp32:\r\n\r\n| Model              | Hardware       | Seq    | `nll`     | `chunked_nll`             |\r\n| ------------------ | -------------- | ------ | --------- | ------------------------- |\r\n| Qwen3-1.7B + LoRA  | 1×H100 80GB    |   2048 | 47.9 GB   | **12.3 GB** (3.9× less)   |\r\n| Qwen3-4B           | 1×H100 80GB    | 16384  | **OOM**   | 63.8 GB                   |\r\n| Qwen3-14B          | 8×H100 FSDP2   | 16384  | 58.9 GB   | **38.9 GB** (1.5× less)   |\r\n| Qwen3-32B          | 8×H100 FSDP2   |   8192 | **OOM**   | 71.2 GB                   |\r\n\r\nEnd-to-end, chunked NLL is **consistently as fast or faster than `nll`** — and it unlocks sequence lengths that don't fit at all under the standard path.\r\n\r\nThe chunked path also supports **VLMs** (https://github.com/huggingface/trl/pull/5684).\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5575, https://github.com/huggingface/trl/pull/5676 and https://github.com/huggingface/trl/pull/5684\r\n\r\n### OpenReward Standard environment adapter (experimental)\r\n\r\nA new `trl.experimental.openreward` adapter plugs any environment speaking the [Open Reward Standard (ORS)](https://openrewardstandard.io) protocol into any TRL trainer accepting an `environment_factory` (`GRPOTrainer`, `AsyncGRPOTrainer`). One identifier wires all three trainer slots — `dataset`, `factory`, `reward_func`:\r\n\r\n```python\r\nfrom trl import GRPOConfig, GRPOTrainer\r\nfrom trl.experimental.openreward import OpenRewardEnv\r\n\r\nenv = OpenRewardEnv(\"Eigent/SETA\")  # or \"http://localhost:8000\"\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=GRPOConfig(...),\r\n    train_dataset=env.dataset,\r\n    environment_factory=env.factory,\r\n    reward_funcs=env.reward_func,\r\n)\r\n```\r\n\r\nTools are bound dynamically from JSON Schema at construction (no per-env wrapper code), and `env.dataset` autoderives task lists from the ORS task endpoints. The same code path works for envs hosted on the OpenReward platform, self-hosted on any container service, or running locally on `localhost`. A SETA training example is included.\r\n\r\nby @adithya-s-k in https://github.com/huggingface/trl/pull/5696\r\n\r\n### Training-invariance test suite\r\n\r\nUnit tests don't catch trainer-level numerical drift (gradient-accumulation normalization bugs, attention-impl divergence (eager ↔ FA2 / kernels)) they silently shift the loss trajectory and users only notice when their run no longer reproduces. (Cf. last year's [`transformers` grad-accum bug](https://github.com/huggingface/transformers/pull/34191), or the [\"We found two bugs in DeepSpeed\"](https://arxiv.org/abs/2604.23747) paper.)\r\n\r\nA new opt-in `pytest -m invariant` suite asserts the `loss` / `grad_norm` trajectory of short end-to-end SFT/DPO runs against committed reference snapshots, with **equivalence classes** for configs that should produce identical trajectories (e.g. `pdb=1, gas=8` ≡ default; eager ≡ FA2 ≡ kernels). Hardware-pinned to H100 80GB, real pretrained model, `full_determinism`, fixed seed. Initial coverage: 2 trainers × 2 invariance axes (grad-accum, attn-impl) × **gradient-checkpointing equivalence**.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5686, https://github.com/huggingface/trl/pull/5688 and https://github.com/huggingface/trl/pull/5689\r\n\r\n### MFU helpers\r\n\r\nThree new pure helpers in `trl.trainer.utils` for measuring training efficiency:\r\n\r\n- `compute_flops_per_token(config, seq_len)` — handles dense and MoE (Mixtral, Qwen3-MoE, DeepSeek-V2)\r\n- `compute_mfu(flops_per_token, tps, world_size, peak_flops)` — Model FLOPs Utilization as a percentage\r\n- `adjusted_mfu(mfu, config, seq_len)` — non-causal → causal-corrected (Llama / DS Ulysses convention)\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/5698\r\n\r\n### GRPO Liger kernel update (Liger 0.8.0)\r\n\r\nGRPO's Liger-kernel integration is updated for [Liger 0.8.0](https://github.com/linkedin/Liger-Kernel/releases/tag/v0.8.0): `delta` two-sided clipping, `use_bias_correction_kl`, and SAPO/VESPO parameters are now forwarded into `LigerFusedLinearGRPOLoss`. The previous `delta` + `use_liger_kernel` guard is removed — both can be combined.\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/5690\r\n\r\n### Length-normalized DPO sigmoid loss\r\n\r\nA new `loss_type=\"sigmoid_norm\"` option for `DPOConfig` implements the per-token (length-normalized) DPO loss used by Tülu 3 / OLMo (paper §5.1.2 eq. 6) to mitigate length bias.\r\n\r\n```python\r\nfrom trl import DPOConfig, DPOTrainer\r\n\r\ntrainer = DPOTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=DPOConfig(loss_type=\"sigmoid_norm\"),\r\n    train_dataset=dataset,\r\n)\r\n```\r\n\r\nby @BrownianNotion in https://github.com/huggingface/trl/pull/5406\r\n\r\n### Even more training chat templates\r\n\r\nFour more model families gain training-compatible chat templates with `{% generation %}` markers (assistant-only loss masking) and/or response schemas (tool-calling parsing):\r\n\r\n- **Cohere** training template by @dschulmeist in https://github.com/huggingface/trl/pull/5627\r\n- **Cohere2** `{% generation %}` markers by @qgallouedec in https://github.com/huggingface/trl/pull/5675\r\n- **Gemma 3** training template by @hwanython in https://github.com/huggingface/trl/pull/5685\r\n- **Qwen3-2507** training template by @SwayamInSync in https://github.com/huggingface/trl/pull/5574\r\n- **Qwen2.5** response schema by @aazizyan in https://github.com/huggingface/trl/pull/5728\r\n\r\n`get_training_chat_template` now also accepts a processor (not just a tokenizer) — useful for VLMs (https://github.com/huggingface/trl/pull/5560).\r\n\r\n### KTO ↔ DPO alignment: closing in on graduation\r\n\r\nAnother batch of alignment PRs this cycle. KTO and DPO are now structurally aligned across PEFT handling, model initialization, training-arg grouping, ref-logp precomputation, and metric handling — promotion of KTO out of `experimental` is imminent.\r\n\r\nPRs (all by @albertvillanova): #5659, #5660, #5661, #5679, #5701, #5702, #5703, #5704, #5705, #5714.\r\n\r\n### Other\r\n\r\n* Reject `parallelism_config` with `cp_size>1` or `sp_size>1` in GRPO/RLOO — fail fast at config init with a clear error instead of mid-training crash. By @kashif in https://github.com/huggingface/trl/pull/5699\r\n* Fail early for unsupported PEFT + Liger Kernel in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5709\r\n* Explicitly set `model_accepts_loss_kwargs=False` in DPO and Reward by @albertvillanova in https://github.com/huggingface/trl/pull/5710\r\n* Set `_tokenizer` attribute in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5566\r\n* Simplify `peft_config` handling in core / experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5673 and https://github.com/huggingface/trl/pull/5674\r\n* Replace `isinstance` with `is_peft_model` / drop redundant `is_peft_available` by @albertvillanova in https://github.com/huggingface/trl/pull/5682 and https://github.com/huggingface/trl/pull/5683\r\n* Reduce inconsistency across trainer test files by @qgallouedec in https://github.com/huggingface/trl/pull/5678\r\n* Refactor tiny-model generation scripts by @qgallouedec in https://github.com/huggingface/trl/pull/5637\r\n* Revert VLM support in `parse_response` by @qgallouedec in https://github.com/huggingface/trl/pull/5561\r\n\r\n## Fixes\r\n\r\n* **5 GB+ CUDA memory leak in activation offloading** — `OffloadActivations.__exit__` now syncs the compute/offload streams and clears the stash dictionaries, preventing orphaned offload tensors from leaking onto a dead stream (~0.2 GiB/step accumulation observed during QLoRA vision training before the fix). By @butterwecksolutions in https://github.com/huggingface/trl/pull/5694 and https://github.com/huggingface/trl/pull/5700\r\n* Fix reverse-KL server path NaN on variable completion length in `DistillationTrainer` by @k1064190 in https://github.com/huggingface/trl/pull/5594\r\n* `GKDTrainer`: fix `return_outputs` in the Liger kernel path by @roycho96 in https://github.com/huggingface/trl/pull/4688\r\n* `GKDTrainer`: fix seq-KD wasted teacher forward by @roycho96 in https://github.com/huggingface/trl/pull/5726\r\n* `GKDTrainer`: fix Liger fused JSD path computing wrong loss by @roycho96 in https://github.com/huggingface/trl/pull/5731\r\n* Fix missing PEFT validation when passing `peft_config` to core / experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5664 and https://github.com/huggingface/trl/pull/5665\r\n* Fix `peft_config` type hint in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5666\r\n* Fix discarded assertion message in trainer parameter checks by @qgallouedec in https://github.com/huggingface/trl/pull/5677\r\n* Fix typo in model name in README by @qgallouedec in https://github.com/huggingface/trl/pull/5711\r\n\r\n## Documentation and Examples\r\n\r\n* Upload testing suite for `DistillationTrainer` by @cmpatino in https://github.com/huggingface/trl/pull/5615\r\n\r\n## CI\r\n\r\n* Fix OOM in CI by reducing batch size in VLM SFT tests by @albertvillanova in https://github.com/huggingface/trl/pull/5687\r\n* Fix OOM in CI by reducing image size of tiny Gemma 3 model by @albertvillanova in https://github.com/huggingface/trl/pull/5680\r\n* Fix OOM in CI test reruns due to GPU memory leak from traceback frame locals by @albertvillanova in https://github.com/huggingface/trl/pull/5681\r\n* Add tiny `Qwen3-4B-Instruct-2507` by @qgallouedec in https://github.com/huggingface/trl/pull/5586\r\n* Align tiny Qwen3 MoE config with `Qwen/Qwen3-30B-A3B` by @qgallouedec in https://github.com/huggingface/trl/pull/5716\r\n* Fix GRPO VLM tests: multimodal training requires conversational prompts by @kaixuanliu in https://github.com/huggingface/trl/pull/5550\r\n* Regenerate invariance data + relax the tolerance by @qgallouedec in https://github.com/huggingface/trl/pull/5688\r\n\r\n## New Contributors\r\n* @dschulmeist made their first contribution in https://github.com/huggingface/trl/pull/5627\r\n* @k1064190 made their first contribution in https://github.com/huggingface/trl/pull/5594\r\n* @butterwecksolutions made their first contribution in https://github.com/huggingface/trl/pull/5694\r\n* @hwanython made their first contribution in https://github.com/huggingface/trl/pull/5685\r\n* @BrownianNotion made their first contribution in https://github.com/huggingface/trl/pull/5406\r\n* @adithya-s-k made their first contribution in https://github.com/huggingface/trl/pull/5696\r\n* @roycho96 made their first contribution in https://github.com/huggingface/trl/pull/4688\r\n* @aazizyan made their first contribution in https://github.com/huggingface/trl/pull/5728\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5648\r\n* Align KTO with DPO: Remove model_init parameter by @albertvillanova in https://github.com/huggingface/trl/pull/5659\r\n* Align KTO with DPO: Remove preprocess_logits_for_metrics parameter by @albertvillanova in https://github.com/huggingface/trl/pull/5660\r\n* Add tiny Qwen3-4B-Instruct-2507 by @qgallouedec in https://github.com/huggingface/trl/pull/5586\r\n* Chunked cross-entropy loss for SFT (up to –50% VRAM) by @qgallouedec in https://github.com/huggingface/trl/pull/5575\r\n* Fix missing PEFT validation when passing peft_config to core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5664\r\n* Fix missing PEFT availability check when passing peft_config to experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5665\r\n* Align KTO with DPO: Align PEFT handling by @albertvillanova in https://github.com/huggingface/trl/pull/5661\r\n* Set _tokenizer attribute in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5566\r\n* Fix peft_config type hint in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5666\r\n* Add Cohere training chat template by @dschulmeist in https://github.com/huggingface/trl/pull/5627\r\n* Simplify peft_config handling in core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5673\r\n* Simplify peft_config handling in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5674\r\n* fix(distillation): reverse-KL server path NaN on variable completion length by @k1064190 in https://github.com/huggingface/trl/pull/5594\r\n* Fix discarded assertion message in trainer parameter checks by @qgallouedec in https://github.com/huggingface/trl/pull/5677\r\n* Align KTO with DPO: Replace direct type check with is_peft_model by @albertvillanova in https://github.com/huggingface/trl/pull/5679\r\n* Remove redundant is_peft_available from core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5682\r\n* Replace isinstance with is_peft_model in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5683\r\n* Upload testing suite for `DistillationTrainer` by @cmpatino in https://github.com/huggingface/trl/pull/5615\r\n* Fix OOM in CI by reducing batch size in VLM SFT tests by @albertvillanova in https://github.com/huggingface/trl/pull/5687\r\n* Fix OOM in CI by reducing image size of tiny Gemma3 model by @albertvillanova in https://github.com/huggingface/trl/pull/5680\r\n* Fix OOM in CI test reruns due to GPU memory leak from traceback frame locals by @albertvillanova in https://github.com/huggingface/trl/pull/5681\r\n* Add training-invariance tests by @qgallouedec in https://github.com/huggingface/trl/pull/5686\r\n* Regenerate invariance data + relax the tolerance by @qgallouedec in https://github.com/huggingface/trl/pull/5688\r\n* fix: prevent RuntimeError crash in activation offloading for non-contiguous tensors by @butterwecksolutions in https://github.com/huggingface/trl/pull/5694\r\n* [GRPO] update Liger-kernel grpo loss (delta, vespo, KL bias correction) by @kashif in https://github.com/huggingface/trl/pull/5690\r\n* Extend invariant suite with gradient-checkpointing equivalence by @qgallouedec in https://github.com/huggingface/trl/pull/5689\r\n* Add Gemma 3 training chat template by @hwanython in https://github.com/huggingface/trl/pull/5685\r\n* Add `{% generation %}` markers for Cohere2 chat template by @qgallouedec in https://github.com/huggingface/trl/pull/5675\r\n* Add length-normalized sigmoid loss type to DPO trainer by @BrownianNotion in https://github.com/huggingface/trl/pull/5406\r\n* Add training chat template for Qwen3-2507 by @SwayamInSync in https://github.com/huggingface/trl/pull/5574\r\n* Align KTO with DPO: Remove enforcement of causal language models by @albertvillanova in https://github.com/huggingface/trl/pull/5701\r\n* Align KTO with DPO: Remove duplicate import of PreTrainedModel by @albertvillanova in https://github.com/huggingface/trl/pull/5702\r\n* Align KTO with DPO: Simplify max_length init logic by @albertvillanova in https://github.com/huggingface/trl/pull/5703\r\n* Align KTO with DPO: Group training arguments by @albertvillanova in https://github.com/huggingface/trl/pull/5704\r\n* Align KTO with DPO: Use _metrics attribute by @albertvillanova in https://github.com/huggingface/trl/pull/5705\r\n* Reduce inconsistency across trainer test files by @qgallouedec in https://github.com/huggingface/trl/pull/5678\r\n* Refactor tiny-model generation scripts by @qgallouedec in https://github.com/huggingface/trl/pull/5637\r\n* Accept processor in `get_training_chat_template` by @qgallouedec in https://github.com/huggingface/trl/pull/5560\r\n* Enable chunked NLL loss with PEFT in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/5676\r\n* Fix GRPO VLM tests: Multimodal training requires conversational prompts by @kaixuanliu in https://github.com/huggingface/trl/pull/5550\r\n* [experimental] Add OpenReward Standard environment adapter by @adithya-s-k in https://github.com/huggingface/trl/pull/5696\r\n* GKDTrainer: Fix return_outputs in Liger kernel path and update tests by @roycho96 in https://github.com/huggingface/trl/pull/4688\r\n* Reject parallelism_config with cp_size>1 or sp_size>1 in GRPO/RLOO by @kashif in https://github.com/huggingface/trl/pull/5699\r\n* Fix typo in model name in README by @qgallouedec in https://github.com/huggingface/trl/pull/5711\r\n* Explicitly set model_accepts_loss_kwargs=False in DPO and Reward by @albertvillanova in https://github.com/huggingface/trl/pull/5710\r\n* Fail early for unsupported PEFT + Liger Kernel in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5709\r\n* Revert VLM support in `parse_response` by @qgallouedec in https://github.com/huggingface/trl/pull/5561\r\n* Align KTO with DPO: Align _precompute_ref_logps by @albertvillanova in https://github.com/huggingface/trl/pull/5714\r\n* fix: prevent 5 GB+ CUDA memory leak in activation offloading by syncing streams and clear stashes in OffloadActivations.__exit__ by @butterwecksolutions in https://github.com/huggingface/trl/pull/5700\r\n* Align tiny Qwen3 MoE config with Qwen/Qwen3-30B-A3B by @qgallouedec in https://github.com/huggingface/trl/pull/5716\r\n* Add MFU helpers by @AmineDiro in https://github.com/huggingface/trl/pull/5698\r\n* [GKD] Fix seq kd wasted teacher forward by @roycho96 in https://github.com/huggingface/trl/pull/5726\r\n* Add Qwen2.5 response schema by @aazizyan in https://github.com/huggingface/trl/pull/5728\r\n* Enable chunked NLL loss with VLM in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/5684\r\n* [GKD] Fix Liger fused JSD path computing wrong loss by @roycho96 in https://github.com/huggingface/trl/pull/5731\r\n* Release: v1.4 by @qgallouedec in https://github.com/huggingface/trl/pull/5732\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.3.0...v1.4.0\r\n","publishedAt":"2026-05-09T00:01:36.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.4.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_aiaSofl7KZgttCpwD70W-","version":"v1.3.0","type":"feature","title":"v1.3.0","summary":"## Features\r\n\r\n### Qwen 3.6 integration\r\n\r\n<img width=\"1536\" height=\"1024\" alt=\"ChatGPT Image Apr 26, 2026 at 11_16_18 AM\" src=\"https://github.com/use...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### Qwen 3.6 integration\r\n\r\n<img width=\"1536\" height=\"1024\" alt=\"ChatGPT Image Apr 26, 2026 at 11_16_18 AM\" src=\"https://github.com/user-attachments/assets/789aad15-03b2-4ece-9828-d5c1dfed1f1e\" />\r\n\r\n\r\nTRL v1.3 ships training support for the new **Qwen 3.6** family (`Qwen/Qwen3.6-27B`, `Qwen/Qwen3.6-35B-A3B`). Qwen 3.6 reuses the `Qwen3_5Moe*` architecture but ships a slightly different chat template (adds a `preserve_thinking` flag, tweaks tool-arg stringification), so exact-string template matching needed updates across the stack.\r\n\r\nWhat landed:\r\n\r\n- **Chat templates**: `qwen3_6.jinja` (verbatim from upstream) and `qwen3_6_training.jinja` (prefix-preserving + `{% generation %}` markers for `assistant_only_loss=True`)\r\n- **Response schema**: routes to the existing `qwen3_5_schema` for tool-call parsing — output format unchanged\r\n- **Tiny test models** for VLM training: `tiny-Qwen3_5MoeForConditionalGeneration-3.6` (with MoE-specific shrinking)\r\n- **Test matrix** updated across SFT/DPO/GRPO/RLOO `test_(train|training)_vlm` cases\r\n\r\n```python\r\nfrom trl import SFTConfig, SFTTrainer\r\n\r\ntrainer = SFTTrainer(\r\n    model=\"Qwen/Qwen3.6-27B\",\r\n    args=SFTConfig(assistant_only_loss=True),  # works out of the box\r\n    train_dataset=dataset,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nTool-calling agent training also works end-to-end via the existing Qwen 3.5 response schema:\r\n\r\n```python\r\nfrom trl import GRPOConfig, GRPOTrainer\r\n\r\ndef multiply(a: int, b: int) -> int:\r\n    \"\"\"\r\n    Multiplies two integers.\r\n\r\n    Args:\r\n        a: The first integer.\r\n        b: The second integer.\r\n\r\n    Returns:\r\n        The product of the two integers.\r\n    \"\"\"\r\n    return a * b\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3.6-27B\",\r\n    reward_funcs=my_reward_fn,\r\n    args=GRPOConfig(...),\r\n    train_dataset=dataset,\r\n    tools=[multiply],\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5642\r\n\r\n### New experimental TPO trainer\r\n\r\n<img width=\"711\" height=\"177\" alt=\"Screenshot 2026-04-26 at 11 37 28 AM\" src=\"https://github.com/user-attachments/assets/6090212e-5c95-45c1-b137-87333d91daa6\" />\r\n\r\nA new experimental `TPOTrainer` implements [Triple Preference Optimization](https://huggingface.co/papers/2405.16681), which augments DPO with a `reference` (gold) completion alongside `chosen`/`rejected`. The paper reports +7-19 points over DPO/SimPO on Arena-Hard, MixEval-Hard, MMLU-Pro and GSM8K, with less data.\r\n\r\n```python\r\nfrom trl.experimental.tpo import TPOConfig, TPOTrainer\r\n\r\ntrainer = TPOTrainer(\r\n    model=\"Qwen/Qwen3-0.6B\",\r\n    args=TPOConfig(output_dir=\"Qwen3-0.6B-TPO\"),\r\n    train_dataset=load_dataset(\"tpo-alignment/triple-preference-ultrafeedback-40K\", split=\"train\"),\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/5506\r\n\r\n### Speculative decoding in `trl vllm-serve`\r\n\r\nA new `--speculative_config` JSON flag exposes vLLM's [speculative decoding](https://docs.vllm.ai/en/latest/features/spec_decode.html) directly through `trl vllm-serve` — works with native MTP heads (Qwen3 Next), Eagle3 drafts, etc. — without forking the serve script.\r\n\r\n```bash\r\n# Qwen3 native MTP (no extra draft model)\r\ntrl vllm-serve --model Qwen/Qwen3-Next-80B-A3B-Instruct \\\r\n    --speculative_config '{\"method\": \"qwen3_next_mtp\", \"num_speculative_tokens\": 5}'\r\n\r\n# Eagle3 draft model\r\ntrl vllm-serve --model Qwen/Qwen3-32B \\\r\n    --speculative_config '{\"model\": \"RedHatAI/Qwen3-32B-speculator.eagle3\", \"method\": \"eagle3\", \"num_speculative_tokens\": 3}'\r\n```\r\n\r\nby @Ofir408 in https://github.com/huggingface/trl/pull/5605\r\n\r\n### KTO ↔ DPO alignment: nearing the finish line\r\n\r\nTwelve more alignment PRs this cycle, bringing `KTOTrainer` and `DPOTrainer` essentially into structural parity. Notable shifts include moving completion assembly out of `_prepare_dataset` into a new `DataCollatorForKTO`, inlining the two-pass tokenization into a single pass, removing BOS/EOS handling, and supporting `IterableDataset` and dict `eval_dataset`. The goal — promoting **KTO out of experimental and into stable** — is now within reach for an upcoming release.\r\n\r\nPRs (all by @albertvillanova): #5582, #5578, #5579, #5583, #5587, #5599, #5601, #5600, #5606, #5612, #5632, #5635\r\n\r\n### More `{% generation %}` training chat templates\r\n\r\nThree more model families gain training-compatible chat templates with `{% generation %}` markers, so `assistant_only_loss=True` works out of the box:\r\n\r\n- **Gemma / Gemma 2** by @ps-abhi in https://github.com/huggingface/trl/pull/5523\r\n- **Phi-3** by @RudrenduPaul in https://github.com/huggingface/trl/pull/5526\r\n- **GLM-4-MoE** by @casinca in https://github.com/huggingface/trl/pull/5519\r\n\r\n### Other\r\n\r\n* Support processor in `maybe_apply_chat_template` by @albertvillanova in https://github.com/huggingface/trl/pull/5567\r\n* Support VLM processors in `is_chat_template_prefix_preserving` by @qgallouedec in https://github.com/huggingface/trl/pull/5558\r\n* Check prefix preservation at the **token** level (not string level) by @qgallouedec in https://github.com/huggingface/trl/pull/5559\r\n* Drop vLLM 0.11 support by @qgallouedec in https://github.com/huggingface/trl/pull/5549\r\n* Remove `forward_masked_logits` by @qgallouedec in https://github.com/huggingface/trl/pull/5626\r\n* Remove dead token attributes from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5565\r\n* Set `_tokenizer` as trainer attribute by @albertvillanova in https://github.com/huggingface/trl/pull/5489\r\n* Use `PreTrainedTokenizerBase` for tokenizer type hints by @qgallouedec in https://github.com/huggingface/trl/pull/5629\r\n* Renaming of internal variables: `async_reward_X` to `async_X` by @qgallouedec in https://github.com/huggingface/trl/pull/5616\r\n\r\n## Fixes\r\n\r\n* **Fix entropy calculation in SFT** — three bugs at once: misaligned by one position (next-token shift), averaged over the wrong tokens (used `attention_mask` instead of `label != -100`), and wrong cross-rank aggregation (unweighted mean instead of sum/count). The reported entropy under `completion_only_loss=True` and sequence parallelism is now correct. Same fix applied to DPO entropy logging. By @qgallouedec in https://github.com/huggingface/trl/pull/5620\r\n* Pass `AsyncGRPOTrainer`'s `processing_class` to `AsyncRolloutWorker` by @xuanduy04 in https://github.com/huggingface/trl/pull/5538\r\n* Fix `generate_tiny_models` for gpt-oss by @albertvillanova in https://github.com/huggingface/trl/pull/5622\r\n* Fix docstring style in vllm-serve script by @albertvillanova in https://github.com/huggingface/trl/pull/5628\r\n* Replace wrong comment about chat template with EOS by @albertvillanova in https://github.com/huggingface/trl/pull/5607\r\n\r\n## Documentation and Examples\r\n\r\n* Add chat templates page to web docs by @sergiopaniego in https://github.com/huggingface/trl/pull/5581\r\n* Update AsyncGRPO example with GSM8K and tested hyperparameters by @sergiopaniego in https://github.com/huggingface/trl/pull/5580\r\n* Update RapidFire AI integration with FSDP and multi-backend tracking by @kamran-rapidfireAI in https://github.com/huggingface/trl/pull/5618\r\n\r\n## CI\r\n\r\n* Add doc-builder style check to pre-commit and CI by @albertvillanova in https://github.com/huggingface/trl/pull/5630\r\n* Align and update doc-builder commit hash in CI GitHub Actions by @albertvillanova in https://github.com/huggingface/trl/pull/5631\r\n* Hotfix CI: Add ruff dependency to doc-builder style check by @albertvillanova in https://github.com/huggingface/trl/pull/5634\r\n* Fix CI with dev dependencies for Llava models by @albertvillanova in https://github.com/huggingface/trl/pull/5499\r\n* Add additional model parameters to `TestSupportsToolCalling` for improved coverage by @qgallouedec in https://github.com/huggingface/trl/pull/5537\r\n* Differentiate Phi-3 and Phi-3.5 in tests by @qgallouedec in https://github.com/huggingface/trl/pull/5546\r\n\r\n## New Contributors\r\n* @Ofir408 made their first contribution in https://github.com/huggingface/trl/pull/5605\r\n* @ps-abhi made their first contribution in https://github.com/huggingface/trl/pull/5523\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5577\r\n* Support processor in maybe_apply_chat_template by @albertvillanova in https://github.com/huggingface/trl/pull/5567\r\n* Remove dead token attributes from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5565\r\n* Support VLM processors in `is_chat_template_prefix_preserving` by @qgallouedec in https://github.com/huggingface/trl/pull/5558\r\n* Align KTO with DPO: Align add_model_tags by @albertvillanova in https://github.com/huggingface/trl/pull/5582\r\n* Align KTO with DPO: Align processing_class initialization by @albertvillanova in https://github.com/huggingface/trl/pull/5578\r\n* Align KTO with DPO: Align _prepare_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/5579\r\n* Align KTO with DPO: Align ref_model preparation for distributed training by @albertvillanova in https://github.com/huggingface/trl/pull/5583\r\n* Align KTO with DPO: Make conditional prompt extraction and unpairing in _prepare_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/5587\r\n* Update AsyncGRPO example with GSM8K and tested hyperparameters by @sergiopaniego in https://github.com/huggingface/trl/pull/5580\r\n* [docs] Add chat templates page to web docs by @sergiopaniego in https://github.com/huggingface/trl/pull/5581\r\n* Add additional model parameters to `TestSupportsToolCalling` for improved coverage by @qgallouedec in https://github.com/huggingface/trl/pull/5537\r\n* Fix CI with dev dependencies for Llava models by @albertvillanova in https://github.com/huggingface/trl/pull/5499\r\n* Differentiate Phi-3 and Phi-3.5 in tests by @qgallouedec in https://github.com/huggingface/trl/pull/5546\r\n* Set _tokenizer as trainer attribute by @albertvillanova in https://github.com/huggingface/trl/pull/5489\r\n* Align KTO with DPO: Support dict eval_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/5599\r\n* Align KTO with DPO: Align tokenization by @albertvillanova in https://github.com/huggingface/trl/pull/5601\r\n* Check prefix preservation at the token level by @qgallouedec in https://github.com/huggingface/trl/pull/5559\r\n* Replace wrong comment about chat template with EOS by @albertvillanova in https://github.com/huggingface/trl/pull/5607\r\n* Align KTO with DPO: Support IterableDataset by @albertvillanova in https://github.com/huggingface/trl/pull/5600\r\n* Drop vLLM 0.11 support by @qgallouedec in https://github.com/huggingface/trl/pull/5549\r\n* Align KTO with DPO: Remove maybe_apply_chat_template by @albertvillanova in https://github.com/huggingface/trl/pull/5606\r\n* [TPO] experimental TPO trainer by @kashif in https://github.com/huggingface/trl/pull/5506\r\n* fix: Pass AsyncGRPOTrainer's processing_class to AsyncRolloutWorker by @xuanduy04 in https://github.com/huggingface/trl/pull/5538\r\n* docs: update RapidFire AI integration with FSDP and multi-backend tracking by @kamran-rapidfireAI in https://github.com/huggingface/trl/pull/5618\r\n* Fix generate_tiny_models for gpt-oss by @albertvillanova in https://github.com/huggingface/trl/pull/5622\r\n* Added speculative_config to vllm-serve by @Ofir408 in https://github.com/huggingface/trl/pull/5605\r\n* feat(glm-4-moe): Add `{% generation %}` markers for training chat template by @casinca in https://github.com/huggingface/trl/pull/5519\r\n* Fix docstring style in vllm-serve script by @albertvillanova in https://github.com/huggingface/trl/pull/5628\r\n* feat: add Gemma/Gemma2 training chat templates with generation markers by @ps-abhi in https://github.com/huggingface/trl/pull/5523\r\n* Align KTO with DPO: Inline tokenization, new output format, DataCollatorForKTO by @albertvillanova in https://github.com/huggingface/trl/pull/5612\r\n* feat: add Phi-3 training chat template with generation markers by @RudrenduPaul in https://github.com/huggingface/trl/pull/5526\r\n* Remove `forward_masked_logits` by @qgallouedec in https://github.com/huggingface/trl/pull/5626\r\n* Use `PreTrainedTokenizerBase` for tokenizer type hints by @qgallouedec in https://github.com/huggingface/trl/pull/5629\r\n* Add doc-builder style check to pre-commit and CI by @albertvillanova in https://github.com/huggingface/trl/pull/5630\r\n* Align and update doc-builder commit hash in CI GitHub Actions by @albertvillanova in https://github.com/huggingface/trl/pull/5631\r\n* Align KTO with DPO: Move completion assembly from _prepare_dataset to data collator by @albertvillanova in https://github.com/huggingface/trl/pull/5632\r\n* Hotfix CI: Add ruff dependency to doc-builder style check by @albertvillanova in https://github.com/huggingface/trl/pull/5634\r\n* Fix entropy calculation in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/5620\r\n* Renaming of internal variables: `async_reward_X` to `async_X` by @qgallouedec in https://github.com/huggingface/trl/pull/5616\r\n* Align KTO with DPO: Remove BOS/EOS handling by @albertvillanova in https://github.com/huggingface/trl/pull/5635\r\n* Qwen3.6 integration by @qgallouedec in https://github.com/huggingface/trl/pull/5642\r\n* Release: v1.3 by @qgallouedec in https://github.com/huggingface/trl/pull/5647\r\n\r\n## New Contributors\r\n* @Ofir408 made their first contribution in https://github.com/huggingface/trl/pull/5605\r\n* @ps-abhi made their first contribution in https://github.com/huggingface/trl/pull/5523\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.2.0...v1.3.0\r\n","publishedAt":"2026-04-26T15:40:07.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.3.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_vSOkDQsjEJAbDUIKmPuqj","version":"v1.2.0","type":"feature","title":"v1.2.0","summary":"## Features\r\n\r\n### New `SSDTrainer` — Simple Self-Distillation\r\n\r\n<img width=\"778\" height=\"334\" alt=\"Screenshot 2026-04-16 at 9 08 04 PM\" src=\"https:/...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### New `SSDTrainer` — Simple Self-Distillation\r\n\r\n<img width=\"778\" height=\"334\" alt=\"Screenshot 2026-04-16 at 9 08 04 PM\" src=\"https://github.com/user-attachments/assets/8ca223f0-6740-48a8-967c-ec10cb262a93\" />\r\n\r\nA new experimental `SSDTrainer` implements the method described in [Embarrassingly Simple Self-Distillation Improves Code Generation](https://huggingface.co/papers/2604.01193). SSD samples completions from the model itself at a training-time temperature/truncation setting, then fine-tunes on those raw, unverified samples with standard cross-entropy loss. No reward model, verifier, teacher model, or RL: just prompts and the model.\r\n\r\n```python\r\nfrom datasets import Dataset\r\nfrom trl.experimental.ssd import SSDConfig, SSDTrainer\r\n\r\ndataset = Dataset.from_dict({\r\n    \"prompt\": [\r\n        [{\"role\": \"user\", \"content\": \"Write a function to add two numbers.\"}],\r\n        [{\"role\": \"user\", \"content\": \"Write a function to check if a number is prime.\"}],\r\n    ],\r\n})\r\n\r\ntrainer = SSDTrainer(\r\n    model=\"Qwen/Qwen3-4B-Instruct\",\r\n    args=SSDConfig(\r\n        output_dir=\"ssd-model\",\r\n        temperature=0.6,      # T_train from the paper\r\n        top_k=20,\r\n        top_p=0.95,\r\n        learning_rate=5e-6,\r\n    ),\r\n    train_dataset=dataset,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @kashif in https://github.com/huggingface/trl/pull/5505\r\n\r\n### Drop, don't truncate, overlong tool results in `GRPOTrainer`\r\n\r\nWhen tool calls produce more tokens than `max_completion_length` allows, `GRPOTrainer` now rolls back the tool messages/images added in the current iteration instead of trying to truncate them. This removes ~80 lines of fragile, image-boundary-aware bookkeeping in favor of a ~15-line snapshot-and-rollback. Since overlong samples almost always get rewarded as failures anyway, the learning signal is effectively unchanged — but the code is dramatically simpler and no longer needs per-VLM-family vision-token lookup tables.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5521\r\n\r\n### Expanded tool-calling model support: LLaMA 3.1 / 3.2 & DeepSeek-V3\r\n\r\nContinuing the effort from v1.1:\r\n\r\n- **LLaMA 3.1 and 3.2** tool-calling response schemas, with dedicated templates for identity matching. Note that these templates only support a single tool call and no content alongside the tool call — limitations inherited from the models' native templates. By @qgallouedec in https://github.com/huggingface/trl/pull/5518\r\n- **DeepSeek-V3** training chat template with `{% generation %}` markers, enabling assistant-only loss masking for `DeepSeek-V3` models. By @RudrenduPaul in https://github.com/huggingface/trl/pull/5527\r\n\r\nAs a result of a tightened detection (see fixes below), the list of templates reported as tool-calling capable is now correct — notably, the basic Llama 3 template is **no longer** falsely classified as tool-calling capable.\r\n\r\n### KTO/DPO alignment push\r\n\r\nA major cleanup sweep keeps `KTOTrainer` and `DPOTrainer` in lockstep, same initialization patterns, same config surface, same precompute behavior:\r\n\r\n- Add `precompute_ref_batch_size` to KTO (https://github.com/huggingface/trl/pull/5530)\r\n- Align `ref_model` initialization (https://github.com/huggingface/trl/pull/5534)\r\n- Align model initialization (https://github.com/huggingface/trl/pull/5533)\r\n- Support `None` args (https://github.com/huggingface/trl/pull/5531)\r\n- Remove `generate_during_eval` (https://github.com/huggingface/trl/pull/5551)\r\n- Remove model and ref adapter names (https://github.com/huggingface/trl/pull/5552)\r\n- Don't load `ref_model` when `precompute_ref_log_probs` is set in DPO/KTO (https://github.com/huggingface/trl/pull/5542)\r\n\r\nAll by @albertvillanova.\r\n\r\n### Other\r\n\r\n* Support messages with images in `prepare_multimodal_messages` by @albertvillanova in https://github.com/huggingface/trl/pull/5474\r\n* Simplify role handling in `prepare_multimodal_messages` by @albertvillanova in https://github.com/huggingface/trl/pull/5508\r\n* Update vLLM version support to 0.18.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5547\r\n\r\n## Fixes\r\n\r\n* Fix `supports_tool_calling` falsely accepting templates that drop assistant `tool_calls` by @qgallouedec in https://github.com/huggingface/trl/pull/5517\r\n* Fix `add_response_schema` for VLM processors — the schema was being set on the outer processor instead of the inner tokenizer, so it had no effect. This also collapses a handful of `__init__`/decode-gate workarounds. By @qgallouedec in https://github.com/huggingface/trl/pull/5520\r\n* Remove xfail condition for Gemma 4 response_schema regex bug by @qgallouedec in https://github.com/huggingface/trl/pull/5510\r\n* Remove unused dependencies for judges from dev requirements by @qgallouedec in https://github.com/huggingface/trl/pull/5515\r\n\r\n## Deprecations\r\n\r\n* **Deprecate `use_transformers_paged`** in `GRPOConfig` and `RLOOConfig` (and remove entirely from experimental `OnlineDPOConfig`, `GOLDConfig`, `SelfDistillationConfig`). Will be removed from the remaining configs in v2.0.0. In a small A/B benchmark (Qwen3-0.6B GRPO), the paged path is ~20% slower and uses ~6x more peak VRAM than the default; it's also superseded by `transformers` continuous batching. By @qgallouedec in https://github.com/huggingface/trl/pull/5544\r\n\r\n## Documentation and Examples\r\n\r\n* Add example script section to experimental trainer docs by @sergiopaniego in https://github.com/huggingface/trl/pull/5543\r\n* [Docs] Fix formatting in SSD training example script by @kashif in https://github.com/huggingface/trl/pull/5548\r\n* Nits in SSD docs by @sergiopaniego in https://github.com/huggingface/trl/pull/5554\r\n* [docs] Add LLaMA 3 / Qwen 2.5 entries to `chat_templates/README` by @qgallouedec in https://github.com/huggingface/trl/pull/5545\r\n* Update CARLA VLM example scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/5557\r\n\r\n## CI\r\n\r\n* Fix CI dependency installs to use a single resolve by @qgallouedec in https://github.com/huggingface/trl/pull/5513\r\n* Set upper transformers version to skip distributed test_rloo after fixed by @albertvillanova in https://github.com/huggingface/trl/pull/5535\r\n* Update tests with zero3 for RLOO and GRPO once fixed in transformers 5.5.4 by @albertvillanova in https://github.com/huggingface/trl/pull/5541\r\n* Bump doc-builder SHA for PR upload workflow by @rtrompier in https://github.com/huggingface/trl/pull/5553\r\n\r\n## What's Changed\r\n\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5525\r\n* Simplify role handling in prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5508\r\n* Fix CI dependency installs to use a single resolve by @qgallouedec in https://github.com/huggingface/trl/pull/5513\r\n* Fix `supports_tool_calling` falsely accepting templates that drop assistant `tool_calls` by @qgallouedec in https://github.com/huggingface/trl/pull/5517\r\n* feat: add DeepSeek-V3 training chat template with generation markers by @RudrenduPaul in https://github.com/huggingface/trl/pull/5527\r\n* Drop, don't truncate, overlong tool results in GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5521\r\n* Set upper transformers version to skip distributed test_rloo after fixed by @albertvillanova in https://github.com/huggingface/trl/pull/5535\r\n* Align KTO with DPO: Add precompute_ref_batch_size by @albertvillanova in https://github.com/huggingface/trl/pull/5530\r\n* Update tests with zero3 for RLOO and GRPO once fixed in transformers 5.5.4 by @albertvillanova in https://github.com/huggingface/trl/pull/5541\r\n* Align KTO with DPO: Align ref_model initialization by @albertvillanova in https://github.com/huggingface/trl/pull/5534\r\n* Align KTO with DPO: Align model initialization by @albertvillanova in https://github.com/huggingface/trl/pull/5533\r\n* Remove unused dependencies for judges from dev requirements by @qgallouedec in https://github.com/huggingface/trl/pull/5515\r\n* Remove xfail condition for Gemma4 response_schema regex bug by @qgallouedec in https://github.com/huggingface/trl/pull/5510\r\n* Align KTO with DPO: Support None args by @albertvillanova in https://github.com/huggingface/trl/pull/5531\r\n* Add example script section to experimental trainer docs by @sergiopaniego in https://github.com/huggingface/trl/pull/5543\r\n* [SSD] Added SSD trainer in experimental by @kashif in https://github.com/huggingface/trl/pull/5505\r\n* [Docs] Fix formatting in SSD training example script by @kashif in https://github.com/huggingface/trl/pull/5548\r\n* Don't load ref_model when precompute_ref_log_probs in DPO/KTO by @albertvillanova in https://github.com/huggingface/trl/pull/5542\r\n* chore: bump doc-builder SHA for PR upload workflow by @rtrompier in https://github.com/huggingface/trl/pull/5553\r\n* Nits is SSD docs by @sergiopaniego in https://github.com/huggingface/trl/pull/5554\r\n* Deprecate `use_transformers_paged` by @qgallouedec in https://github.com/huggingface/trl/pull/5544\r\n* Update vLLM version support to 0.18.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5547\r\n* Align KTO with DPO: Remove generate_during_eval by @albertvillanova in https://github.com/huggingface/trl/pull/5551\r\n* Align KTO with DPO: Remove model and ref adapter names by @albertvillanova in https://github.com/huggingface/trl/pull/5552\r\n* Support messages with images in prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5474\r\n* Update CARLA VLM example scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/5557\r\n* Fix `add_response_schema` for VLM processors by @qgallouedec in https://github.com/huggingface/trl/pull/5520\r\n* [docs] Add LLaMA 3 / Qwen 2.5 entries to `chat_templates/README` by @qgallouedec in https://github.com/huggingface/trl/pull/5545\r\n* Add LLaMA 3.1 and 3.2 tool calling support by @qgallouedec in https://github.com/huggingface/trl/pull/5518\r\n* Release: v1.2 by @qgallouedec in https://github.com/huggingface/trl/pull/5576\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.1.0...v1.2.0\r\n","publishedAt":"2026-04-17T01:13:05.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.2.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_Ozg1XLmUA2NW-hdYxL0RF","version":"v0.19.1","type":"feature","title":"v0.19.1","summary":"A small patch release containing these fixes:\r\n\r\n- #3161\r\n- #3165\r\n\r\n**Full Changelog**: https://github.com/huggingface/peft/compare/v0.19.0...v0.19.1","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"A small patch release containing these fixes:\r\n\r\n- #3161\r\n- #3165\r\n\r\n**Full Changelog**: https://github.com/huggingface/peft/compare/v0.19.0...v0.19.1","publishedAt":"2026-04-16T15:50:38.000Z","url":"https://github.com/huggingface/peft/releases/tag/v0.19.1","media":[],"prerelease":false,"source":{"slug":"peft","name":"PEFT","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_jZk5p9LTO1zAqiGEpZa5P","version":"v0.19.0","type":"feature","title":"v0.19.0","summary":"# Highlights\r\n\r\nThis PEFT release contains no less than nine new PEFT methods, described below. It also contains numerous enhancements that should mak...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"# Highlights\r\n\r\nThis PEFT release contains no less than nine new PEFT methods, described below. It also contains numerous enhancements that should make PEFT more useful to many users.\r\n\r\n<img width=\"1248\" height=\"560\" alt=\"peft-v0 19 0\" src=\"https://github.com/user-attachments/assets/f2878d0d-b1a1-46d0-9b61-55ab6097694c\" />\r\n\r\n## New Methods\r\n\r\n### GraLoRA\r\n\r\n@yeonjoon-jung01 added [\"GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning\"](https://arxiv.org/abs/2505.20355) to PEFT (#2851). This method subdivides the base weight into smaller blocks and applies LoRA to those. This more granular adaptation promises to increase expressiveness and improve performance, especially at higher ranks (64+), closing the gap to full fine-tuning.\r\n\r\n### BD-LoRA\r\n\r\n@Conzel contributed BD-LoRA: [\"Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving\"](https://openreview.net/forum?id=1cjLvtFOmL) (#2895). With BD-LoRA, the LoRA weights are implemented in a block-diagonal way. This allows to reduce communication overhead when using tensor parallelism (TP) and thus faster serving.\r\n\r\nThere is an experiment branch for BD-LoRA support in vLLM: vllm-project/vllm#28136.\r\n\r\n### Cartridges\r\n\r\nThanks to @kashif, PEFT now also supports [Cartridges](https://arxiv.org/abs/2506.06266) (#2953). The main purpose of this method is to train a prefix to [compress a long context to a short size](https://hazyresearch.stanford.edu/blog/2025-06-08-cartridges) and thus save on tokens. On a low level, this is similar to [prefix tuning](https://huggingface.co/docs/peft/package_reference/prefix_tuning). The PR also added an [example recipe](https://github.com/huggingface/peft/tree/main/examples/cartridge_self_study) to quickly get started.\r\n\r\n### PVeRA\r\n\r\n[\"PVeRA: Probabilistic Vector-Based Random Matrix Adaptation\"](https://arxiv.org/abs/2512.07703) was added to PEFT by @leofillioux in #2952. It is an extension of [VeRA](https://huggingface.co/docs/peft/package_reference/vera), a PEFT method that uses weight sharing between layers to be especially parameter efficient. PVeRA builds on top of that by adding a probabilistic element, sampling from the shared parameters and promising better performance overall.\r\n\r\n### PSOFT\r\n\r\n@fei407 added PSOFT, [\"Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation\"](https://openreview.net/forum?id=FSHrinMArK), to PEFT in #3037. Orthogonal fine-tuning techniques like [OFT](https://huggingface.co/docs/peft/package_reference/oft) and [BOFT](https://huggingface.co/docs/peft/package_reference/boft) are good at preserving the structure and thus capabilities of the underlying base model. PSOFT improves efficiency of this technique by constraining the adaptation to low-rank principal subspace.\r\n\r\n### Lily\r\n\r\n@yibozhong added Lily: [\"Low-Rank Interconnected Adaptation across Layers\"](https://arxiv.org/abs/2407.09946) to PEFT in #2563. Lily is on the surface similar to LoRA but has a sophisticated parameter sharing scheme. The A parameters are shared blockwise (e.g. 4 consecutive q_proj layers share the same A). There is a pool of B parameters that is shared globally, the actual B's are chosen in a data-dependent way through a router. This allows Lily to use higher ranks than LoRA while maintaining a low trainable parameter count.\r\n\r\n### PEANuT\r\n\r\nIn #3084, [\"PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers\"](https://arxiv.org/abs/2410.01870) was added to PEFT, again by @yibozhong. PEANuT adds a small, neural net (so called weight-aware neural tweakers) to the base model. Compared to LoRA, this increases expressivity for the same trainable parameter count or allows to greatly lower the parameter count without sacrificing expressivity. This comes at the expensive of a higher memory requirement for the same parameter count and decreased speed.\r\n\r\n### TinyLoRA\r\n\r\nWe have another serial contributor in @kashif, who also contributed [TinyLoRA: \"Learning to Reason in 13 Parameters\"](https://arxiv.org/abs/2602.04118) in #3024. This is a PEFT method that allows to train an extremely small number of parameters, much lower than what could be achieved even with LoRA rank 1. The paper shows that in particular with reinforcement learning, it can often be enough to train just a few parameters to achieve good results.\r\n\r\n### AdaMSS\r\n\r\n@LonglongaaaGo added [\"AdaMSS: Adaptive Multi-Subspace Approach for Parameter-Efficient Fine-Tuning\"](https://neurips.cc/virtual/2025/loc/san-diego/poster/119606) to PEFT. This method segments the base weights of the model into smaller subspaces that are targeted for fine-tuning. Moreover, it's possible to dynamically assign a lower parameter budget to less important subspaces during training, similar to what [AdaLoRA](https://huggingface.co/docs/peft/package_reference/adalora) does. This promises to provide higher expressiveness and better generalization than similar PEFT methods.\r\n\r\n## Enhancements\r\n\r\n### Convert non-LoRA adapters to LoRA\r\n\r\nIn #2939, we added functions to PEFT to allow [converting checkpoints of many non-LoRA methods into LoRA checkpoints](https://huggingface.co/docs/peft/main/en/package_reference/lora_conversion). This can be useful because many other packages support only LoRA but not other PEFT methods, e.g. [Diffusers](https://huggingface.co/docs/diffusers/v0.37.1/en/api/loaders/lora) and [vLLM](https://docs.vllm.ai/en/latest/features/lora/). With the new conversions tools, more PEFT methods than just LoRA can thus be used with those packages. Conversion is lossy but empirical testing showed that with a sufficiently high LoRA rank, the error can be quite low.\r\n\r\n### LoRA-GA\r\n\r\n@sambhavnoobcoder added a new way to initialize LoRA weights with [\"LoRA-GA: Low-Rank Adaptation with Gradient Approximation\"](https://arxiv.org/abs/2407.05000) (#2926). This allows you to initialize the LoRA weights in a way that aligns the gradients with full fine-tuning and should lead to faster training convergence.\r\n\r\n### Reducing intruder dimensions\r\n\r\nIn [\"LoRA vs Full Fine-tuning: An Illusion of Equivalence\"](https://huggingface.co/papers/2410.21228), the authors showed that LoRA fine-tuning can introduce so-called \"intruder dimensions\" which contribute to forgetting. We now have a [utility function to remove intruder dimension](https://huggingface.co/docs/peft/main/en/developer_guides/lora#recovering-base-model-performance-via-intruder-dimension-reduction) in PEFT, `reduce_intruder_dimension`. When calling this on a fine-tuned LoRA model, forgetting should be reduced while the fine-tuned task performance should remain almost the same.\r\n\r\n### Transformer Engine\r\n\r\nIn #3048, @balvisio added support for [Transformer Engine](https://github.com/NVIDIA/TransformerEngine), a quantization method by NVIDIA, to PEFT.\r\n\r\n### Tensor Parallel Support\r\n\r\nIn a series of PRs (#3079, #3091, #3096), @michaelbenayoun added support for [Tensor Parallelism](https://huggingface.co/docs/transformers/v5.4.0/en/perf_infer_gpu_multi#tensor-parallelism) to LoRA.\r\n\r\n### Weight tying improvements\r\n\r\nIn many LLMs, the embedding and the LM head have tied weights to save on parameter count. This can, however, lead to tricky situations when trying to fine-tune those layers. Through a series of PRs (#2803, #2922, #2870, #2879, #3126), we improved the user experience when doing so. Most notably, users can now pass `ensure_weight_tying=True` to their PEFT config to force weight tying to be upheld. Please check the [PEFT weight tying docs](https://huggingface.co/docs/peft/main/en/developer_guides/lora#weight-tying) for how weight tying is now being handled. Thanks to @romitjain, @sambhavnoobcoder, and @Cursx for their contributions.\r\n\r\n### Low precsion floating type support\r\n\r\n#3055 makes LoRA work with base models that use very low precision floats like `torch.float8_e4m3fn`. An example of that would be MiniMax-M2.5.\r\n\r\n### Zero init for PrefixTuning\r\n\r\n#3128 introduces zero init to Prefix Tuning which, according to our benchmarks, reduced the result variance significantly and yielded good task accuracy without the need for prompt engineering.\r\n\r\n### LoftQ + int8 quantization\r\n\r\nWith #3088 the LoftQ implementation now supports correcting errors for int8 quantization without utilizing activation thresholding alongside the already existing nf4 quantization.\r\n\r\n## Changes\r\n\r\n### Removal of Bone\r\n\r\nThe Bone PEFT method was removed in #3115. Users are directed to use [MiSS](https://huggingface.co/docs/peft/package_reference/miss) instead, which is the improved replacement for Bone. Use this [Bone-to-MiSS conversion script](https://github.com/huggingface/peft/blob/main/scripts/convert-bone-to-miss.py) if you want to port old Bone checkpoints.\r\n\r\n### AutoGPTQ and AutoAWQ\r\n\r\nThese two quantization methods now use [GPTQModel](https://github.com/ModelCloud/GPTQModel) as their backend (#2932) thanks to @ZX-ModelCloud.\r\n\r\n### Handling of `requires_grad` in `modules_to_save`\r\n\r\nPreviously, PEFT would enable `requires_grad` on the original module if the corresponding `modules_to_save` was disabled. This is almost never desirable and was thus fixed. Although this change is technically backwards-incompatible, it's an extreme niche case, so we don't expect any user to be negatively affected by it.\r\n\r\n# All Changes\r\n\r\n* FIX SFT example (8bit quant, trl) by @BenjaminBossan in https://github.com/huggingface/peft/pull/2857\r\n* TST Add GPU training tests for p-tuning & prefix tuning by @BenjaminBossan in https://github.com/huggingface/peft/pull/2844\r\n* CHORE: Bump Python version in pyproject.toml by @BenjaminBossan in https://github.com/huggingface/peft/pull/2865\r\n* MNT: Clean up unused method set_auxiliary_adapters by @BenjaminBossan in https://github.com/huggingface/peft/pull/2876\r\n* ENH: Improve MetaMath training script runtime by @BenjaminBossan in https://github.com/huggingface/peft/pull/2894\r\n* CI: Install fbgemm package needed by torchao, update test by @BenjaminBossan in https://github.com/huggingface/peft/pull/2887\r\n* Resolve #2431: Remove macos-13 from tests by @githubnemo in https://github.com/huggingface/peft/pull/2906\r\n* FEAT add GraLoRA by @yeonjoon-jung01 in https://github.com/huggingface/peft/pull/2851\r\n* TST FIX: Issue with pickle models and caching by @BenjaminBossan in https://github.com/huggingface/peft/pull/2913\r\n* Bump version to 0.18.1.dev0 after release by @BenjaminBossan in https://github.com/huggingface/peft/pull/2910\r\n* FIX Move further models to safetensors by @BenjaminBossan in https://github.com/huggingface/peft/pull/2920\r\n* Add Marian to author list by @BenjaminBossan in https://github.com/huggingface/peft/pull/2909\r\n* FIX Load quantized weights with PEFT mixed model by @BenjaminBossan in https://github.com/huggingface/peft/pull/2915\r\n* FIX Bug when merging negatively weighted adapters by @BenjaminBossan in https://github.com/huggingface/peft/pull/2918\r\n* FIX Beam search w/ mixed adapter batches & encoder by @BenjaminBossan in https://github.com/huggingface/peft/pull/2921\r\n* Deal with weight tying in transformers >=5 by @githubnemo in https://github.com/huggingface/peft/pull/2922\r\n* Fix caching for LoRA parametrizations on nn.Parameter by @jonnyli1125 in https://github.com/huggingface/peft/pull/2912\r\n* MetaMath: Add forgetting metric by @BenjaminBossan in https://github.com/huggingface/peft/pull/2925\r\n* Fix EETQ GPU Docker image build by @githubnemo in https://github.com/huggingface/peft/pull/2935\r\n* FIX Transformers v5 fixes by @BenjaminBossan in https://github.com/huggingface/peft/pull/2934\r\n* ENH: Improve torch.compile support in MetaMath by @BenjaminBossan in https://github.com/huggingface/peft/pull/2900\r\n* TST: Clean up testing by @BenjaminBossan in https://github.com/huggingface/peft/pull/2846\r\n* FIX: Some GPU tests failing due to transformers v5 by @BenjaminBossan in https://github.com/huggingface/peft/pull/2937\r\n* FIX Don't set requires_grad on original module by @BenjaminBossan in https://github.com/huggingface/peft/pull/2936\r\n* [FEAT] Integrate BD-LoRA into PEFT by @Conzel in https://github.com/huggingface/peft/pull/2895\r\n* Test cleaning pytest caches by @githubnemo in https://github.com/huggingface/peft/pull/2938\r\n* FIX Migrate method comparison space to Gradio 6 by @BenjaminBossan in https://github.com/huggingface/peft/pull/2947\r\n* TST Remove unnecessary PREFIXES constant by @BenjaminBossan in https://github.com/huggingface/peft/pull/2942\r\n* TST: Remove unnecessary prefix tuning dtype test by @BenjaminBossan in https://github.com/huggingface/peft/pull/2955\r\n* detect if torch.distributed is available by @vladmandic in https://github.com/huggingface/peft/pull/2963\r\n* FIX: Inject from state dict into compiled model by @BenjaminBossan in https://github.com/huggingface/peft/pull/2962\r\n* FIX: Correct adapter dtype with bnb weights by @BenjaminBossan in https://github.com/huggingface/peft/pull/2893\r\n* CI For transformers main tests, clear disk space by @BenjaminBossan in https://github.com/huggingface/peft/pull/2956\r\n* Add cartridges to PEFT by @kashif in https://github.com/huggingface/peft/pull/2953\r\n* fix oft gptq forward by @jiqing-feng in https://github.com/huggingface/peft/pull/2978\r\n* FIX Don't implicitly require transformers v4.52 by @BenjaminBossan in https://github.com/huggingface/peft/pull/2976\r\n* FEAT Add function to convert non-LoRA PEFT adapters to LoRA by @BenjaminBossan in https://github.com/huggingface/peft/pull/2939\r\n* FIX Bug in how forgetting metric treats padding by @BenjaminBossan in https://github.com/huggingface/peft/pull/2986\r\n* Upgrade GitHub Actions to latest versions by @salmanmkc in https://github.com/huggingface/peft/pull/2966\r\n* Implement ensure_weight_tying for trainable_token_indices (#2864) by @sambhavnoobcoder in https://github.com/huggingface/peft/pull/2870\r\n* fix device map check by @jiqing-feng in https://github.com/huggingface/peft/pull/2979\r\n* Updated MetaMathQA results by @githubnemo in https://github.com/huggingface/peft/pull/2984\r\n* add Intel XPU platform support for cartridge_self_study example by @kaixuanliu in https://github.com/huggingface/peft/pull/2990\r\n* Add Conv1D support to CoRDA for GPT-2 compatibility #2991 by @sambhavnoobcoder in https://github.com/huggingface/peft/pull/2992\r\n* Update prompt_based_methods.md - remove eval_preds by @maerory in https://github.com/huggingface/peft/pull/2994\r\n* Bump version to 0.18.2.dev0 after release by @BenjaminBossan in https://github.com/huggingface/peft/pull/2985\r\n* DOC Prefix tuning for encoder-decoder models by @BenjaminBossan in https://github.com/huggingface/peft/pull/2989\r\n* TST Remove tests that are completely skipped by @BenjaminBossan in https://github.com/huggingface/peft/pull/2965\r\n* TST Fix tolerance issue with GPT-OSS and transformers v5 by @BenjaminBossan in https://github.com/huggingface/peft/pull/2982\r\n* TST Small clean up regarding weight initialization by @BenjaminBossan in https://github.com/huggingface/peft/pull/2961\r\n* ENH Cache DoRA weight norm for inference by @BenjaminBossan in https://github.com/huggingface/peft/pull/2661\r\n* Add OSF continual learning example by @NikhilNayak-debug in https://github.com/huggingface/peft/pull/2897\r\n* LoRA-GA Integration by @sambhavnoobcoder in https://github.com/huggingface/peft/pull/2926\r\n* FIX Don't warn about unknown layer type when using target parameters by @BenjaminBossan in https://github.com/huggingface/peft/pull/2997\r\n* lower tol for specific test by @jiqing-feng in https://github.com/huggingface/peft/pull/2996\r\n* Refactor layer initialization to use PEFT config directly by @BenjaminBossan in https://github.com/huggingface/peft/pull/2960\r\n* CI: Add FSDP tests on multi GPU machine by @BenjaminBossan in https://github.com/huggingface/peft/pull/2856\r\n* Bugfix turned into restructuring by @githubnemo in https://github.com/huggingface/peft/pull/3003\r\n* Intruder dimension reduction for LoRA by @githubnemo in https://github.com/huggingface/peft/pull/2999\r\n* [LoRA] Document support for effective rank for LoRA on MOE experts by @kashif in https://github.com/huggingface/peft/pull/3007\r\n* Fix fbgemm exception in nightly CI by @githubnemo in https://github.com/huggingface/peft/pull/3010\r\n* Ignore BPE errors in tests by @githubnemo in https://github.com/huggingface/peft/pull/3011\r\n* Fix initialization bug introduced in #2960 by @githubnemo in https://github.com/huggingface/peft/pull/3006\r\n* Fully deprecate AutoGPTQ and AutoAWQ for GPT-QModel by @ZX-ModelCloud in https://github.com/huggingface/peft/pull/2932\r\n* Fix two issues introduced in AutoGPTQ deprecation by @githubnemo in https://github.com/huggingface/peft/pull/3014\r\n* Fix docker GPU build for gptqmodel by @githubnemo in https://github.com/huggingface/peft/pull/3018\r\n* Fix docker CPU build by @githubnemo in https://github.com/huggingface/peft/pull/3023\r\n* nightly-gpu: Make sure that all steps are run by @githubnemo in https://github.com/huggingface/peft/pull/3030\r\n* Fix various test errors in the single GPU case by @githubnemo in https://github.com/huggingface/peft/pull/3031\r\n* FIX: warmup_ratio deprecated (fixes #2949) by @shantanugupta2004 in https://github.com/huggingface/peft/pull/2950\r\n* Upgrade GitHub Actions for Node 24 compatibility by @salmanmkc in https://github.com/huggingface/peft/pull/3008\r\n* Improve LoftQ documentation by @githubnemo in https://github.com/huggingface/peft/pull/3041\r\n* `no_split_modules` now captures values recursively by @githubnemo in https://github.com/huggingface/peft/pull/3032\r\n* FIX Issue with disable adapter test by @BenjaminBossan in https://github.com/huggingface/peft/pull/3045\r\n* Fix error of PEFT with disable adapters and FSDP by @Isalia20 in https://github.com/huggingface/peft/pull/3001\r\n* Add Dependabot configuration for GitHub Actions by @salmanmkc in https://github.com/huggingface/peft/pull/3040\r\n* Bump the ci-actions group with 2 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3060\r\n* Bump the third-party-actions group with 7 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3061\r\n* Integration of PVeRA by @leofillioux in https://github.com/huggingface/peft/pull/2952\r\n* FIX OPT regression test after dtype change from v5 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3053\r\n* CI: Dependabot PRs don't trigger unit tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/3062\r\n* CHORE: Remove deprecated Bone method by @BenjaminBossan in https://github.com/huggingface/peft/pull/3051\r\n* FIX Multiple failing nightly GPU tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/3052\r\n* TST Improve speed of X-LoRA, Eva, and Poly tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/3046\r\n* FIX Syntax error in test workflow file by @BenjaminBossan in https://github.com/huggingface/peft/pull/3065\r\n* ENH: Tie weights for target_modules in Lora (#2864) by @romitjain in https://github.com/huggingface/peft/pull/2879\r\n* FIX Two transformers warnings when generating in MetaMath train script by @BenjaminBossan in https://github.com/huggingface/peft/pull/3064\r\n* FIX Flaky X-LoRA test after adding caching by @BenjaminBossan in https://github.com/huggingface/peft/pull/3068\r\n* fix: clean up peft_config from model on unload() and merge_and_unload() by @zamal-db in https://github.com/huggingface/peft/pull/3067\r\n* Add PSOFT tuner implementation by @fei407 in https://github.com/huggingface/peft/pull/3037\r\n* FEAT: add more generic device support for pvera by @kaixuanliu in https://github.com/huggingface/peft/pull/3074\r\n* Add support for LoRA with Transformer Engine by @balvisio in https://github.com/huggingface/peft/pull/3048\r\n* Fix adalora layer init refactor 2960 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3070\r\n* CHORE: Bump 3rd party GH actions by @BenjaminBossan in https://github.com/huggingface/peft/pull/3076\r\n* Fix: Respect `inference_mode` when setting adapters with `modules_to_save` (Issue #2928) by @ada-ggf25 in https://github.com/huggingface/peft/pull/2931\r\n* [Lily] implementations for peft integration by @yibozhong in https://github.com/huggingface/peft/pull/3036\r\n* [feature] Tiny modification to enable OFT for finetuning embedding layers by @zqiu24 in https://github.com/huggingface/peft/pull/3005\r\n* FIX Error with PSOFT fp16/bf16 on GPU by @BenjaminBossan in https://github.com/huggingface/peft/pull/3087\r\n* Ensure that te.pytorch exists by @githubnemo in https://github.com/huggingface/peft/pull/3081\r\n* FIX Add guard when detecting the optimum version by @BenjaminBossan in https://github.com/huggingface/peft/pull/3042\r\n* Partial fix for LoftQ + int8 quantization by @githubnemo in https://github.com/huggingface/peft/pull/3088\r\n* Update contributing guidelines regarding typos by @githubnemo in https://github.com/huggingface/peft/pull/3094\r\n* FIX Distributed training tests and extend them by @BenjaminBossan in https://github.com/huggingface/peft/pull/3092\r\n* [AdaLoRA] fix update_layer api by @kashif in https://github.com/huggingface/peft/pull/3099\r\n* [FEAT] Add PEANuT to peft by @yibozhong in https://github.com/huggingface/peft/pull/3084\r\n* FIX GraLoRA: Use its own target module mapping by @BenjaminBossan in https://github.com/huggingface/peft/pull/3105\r\n* docs: replace deprecated financial_phrasebank dataset in IA3 tutorial by @dhruvildarji in https://github.com/huggingface/peft/pull/3058\r\n* LoRA and Transformers TP by @michaelbenayoun in https://github.com/huggingface/peft/pull/3079\r\n* CHORE Remove deprecated Bone experiments by @BenjaminBossan in https://github.com/huggingface/peft/pull/3115\r\n* Embeddings LoRA & TP by @michaelbenayoun in https://github.com/huggingface/peft/pull/3091\r\n* Improve DeloRA: add config validation, dedicated tests, and fix typos by @joshuaswanson in https://github.com/huggingface/peft/pull/3097\r\n* DOC Improve LoRA conversion docs by @BenjaminBossan in https://github.com/huggingface/peft/pull/3118\r\n* FIX Deal with missing attribute on model config by @BenjaminBossan in https://github.com/huggingface/peft/pull/3109\r\n* CHORE Zizmor: branch protection for GH workflows by @BenjaminBossan in https://github.com/huggingface/peft/pull/3103\r\n* miss update by @Joluck in https://github.com/huggingface/peft/pull/3122\r\n* FIX Broken tests with torchao >= 0.15 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3101\r\n* Bump actions/cache from 5.0.3 to 5.0.4 in the ci-actions group by @dependabot[bot] in https://github.com/huggingface/peft/pull/3124\r\n* Changes for transformers 5 weight conversion by @BenjaminBossan in https://github.com/huggingface/peft/pull/3083\r\n* [TinyLoRA]tinylora implementation by @kashif in https://github.com/huggingface/peft/pull/3024\r\n* FIX Cache position is None with transformers v5.4 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3120\r\n* FIX Issues with transformer weight conversion code by @BenjaminBossan in https://github.com/huggingface/peft/pull/3127\r\n* Add AdaMSS tuner with Adaptive Subspace Allocation (ASA) by @LonglongaaaGo in https://github.com/huggingface/peft/pull/2987\r\n* CI FIX Some tests require torchvision by @BenjaminBossan in https://github.com/huggingface/peft/pull/3135\r\n* Add zero init support in Prefix Tuning by @githubnemo in https://github.com/huggingface/peft/pull/3128\r\n* DOC Update contribution guidelines by @BenjaminBossan in https://github.com/huggingface/peft/pull/3119\r\n* Fix save_pretrained writing incorrect tie_word_embeddings=True config after PEFT merge by @Cursx in https://github.com/huggingface/peft/pull/3126\r\n* Enable XPU support for GPTQ tests in PEFT by @jiqing-feng in https://github.com/huggingface/peft/pull/3137\r\n* DOC: Info about runtime performance of LoRA on MoE by @BenjaminBossan in https://github.com/huggingface/peft/pull/3138\r\n* Remove references to torchao's AffineQuantizedTensor by @andrewor14 in https://github.com/huggingface/peft/pull/3140\r\n* Enh add default target modules for gemma4 and tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/3136\r\n* DOC: Section on weight tying with LoRA by @BenjaminBossan in https://github.com/huggingface/peft/pull/3066\r\n* FIX CI Remove invalid arg in nightly GPU test call by @BenjaminBossan in https://github.com/huggingface/peft/pull/3104\r\n* CI Move slow EVA tests to nightly GPU CI by @BenjaminBossan in https://github.com/huggingface/peft/pull/3108\r\n* enable arrow xpu tests by @jiqing-feng in https://github.com/huggingface/peft/pull/3145\r\n* Save checkpoint with TP by @michaelbenayoun in https://github.com/huggingface/peft/pull/3096\r\n* Bump the third-party-actions group with 8 updates by @dependabot[bot] in https://github.com/huggingface/peft/pull/3125\r\n* Fix DARE rescaling no-op in random_pruning by @Chessing234 in https://github.com/huggingface/peft/pull/3152\r\n* ENH Support models with low precision float dtypes by @BenjaminBossan in https://github.com/huggingface/peft/pull/3055\r\n* FIX Explicit weight conversion map for Mixtral by @BenjaminBossan in https://github.com/huggingface/peft/pull/3146\r\n* Release 0.19.0 by @BenjaminBossan in https://github.com/huggingface/peft/pull/3155\r\n\r\n## New Contributors\r\n* @yeonjoon-jung01 made their first contribution in https://github.com/huggingface/peft/pull/2851\r\n* @jonnyli1125 made their first contribution in https://github.com/huggingface/peft/pull/2912\r\n* @Conzel made their first contribution in https://github.com/huggingface/peft/pull/2895\r\n* @vladmandic made their first contribution in https://github.com/huggingface/peft/pull/2963\r\n* @salmanmkc made their first contribution in https://github.com/huggingface/peft/pull/2966\r\n* @maerory made their first contribution in https://github.com/huggingface/peft/pull/2994\r\n* @ZX-ModelCloud made their first contribution in https://github.com/huggingface/peft/pull/2932\r\n* @Isalia20 made their first contribution in https://github.com/huggingface/peft/pull/3001\r\n* @dependabot[bot] made their first contribution in https://github.com/huggingface/peft/pull/3060\r\n* @leofillioux made their first contribution in https://github.com/huggingface/peft/pull/2952\r\n* @zamal-db made their first contribution in https://github.com/huggingface/peft/pull/3067\r\n* @fei407 made their first contribution in https://github.com/huggingface/peft/pull/3037\r\n* @balvisio made their first contribution in https://github.com/huggingface/peft/pull/3048\r\n* @ada-ggf25 made their first contribution in https://github.com/huggingface/peft/pull/2931\r\n* @yibozhong made their first contribution in https://github.com/huggingface/peft/pull/3036\r\n* @dhruvildarji made their first contribution in https://github.com/huggingface/peft/pull/3058\r\n* @michaelbenayoun made their first contribution in https://github.com/huggingface/peft/pull/3079\r\n* @joshuaswanson made their first contribution in https://github.com/huggingface/peft/pull/3097\r\n* @LonglongaaaGo made their first contribution in https://github.com/huggingface/peft/pull/2987\r\n* @Cursx made their first contribution in https://github.com/huggingface/peft/pull/3126\r\n* @andrewor14 made their first contribution in https://github.com/huggingface/peft/pull/3140\r\n* @Chessing234 made their first contribution in https://github.com/huggingface/peft/pull/3152\r\n\r\n**Full Changelog**: https://github.com/huggingface/peft/compare/v0.18.1...v0.19.0","publishedAt":"2026-04-14T14:05:11.000Z","url":"https://github.com/huggingface/peft/releases/tag/v0.19.0","media":[],"prerelease":false,"source":{"slug":"peft","name":"PEFT","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_-_Lo5QLOolB09xyuoL6kA","version":"v1.1.0","type":"feature","title":"v1.1.0","summary":"## Features\r\n\r\n### `DistillationTrainer` for efficient on-policy distillation\r\n\r\nRead the blog post: https://huggingface.co/spaces/HuggingFaceTB/trl-d...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### `DistillationTrainer` for efficient on-policy distillation\r\n\r\nRead the blog post: https://huggingface.co/spaces/HuggingFaceTB/trl-distillation-trainer\r\n\r\n![off_vs_on_policy_distillation yePX-mwe_1umXK5](https://github.com/user-attachments/assets/73b4d47c-adea-440e-ab39-9c5df40b9915)\r\n\r\nThe new `DistillationTrainer` implements on-policy knowledge distillation as described in [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://huggingface.co/papers/2306.13649). It extends the ideas from the `GKDTrainer` with three key optimizations: a **generation buffer** that decouples the training microbatch size from the generation batch size (up to 40x speedup), **external teacher server support** so the teacher doesn't need to fit on training GPUs, and **binary-encoded logprob payloads** that shrink transfer payloads by ~5x.\r\n\r\n```python\r\nfrom datasets import load_dataset\r\nfrom trl.experimental.distillation import DistillationConfig, DistillationTrainer\r\n\r\ndataset = load_dataset(\"openai/gsm8k\", \"main\", split=\"train\")\r\ndataset = dataset.map(\r\n    lambda x: {\"messages\": [{\"role\": \"user\", \"content\": x[\"question\"]}]},\r\n    remove_columns=dataset.column_names,\r\n)\r\n\r\ntrainer = DistillationTrainer(\r\n    model=\"Qwen/Qwen2.5-1.5B-Instruct\",\r\n    teacher_model=\"Qwen/Qwen2.5-7B-Instruct\",\r\n    args=DistillationConfig(\r\n        output_dir=\"results/distill-qwen-gsm8k\",\r\n        lmbda=1.0,                   # fully on-policy (student generates)\r\n        beta=1.0,                    # reverse KL\r\n        teacher_model_init_kwargs={\"torch_dtype\": \"bfloat16\"},\r\n    ),\r\n    train_dataset=dataset,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @cmpatino in https://github.com/huggingface/trl/pull/5407, https://github.com/huggingface/trl/pull/5500 and https://github.com/huggingface/trl/pull/5501\r\n\r\n### Chunked LM head for memory-efficient log-prob computation in `AsyncGRPOTrainer`\r\n\r\n`AsyncGRPOTrainer` now supports a chunked LM-head path that computes per-token log-probs and entropy via online `logsumexp` without materializing the full `[N, V]` logits tensor. Combined with `completion_mask` filtering to skip prompt tokens, this brings massive memory savings on long sequences — up to **44x** lower peak-allocated memory on an 8192-token sequence:\r\n\r\n| `chunk_lm_head_size` | Peak Alloc (GB) | Reduction | Wall Time (ms) |\r\n| :--- | :--- | :--- | :--- |\r\n| `None` (baseline) | 18.55 | 1.00x | 808.7 |\r\n| `4096` | 0.42 | **44.32x** | 459.0 |\r\n| `8192` | 0.76 | **24.34x** | 393.0 |\r\n\r\nEnable it via the new `chunk_lm_head_size` option in `AsyncGRPOConfig`:\r\n\r\n```python\r\nfrom trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer\r\n\r\ntrainer = AsyncGRPOTrainer(\r\n    model=\"Qwen/Qwen2.5-0.5B-Instruct\",\r\n    args=AsyncGRPOConfig(chunk_lm_head_size=4096),\r\n    ...\r\n)\r\n```\r\n\r\nNote: mutually exclusive with `use_liger_kernel` (both replace the LM head forward pass).\r\n\r\nby @AmineDiro in https://github.com/huggingface/trl/pull/5349\r\n\r\n### `{% generation %}` support in training chat templates\r\n\r\nSFT with `assistant_only_loss=True` requires chat templates to include `{% generation %}` / `{% endgeneration %}` markers so that `return_assistant_tokens_mask=True` produces correct masks. Very few models ship these markers natively, so users hit a cryptic error when enabling assistant-only loss with models like Qwen3, Llama 3 or GPT-OSS.\r\n\r\n`SFTTrainer` now automatically swaps in a patched *training chat template* when the original template lacks generation markers — no manual template surgery required. Training templates are shipped for **Qwen2.5**, **Qwen3**, **Llama 3** and **GPT-OSS**, stored as standalone `.jinja` files under `trl/chat_templates/` for readability, diffability, and editor syntax highlighting.\r\n\r\n```python\r\nfrom trl import SFTConfig, SFTTrainer\r\n\r\ntrainer = SFTTrainer(\r\n    model=\"Qwen/Qwen3-4B\",\r\n    args=SFTConfig(assistant_only_loss=True),  # now just works\r\n    train_dataset=dataset,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5459, https://github.com/huggingface/trl/pull/5470, by @RudrenduPaul in https://github.com/huggingface/trl/pull/5493 and https://github.com/huggingface/trl/pull/5522, and by @casinca in https://github.com/huggingface/trl/pull/5484\r\n\r\n### Expanded tool-calling model support\r\n\r\nAgent training now supports a broader family of models via native tool-call response schemas:\r\n\r\n- **GPT-OSS** (https://github.com/huggingface/trl/pull/5464)\r\n- **GLM-4-MoE** (https://github.com/huggingface/trl/pull/5463)\r\n- **Qwen3-VL** (https://github.com/huggingface/trl/pull/5469)\r\n- **Gemma 4** — the first model to natively ship a response schema (https://github.com/huggingface/trl/pull/5454)\r\n\r\nA new `supports_tool_calling()` utility detects whether a tokenizer/processor can render a full tool-calling turn, and `GRPOTrainer` now validates tool support at initialization — raising a clear error upfront instead of failing cryptically mid-training.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5462, https://github.com/huggingface/trl/pull/5464, https://github.com/huggingface/trl/pull/5463, https://github.com/huggingface/trl/pull/5469 and https://github.com/huggingface/trl/pull/5454\r\n\r\n### Multimodal tool responses for VLM training\r\n\r\n`environment_factory` tool methods can now return multimodal content blocks (images + text) for VLM training. Previously, tool responses were always converted to `str(result)`, discarding any visual information. Now tools can return content block lists with images, and the trainer handles them end-to-end through tokenization, generation, and the forward pass — including correct `pixel_values` plumbing.\r\n\r\n```python\r\nclass ScreenshotEnv:\r\n    def take_screenshot(self) -> list[dict]:\r\n        return [\r\n            {\"type\": \"image\", \"image\": self.browser.screenshot()},\r\n            {\"type\": \"text\", \"text\": \"Current page state\"},\r\n        ]\r\n```\r\n\r\nThe OpenEnv `browsergym.py` example has been migrated to this pattern, and a new `carla_vlm.py` example demonstrates VLM training against CARLA with camera-image tool responses.\r\n\r\nby @sergiopaniego in https://github.com/huggingface/trl/pull/5323 and https://github.com/huggingface/trl/pull/5437, and by @qgallouedec in https://github.com/huggingface/trl/pull/5448\r\n\r\n### Built-in reward functions now log extra columns\r\n\r\n`accuracy_reward` and `reasoning_accuracy_reward` now emit extra diagnostic columns (`solution`, `gold_parsed`, `answer_parsed`) via the `log_extra` callback introduced in v1.0.0. These show up in the rich completions table, making it much easier to debug why a reward was (or wasn't) assigned.\r\n\r\n<img width=\"1627\" alt=\"accuracy reward with extra columns\" src=\"https://github.com/user-attachments/assets/d7f6e9c2-4d7b-4886-ba7a-f58f0ccfcb9b\" />\r\n\r\n```python\r\nfrom datasets import load_dataset\r\nfrom trl import GRPOConfig, GRPOTrainer\r\nfrom trl.rewards import accuracy_reward\r\n\r\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen2-0.5B-Instruct\",\r\n    reward_funcs=accuracy_reward,\r\n    args=GRPOConfig(log_completions=True),\r\n    train_dataset=dataset,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5308\r\n\r\n### Other\r\n\r\n* Align KTO with DPO: precompute reference log probs at init by @albertvillanova in https://github.com/huggingface/trl/pull/5447\r\n* Align KTO with DPO: reorganize `KTOConfig` by @albertvillanova in https://github.com/huggingface/trl/pull/5477\r\n* Use generic VLM key passthrough in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5468\r\n* Make images optional in `prepare_multimodal_messages` by @albertvillanova in https://github.com/huggingface/trl/pull/5424\r\n* Avoid image deepcopy in `prepare_multimodal_messages` by @albertvillanova in https://github.com/huggingface/trl/pull/5475\r\n* Replace `pixel_position_ids` with `image_position_ids` for Gemma 4 support by @qgallouedec in https://github.com/huggingface/trl/pull/5452\r\n* Update vLLM minimum supported version to 0.11.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5443\r\n* Remove dead token attributes from trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5483\r\n* Remove unnecessary `isinstance(part, dict)` checks in image extraction by @qgallouedec in https://github.com/huggingface/trl/pull/5439\r\n* Simplify `_get_tool_suffix_ids` by @qgallouedec in https://github.com/huggingface/trl/pull/5440\r\n* Narrow prefix-preserving check to the actual requirement by @qgallouedec in https://github.com/huggingface/trl/pull/5458\r\n* Better test consistency RLOO vs GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/5396\r\n* Remove duplicated `prepare_deepspeed` by @albertvillanova in https://github.com/huggingface/trl/pull/5414\r\n\r\n## Fixes\r\n\r\n* Fix targeting fused parameters with LoRA by @BenjaminBossan in https://github.com/huggingface/trl/pull/5430\r\n* Fix `ImportError` with vllm-0.10.2 in OnlineDPO and OpenEnv by @albertvillanova in https://github.com/huggingface/trl/pull/5423\r\n* Fix `_get_per_token_logps_and_entropies` return type by @kashif in https://github.com/huggingface/trl/pull/5456\r\n* Fix SFT deprecation warning by @albertvillanova in https://github.com/huggingface/trl/pull/5466\r\n* Fix broken validation of user-specified tokens by @albertvillanova in https://github.com/huggingface/trl/pull/5482\r\n* Fix `prepare_multimodal_messages` not normalizing empty string content for assistant/tool roles by @albertvillanova in https://github.com/huggingface/trl/pull/5496\r\n* Remove redundant alignment of `pad_token_id` by @albertvillanova in https://github.com/huggingface/trl/pull/5487\r\n* Replace deprecated `huggingface-cli` references with `hf` by @hanouticelina in https://github.com/huggingface/trl/pull/5486\r\n* Remove unused `truncation_mode` from experimental `truncate_dataset` by @albertvillanova in https://github.com/huggingface/trl/pull/5467\r\n* Fix PR template check bot reopen loop by @qgallouedec in https://github.com/huggingface/trl/pull/5488\r\n\r\n## Deprecations and Removals\r\n\r\n* **Deprecate `keep_end` truncation mode** in `DPOConfig` and `SFTConfig` — will be removed in v2.0.0. Use `keep_start` instead. By @albertvillanova in https://github.com/huggingface/trl/pull/5465\r\n* **Deprecate `pad_token` config parameter** in `DPOConfig`, `SFTConfig`, and `RewardConfig` — will be removed in v2.0.0. Set `tokenizer.pad_token` directly on the `processing_class` instead. By @albertvillanova in https://github.com/huggingface/trl/pull/5480\r\n* **Remove `trl.experimental.judges` module and all judge support from trainers.** Judges were experimental, unused in practice, and `llm-blender` (backing `PairRMJudge`) was unmaintained and incompatible with transformers v5 — actively blocking v5 adoption. Everything judges did can be achieved with `reward_funcs`. `OnlineDPOTrainer`, `NashMDTrainer`, and `XPOTrainer` are now unified on reward-model scoring only. By @qgallouedec in https://github.com/huggingface/trl/pull/5485\r\n\r\n## Documentation and Examples\r\n\r\n* Update \"What's New\": TRL v1 blog post by @qgallouedec in https://github.com/huggingface/trl/pull/5385\r\n* New `carla_vlm` OpenEnv example by @sergiopaniego in https://github.com/huggingface/trl/pull/5437\r\n* Add code example for `completion_only_loss` in SFT trainer docs by @RudrenduPaul in https://github.com/huggingface/trl/pull/5494\r\n* Add docs and good defaults for `DistillationTrainer` by @cmpatino in https://github.com/huggingface/trl/pull/5500\r\n* Add test and docs for multimodal tool responses by @qgallouedec in https://github.com/huggingface/trl/pull/5448\r\n* Add tests for Gemma pixel splitting by @qgallouedec in https://github.com/huggingface/trl/pull/5450\r\n* Update docstring about tool messages in `prepare_multimodal_messages` by @albertvillanova in https://github.com/huggingface/trl/pull/5476\r\n* Run `make precommit` to fix docstring style by @albertvillanova in https://github.com/huggingface/trl/pull/5436\r\n\r\n## CI\r\n\r\n* Pin GitHub Actions to commit SHAs by @paulinebm in https://github.com/huggingface/trl/pull/5435\r\n* Update GitHub Action to use specific version of github-script by @qgallouedec in https://github.com/huggingface/trl/pull/5491\r\n* Generic device support for CI tests by @kaixuanliu in https://github.com/huggingface/trl/pull/5357\r\n* CI: Gemma 4 support by @qgallouedec in https://github.com/huggingface/trl/pull/5453\r\n* Fix CI slow-tests cannot remove: No such file or directory by @albertvillanova in https://github.com/huggingface/trl/pull/5401\r\n* Remove xfail for Qwen3VL CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/5402\r\n* Fix flaky CI `test_rloo[fsdp2]`: replace non-deterministic xfail with skipif for transformers 5.4.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5403\r\n* Mark as strict the xfail tests with zero3 for RLOO and GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/5404\r\n* Hotfix CI: mark tests as xfail due to missing `input_ids` or `inputs_embeds` by @albertvillanova in https://github.com/huggingface/trl/pull/5422\r\n* Update tests to not pass `eval_strategy` by @SunMarc in https://github.com/huggingface/trl/pull/5426\r\n* Hotfix CI: mark tests as xfail with transformers dev due to `TypeError: 'NoneType' object is not iterable` by @albertvillanova in https://github.com/huggingface/trl/pull/5427\r\n* Revert hotfix CI for `TypeError: 'NoneType' object is not iterable` by @albertvillanova in https://github.com/huggingface/trl/pull/5438\r\n* Update tests with zero3 for RLOO and GRPO as xfail only with transformers >= v5 by @albertvillanova in https://github.com/huggingface/trl/pull/5420\r\n* Hotfix CI: update skipif for `test_rloo[fsdp2]` after transformers 5.5.0 release by @albertvillanova in https://github.com/huggingface/trl/pull/5442\r\n* Remove xfail for ZeRO 2 and 3 + SFT + PEFT test by @qgallouedec in https://github.com/huggingface/trl/pull/5383\r\n* Hotfix CI: mark tests as xfail with transformers dev for Llava models by @albertvillanova in https://github.com/huggingface/trl/pull/5504\r\n* Restrict VLM padding workaround to transformers 5.3.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5503\r\n\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5410\r\n* Update \"What's New\": TRL v1 blog post by @qgallouedec in https://github.com/huggingface/trl/pull/5385\r\n* Fix CI slow-tests cannot remove: No such file or directory by @albertvillanova in https://github.com/huggingface/trl/pull/5401\r\n* Remove xfail for Qwen3VL CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/5402\r\n* Fix flaky CI test_rloo[fsdp2]: Replace non-deterministic xfail with skipif for transformers 5.4.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5403\r\n* Mark as strict the xfail tests with zero3 for RLOO and GRPO by @albertvillanova in https://github.com/huggingface/trl/pull/5404\r\n* Remove duplicated prepare_deepspeed by @albertvillanova in https://github.com/huggingface/trl/pull/5414\r\n* Hotfix CI: Mark tests as xfail due to missing input_ids or inputs_embeds by @albertvillanova in https://github.com/huggingface/trl/pull/5422\r\n* Update tests to not pass `eval_strategy` by @SunMarc in https://github.com/huggingface/trl/pull/5426\r\n* Hotfix CI: Mark tests as xfail with transformers dev due to TypeError: 'NoneType' object is not iterable by @albertvillanova in https://github.com/huggingface/trl/pull/5427\r\n* FIX CI: Targeting fused parameters with LoRA by @BenjaminBossan in https://github.com/huggingface/trl/pull/5430\r\n* Support multimodal tool responses in `environment_factory` for VLM training by @sergiopaniego in https://github.com/huggingface/trl/pull/5323\r\n* 🔒 Pin GitHub Actions to commit SHAs by @paulinebm in https://github.com/huggingface/trl/pull/5435\r\n* New carla vlm example by @sergiopaniego in https://github.com/huggingface/trl/pull/5437\r\n* Revert hotfix CI for TypeError: 'NoneType' object is not iterable by @albertvillanova in https://github.com/huggingface/trl/pull/5438\r\n* Run make precommit to fix docstring style by @albertvillanova in https://github.com/huggingface/trl/pull/5436\r\n* Fix ImportError with vllm-0.10.2 in OnlineDPO and OpenEnv by @albertvillanova in https://github.com/huggingface/trl/pull/5423\r\n* Add chunked LM head for memory-efficient log-prob computation  for AsyncGRPOTrainer by @AmineDiro in https://github.com/huggingface/trl/pull/5349\r\n* Update tests with zero3 for RLOO and GRPO as xfail only with transformers >= v5 by @albertvillanova in https://github.com/huggingface/trl/pull/5420\r\n* Make images optional in prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5424\r\n* Hotfix CI: Update skipif for test_rloo[fsdp2] after transformers 5.5.0 release by @albertvillanova in https://github.com/huggingface/trl/pull/5442\r\n* Update vLLM minimum supported version to 0.11.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5443\r\n* Better test consistency RLOO vs GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/5396\r\n* Align KTO with DPO: Precompute reference log probs at init by @albertvillanova in https://github.com/huggingface/trl/pull/5447\r\n* Add support for logging extra columns in reward functions and update related tests by @qgallouedec in https://github.com/huggingface/trl/pull/5308\r\n* Remove unnecessary `isinstance(part, dict)` checks in image extraction by @qgallouedec in https://github.com/huggingface/trl/pull/5439\r\n* Remove xfail for ZeRO 2 and 3 + SFT + PEFT test by @qgallouedec in https://github.com/huggingface/trl/pull/5383\r\n* Replace `pixel_position_ids` with `image_position_ids` for Gemma4 support by @qgallouedec in https://github.com/huggingface/trl/pull/5452\r\n* Add test and docs for multimodal tool responses by @qgallouedec in https://github.com/huggingface/trl/pull/5448\r\n* Add tests for Gemma pixel splitting by @qgallouedec in https://github.com/huggingface/trl/pull/5450\r\n* Generic device support for CI tests by @kaixuanliu in https://github.com/huggingface/trl/pull/5357\r\n* Revert speculative argument parsing and add Gemma4 agent support by @qgallouedec in https://github.com/huggingface/trl/pull/5454\r\n* fix _get_per_token_logps_and_entropies return type by @kashif in https://github.com/huggingface/trl/pull/5456\r\n* Deprecate keep_end truncation mode by @albertvillanova in https://github.com/huggingface/trl/pull/5465\r\n* Fix SFT deprecation warning by @albertvillanova in https://github.com/huggingface/trl/pull/5466\r\n* Remove unused truncation_mode from experimental truncate_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/5467\r\n* Use generic VLM key passthrough in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5468\r\n* Narrow prefix-preserving check to the actual requirement by @qgallouedec in https://github.com/huggingface/trl/pull/5458\r\n* Simplify `_get_tool_suffix_ids` by @qgallouedec in https://github.com/huggingface/trl/pull/5440\r\n* Update docstring about tool messages in prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5476\r\n* CI Gemma 4 support by @qgallouedec in https://github.com/huggingface/trl/pull/5453\r\n* Move chat templates from inline strings to `.jinja` files by @qgallouedec in https://github.com/huggingface/trl/pull/5459\r\n* Align KTO with DPO: Reorganize KTOConfig by @albertvillanova in https://github.com/huggingface/trl/pull/5477\r\n* Add `supports_tool_calling` utility and validate tool support at init by @qgallouedec in https://github.com/huggingface/trl/pull/5462\r\n* Add GPT-OSS tool calling support by @qgallouedec in https://github.com/huggingface/trl/pull/5464\r\n* Add `{% generation %}` support to training chat templates by @qgallouedec in https://github.com/huggingface/trl/pull/5470\r\n* Avoid image deepcopy in prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5475\r\n* Remove dead token attributes from trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5483\r\n* Add `DistillationTrainer` for efficient on-policy distillation by @cmpatino in https://github.com/huggingface/trl/pull/5407\r\n* Replace deprecated `huggingface-cli` references with `hf` by @hanouticelina in https://github.com/huggingface/trl/pull/5486\r\n* Fix broken validation of user-specified tokens by @albertvillanova in https://github.com/huggingface/trl/pull/5482\r\n* Deprecate pad_token config parameter by @albertvillanova in https://github.com/huggingface/trl/pull/5480\r\n* Remove redundant alignment of pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/5487\r\n* Fix PR template check bot reopen loop by @qgallouedec in https://github.com/huggingface/trl/pull/5488\r\n* feat(gpt-oss): Add `{% generation %}` markers for training chat template by @casinca in https://github.com/huggingface/trl/pull/5484\r\n* Remove the `trl.experimental.judges` module and all judge support from trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5485\r\n* Hotfix CI: Mark tests as xfail with transformers dev for Llava models by @albertvillanova in https://github.com/huggingface/trl/pull/5504\r\n* Restrict VLM padding workaround to transformers 5.3.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5503\r\n* Update GitHub Action to use specific version of github-script by @qgallouedec in https://github.com/huggingface/trl/pull/5491\r\n* [docs] Add code example for completion_only_loss in SFT trainer docs by @RudrenduPaul in https://github.com/huggingface/trl/pull/5494\r\n* Fix prepare_multimodal_messages not normalizing empty string content for assistant/tool roles by @albertvillanova in https://github.com/huggingface/trl/pull/5496\r\n* Add trackio support to `DistillationTrainer` by @cmpatino in https://github.com/huggingface/trl/pull/5501\r\n* feat: add Llama 3 training chat template with generation markers by @RudrenduPaul in https://github.com/huggingface/trl/pull/5493\r\n* Add GLM-4-MoE tool calling support by @qgallouedec in https://github.com/huggingface/trl/pull/5463\r\n* Add Qwen3-VL tool calling support by @qgallouedec in https://github.com/huggingface/trl/pull/5469\r\n* Add docs and good defaults for `DistillationTrainer` by @cmpatino in https://github.com/huggingface/trl/pull/5500\r\n* feat: add Qwen2.5 training chat template with generation markers by @RudrenduPaul in https://github.com/huggingface/trl/pull/5522\r\n* Release: v1.1 by @qgallouedec in https://github.com/huggingface/trl/pull/5524\r\n\r\n## New Contributors\r\n* @BenjaminBossan made their first contribution in https://github.com/huggingface/trl/pull/5430\r\n* @hanouticelina made their first contribution in https://github.com/huggingface/trl/pull/5486\r\n* @RudrenduPaul made their first contribution in https://github.com/huggingface/trl/pull/5494\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v1.0.0...v1.1.0","publishedAt":"2026-04-12T02:15:56.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.1.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_p4jBIBiR7l1UL1yoc7ZSF","version":"v1.0.0","type":"feature","title":"v1.0.0","summary":"<img width=\"1800\" height=\"1013\" alt=\"thumbnail-2\" src=\"https://github.com/user-attachments/assets/5c55b86a-0600-4f70-bf37-41ab240af851\" />\r\n\r\nRead our...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"<img width=\"1800\" height=\"1013\" alt=\"thumbnail-2\" src=\"https://github.com/user-attachments/assets/5c55b86a-0600-4f70-bf37-41ab240af851\" />\r\n\r\nRead our [blog post](https://hf.co/blog/trl-v1) for an overview of TRL v1.\r\n\r\n## Features\r\n\r\n### Asynchronous GRPO\r\n\r\nAsynchronous GRPO decouples generation from the gradient update loop by offloading rollouts to an external vLLM server. Generation runs in parallel while training continues, eliminating idle GPU time and improving hardware utilization.\r\n\r\n```python\r\nfrom trl.experimental.async_grpo import AsyncGRPOTrainer\r\nfrom trl.rewards import accuracy_reward\r\nfrom datasets import load_dataset\r\n\r\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\r\n\r\ntrainer = AsyncGRPOTrainer(\r\n    model=\"Qwen/Qwen2.5-0.5B-Instruct\",\r\n    reward_funcs=accuracy_reward,\r\n    train_dataset=dataset,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5293\r\n\r\n### Variational Sequence-Level Soft Policy Optimization (VESPO)\r\n\r\n<img width=\"465\" height=\"279\" alt=\"Screenshot 2026-03-20 at 5 49 50 PM\" src=\"https://github.com/user-attachments/assets/b60c9697-6eb7-498e-95b3-df78c367f5fa\" />\r\n\r\n[VESPO](https://huggingface.co/papers/2602.10693) addresses training instability in off-policy RL caused by policy staleness, asynchronous updates, and train-inference mismatches. Rather than relying on heuristic token-level clipping (GRPO) or sequence-length normalization (GSPO), VESPO derives a principled reshaping kernel from a variational framework. In practice, this yields a smooth, asymmetric Gamma weighting function that gracefully suppresses extreme sequence-level importance weights without introducing length bias. It can be enabled via the `loss_type` parameter of `GRPOConfig`:\r\n\r\n```python\r\nfrom trl import GRPOConfig, GRPOTrainer\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-0.6B\",\r\n    args=GRPOConfig(loss_type=\"vespo\"),\r\n    ...\r\n)\r\n```\r\n\r\nby @casinca in https://github.com/huggingface/trl/pull/5199\r\n\r\n### Divergence Proximal Policy Optimization (DPPO)\r\n\r\n<img width=\"3180\" height=\"1187\" alt=\"z_TXYw37xZqsQ21YiDkYL\" src=\"https://github.com/user-attachments/assets/40f1d538-82b3-4097-91c6-119ea9f7797b\" />\r\n<img width=\"1189\" height=\"490\" alt=\"SfgWotuuuRKPkg-0bxWv1\" src=\"https://github.com/user-attachments/assets/2b090df3-0bfb-42e4-9f94-15943736e689\" />\r\n\r\n[DPPO](https://huggingface.co/papers/2602.04879) is a new experimental trainer that replaces the standard PPO clipping mechanism with divergence constraints, providing more principled trust-region updates.\r\n\r\nby @LeonEricsson in https://github.com/huggingface/trl/pull/5117\r\n\r\n### Self-Distillation Policy Optimization (SDPO)\r\n\r\n[SDPO](https://huggingface.co/papers/2601.20802) is a new experimental trainer that augments on-policy RL with self-distillation from the model's own high-reward trajectories. Instead of using an external teacher, SDPO treats the current model conditioned on feedback as a self-teacher, distilling its feedback-informed predictions back into the policy.\r\n\r\n```python\r\nfrom trl.experimental import SDPOTrainer, SDPOConfig\r\n\r\nconfig = SDPOConfig(\r\n    output_dir=\"./results\",\r\n    num_generations=8,\r\n    success_reward_threshold=1.0,\r\n    use_successful_as_teacher=True,\r\n)\r\n\r\ntrainer = SDPOTrainer(\r\n    model=\"Qwen/Qwen2.5-Math-1.5B-Instruct\",\r\n    reward_funcs=[accuracy_reward],\r\n    args=config,\r\n    train_dataset=dataset,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @MengAiDev in https://github.com/huggingface/trl/pull/4935\r\n\r\n### Reward functions can now log extra columns and scalar metrics\r\n\r\nReward functions can return a dictionary of extra values (scalars or per-sample columns) that will be logged alongside the reward. This makes it easier to track intermediate signals without writing custom callbacks.\r\n\r\n```python\r\ndef my_reward_fn(completions, answer, log_extra=None, log_metric=None, **kwargs):\r\n    extracted = [extract_answer(c) for c in completions]\r\n    rewards = [1.0 if e == a else 0.0 for e, a in zip(extracted, answer)]\r\n\r\n    if log_extra:\r\n        log_extra(\"golden_answer\", list(answer))\r\n        log_extra(\"extracted_answer\", extracted)\r\n\r\n    if log_metric:\r\n        log_metric(\"accuracy\", sum(rewards) / len(rewards))\r\n\r\n    return rewards\r\n```\r\n\r\n<img width=\"1400\" height=\"407\" alt=\"image\" src=\"https://github.com/user-attachments/assets/d345b0ac-0d3c-446f-9321-a26e73ee16b4\" />\r\n<img width=\"1353\" height=\"673\" alt=\"image\" src=\"https://github.com/user-attachments/assets/b4c0302b-f69a-4715-9aad-278b4ad13299\" />\r\n\r\nby @manueldeprada in https://github.com/huggingface/trl/pull/5233\r\n\r\n### Tool calling support in `VLLMClient.chat()`\r\n\r\n`VLLMClient.chat()` now supports tool calling, enabling agentic workflows directly through the vLLM client interface.\r\n\r\nby @kansalaman in https://github.com/huggingface/trl/pull/4889\r\n\r\n### 35% faster packing\r\n\r\nBFD packing is 35% faster. The `\"bfd-requeue\"` packing strategy has also been renamed to `\"bfd_split\"`. See [MIGRATION.md](MIGRATION.md) for details.\r\n\r\n<img width=\"1784\" height=\"732\" alt=\"benchmark_results\" src=\"https://github.com/user-attachments/assets/8f0a35ad-cf1a-4fe1-a1f4-9b102637bdca\" />\r\n\r\nby @mariosasko in https://github.com/huggingface/trl/pull/5189\r\n\r\n### [GKD] Buffer implementation and vLLM inference for distillation trainer\r\n\r\nThe GKD/GOLD trainer now supports buffered rollout generation, decoupling generation from gradient updates for more efficient distillation. vLLM inference support has also been added to the base self-distillation trainer.\r\n\r\nby @cmpatino in https://github.com/huggingface/trl/pull/5137 and https://github.com/huggingface/trl/pull/5388\r\n\r\n### v0 → v1 migration guide\r\n\r\nA [`MIGRATION.md`](MIGRATION.md) guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are minimal.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5255\r\n\r\n### Other\r\n\r\n* Change default `vllm_mode` to `\"colocate\"` by @qgallouedec in https://github.com/huggingface/trl/pull/5255\r\n* Support `truncation_mode` in SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5306\r\n* Support `max_length` in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5284\r\n* Add `pad_to_multiple_of` to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180\r\n* Support sequence sampling in Liger Kernel by @michaelroyzen in https://github.com/huggingface/trl/pull/5190\r\n* Add tool calling support to VLLMClient.chat() by @kansalaman in https://github.com/huggingface/trl/pull/4889\r\n* Add support for raw token IDs in vLLM client prompts by @qgallouedec in https://github.com/huggingface/trl/pull/5225\r\n* Add VLM support when passing raw token IDs to vLLM client by @qgallouedec in https://github.com/huggingface/trl/pull/5227\r\n* Enhance `print_prompt_completions_sample` to include reasoning content by @qgallouedec in https://github.com/huggingface/trl/pull/5327\r\n* Add support for `pixel_position_ids` vision key by @qgallouedec in https://github.com/huggingface/trl/pull/5374\r\n* Add second version of Qwen 3.5 chat template by @apardyl in https://github.com/huggingface/trl/pull/5405\r\n* Pass tools as `None` to `apply_chat_template` when it is an empty list by @rabinadk1 in https://github.com/huggingface/trl/pull/5380\r\n\r\n## Fixes\r\n\r\n* Fix DPOTrainer collators to truncate sequences before padding by @albertvillanova in https://github.com/huggingface/trl/pull/5305\r\n* Prevent corruption of DPO VLM training if \"keep_end\" truncation_mode by @albertvillanova in https://github.com/huggingface/trl/pull/5286\r\n* Fix mm_token_type_ids silently dropped in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5279\r\n* Fix UNEXPECTED lm_head.weight warning when loading a CausalLM as a reward model by @albertvillanova in https://github.com/huggingface/trl/pull/5295\r\n* Fix `accuracy_reward` crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281\r\n* Fix GRPOTrainer attribute access for vLLM model config by @falcondai in https://github.com/huggingface/trl/pull/5302\r\n* [GRPO] Fix re-tokenization bug in tool-calling loop by @qgallouedec in https://github.com/huggingface/trl/pull/5242\r\n* [CPO/ORPO] Fix handling of different length chosen/rejected prompts by @davmels in https://github.com/huggingface/trl/pull/4639\r\n* Fix `RewardFunc` type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246\r\n* fix(ppo): add gradient_checkpointing_enable/disable to PolicyAndValueWrapper by @s-zx in https://github.com/huggingface/trl/pull/5245\r\n* Fix `prepare_multimodal_messages` to support `tool_calls` and `tool` role by @alvarobartt in https://github.com/huggingface/trl/pull/5212\r\n* Fix support for model_init_kwargs when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5230\r\n* Fix support for model_init_kwargs in MiniLLM when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5274\r\n* Fix support for model_init_kwargs in GKD/GOLD when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5266\r\n* Sync entire prompt/completion token tensors before indexing by @shawnghu in https://github.com/huggingface/trl/pull/5218\r\n* Clean up model update group on worker exit by @AmineDiro in https://github.com/huggingface/trl/pull/5325\r\n* Fix prefix EOS slicing for tool suffix (with Qwen3/3.5 chat templates) by @casinca in https://github.com/huggingface/trl/pull/5330\r\n* Fix: apply reward_weights to logged reward/reward_std in GRPOTrainer by @lailanelkoussy in https://github.com/huggingface/trl/pull/5353\r\n* Fix IDs shape mismatch in SFT for VLMs with text-only by @albertvillanova in https://github.com/huggingface/trl/pull/5354\r\n\r\n## Documentation and Examples\r\n\r\n* Add minimal CARLA example script by @sergiopaniego in https://github.com/huggingface/trl/pull/5161\r\n* Nemotron 3 examples added by @sergiopaniego in https://github.com/huggingface/trl/pull/5272\r\n* Align docs about tool calling in trainers with dataset format by @albertvillanova in https://github.com/huggingface/trl/pull/5311\r\n* Add repository-specific guidance for agents (`AGENTS.md`) by @qgallouedec in https://github.com/huggingface/trl/pull/5236\r\n* Align documentation with the intended public API by @qgallouedec in https://github.com/huggingface/trl/pull/5162\r\n* Update openenv examples to use `environment_factory` by @sergiopaniego in https://github.com/huggingface/trl/pull/5235\r\n* Add \"It Takes Two: Your GRPO Is Secretly DPO\" paper to GRPOTrainer by @DhruvvArora in https://github.com/huggingface/trl/pull/5347\r\n* Centralize AI agent templates in `.ai` by @qgallouedec in https://github.com/huggingface/trl/pull/5268\r\n\r\n## What's Changed\r\n\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5182\r\n* Handle mm_token_type_ids in SFT/GRPO/RLOO to fix IndexError by @albertvillanova in https://github.com/huggingface/trl/pull/5178\r\n* Document parameters with differing default values in core configs by @albertvillanova in https://github.com/huggingface/trl/pull/5168\r\n* Make _BaseConfig and _BaseTrainer explicitly private by @albertvillanova in https://github.com/huggingface/trl/pull/5169\r\n* Refactor CLI [4/N]: Replace top-level TrlParser with ArgumentParser by @albertvillanova in https://github.com/huggingface/trl/pull/5170\r\n* Add minimal CARLA example script by @sergiopaniego in https://github.com/huggingface/trl/pull/5161\r\n* Align documentation with the intended public API by @qgallouedec in https://github.com/huggingface/trl/pull/5162\r\n* Fix deprecation warning of create_reference_model by @albertvillanova in https://github.com/huggingface/trl/pull/5184\r\n* Fix deprecation warning of fork in multi-threaded process by @albertvillanova in https://github.com/huggingface/trl/pull/5185\r\n* Refactor CLI [5/N]: Refactor TrainingCommand with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5186\r\n* Refactor CLI [6/N]: Refactor env/vllm-serve commands with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5187\r\n* Fix CI tests patching BaseTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/5192\r\n* Add `pad_to_multiple_of` to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180\r\n* Re-add liger-kernel to dev deps by @qgallouedec in https://github.com/huggingface/trl/pull/5164\r\n* Set CI PYTORCH_ALLOC_CONF env variable to avoid OOM by @albertvillanova in https://github.com/huggingface/trl/pull/5197\r\n* Support sequence sampling in Liger Kernel and pass importance_samplin… by @michaelroyzen in https://github.com/huggingface/trl/pull/5190\r\n* Mark CI test_training_vlm_and_liger as xfail by @albertvillanova in https://github.com/huggingface/trl/pull/5202\r\n* Decouple rollout dispatch from vLLM backend in GRPO _generate_single_turn by @albertvillanova in https://github.com/huggingface/trl/pull/5122\r\n* CI: Add Qwen 3.5 tiny model to tests by @qgallouedec in https://github.com/huggingface/trl/pull/5204\r\n* Add support for Qwen3.5 for agent training by @qgallouedec in https://github.com/huggingface/trl/pull/5205\r\n* Update vLLM version support to include 0.13.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5206\r\n* feat: Add tool calling support to VLLMClient.chat() by @kansalaman in https://github.com/huggingface/trl/pull/4889\r\n* Refactor CLI [7/N]: Move patching to compat and import transformers conditionally by @albertvillanova in https://github.com/huggingface/trl/pull/5208\r\n* Update vLLM version support to include 0.14.0 and 0.14.1 by @qgallouedec in https://github.com/huggingface/trl/pull/5214\r\n* Refactor CLI [8/N]: Refactor scripts/utils with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5209\r\n* Simplify logic for structured outputs across vLLM versions by @albertvillanova in https://github.com/huggingface/trl/pull/5215\r\n* Refactor CLI [9/N]: Replace HfArgumentParser from transformers with local by @albertvillanova in https://github.com/huggingface/trl/pull/5210\r\n* Refactor CLI [10/N]: Refactor scripts with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5219\r\n* Refactor CLI [11/N]: Refactor scripts/vllm_serve with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5220\r\n* Refactor CLI [12/N]: Fix command name in scripts help usage by @albertvillanova in https://github.com/huggingface/trl/pull/5221\r\n* Refactor CLI [13/N]: Pass clean training args to scripts by @albertvillanova in https://github.com/huggingface/trl/pull/5223\r\n* Fix `prepare_multimodal_messages` to support `tool_calls` and `tool` role by @alvarobartt in https://github.com/huggingface/trl/pull/5212\r\n* Fix link to Hugging Face Hub in OpenEnv documentation by @thesteve0 in https://github.com/huggingface/trl/pull/5229\r\n* Fix type for model_init_kwargs when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5230\r\n* Add repository-specific guidance for agents (`AGENTS.md`) by @qgallouedec in https://github.com/huggingface/trl/pull/5236\r\n* Add support for raw ids in `prompts` in vLLM client and server by @qgallouedec in https://github.com/huggingface/trl/pull/5225\r\n* Deprecate `truncate_prompt_tokens` for vLLM 0.17.0 by @winglian in https://github.com/huggingface/trl/pull/5248\r\n* Add VLM support when passing raw token IDs to vLLM client by @qgallouedec in https://github.com/huggingface/trl/pull/5227\r\n* Move `rollout_func` from `_generate_single_turn` to `_generate` by @qgallouedec in https://github.com/huggingface/trl/pull/5232\r\n* Fix `RewardFunc` type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246\r\n* [GRPO] In-place temperature scaling operation by @winglian in https://github.com/huggingface/trl/pull/5254\r\n* Update vLLM version support to 0.15.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5251\r\n* Sync entire prompt/completion token tensors before indexing by @shawnghu in https://github.com/huggingface/trl/pull/5218\r\n* Update vLLM version support to 0.16.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5252\r\n* Update vLLM version support to 0.17.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5253\r\n* [GRPO/RLOO] Tokenize before vLLM generation call by @qgallouedec in https://github.com/huggingface/trl/pull/5238\r\n* Refactor CLI [14/N] : Remove TrainingArguments import from core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5257\r\n* Support JSON string parsing of teacher_model_init_kwargs in MiniLLMConfig by @albertvillanova in https://github.com/huggingface/trl/pull/5259\r\n* Fix typo in docstring for teacher_model_init_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/5260\r\n* Remove extra_fields dead code [1/N]: Remove extra_fields handling from VLLMGeneration.generate by @albertvillanova in https://github.com/huggingface/trl/pull/5262\r\n* [GRPO/RLOO] Unify tokenization across all generation backends in `_generate_single_turn` by @qgallouedec in https://github.com/huggingface/trl/pull/5239\r\n* Remove extra_fields dead code [2/N]: Remove extra_fields from VLLMGeneration.generate return value by @albertvillanova in https://github.com/huggingface/trl/pull/5263\r\n* Remove extra_fields dead code [3/N]: Remove extra_fields from GRPOTrainer._generate_single_turn return value by @albertvillanova in https://github.com/huggingface/trl/pull/5264\r\n* fix(ppo): add gradient_checkpointing_enable/disable to PolicyAndValueWrapper by @s-zx in https://github.com/huggingface/trl/pull/5245\r\n* [GRPO/RLOO] Extract tokenize prompts from `_generate_single_turn` by @qgallouedec in https://github.com/huggingface/trl/pull/5240\r\n* [CPO/ORPO] Fix handling of different length chosen/rejected prompts. by @davmels in https://github.com/huggingface/trl/pull/4639\r\n* Fix type for teacher_model_init_kwargs when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5258\r\n* Align GOLDConfig docstrings for optional params with None default by @albertvillanova in https://github.com/huggingface/trl/pull/5261\r\n* Fix support for model_init_kwargs in GKD/GOLD when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5266\r\n* Update TRL banner to support light/dark mode by @qgallouedec in https://github.com/huggingface/trl/pull/5270\r\n* Fix error message in OnlineDPO by @qgallouedec in https://github.com/huggingface/trl/pull/5237\r\n* Fix title consistency from \"Transformer Reinforcement Learning\" to \"Transformers Reinforcement Learning\" by @qgallouedec in https://github.com/huggingface/trl/pull/5183\r\n* Nemotron 3 examples added by @sergiopaniego in https://github.com/huggingface/trl/pull/5272\r\n* Fix mm_token_type_ids silently dropped in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5279\r\n* Simplify get_train_dataloader in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/5276\r\n* Raise ValueError for None train_dataset in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5275\r\n* 35% faster packing + rename `bfd-requeue` to `bfd_split` by @mariosasko in https://github.com/huggingface/trl/pull/5189\r\n* Change default `vllm_mode` to `\"colocate\"` and add v0→v1 migration guide by @qgallouedec in https://github.com/huggingface/trl/pull/5255\r\n* Allow nullable logprobs in vLLM serve responses  by @LeonEricsson in https://github.com/huggingface/trl/pull/5203\r\n* feat(`grpo_trainer.py`): Variational Sequence-Level Soft Policy Optimization (VESPO) by @casinca in https://github.com/huggingface/trl/pull/5199\r\n* Simplify structured outputs logic across vLLM versions in scripts/vllm_serve by @albertvillanova in https://github.com/huggingface/trl/pull/5273\r\n* Fix support for model_init_kwargs in MiniLLM when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5274\r\n* Fix `accuracy_reward` crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281\r\n* Remove TrainingArguments import from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5290\r\n* Remove custom get_train/eval_dataloader from OnlineDPO by @albertvillanova in https://github.com/huggingface/trl/pull/5291\r\n* [GKD] Buffer Implementation for Distillation Trainer by @cmpatino in https://github.com/huggingface/trl/pull/5137\r\n* Support max_length in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5284\r\n* Prevent corruption of DPO VLM training if \"keep_end\" truncation_mode by @albertvillanova in https://github.com/huggingface/trl/pull/5286\r\n* Fix UNEXPECTED lm_head.weight warning when loading a CausalLM as a reward model by @albertvillanova in https://github.com/huggingface/trl/pull/5295\r\n* Apply docstyle by @qgallouedec in https://github.com/huggingface/trl/pull/5296\r\n* Add guidance to avoid `hasattr` and `getattr` with defaults in `AGENTS.md` by @qgallouedec in https://github.com/huggingface/trl/pull/5294\r\n* Fix DPOTrainer collators to truncate sequences before padding by @albertvillanova in https://github.com/huggingface/trl/pull/5305\r\n* Update `RewardFunc` type annotation to allow `None`values in reward list by @qgallouedec in https://github.com/huggingface/trl/pull/5297\r\n* Suggest the `Json()` type for tool calling dataset format by @lhoestq in https://github.com/huggingface/trl/pull/5307\r\n* Allow reward functions to log extra columns and scalar metrics by @manueldeprada in https://github.com/huggingface/trl/pull/5233\r\n* Fix GRPOTrainer attribute access for vLLM model config by @falcondai in https://github.com/huggingface/trl/pull/5302\r\n* Support truncation_mode in SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5306\r\n* 🔌 Asynchronous GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/5293\r\n* Fix datasets version supporting Json dtype in docs about tool calling dataset format by @albertvillanova in https://github.com/huggingface/trl/pull/5310\r\n* Align docs about tool calling in trainers with dataset format by @albertvillanova in https://github.com/huggingface/trl/pull/5311\r\n* [GRPO] Fix re-tokenization bug in tool-calling loop by concatenating token IDs by @qgallouedec in https://github.com/huggingface/trl/pull/5242\r\n* feat(experimental): Divergence Proximal Policy Optimization by @LeonEricsson in https://github.com/huggingface/trl/pull/5117\r\n* Clean up model update group on worker exit by @AmineDiro in https://github.com/huggingface/trl/pull/5325\r\n* Fix style in DPPO docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/5326\r\n* `GRPOTrainer/async`: fix prefix EOS slicing for tool suffix (with Qwen3/3.5 type of chat templates) by @casinca in https://github.com/huggingface/trl/pull/5330\r\n* refactor(async_rollout_worker): renamed tool variables to mirror `grpo_trainer.py` by @casinca in https://github.com/huggingface/trl/pull/5332\r\n* Add truncation to SFT DataCollatorForLanguageModeling by @albertvillanova in https://github.com/huggingface/trl/pull/5315\r\n* Add SDPO (Self-Distillation Policy Optimization) trainer by @MengAiDev in https://github.com/huggingface/trl/pull/4935\r\n* Update openenv examples to use `environment_factory` by @sergiopaniego in https://github.com/huggingface/trl/pull/5235\r\n* Enhance `print_prompt_completions_sample` to include reasoning content by @qgallouedec in https://github.com/huggingface/trl/pull/5327\r\n* Add Cursor Bugbot rules from `AGENTS.md` by @qgallouedec in https://github.com/huggingface/trl/pull/5280\r\n* Change model dtype from bfloat16 to float32 in AsyncGRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5333\r\n* docs: Add \"It Takes Two: Your GRPO Is Secretly DPO\" paper to GRPOTrainer by @DhruvvArora in https://github.com/huggingface/trl/pull/5347\r\n* fix: apply reward_weights to logged reward/reward_std in GRPOTrainer by @lailanelkoussy in https://github.com/huggingface/trl/pull/5353\r\n* Remove post-collation truncation from DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5350\r\n* Remove unused flush_right by @albertvillanova in https://github.com/huggingface/trl/pull/5358\r\n* Fix IDs shape mismatch in SFT for VLMs with text-only by @albertvillanova in https://github.com/huggingface/trl/pull/5354\r\n* Remove post-collation truncation from SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5359\r\n* Simplify DPO DataCollatorForPreference by @albertvillanova in https://github.com/huggingface/trl/pull/5362\r\n* Simplify SFT tokenization by @albertvillanova in https://github.com/huggingface/trl/pull/5363\r\n* Simplify SFT DataCollatorForLanguageModeling by @albertvillanova in https://github.com/huggingface/trl/pull/5360\r\n* Use BaseConfig post_init in experimental KTO and MiniLLM configs by @albertvillanova in https://github.com/huggingface/trl/pull/5371\r\n* Move truncate_dataset to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5370\r\n* Simplify DPO tokenization by @albertvillanova in https://github.com/huggingface/trl/pull/5369\r\n* Kd vllm generation by @cmpatino in https://github.com/huggingface/trl/pull/5351\r\n* Adds support for the `pixel_position_ids` vision key by @qgallouedec in https://github.com/huggingface/trl/pull/5374\r\n* Minor diff reduction between RLOO and GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/5368\r\n* Remove requirements.txt by @albertvillanova in https://github.com/huggingface/trl/pull/5377\r\n* Remove dead truncation_mode from experimental BCO, CPO and ORPO by @albertvillanova in https://github.com/huggingface/trl/pull/5378\r\n* Centralize AI agent templates in `.ai` by @qgallouedec in https://github.com/huggingface/trl/pull/5268\r\n* Pass tools as None to `apply_chat_template` when it is an empty list by @rabinadk1 in https://github.com/huggingface/trl/pull/5380\r\n* Require datasets>=4.7.0 for Json dtype to prevent insertion of None values by @albertvillanova in https://github.com/huggingface/trl/pull/5376\r\n* Remove deprecated `TRACKIO_SPACE_ID` env var from all scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/5365\r\n* Mark test_rloo[fsdp2] as xfail for transformers 5.4.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5387\r\n* Enforce PR template for first-time contributors and document AI usage policy by @qgallouedec in https://github.com/huggingface/trl/pull/5356\r\n* Enhance PR template check to exclude reopened PRs from first-time contributor validation by @qgallouedec in https://github.com/huggingface/trl/pull/5392\r\n* chore: update `pr_template_check.yml` by @qgallouedec in https://github.com/huggingface/trl/pull/5393\r\n* Move `disable_config=True` from `generate` to `GenerationConfig` by @qgallouedec in https://github.com/huggingface/trl/pull/5384\r\n* Add vLLM inference to the Base Self-Distillation Trainer by @cmpatino in https://github.com/huggingface/trl/pull/5388\r\n* Add HF_TOKEN environment variable to workflow files by @qgallouedec in https://github.com/huggingface/trl/pull/5397\r\n* Add second version of Qwen 3.5 chat template to chat_template_utils by @apardyl in https://github.com/huggingface/trl/pull/5405\r\n* Release: v1.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5409\r\n\r\n## New Contributors\r\n\r\n* @czkkkkkk made their first contribution in https://github.com/huggingface/trl/pull/5180\r\n* @michaelroyzen made their first contribution in https://github.com/huggingface/trl/pull/5190\r\n* @thesteve0 made their first contribution in https://github.com/huggingface/trl/pull/5229\r\n* @s-zx made their first contribution in https://github.com/huggingface/trl/pull/5246\r\n* @shawnghu made their first contribution in https://github.com/huggingface/trl/pull/5218\r\n* @davmels made their first contribution in https://github.com/huggingface/trl/pull/4639\r\n* @manueldeprada made their first contribution in https://github.com/huggingface/trl/pull/5233\r\n* @falcondai made their first contribution in https://github.com/huggingface/trl/pull/5302\r\n* @AmineDiro made their first contribution in https://github.com/huggingface/trl/pull/5325\r\n* @DhruvvArora made their first contribution in https://github.com/huggingface/trl/pull/5347\r\n* @lailanelkoussy made their first contribution in https://github.com/huggingface/trl/pull/5353\r\n* @rabinadk1 made their first contribution in https://github.com/huggingface/trl/pull/5380\r\n* @apardyl made their first contribution in https://github.com/huggingface/trl/pull/5405\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v0.29.0...v1.0.0\r\n","publishedAt":"2026-03-31T14:15:06.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.0.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_uH5Du5_0GLzI_59w8HPxR","version":"v1.0.0rc1","type":"feature","title":"v1.0.0rc1","summary":"## Features\r\n\r\n### Variational Sequence-Level Soft Policy Optimization (VESPO)\r\n\r\n<img width=\"465\" height=\"279\" alt=\"Screenshot 2026-03-20 at 5 49 50 ...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### Variational Sequence-Level Soft Policy Optimization (VESPO)\r\n\r\n<img width=\"465\" height=\"279\" alt=\"Screenshot 2026-03-20 at 5 49 50 PM\" src=\"https://github.com/user-attachments/assets/b60c9697-6eb7-498e-95b3-df78c367f5fa\" />\r\n\r\n[VESPO](https://huggingface.co/papers/2602.10693) addresses training instability in off-policy RL caused by policy staleness, asynchronous updates, and train-inference mismatches. Rather than relying on heuristic token-level clipping (GRPO) or sequence-length normalization (GSPO), VESPO derives a principled reshaping kernel from a variational framework. In practice, this yields a smooth, asymmetric Gamma weighting function that gracefully suppresses extreme sequence-level importance weights without introducing length bias. It can be enabled via the `loss_type` parameter of `GRPOConfig`:\r\n\r\n```python\r\nfrom trl import GRPOConfig, GRPOTrainer\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-0.6B\",\r\n    args=GRPOConfig(loss_type=\"vespo\"),\r\n    ...\r\n)\r\n```\r\n\r\nby @casinca in https://github.com/huggingface/trl/pull/5199\r\n\r\n### Divergence Proximal Policy Optimization (DPPO)\r\n\r\n<img width=\"3180\" height=\"1187\" alt=\"z_TXYw37xZqsQ21YiDkYL\" src=\"https://github.com/user-attachments/assets/40f1d538-82b3-4097-91c6-119ea9f7797b\" />\r\n<img width=\"1189\" height=\"490\" alt=\"SfgWotuuuRKPkg-0bxWv1\" src=\"https://github.com/user-attachments/assets/2b090df3-0bfb-42e4-9f94-15943736e689\" />\r\n\r\n[DPPO](https://huggingface.co/papers/2602.04879) is a new experimental trainer that replaces the standard PPO clipping mechanism with divergence constraints, providing more principled trust-region updates.\r\n\r\nby @LeonEricsson in https://github.com/huggingface/trl/pull/5117\r\n\r\n### Reward functions can now log extra columns and scalar metrics\r\n\r\nReward functions can return a dictionary of extra values (scalars or per-sample columns) that will be logged alongside the reward. This makes it easier to track intermediate signals without writing custom callbacks.\r\n\r\n```python\r\ndef my_reward_fn(completions, answer, log_extra=None, log_metric=None, **kwargs):\r\n    extracted = [extract_answer(c) for c in completions]\r\n    rewards = [1.0 if e == a else 0.0 for e, a in zip(extracted, answer)]\r\n\r\n    if log_extra:\r\n        log_extra(\"golden_answer\", list(answer))\r\n        log_extra(\"extracted_answer\", extracted)\r\n\r\n    if log_metric:\r\n        log_metric(\"accuracy\", sum(rewards) / len(rewards))\r\n\r\n    return rewards\r\n```\r\n\r\n<img width=\"1400\" height=\"407\" alt=\"image\" src=\"https://github.com/user-attachments/assets/d345b0ac-0d3c-446f-9321-a26e73ee16b4\" />\r\n<img width=\"1353\" height=\"673\" alt=\"image\" src=\"https://github.com/user-attachments/assets/b4c0302b-f69a-4715-9aad-278b4ad13299\" />\r\n\r\nby @manueldeprada in https://github.com/huggingface/trl/pull/5233\r\n\r\n### Tool calling support in `VLLMClient.chat()`\r\n\r\n`VLLMClient.chat()` now supports tool calling, enabling agentic workflows directly through the vLLM client interface.\r\n\r\nby @kansalaman in https://github.com/huggingface/trl/pull/4889\r\n\r\n### 35% faster packing\r\n\r\nBFD packing is 35% faster. The `\"bfd-requeue\"` packing strategy has also been renamed to `\"bfd_split\"`. See [MIGRATION.md](MIGRATION.md) for details.\r\n\r\n<img width=\"1784\" height=\"732\" alt=\"benchmark_results\" src=\"https://github.com/user-attachments/assets/8f0a35ad-cf1a-4fe1-a1f4-9b102637bdca\" />\r\n\r\n\r\nby @mariosasko in https://github.com/huggingface/trl/pull/5189\r\n\r\n### [GKD] Buffer implementation for distillation trainer\r\n\r\nThe GKD/GOLD trainer now supports buffered rollout generation, decoupling generation from gradient updates for more efficient distillation.\r\n\r\nby @cmpatino in https://github.com/huggingface/trl/pull/5137\r\n\r\n### v0 → v1 migration guide\r\n\r\nA [`MIGRATION.md`](MIGRATION.md) guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are minimal.\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5255\r\n\r\n### Other\r\n\r\n* Change default `vllm_mode` to `\"colocate\"` by @qgallouedec in https://github.com/huggingface/trl/pull/5255\r\n* Support `truncation_mode` in SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5306\r\n* Support `max_length` in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5284\r\n* Add `pad_to_multiple_of` to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180\r\n* Support sequence sampling in Liger Kernel by @michaelroyzen in https://github.com/huggingface/trl/pull/5190\r\n* Add tool calling support to VLLMClient.chat() by @kansalaman in https://github.com/huggingface/trl/pull/4889\r\n* Add support for raw token IDs in vLLM client prompts by @qgallouedec in https://github.com/huggingface/trl/pull/5225\r\n* Add VLM support when passing raw token IDs to vLLM client by @qgallouedec in https://github.com/huggingface/trl/pull/5227\r\n\r\n## Fixes\r\n\r\n* Fix DPOTrainer collators to truncate sequences before padding by @albertvillanova in https://github.com/huggingface/trl/pull/5305\r\n* Prevent corruption of DPO VLM training if \"keep_end\" truncation_mode by @albertvillanova in https://github.com/huggingface/trl/pull/5286\r\n* Fix mm_token_type_ids silently dropped in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5279\r\n* Fix UNEXPECTED lm_head.weight warning when loading a CausalLM as a reward model by @albertvillanova in https://github.com/huggingface/trl/pull/5295\r\n* Fix `accuracy_reward` crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281\r\n* Fix GRPOTrainer attribute access for vLLM model config by @falcondai in https://github.com/huggingface/trl/pull/5302\r\n* [GRPO] Fix re-tokenization bug in tool-calling loop by @qgallouedec in https://github.com/huggingface/trl/pull/5242\r\n* [CPO/ORPO] Fix handling of different length chosen/rejected prompts by @davmels in https://github.com/huggingface/trl/pull/4639\r\n* Fix `RewardFunc` type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246\r\n* fix(ppo): add gradient_checkpointing_enable/disable to PolicyAndValueWrapper by @s-zx in https://github.com/huggingface/trl/pull/5245\r\n* Fix `prepare_multimodal_messages` to support `tool_calls` and `tool` role by @alvarobartt in https://github.com/huggingface/trl/pull/5212\r\n* Fix support for model_init_kwargs when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5230\r\n* Fix support for model_init_kwargs in MiniLLM when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5274\r\n* Fix support for model_init_kwargs in GKD/GOLD when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5266\r\n* Sync entire prompt/completion token tensors before indexing by @shawnghu in https://github.com/huggingface/trl/pull/5218\r\n* Clean up model update group on worker exit by @AmineDiro in https://github.com/huggingface/trl/pull/5325\r\n\r\n## Documentation and Examples\r\n\r\n* Add minimal CARLA example script by @sergiopaniego in https://github.com/huggingface/trl/pull/5161\r\n* Nemotron 3 examples added by @sergiopaniego in https://github.com/huggingface/trl/pull/5272\r\n* Align docs about tool calling in trainers with dataset format by @albertvillanova in https://github.com/huggingface/trl/pull/5311\r\n* Add repository-specific guidance for agents (`AGENTS.md`) by @qgallouedec in https://github.com/huggingface/trl/pull/5236\r\n* Align documentation with the intended public API by @qgallouedec in https://github.com/huggingface/trl/pull/5162\r\n\r\n## What's Changed\r\n\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5182\r\n* Handle mm_token_type_ids in SFT/GRPO/RLOO to fix IndexError by @albertvillanova in https://github.com/huggingface/trl/pull/5178\r\n* Document parameters with differing default values in core configs by @albertvillanova in https://github.com/huggingface/trl/pull/5168\r\n* Make _BaseConfig and _BaseTrainer explicitly private by @albertvillanova in https://github.com/huggingface/trl/pull/5169\r\n* Refactor CLI [4/N]: Replace top-level TrlParser with ArgumentParser by @albertvillanova in https://github.com/huggingface/trl/pull/5170\r\n* Add minimal CARLA example script by @sergiopaniego in https://github.com/huggingface/trl/pull/5161\r\n* Align documentation with the intended public API by @qgallouedec in https://github.com/huggingface/trl/pull/5162\r\n* Fix deprecation warning of create_reference_model by @albertvillanova in https://github.com/huggingface/trl/pull/5184\r\n* Fix deprecation warning of fork in multi-threaded process by @albertvillanova in https://github.com/huggingface/trl/pull/5185\r\n* Refactor CLI [5/N]: Refactor TrainingCommand with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5186\r\n* Refactor CLI [6/N]: Refactor env/vllm-serve commands with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5187\r\n* Fix CI tests patching BaseTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/5192\r\n* Add `pad_to_multiple_of` to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180\r\n* Re-add liger-kernel to dev deps by @qgallouedec in https://github.com/huggingface/trl/pull/5164\r\n* Set CI PYTORCH_ALLOC_CONF env variable to avoid OOM by @albertvillanova in https://github.com/huggingface/trl/pull/5197\r\n* Support sequence sampling in Liger Kernel and pass importance_samplin… by @michaelroyzen in https://github.com/huggingface/trl/pull/5190\r\n* Mark CI test_training_vlm_and_liger as xfail by @albertvillanova in https://github.com/huggingface/trl/pull/5202\r\n* Decouple rollout dispatch from vLLM backend in GRPO _generate_single_turn by @albertvillanova in https://github.com/huggingface/trl/pull/5122\r\n* CI: Add Qwen 3.5 tiny model to tests by @qgallouedec in https://github.com/huggingface/trl/pull/5204\r\n* Add support for Qwen3.5 for agent training by @qgallouedec in https://github.com/huggingface/trl/pull/5205\r\n* Update vLLM version support to include 0.13.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5206\r\n* feat: Add tool calling support to VLLMClient.chat() by @kansalaman in https://github.com/huggingface/trl/pull/4889\r\n* Refactor CLI [7/N]: Move patching to compat and import transformers conditionally by @albertvillanova in https://github.com/huggingface/trl/pull/5208\r\n* Update vLLM version support to include 0.14.0 and 0.14.1 by @qgallouedec in https://github.com/huggingface/trl/pull/5214\r\n* Refactor CLI [8/N]: Refactor scripts/utils with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5209\r\n* Simplify logic for structured outputs across vLLM versions by @albertvillanova in https://github.com/huggingface/trl/pull/5215\r\n* Refactor CLI [9/N]: Replace HfArgumentParser from transformers with local by @albertvillanova in https://github.com/huggingface/trl/pull/5210\r\n* Refactor CLI [10/N]: Refactor scripts with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5219\r\n* Refactor CLI [11/N]: Refactor scripts/vllm_serve with delayed imports by @albertvillanova in https://github.com/huggingface/trl/pull/5220\r\n* Refactor CLI [12/N]: Fix command name in scripts help usage by @albertvillanova in https://github.com/huggingface/trl/pull/5221\r\n* Refactor CLI [13/N]: Pass clean training args to scripts by @albertvillanova in https://github.com/huggingface/trl/pull/5223\r\n* Fix `prepare_multimodal_messages` to support `tool_calls` and `tool` role by @alvarobartt in https://github.com/huggingface/trl/pull/5212\r\n* Fix link to Hugging Face Hub in OpenEnv documentation by @thesteve0 in https://github.com/huggingface/trl/pull/5229\r\n* Fix type for model_init_kwargs when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5230\r\n* Add repository-specific guidance for agents (`AGENTS.md`) by @qgallouedec in https://github.com/huggingface/trl/pull/5236\r\n* Add support for raw ids in `prompts` in vLLM client and server by @qgallouedec in https://github.com/huggingface/trl/pull/5225\r\n* Deprecate `truncate_prompt_tokens` for vLLM 0.17.0 by @winglian in https://github.com/huggingface/trl/pull/5248\r\n* Add VLM support when passing raw token IDs to vLLM client by @qgallouedec in https://github.com/huggingface/trl/pull/5227\r\n* Move `rollout_func` from `_generate_single_turn` to `_generate` by @qgallouedec in https://github.com/huggingface/trl/pull/5232\r\n* Fix `RewardFunc` type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246\r\n* [GRPO] In-place temperature scaling operation by @winglian in https://github.com/huggingface/trl/pull/5254\r\n* Update vLLM version support to 0.15.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5251\r\n* Sync entire prompt/completion token tensors before indexing by @shawnghu in https://github.com/huggingface/trl/pull/5218\r\n* Update vLLM version support to 0.16.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5252\r\n* Update vLLM version support to 0.17.0 by @qgallouedec in https://github.com/huggingface/trl/pull/5253\r\n* [GRPO/RLOO] Tokenize before vLLM generation call by @qgallouedec in https://github.com/huggingface/trl/pull/5238\r\n* Refactor CLI [14/N] : Remove TrainingArguments import from core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5257\r\n* Support JSON string parsing of teacher_model_init_kwargs in MiniLLMConfig by @albertvillanova in https://github.com/huggingface/trl/pull/5259\r\n* Fix typo in docstring for teacher_model_init_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/5260\r\n* Remove extra_fields dead code [1/N]: Remove extra_fields handling from VLLMGeneration.generate by @albertvillanova in https://github.com/huggingface/trl/pull/5262\r\n* [GRPO/RLOO] Unify tokenization across all generation backends in `_generate_single_turn` by @qgallouedec in https://github.com/huggingface/trl/pull/5239\r\n* Remove extra_fields dead code [2/N]: Remove extra_fields from VLLMGeneration.generate return value by @albertvillanova in https://github.com/huggingface/trl/pull/5263\r\n* Remove extra_fields dead code [3/N]: Remove extra_fields from GRPOTrainer._generate_single_turn return value by @albertvillanova in https://github.com/huggingface/trl/pull/5264\r\n* fix(ppo): add gradient_checkpointing_enable/disable to PolicyAndValueWrapper by @s-zx in https://github.com/huggingface/trl/pull/5245\r\n* [GRPO/RLOO] Extract tokenize prompts from `_generate_single_turn` by @qgallouedec in https://github.com/huggingface/trl/pull/5240\r\n* [CPO/ORPO] Fix handling of different length chosen/rejected prompts. by @davmels in https://github.com/huggingface/trl/pull/4639\r\n* Fix type for teacher_model_init_kwargs when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5258\r\n* Align GOLDConfig docstrings for optional params with None default by @albertvillanova in https://github.com/huggingface/trl/pull/5261\r\n* Fix support for model_init_kwargs in GKD/GOLD when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5266\r\n* Update TRL banner to support light/dark mode by @qgallouedec in https://github.com/huggingface/trl/pull/5270\r\n* Fix error message in OnlineDPO by @qgallouedec in https://github.com/huggingface/trl/pull/5237\r\n* Fix title consistency from \"Transformer Reinforcement Learning\" to \"Transformers Reinforcement Learning\" by @qgallouedec in https://github.com/huggingface/trl/pull/5183\r\n* Nemotron 3 examples added by @sergiopaniego in https://github.com/huggingface/trl/pull/5272\r\n* Fix mm_token_type_ids silently dropped in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5279\r\n* Simplify get_train_dataloader in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/5276\r\n* Raise ValueError for None train_dataset in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5275\r\n* 35% faster packing + rename `bfd-requeue` to `bfd_split` by @mariosasko in https://github.com/huggingface/trl/pull/5189\r\n* Change default `vllm_mode` to `\"colocate\"` and add v0→v1 migration guide by @qgallouedec in https://github.com/huggingface/trl/pull/5255\r\n* Allow nullable logprobs in vLLM serve responses  by @LeonEricsson in https://github.com/huggingface/trl/pull/5203\r\n* feat(`grpo_trainer.py`): Variational Sequence-Level Soft Policy Optimization (VESPO) by @casinca in https://github.com/huggingface/trl/pull/5199\r\n* Simplify structured outputs logic across vLLM versions in scripts/vllm_serve by @albertvillanova in https://github.com/huggingface/trl/pull/5273\r\n* Fix support for model_init_kwargs in MiniLLM when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5274\r\n* Fix `accuracy_reward` crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281\r\n* Remove TrainingArguments import from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5290\r\n* Remove custom get_train/eval_dataloader from OnlineDPO by @albertvillanova in https://github.com/huggingface/trl/pull/5291\r\n* [GKD] Buffer Implementation for Distillation Trainer by @cmpatino in https://github.com/huggingface/trl/pull/5137\r\n* Support max_length in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5284\r\n* Prevent corruption of DPO VLM training if \"keep_end\" truncation_mode by @albertvillanova in https://github.com/huggingface/trl/pull/5286\r\n* Fix UNEXPECTED lm_head.weight warning when loading a CausalLM as a reward model by @albertvillanova in https://github.com/huggingface/trl/pull/5295\r\n* Apply docstyle by @qgallouedec in https://github.com/huggingface/trl/pull/5296\r\n* Add guidance to avoid `hasattr` and `getattr` with defaults in `AGENTS.md` by @qgallouedec in https://github.com/huggingface/trl/pull/5294\r\n* Fix DPOTrainer collators to truncate sequences before padding by @albertvillanova in https://github.com/huggingface/trl/pull/5305\r\n* Update `RewardFunc` type annotation to allow `None`values in reward list by @qgallouedec in https://github.com/huggingface/trl/pull/5297\r\n* Suggest the `Json()` type for tool calling dataset format by @lhoestq in https://github.com/huggingface/trl/pull/5307\r\n* Allow reward functions to log extra columns and scalar metrics by @manueldeprada in https://github.com/huggingface/trl/pull/5233\r\n* Fix GRPOTrainer attribute access for vLLM model config by @falcondai in https://github.com/huggingface/trl/pull/5302\r\n* Support truncation_mode in SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5306\r\n* 🔌 Asynchronous GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/5293\r\n* Fix datasets version supporting Json dtype in docs about tool calling dataset format by @albertvillanova in https://github.com/huggingface/trl/pull/5310\r\n* Align docs about tool calling in trainers with dataset format by @albertvillanova in https://github.com/huggingface/trl/pull/5311\r\n* [GRPO] Fix re-tokenization bug in tool-calling loop by concatenating token IDs by @qgallouedec in https://github.com/huggingface/trl/pull/5242\r\n* feat(experimental): Divergence Proximal Policy Optimization by @LeonEricsson in https://github.com/huggingface/trl/pull/5117\r\n* Clean up model update group on worker exit by @AmineDiro in https://github.com/huggingface/trl/pull/5325\r\n* Fix style in DPPO docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/5326\r\n\r\n## New Contributors\r\n\r\n* @czkkkkkk made their first contribution in https://github.com/huggingface/trl/pull/5180\r\n* @michaelroyzen made their first contribution in https://github.com/huggingface/trl/pull/5190\r\n* @thesteve0 made their first contribution in https://github.com/huggingface/trl/pull/5229\r\n* @s-zx made their first contribution in https://github.com/huggingface/trl/pull/5246\r\n* @shawnghu made their first contribution in https://github.com/huggingface/trl/pull/5218\r\n* @davmels made their first contribution in https://github.com/huggingface/trl/pull/4639\r\n* @manueldeprada made their first contribution in https://github.com/huggingface/trl/pull/5233\r\n* @falcondai made their first contribution in https://github.com/huggingface/trl/pull/5302\r\n* @AmineDiro made their first contribution in https://github.com/huggingface/trl/pull/5325\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v0.29.0...v1.0.0rc1","publishedAt":"2026-03-20T23:55:04.000Z","url":"https://github.com/huggingface/trl/releases/tag/v1.0.0rc1","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_7TW8tyroMX0X7yLx91KP6","version":"v0.29.1","type":"feature","title":"v0.29.1","summary":"## What's Changed\r\n\r\n* Handle mm_token_type_ids in SFT/GRPO/RLOO to fix IndexError by @albertvillanova in https://github.com/huggingface/trl/pull/5178...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## What's Changed\r\n\r\n* Handle mm_token_type_ids in SFT/GRPO/RLOO to fix IndexError by @albertvillanova in https://github.com/huggingface/trl/pull/5178\r\n* Fix `prepare_multimodal_messages` to support `tool_calls` and `tool` role by @alvarobartt in https://github.com/huggingface/trl/pull/5212\r\n* Fix type for model_init_kwargs when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5230\r\n* Decouple rollout dispatch from vLLM backend in GRPO _generate_single_turn by @albertvillanova in https://github.com/huggingface/trl/pull/5122\r\n* Simplify logic for structured outputs across vLLM versions by @albertvillanova in https://github.com/huggingface/trl/pull/5215\r\n* Add support for raw ids in `prompts` in vLLM client and server by @qgallouedec in https://github.com/huggingface/trl/pull/5225\r\n* Add VLM support when passing raw token IDs to vLLM client by @qgallouedec in https://github.com/huggingface/trl/pull/5227\r\n* Move `rollout_func` from `_generate_single_turn` to `_generate` by @qgallouedec in https://github.com/huggingface/trl/pull/5232\r\n* [GRPO/RLOO] Tokenize before vLLM generation call by @qgallouedec in https://github.com/huggingface/trl/pull/5238\r\n* Support JSON string parsing of teacher_model_init_kwargs in MiniLLMConfig by @albertvillanova in https://github.com/huggingface/trl/pull/5259\r\n* [GRPO/RLOO] Unify tokenization across all generation backends in `_generate_single_turn` by @qgallouedec in https://github.com/huggingface/trl/pull/5239\r\n* [GRPO/RLOO] Extract tokenize prompts from `_generate_single_turn` by @qgallouedec in https://github.com/huggingface/trl/pull/5240\r\n* [CPO/ORPO] Fix handling of different length chosen/rejected prompts. by @davmels in https://github.com/huggingface/trl/pull/4639\r\n* Fix type for teacher_model_init_kwargs when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5258\r\n* Fix support for model_init_kwargs in GKD/GOLD when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5266\r\n* Fix mm_token_type_ids silently dropped in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5279\r\n* Fix support for model_init_kwargs in MiniLLM when passed as CLI JSON string by @albertvillanova in https://github.com/huggingface/trl/pull/5274\r\n* Fix GRPOTrainer attribute access for vLLM model config by @falcondai in https://github.com/huggingface/trl/pull/5302\r\n* [GRPO] Fix re-tokenization bug in tool-calling loop by concatenating token IDs by @qgallouedec in https://github.com/huggingface/trl/pull/5242\r\n\r\n## New Contributors\r\n\r\n* @davmels made their first contribution in https://github.com/huggingface/trl/pull/4639\r\n* @falcondai made their first contribution in https://github.com/huggingface/trl/pull/5302\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v0.29.0...v0.29.1\r\n","publishedAt":"2026-03-20T03:57:13.000Z","url":"https://github.com/huggingface/trl/releases/tag/v0.29.1","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_sd0_rtFVZKLv1UVYmn474","version":"v0.29.0","type":"feature","title":"v0.29.0","summary":"## Features\r\n\r\n### Add `environment_factory` to `GRPOTrainer`\r\n\r\n`GRPOTrainer` now accepts an `environment_factory` argument, allowing users to specif...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## Features\r\n\r\n### Add `environment_factory` to `GRPOTrainer`\r\n\r\n`GRPOTrainer` now accepts an `environment_factory` argument, allowing users to specify a custom environment class for training. This enables more flexible and diverse training scenarios by letting users define their own environments with specific dynamics and reward structures.\r\n\r\n```python\r\nfrom datasets import Dataset\r\nfrom trl import GRPOConfig, GRPOTrainer\r\n\r\ndataset = Dataset.from_dict({\r\n    \"prompt\": [[{\"role\": \"user\", \"content\": f\"Increment the counter by {i}.\"}] for i in range(1, 7)]\r\n})\r\n\r\ndef reward_func(environments, **kwargs):\r\n    return [env.counter for env in environments]\r\n\r\nclass IncrementEnv:\r\n    def reset(self):\r\n        self.counter = 0\r\n\r\n    def increment(self, step: int) -> int:\r\n        \"\"\"\r\n        Increment the internal counter.\r\n\r\n        Args:\r\n            step: Value to add to the counter.\r\n\r\n        Returns:\r\n            The updated counter value.\r\n        \"\"\"\r\n        self.counter += step\r\n        return self.counter\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen/Qwen3-0.6B\",\r\n    args=GRPOConfig(chat_template_kwargs={\"enable_thinking\": False}),\r\n    train_dataset=dataset,\r\n    reward_funcs=reward_func,\r\n    environment_factory=IncrementEnv,\r\n)\r\ntrainer.train()\r\n```\r\n\r\nby @qgallouedec in https://github.com/huggingface/trl/pull/5093\r\n\r\n### Skills\r\n\r\nTRL introduces agent-native CLI Integration: trl-training, a first-class Agent Skill that exposes TRL’s training workflows (SFT, DPO, GRPO, etc.) in a structured, agent-readable format. The skill is packaged directly with the trl library and can be installed via the CLI:\r\n\r\n```bash\r\n# Install into the project's agent directory (default scope=project), by agent name: claude, codex, opencode\r\ntrl skills install trl-training --target <agent>\r\n```\r\n\r\nThis enables AI agents to safely and reproducibly execute TRL training workflows using a well-defined interface.\r\n\r\nSkills can be installed at the project or global scope, and support explicit targets and overwrite controls.\r\n\r\n* Implement Agent Skills [1/N]: Create training skill (MVP) by @albertvillanova in https://github.com/huggingface/trl/pull/5096\r\n* Implement Agent Skills [2/N]:  Create skills module by @albertvillanova in https://github.com/huggingface/trl/pull/5097\r\n* Implement Agent Skills [3/N]: Create skills installer by @albertvillanova in https://github.com/huggingface/trl/pull/5100\r\n* Implement Agent Skills [4/N]: Create skills CLI by @albertvillanova in https://github.com/huggingface/trl/pull/5103\r\n\r\n### Other \r\n* Pass vllm_is_ratio to LigerFusedLinearGRPOLoss in compute_liger_loss by @yukiu00 in https://github.com/huggingface/trl/pull/5031\r\n* feature: top_k selective_log_softmax by @LeonEricsson in https://github.com/huggingface/trl/pull/5104\r\n* Add Trackio integration for model card visualization by @qgallouedec in https://github.com/huggingface/trl/pull/5101\r\n* Update tool handling to support JSON string schemas in trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5118\r\n* Refactor DPO by @qgallouedec in https://github.com/huggingface/trl/pull/3906\r\n* Add support for Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/4225\r\n* Fix default learning_rate in PPO according to paper by @albertvillanova in https://github.com/huggingface/trl/pull/5174\r\n* Fix default learning_rate in BCO according to paper by @albertvillanova in https://github.com/huggingface/trl/pull/5173\r\n* feature: Configurable num logprobs in vLLM generation by @LeonEricsson in https://github.com/huggingface/trl/pull/5107\r\n\r\n## Fixes\r\n\r\n* [GRPO] fix: remove SAPO temperature check by @LeonEricsson in https://github.com/huggingface/trl/pull/5042\r\n* fix: Use `launch_args` for all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5059\r\n* Fix GRPO multi-turn training with liger kernels by @albertvillanova in https://github.com/huggingface/trl/pull/4975\r\n* fix: Set `num_labels` to 1 in causal model initialization for RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5066\r\n* [SFT] Fix high vRAM consumption during eval with liger kernel by @LoganVegnaSHOP in https://github.com/huggingface/trl/pull/5069\r\n* Fix BFD packing for SFT datasets by @albertvillanova in https://github.com/huggingface/trl/pull/5076\r\n* Fix DPO and RLOO incompatibility with FSDP2 by @flutist in https://github.com/huggingface/trl/pull/4838\r\n* Fix SFT loss type rewards being overwritten in dpo_loss() by @Mr-Neutr0n in https://github.com/huggingface/trl/pull/5079\r\n* Fix Qwen3 schema by @qgallouedec in https://github.com/huggingface/trl/pull/5111\r\n* Add check for `None` in `get_trackio_space_url()` to prevent errors by @qgallouedec in https://github.com/huggingface/trl/pull/5115\r\n* Fix `trl <command> --help` TypeError caused by unescaped `%` in `TrainingArguments` help strings by @albertvillanova in https://github.com/huggingface/trl/pull/5135\r\n* Fix PPOTrainer.save_model by @albertvillanova in https://github.com/huggingface/trl/pull/5151\r\n* Fix `SFTTrainer` support for single-image data by @qgallouedec in https://github.com/huggingface/trl/pull/5132\r\n* Fix structured_outputs handling and tool normalization in vLLM backend by @ehofm in https://github.com/huggingface/trl/pull/5155\r\n* fix: wake up vLLM weights before sync to prevent writes to freed memory by @bledden in https://github.com/huggingface/trl/pull/5147\r\n* Accept mm_token_type_ids in GRPO/RLOO _get_per_token_logps_and_entropies by @albertvillanova in https://github.com/huggingface/trl/pull/5176\r\n\r\n## Documentation and Examples\r\n\r\n* [minor] docs: typo in `grpo_trainer.md` by @casinca in https://github.com/huggingface/trl/pull/5047\r\n* docs: add DeepSeek-R1 training dynamics and GRPO example by @JenWei0312 in https://github.com/huggingface/trl/pull/5053\r\n* docs: Add INTELLECT-2 (2505.07291) to Paper Index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5061\r\n* docs: Add REINFORCE++ (2501.03262) to Paper Index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5062\r\n* docs: Add XPO (2405.21046) to Paper Index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5068\r\n* docs: Add RPO paper (2405.16436) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5070\r\n* docs: Add SimPO paper (2405.14734) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5071\r\n* docs: Add TR-DPO paper (2404.09656) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5078\r\n* docs: Add ORPO paper (2403.07691) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5080\r\n* docs: Add CPO paper (2401.08417) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5081\r\n* docs: Add GKD paper (2306.13649) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5082\r\n* docs: Add PRM paper (2211.14275) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5083\r\n* docs: Add T5 packing paper (1910.10683) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5084\r\n* docs: Add PPO paper (1707.06347) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5085\r\n* docs: Add MPO paper (2411.10442) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5089\r\n* docs: add Multi-Node Training subsection (#4384) by @nabin2004 in https://github.com/huggingface/trl/pull/5091\r\n* docs: Unify model examples to use trl-lib namespace by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4431\r\n* Add Tiny Aya tool calling examples (script/notebook) by @sergiopaniego in https://github.com/huggingface/trl/pull/5123\r\n* Fix wording in DPO and SFT trainer documentation for clarity by @qgallouedec in https://github.com/huggingface/trl/pull/5140\r\n* Fix type of TrainingArguments.logging_steps in docs by @albertvillanova in https://github.com/huggingface/trl/pull/5149\r\n* Fix Liquid syntax error in DPO trainer docs caused by double braces in LaTeX by @albertvillanova in https://github.com/huggingface/trl/pull/5153\r\n* Document parameters with differing default values in experimental configs by @albertvillanova in https://github.com/huggingface/trl/pull/5172\r\n\r\n## Deprecations\r\n\r\n* Remove deprecated BCO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5045\r\n* Remove deprecated CPO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5046\r\n* Remove deprecated Judges after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5048\r\n* Remove deprecated ORPO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5050\r\n* Remove deprecated PPO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5051\r\n* Remove deprecated PRM after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5052\r\n* Remove deprecated XPO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5055\r\n* Remove deprecated RLOOConfig.max_prompt_length by @albertvillanova in https://github.com/huggingface/trl/pull/5056\r\n* Remove deprecated classes moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5044\r\n* Remove deprecated mergekit_utils moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5057\r\n* Rename input keys in `RewardTrainer` collator from `chosen/rejected_input_ids` to `chosen/rejected_ids` by @qgallouedec in https://github.com/huggingface/trl/pull/5179\r\n\r\n## CI Improvements\r\n\r\n* Upgrade GitHub Actions to latest versions by @salmanmkc in https://github.com/huggingface/trl/pull/4893\r\n* Remove duplicated tests for SFT and add gradient checkpointing tests by @qgallouedec in https://github.com/huggingface/trl/pull/5054\r\n* Update model from SequenceClassification to CausalLM in `RewardTrainer` tests by @qgallouedec in https://github.com/huggingface/trl/pull/5060\r\n* Fix CI ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?) by @albertvillanova in https://github.com/huggingface/trl/pull/5074\r\n* Add more tests for `get_training_chat_template` by @qgallouedec in https://github.com/huggingface/trl/pull/5108\r\n* Add test for Cohere2 models by @qgallouedec in https://github.com/huggingface/trl/pull/5116\r\n* Remove revision references in dataset loading for toolcall tests by @qgallouedec in https://github.com/huggingface/trl/pull/5133\r\n* Fix NameError: name 'importlib' is not defined by @albertvillanova in https://github.com/huggingface/trl/pull/5134\r\n* Fix CI by removing liger-kernel from dev deps by @qgallouedec in https://github.com/huggingface/trl/pull/5163\r\n* Fix experimental TestUpdateWithReplayBuffer: ValueError: `train_dataset` is required by @albertvillanova in https://github.com/huggingface/trl/pull/5171\r\n* Update upstream tracking info about CI PyTorch JIT deprecation warnings by @albertvillanova in https://github.com/huggingface/trl/pull/5166\r\n\r\n## Miscellaneous\r\n\r\n* Fix logging warning suppression with scoped override for seq-clf head key by @qgallouedec in https://github.com/huggingface/trl/pull/5058\r\n* Fix logging warning suppression for transformers 4.56.2 by @albertvillanova in https://github.com/huggingface/trl/pull/5077\r\n* Validate reward model has 1 num_labels by @albertvillanova in https://github.com/huggingface/trl/pull/5087\r\n* Fix style by @albertvillanova in https://github.com/huggingface/trl/pull/5106\r\n* Remove outdated liger-kernel compatibility checks and warnings in tests and SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5105\r\n* Add validation for conversational prompts in multimodal training by @qgallouedec in https://github.com/huggingface/trl/pull/5067\r\n* Update version check for transformers to 5.2.0 in online_dpo_trainer.py by @qgallouedec in https://github.com/huggingface/trl/pull/5110\r\n* Add GLM-4.5 model to tests by @qgallouedec in https://github.com/huggingface/trl/pull/5114\r\n* Fix import latency [1/N]: Extract _LazyModule to dedicated module by @albertvillanova in https://github.com/huggingface/trl/pull/5128\r\n* Fix import latency [2/N]: Implement native _is_package_available by @albertvillanova in https://github.com/huggingface/trl/pull/5129\r\n* refactor(gkd_trainer): small optim by @casinca in https://github.com/huggingface/trl/pull/5143\r\n* Move common fields from stable trainer configs to BaseConfig by @albertvillanova in https://github.com/huggingface/trl/pull/5136\r\n* Use BaseConfig in all experimental configs by @albertvillanova in https://github.com/huggingface/trl/pull/5148\r\n* Raise ValueError for None train_dataset in core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5157\r\n* Revert changes in vLLM client/server by @qgallouedec in https://github.com/huggingface/trl/pull/5165\r\n\r\n### Refactor CLI\r\n\r\n* Refactor CLI [1/N]: Refactor into modular command architecture by @albertvillanova in https://github.com/huggingface/trl/pull/5124\r\n* Refactor CLI [2/N]: Move accelerate concerns into TrainingCommand by @albertvillanova in https://github.com/huggingface/trl/pull/5159\r\n* Refactor CLI [3/N]: Self-contain VllmServeCommand argument parsing by @albertvillanova in https://github.com/huggingface/trl/pull/5160\r\n\r\n## What's Changed\r\n\r\n* [minor] docs: typo in `grpo_trainer.md` by @casinca in https://github.com/huggingface/trl/pull/5047\r\n* ⬆️  Bump dev version by @albertvillanova in https://github.com/huggingface/trl/pull/5049\r\n* [GRPO] fix: remove SAPO temperature check by @LeonEricsson in https://github.com/huggingface/trl/pull/5042\r\n* Remove deprecated BCO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5045\r\n* Remove deprecated CPO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5046\r\n* Remove deprecated Judges after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5048\r\n* Upgrade GitHub Actions to latest versions by @salmanmkc in https://github.com/huggingface/trl/pull/4893\r\n* Remove deprecated ORPO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5050\r\n* Remove deprecated PPO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5051\r\n* Remove deprecated PRM after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5052\r\n* Remove deprecated XPO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5055\r\n* Remove deprecated RLOOConfig.max_prompt_length by @albertvillanova in https://github.com/huggingface/trl/pull/5056\r\n* Remove deprecated classes moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5044\r\n* Remove duplicated tests for SFT and add gradient checkpointing tests by @qgallouedec in https://github.com/huggingface/trl/pull/5054\r\n* Remove deprecated mergekit_utils moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5057\r\n* docs: add DeepSeek-R1 training dynamics and GRPO example by @JenWei0312 in https://github.com/huggingface/trl/pull/5053\r\n* docs: Add INTELLECT-2 (2505.07291) to Paper Index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5061\r\n* docs: Add REINFORCE++ (2501.03262) to Paper Index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5062\r\n* docs: Add XPO (2405.21046) to Paper Index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5068\r\n* docs: Add RPO paper (2405.16436) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5070\r\n* docs: Add SimPO paper (2405.14734) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5071\r\n* Fix logging warning suppression with scoped override for seq-clf head key by @qgallouedec in https://github.com/huggingface/trl/pull/5058\r\n* fix: Use `launch_args` for all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5059\r\n* Fix GRPO multi-turn training with liger kernels by @albertvillanova in https://github.com/huggingface/trl/pull/4975\r\n* fix: Set `num_labels` to 1 in causal model initialization for RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5066\r\n* Fix logging warning suppression for transformers 4.56.2 by @albertvillanova in https://github.com/huggingface/trl/pull/5077\r\n* Update model from SequenceClassification to CausalLM in `RewardTrainer` tests by @qgallouedec in https://github.com/huggingface/trl/pull/5060\r\n* Fix CI ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?) by @albertvillanova in https://github.com/huggingface/trl/pull/5074\r\n* [SFT] Fix high vRAM consumption during eval with liger kernel by @LoganVegnaSHOP in https://github.com/huggingface/trl/pull/5069\r\n* docs: Add TR-DPO paper (2404.09656) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5078\r\n* docs: Add ORPO paper (2403.07691) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5080\r\n* docs: Add CPO paper (2401.08417) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5081\r\n* docs: Add GKD paper (2306.13649) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5082\r\n* docs: Add PRM paper (2211.14275) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5083\r\n* docs: Add T5 packing paper (1910.10683) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5084\r\n* docs: Add PPO paper (1707.06347) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5085\r\n* Fix BFD packing for SFT datasets by @albertvillanova in https://github.com/huggingface/trl/pull/5076\r\n* Validate reward model has 1 num_labels by @albertvillanova in https://github.com/huggingface/trl/pull/5087\r\n* docs: Add MPO paper (2411.10442) to paper index by @behroozazarkhalili in https://github.com/huggingface/trl/pull/5089\r\n* docs: add Multi-Node Training subsection (#4384) by @nabin2004 in https://github.com/huggingface/trl/pull/5091\r\n* docs: Unify model examples to use trl-lib namespace by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4431\r\n* Implement Agent Skills [1/N]: Create training skill (MVP) by @albertvillanova in https://github.com/huggingface/trl/pull/5096\r\n* Pass vllm_is_ratio to LigerFusedLinearGRPOLoss in compute_liger_loss by @yukiu00 in https://github.com/huggingface/trl/pull/5031\r\n* Fix DPO and RLOO incompatibility with FSDP2 by @flutist in https://github.com/huggingface/trl/pull/4838\r\n* feature: top_k selective_log_softmax by @LeonEricsson in https://github.com/huggingface/trl/pull/5104\r\n* Implement Agent Skills [2/N]:  Create skills module by @albertvillanova in https://github.com/huggingface/trl/pull/5097\r\n* Fix style by @albertvillanova in https://github.com/huggingface/trl/pull/5106\r\n* Add Trackio integration for model card visualization by @qgallouedec in https://github.com/huggingface/trl/pull/5101\r\n* Fix SFT loss type rewards being overwritten in dpo_loss() by @Mr-Neutr0n in https://github.com/huggingface/trl/pull/5079\r\n* Remove outdated liger-kernel compatibility checks and warnings in tests and SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5105\r\n* Implement Agent Skills [3/N]: Create skills installer by @albertvillanova in https://github.com/huggingface/trl/pull/5100\r\n* Add validation for conversational prompts in multimodal training by @qgallouedec in https://github.com/huggingface/trl/pull/5067\r\n* Update version check for transformers to 5.2.0 in online_dpo_trainer.py by @qgallouedec in https://github.com/huggingface/trl/pull/5110\r\n* Add more tests for `get_training_chat_template` by @qgallouedec in https://github.com/huggingface/trl/pull/5108\r\n* Add test for Cohere2 models by @qgallouedec in https://github.com/huggingface/trl/pull/5116\r\n* Fix Qwen3 schema by @qgallouedec in https://github.com/huggingface/trl/pull/5111\r\n* Add check for `None` in `get_trackio_space_url()` to prevent errors by @qgallouedec in https://github.com/huggingface/trl/pull/5115\r\n* Add GLM-4.5 model to tests by @qgallouedec in https://github.com/huggingface/trl/pull/5114\r\n* Add Tiny Aya tool calling examples (script/notebook) by @sergiopaniego in https://github.com/huggingface/trl/pull/5123\r\n* Update tool handling to support JSON string schemas in trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5118\r\n* Implement Agent Skills [4/N]: Create skills CLI by @albertvillanova in https://github.com/huggingface/trl/pull/5103\r\n* Refactor CLI [1/N]: Refactor into modular command architecture by @albertvillanova in https://github.com/huggingface/trl/pull/5124\r\n* Remove revision references in dataset loading for toolcall tests by @qgallouedec in https://github.com/huggingface/trl/pull/5133\r\n* Refactor DPO by @qgallouedec in https://github.com/huggingface/trl/pull/3906\r\n* Fix import latency [1/N]: Extract _LazyModule to dedicated module by @albertvillanova in https://github.com/huggingface/trl/pull/5128\r\n* Fix import latency [2/N]: Implement native _is_package_available by @albertvillanova in https://github.com/huggingface/trl/pull/5129\r\n* Fix NameError: name 'importlib' is not defined by @albertvillanova in https://github.com/huggingface/trl/pull/5134\r\n* Fix `trl <command> --help` TypeError caused by unescaped `%` in `TrainingArguments` help strings by @albertvillanova in https://github.com/huggingface/trl/pull/5135\r\n* Add `environment_factory` to `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/5093\r\n* refactor(gkd_trainer): small optim by @casinca in https://github.com/huggingface/trl/pull/5143\r\n* Move common fields from stable trainer configs to BaseConfig by @albertvillanova in https://github.com/huggingface/trl/pull/5136\r\n* Fix wording in DPO and SFT trainer documentation for clarity by @qgallouedec in https://github.com/huggingface/trl/pull/5140\r\n* Fix PPOTrainer.save_model by @albertvillanova in https://github.com/huggingface/trl/pull/5151\r\n* Use BaseConfig in all experimental configs by @albertvillanova in https://github.com/huggingface/trl/pull/5148\r\n* Fix type of TrainingArguments.logging_steps in docs by @albertvillanova in https://github.com/huggingface/trl/pull/5149\r\n* Add support for Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/4225\r\n* Fix `SFTTrainer` support for single-image data by @qgallouedec in https://github.com/huggingface/trl/pull/5132\r\n* Fix CI by removing liger-kernel from dev deps by @qgallouedec in https://github.com/huggingface/trl/pull/5163\r\n* Fix structured_outputs handling and tool normalization in vLLM backend by @ehofm in https://github.com/huggingface/trl/pull/5155\r\n* fix: wake up vLLM weights before sync to prevent writes to freed memory by @bledden in https://github.com/huggingface/trl/pull/5147\r\n* Fix Liquid syntax error in DPO trainer docs caused by double braces in LaTeX by @albertvillanova in https://github.com/huggingface/trl/pull/5153\r\n* Raise ValueError for None train_dataset in core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5157\r\n* Refactor CLI [2/N]: Move accelerate concerns into TrainingCommand by @albertvillanova in https://github.com/huggingface/trl/pull/5159\r\n* Refactor CLI [3/N]: Self-contain VllmServeCommand argument parsing by @albertvillanova in https://github.com/huggingface/trl/pull/5160\r\n* Revert changes in vLLM client/server by @qgallouedec in https://github.com/huggingface/trl/pull/5165\r\n* Fix experimental TestUpdateWithReplayBuffer: ValueError: `train_dataset` is required by @albertvillanova in https://github.com/huggingface/trl/pull/5171\r\n* Fix default learning_rate in PPO according to paper by @albertvillanova in https://github.com/huggingface/trl/pull/5174\r\n* Accept mm_token_type_ids in GRPO/RLOO _get_per_token_logps_and_entropies by @albertvillanova in https://github.com/huggingface/trl/pull/5176\r\n* Fix default learning_rate in BCO according to paper by @albertvillanova in https://github.com/huggingface/trl/pull/5173\r\n* Document parameters with differing default values in experimental configs by @albertvillanova in https://github.com/huggingface/trl/pull/5172\r\n* Update upstream tracking info about CI PyTorch JIT deprecation warnings by @albertvillanova in https://github.com/huggingface/trl/pull/5166\r\n* Rename input keys in `RewardTrainer` collator from `chosen/rejected_input_ids` to `chosen/rejected_ids` by @qgallouedec in https://github.com/huggingface/trl/pull/5179\r\n* feature: Configurable num logprobs in vLLM generation by @LeonEricsson in https://github.com/huggingface/trl/pull/5107\r\n* Release: v0.29 by @qgallouedec in https://github.com/huggingface/trl/pull/5181\r\n\r\n## New Contributors\r\n\r\n* @LoganVegnaSHOP made their first contribution in https://github.com/huggingface/trl/pull/5069\r\n* @yukiu00 made their first contribution in https://github.com/huggingface/trl/pull/5031\r\n* @flutist made their first contribution in https://github.com/huggingface/trl/pull/4838\r\n* @Mr-Neutr0n made their first contribution in https://github.com/huggingface/trl/pull/5079\r\n* @ehofm made their first contribution in https://github.com/huggingface/trl/pull/5155\r\n* @bledden made their first contribution in https://github.com/huggingface/trl/pull/5147\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v0.28.0...v0.29.0\r\n","publishedAt":"2026-02-25T22:38:09.000Z","url":"https://github.com/huggingface/trl/releases/tag/v0.29.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_jAjl59S0OzPM70FGxKmgd","version":"v0.28.0","type":"feature","title":" v0.28.0","summary":"## Features\r\n* [GRPOTrainer]: Agent Training Supports Async Tool Calls by @pramodith in https://github.com/huggingface/trl/pull/4742\r\n* Add retry stra...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## Features\r\n* [GRPOTrainer]: Agent Training Supports Async Tool Calls by @pramodith in https://github.com/huggingface/trl/pull/4742\r\n* Add retry strategy to vLLM Client for increased robustness by @apalmas-saifh in https://github.com/huggingface/trl/pull/4845\r\n* Enable vLLM sleep mode for generation in Online DPO by @winglian in https://github.com/huggingface/trl/pull/4882\r\n* Support tool call data in `is_conversational` by @qgallouedec in https://github.com/huggingface/trl/pull/4923\r\n* [GRPO] Add parquet logging for completions with individual rewards by @qgallouedec in https://github.com/huggingface/trl/pull/4818\r\n* Update wordle.py example with masking of env tokens by @sergiopaniego in https://github.com/huggingface/trl/pull/4895\r\n* NeMo-Gym Integration by @cmunley1 in https://github.com/huggingface/trl/pull/4848\r\n\r\n## Experimental\r\n* Refactor KTO coordinated with DPO [c/N]: Remove ref_model_init_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4837\r\n* Refactor KTO coordinated with DPO [e/N]: Remove label_pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/4875\r\n* Refactor KTO coordinated with DPO [d/N]: Remove base_model_attribute_name by @albertvillanova in https://github.com/huggingface/trl/pull/4862\r\n* Fix type hint in `openenv/utils.py`: fallback for no vLLM installed case by @Datta0 in https://github.com/huggingface/trl/pull/4868\r\n* Remove label_pad_token_id from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4878\r\n* GOLD training speed up by @141forever in https://github.com/huggingface/trl/pull/4888\r\n* Remove ref_model_init_kwargs from experimental BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4946\r\n* Remove max_prompt_length from experimental PRM by @albertvillanova in https://github.com/huggingface/trl/pull/4963\r\n* Remove max_prompt_length from experimental BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4964\r\n* Remove max_prompt_length from experimental CPO by @albertvillanova in https://github.com/huggingface/trl/pull/4965\r\n* Remove max_prompt_length from experimental ORPO by @albertvillanova in https://github.com/huggingface/trl/pull/4966\r\n* Remove padding_value from experimental CPO and use pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/4962\r\n\r\n## Fixes\r\n* Fix _patch_transformers_hybrid_cache for peft by @albertvillanova in https://github.com/huggingface/trl/pull/4844\r\n* Refactor KTO [4/N]: Remove unused padding_value by @albertvillanova in https://github.com/huggingface/trl/pull/4839\r\n* Fix: undefined `current_gradient_accumulation_steps` by @qgallouedec in https://github.com/huggingface/trl/pull/4852\r\n* fix(DeepSeek OPSM): passing correct (vLLM) logprobs by @casinca in https://github.com/huggingface/trl/pull/4857\r\n* Fix SFT training for prompt-completion type and transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4880\r\n* Bugfix: Logprob drift in vLLM serving mode (compared to colocate mode) by @kdubovikov in https://github.com/huggingface/trl/pull/4873\r\n* Fix import path for `get_open_port` based on vLLM version by @qgallouedec in https://github.com/huggingface/trl/pull/4883\r\n* Fix RewardTrainer's results not reproducible by @liyc-ai in https://github.com/huggingface/trl/pull/4887\r\n* `device_map` init consistency in GRPO/RLOO/KTO by @qgallouedec in https://github.com/huggingface/trl/pull/4909\r\n* Fix extra EOS appended in DPO preprocessing for conversational data by @qgallouedec in https://github.com/huggingface/trl/pull/4908\r\n* Fix SFTTrainer init logic: remove TrainingArguments.push_to_hub_token only for transformers < v5 by @albertvillanova in https://github.com/huggingface/trl/pull/4942\r\n* Fix PPO run_name parameter not taking effect by @mel3c in https://github.com/huggingface/trl/pull/4945\r\n* Remove access to `warnings_issued` by @qgallouedec in https://github.com/huggingface/trl/pull/4960\r\n* Revert change in GRPO from NeMo-Gym Integration by @qgallouedec in https://github.com/huggingface/trl/pull/4970\r\n\r\n## Documentation and Examples\r\n* Add Nash Learning from Human Feedback paper to paper index by @kansalaman in https://github.com/huggingface/trl/pull/4860\r\n* Update OpenEnv dependency to new version for hf jobs scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/4843\r\n* Enhance GRPO documentation with scaling notes by @javadtaghia in https://github.com/huggingface/trl/pull/4849\r\n* Created new PTT integration docs as requested by @adityachallapally in https://github.com/huggingface/trl/pull/4907\r\n* docs: add DoRA (2402.09353) to Paper Index by @billycrapediem in https://github.com/huggingface/trl/pull/4892\r\n\r\n## Deprecations\r\n* Remove unused padding_value from BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4846\r\n* Remove deprecated parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4847\r\n* Deprecate parameters in `DPOConfig` by @qgallouedec in https://github.com/huggingface/trl/pull/4969\r\n* Replace `warmup_ratio` with `warmup_steps` by @qgallouedec in https://github.com/huggingface/trl/pull/4983\r\n\r\n## CI Improvements\r\n* Support triggering CI via push to ci-* branches by @albertvillanova in https://github.com/huggingface/trl/pull/4840\r\n* Revert CI hotfix pinning transformers 4.57.4 after tiny model regeneration by @albertvillanova in https://github.com/huggingface/trl/pull/4833\r\n* Use pytest-datadir in CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/4836\r\n* Fix CI with dev dependencies: Mark Qwen3-VL tests as xfail by @albertvillanova in https://github.com/huggingface/trl/pull/4851\r\n* Use pytest-datadir for accelerate config files by @albertvillanova in https://github.com/huggingface/trl/pull/4861\r\n* Update transformer version checks and documentation for lr_scheduler_kwargs workaround by @qgallouedec in https://github.com/huggingface/trl/pull/4876\r\n* Test distributed training for `RewardTrainer`, `RLOOTrainer` and `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4823\r\n* Mark ZeRO 2 as xfail in distributed tests due to current failure by @qgallouedec in https://github.com/huggingface/trl/pull/4885\r\n* Transformers v5 release: extend xfail condition for `TestGRPOTrainer.test_training_vlm_and_liger` and update version checks by @qgallouedec in https://github.com/huggingface/trl/pull/4898\r\n* Fix CI NotImplementedError for bfloat16 by @albertvillanova in https://github.com/huggingface/trl/pull/4902\r\n* Fix CI AssertionError: Parameter has not changed by @albertvillanova in https://github.com/huggingface/trl/pull/4904\r\n* Fix CI TypeError in llm-blender tests by @albertvillanova in https://github.com/huggingface/trl/pull/4919\r\n* Fix CI AssertionError: assert not True by @albertvillanova in https://github.com/huggingface/trl/pull/4921\r\n* Fix CI ValueError for 0 temperature by @albertvillanova in https://github.com/huggingface/trl/pull/4916\r\n* Set model dtype to float32 in tests of trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4924\r\n* Set model dtype to float32 in experimental tests of trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4925\r\n* Add test for training with `compute_metrics` in `SFTTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4950\r\n* Add test for tool call data in `RewardTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4959\r\n* Add test for training with `compute_metrics` in `RewardTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4958\r\n* Fix test_train_with_chat_template_kwargs by @qgallouedec in https://github.com/huggingface/trl/pull/4971\r\n\r\n## Miscellaneous\r\n* Update `CITATION.cff` by @qgallouedec in https://github.com/huggingface/trl/pull/4856\r\n* Update generate_tiny_models.py: CohereForAI -> CohereLabs by @Michellehbn in https://github.com/huggingface/trl/pull/4877\r\n* Refactor vLLM generation [1/N]: Extract vLLM generation by @albertvillanova in https://github.com/huggingface/trl/pull/4700\r\n* Rearrange variable assignments in `DataCollatorForVisionLanguageModeling` by @qgallouedec in https://github.com/huggingface/trl/pull/4911\r\n* Fix help text formatting for `max_length` in `RewardConfig` and `SFTConfig` by @qgallouedec in https://github.com/huggingface/trl/pull/4910\r\n* Comment about overriding prediction_step in GRPOTrainer and RLOOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4913\r\n* Remove gradient checkpointing option from various training scripts  by @qgallouedec in https://github.com/huggingface/trl/pull/4905\r\n* Remove chat template setup in dpo_vlm.py by @qgallouedec in https://github.com/huggingface/trl/pull/4906\r\n* Update learning rate comments and add assertions for reference model parameters in GRPO and RLOO tests by @qgallouedec in https://github.com/huggingface/trl/pull/4914\r\n* Add validation for `sync_ref_model` in `GRPOTrainer` and `RLOOTrainer` when using PEFT models by @qgallouedec in https://github.com/huggingface/trl/pull/4912\r\n* Require transformers<5 with PairRMJudge by @albertvillanova in https://github.com/huggingface/trl/pull/4926\r\n* Move VLLMClient to generation module by @albertvillanova in https://github.com/huggingface/trl/pull/4928\r\n* Fix profiling of VLLMGeneration.sync_weights by @albertvillanova in https://github.com/huggingface/trl/pull/4931\r\n* Fix import statement for import_utils in vllm_client.py by @qgallouedec in https://github.com/huggingface/trl/pull/4932\r\n* Set default top_k to 0 in VLLMClient by @albertvillanova in https://github.com/huggingface/trl/pull/4927\r\n* Minor fix docs style by @albertvillanova in https://github.com/huggingface/trl/pull/4953\r\n\r\n## What's Changed\r\n* ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/4835\r\n* Support triggering CI via push to ci-* branches by @albertvillanova in https://github.com/huggingface/trl/pull/4840\r\n* Revert CI hotfix pinning transformers 4.57.4 after tiny model regeneration by @albertvillanova in https://github.com/huggingface/trl/pull/4833\r\n* Use pytest-datadir in CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/4836\r\n* Refactor KTO coordinated with DPO [c/N]: Remove ref_model_init_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4837\r\n* Fix _patch_transformers_hybrid_cache for peft by @albertvillanova in https://github.com/huggingface/trl/pull/4844\r\n* Refactor KTO [4/N]: Remove unused padding_value by @albertvillanova in https://github.com/huggingface/trl/pull/4839\r\n* Remove unused padding_value from BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4846\r\n* Fix CI with dev dependencies: Mark Qwen3-VL tests as xfail by @albertvillanova in https://github.com/huggingface/trl/pull/4851\r\n* Fix: undefined `current_gradient_accumulation_steps` by @qgallouedec in https://github.com/huggingface/trl/pull/4852\r\n* Remove deprecated parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4847\r\n* Add Nash Learning from Human Feedback paper to paper index by @kansalaman in https://github.com/huggingface/trl/pull/4860\r\n* Use pytest-datadir for accelerate config files by @albertvillanova in https://github.com/huggingface/trl/pull/4861\r\n* Update OpenEnv dependency to new version for hf jobs scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/4843\r\n* Update `CITATION.cff` by @qgallouedec in https://github.com/huggingface/trl/pull/4856\r\n* [GRPOTrainer]: Agent Training Supports Async Tool Calls by @pramodith in https://github.com/huggingface/trl/pull/4742\r\n* Enhance GRPO documentation with scaling notes by @javadtaghia in https://github.com/huggingface/trl/pull/4849\r\n* Add retry strategy to vLLM Client for increased robustness by @apalmas-saifh in https://github.com/huggingface/trl/pull/4845\r\n* Update generate_tiny_models.py: CohereForAI -> CohereLabs by @Michellehbn in https://github.com/huggingface/trl/pull/4877\r\n* Refactor KTO coordinated with DPO [e/N]: Remove label_pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/4875\r\n* Refactor KTO coordinated with DPO [d/N]: Remove base_model_attribute_name by @albertvillanova in https://github.com/huggingface/trl/pull/4862\r\n* Fix type hint in `openenv/utils.py`: fallback for no vLLM installed case by @Datta0 in https://github.com/huggingface/trl/pull/4868\r\n* Update transformer version checks and documentation for lr_scheduler_kwargs workaround by @qgallouedec in https://github.com/huggingface/trl/pull/4876\r\n* fix(DeepSeek OPSM): passing correct (vLLM) logprobs by @casinca in https://github.com/huggingface/trl/pull/4857\r\n* Remove label_pad_token_id from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4878\r\n* Fix SFT training for prompt-completion type and transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4880\r\n* Bugfix: Logprob drift in vLLM serving mode (compared to colocate mode) by @kdubovikov in https://github.com/huggingface/trl/pull/4873\r\n* Enable vLLM sleep mode for generation in Online DPO by @winglian in https://github.com/huggingface/trl/pull/4882\r\n* Test distributed training for `RewardTrainer`, `RLOOTrainer` and `GRPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4823\r\n* Mark ZeRO 2 as xfail in distributed tests due to current failure by @qgallouedec in https://github.com/huggingface/trl/pull/4885\r\n* Fix import path for `get_open_port` based on vLLM version by @qgallouedec in https://github.com/huggingface/trl/pull/4883\r\n* Fix RewardTrainer's results not reproducible by @liyc-ai in https://github.com/huggingface/trl/pull/4887\r\n* GOLD training speed up by @141forever in https://github.com/huggingface/trl/pull/4888\r\n* Transformers v5 release: extend xfail condition for `TestGRPOTrainer.test_training_vlm_and_liger` and update version checks by @qgallouedec in https://github.com/huggingface/trl/pull/4898\r\n* Fix CI NotImplementedError for bfloat16 by @albertvillanova in https://github.com/huggingface/trl/pull/4902\r\n* Fix CI AssertionError: Parameter has not changed by @albertvillanova in https://github.com/huggingface/trl/pull/4904\r\n* Refactor vLLM generation [1/N]: Extract vLLM generation by @albertvillanova in https://github.com/huggingface/trl/pull/4700\r\n* Created new PTT integration docs as requested by @adityachallapally in https://github.com/huggingface/trl/pull/4907\r\n* Fix CI TypeError in llm-blender tests by @albertvillanova in https://github.com/huggingface/trl/pull/4919\r\n* Rearrange variable assignments in `DataCollatorForVisionLanguageModeling` by @qgallouedec in https://github.com/huggingface/trl/pull/4911\r\n* Fix help text formatting for `max_length` in `RewardConfig` and `SFTConfig` by @qgallouedec in https://github.com/huggingface/trl/pull/4910\r\n* `device_map` init consistency in GRPO/RLOO/KTO by @qgallouedec in https://github.com/huggingface/trl/pull/4909\r\n* Comment about overriding prediction_step in GRPOTrainer and RLOOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4913\r\n* Remove gradient checkpointing option from various training scripts  by @qgallouedec in https://github.com/huggingface/trl/pull/4905\r\n* docs: add DoRA (2402.09353) to Paper Index by @billycrapediem in https://github.com/huggingface/trl/pull/4892\r\n* Fix CI AssertionError: assert not True by @albertvillanova in https://github.com/huggingface/trl/pull/4921\r\n* Fix CI ValueError for 0 temperature by @albertvillanova in https://github.com/huggingface/trl/pull/4916\r\n* Fix extra EOS appended in DPO preprocessing for conversational data by @qgallouedec in https://github.com/huggingface/trl/pull/4908\r\n* Remove chat template setup in dpo_vlm.py by @qgallouedec in https://github.com/huggingface/trl/pull/4906\r\n* Update learning rate comments and add assertions for reference model parameters in GRPO and RLOO tests by @qgallouedec in https://github.com/huggingface/trl/pull/4914\r\n* Add validation for `sync_ref_model` in `GRPOTrainer` and `RLOOTrainer` when using PEFT models by @qgallouedec in https://github.com/huggingface/trl/pull/4912\r\n* Support tool call data in `is_conversational` by @qgallouedec in https://github.com/huggingface/trl/pull/4923\r\n* Set model dtype to float32 in tests of trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4924\r\n* Require transformers<5 with PairRMJudge by @albertvillanova in https://github.com/huggingface/trl/pull/4926\r\n* Move VLLMClient to generation module by @albertvillanova in https://github.com/huggingface/trl/pull/4928\r\n* Set model dtype to float32 in experimental tests of trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4925\r\n* Fix profiling of VLLMGeneration.sync_weights by @albertvillanova in https://github.com/huggingface/trl/pull/4931\r\n* Fix import statement for import_utils in vllm_client.py by @qgallouedec in https://github.com/huggingface/trl/pull/4932\r\n* Set default top_k to 0 in VLLMClient by @albertvillanova in https://github.com/huggingface/trl/pull/4927\r\n* [GRPO] Add parquet logging for completions with individual rewards by @qgallouedec in https://github.com/huggingface/trl/pull/4818\r\n* Fix SFTTrainer init logic: remove TrainingArguments.push_to_hub_token only for transformers < v5 by @albertvillanova in https://github.com/huggingface/trl/pull/4942\r\n* Remove ref_model_init_kwargs from experimental BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4946\r\n* Update wordle.py example with masking of env tokens by @sergiopaniego in https://github.com/huggingface/trl/pull/4895\r\n* Fix PPO run_name parameter not taking effect by @mel3c in https://github.com/huggingface/trl/pull/4945\r\n* Minor fix docs style by @albertvillanova in https://github.com/huggingface/trl/pull/4953\r\n* Add test for training with `compute_metrics` in `SFTTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4950\r\n* Remove access to `warnings_issued` by @qgallouedec in https://github.com/huggingface/trl/pull/4960\r\n* NeMo-Gym Integration by @cmunley1 in https://github.com/huggingface/trl/pull/4848\r\n* Add test for tool call data in `RewardTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4959\r\n* Add test for training with `compute_metrics` in `RewardTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4958\r\n* Remove max_prompt_length from experimental PRM by @albertvillanova in https://github.com/huggingface/trl/pull/4963\r\n* Remove max_prompt_length from experimental BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4964\r\n* Remove max_prompt_length from experimental CPO by @albertvillanova in https://github.com/huggingface/trl/pull/4965\r\n* Remove max_prompt_length from experimental ORPO by @albertvillanova in https://github.com/huggingface/trl/pull/4966\r\n* Revert change in GRPO from NeMo-Gym Integration by @qgallouedec in https://github.com/huggingface/trl/pull/4970\r\n* Fix test_train_with_chat_template_kwargs by @qgallouedec in https://github.com/huggingface/trl/pull/4971\r\n* Remove padding_value from experimental CPO and use pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/4962\r\n\r\n* Remove truncation from tokenizer calls if no max_length by @albertvillanova in https://github.com/huggingface/trl/pull/4972\r\n* Set specific OpenEnv version when installed by @sergiopaniego in https://github.com/huggingface/trl/pull/4978\r\n* Fix add_column in test_train_with_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4979\r\n* Support truncated completions in GRPO multi-turn training by @albertvillanova in https://github.com/huggingface/trl/pull/4976\r\n* Replace `torch.allclose` with `torch.testing.assert_close` by @qgallouedec in https://github.com/huggingface/trl/pull/4977\r\n* Simplify instructions of installation of OpenEnv  by @sergiopaniego in https://github.com/huggingface/trl/pull/4980\r\n\r\n* Deprecate parameters in `DPOConfig` by @qgallouedec in https://github.com/huggingface/trl/pull/4969\r\n\r\n* [CI] Disallow installation of transformers 5.1.0 due to compatibility issues with DeepSpeed by @qgallouedec in https://github.com/huggingface/trl/pull/4982\r\n\r\n* Replace `warmup_ratio` with `warmup_steps` by @qgallouedec in https://github.com/huggingface/trl/pull/4983\r\n\r\n* Pin transformers!=5.1.0 in deepspeed extra due to incompatibility by @albertvillanova in https://github.com/huggingface/trl/pull/4985\r\n* Fix passing tokenizer in test_train_with_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4987\r\n* Update dataset configuration name in toolcall dataset loading by @qgallouedec in https://github.com/huggingface/trl/pull/4984\r\n* Use local variable instead of attribute in collator tests by @qgallouedec in https://github.com/huggingface/trl/pull/4957\r\n* Fix import of AutoModelForCausalLMWithValueHead from experimental by @albertvillanova in https://github.com/huggingface/trl/pull/4990\r\n* Assert chat_template is applied in test_train_with_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4991\r\n* Fix deprecation of DPOConfig.max_completion_length by @albertvillanova in https://github.com/huggingface/trl/pull/4992\r\n* Fix post_init warning stacklevel to 3 by @albertvillanova in https://github.com/huggingface/trl/pull/4993\r\n* Fix ZeRO-3 + PEFT + gradient checkpointing by @qgallouedec in https://github.com/huggingface/trl/pull/4951\r\n* Add GitHub Actions workflow for testing against Transformers branch by @qgallouedec in https://github.com/huggingface/trl/pull/4995\r\n* Add distributed smoke tests workflow for Transformers branch by @qgallouedec in https://github.com/huggingface/trl/pull/4996\r\n* Update NeMo-Gym to use `env_mask` by @cmunley1 in https://github.com/huggingface/trl/pull/4986\r\n* Update sampling mode to token level for safety by @sergiopaniego in https://github.com/huggingface/trl/pull/4989\r\n* perf: Qwen SAPO loss optimization by @casinca in https://github.com/huggingface/trl/pull/4956\r\n* Fix GRPO tool calling for corrupted tool calls by @akshayballal95 in https://github.com/huggingface/trl/pull/4890\r\n* Add `sanitize_logprob` function for NaN handling in vLLM log probabilities by @qgallouedec in https://github.com/huggingface/trl/pull/5001\r\n* [tests] Remove xfail for transformers version >= 5.0.0 due to upstream bug resolution by @qgallouedec in https://github.com/huggingface/trl/pull/5000\r\n* docs: add CGPO/Mixture of Judges (2409.20370) to Paper Index + link ref to AllTrueJudge by @nabin2004 in https://github.com/huggingface/trl/pull/5002\r\n* Filter CI SWIG deprecation warnings by @albertvillanova in https://github.com/huggingface/trl/pull/5004\r\n* Fix CI TRLExperimentalWarning in regular tests by @albertvillanova in https://github.com/huggingface/trl/pull/5007\r\n* Add support for `nested_gather` in OnlineDPOTrainer for transformers v5.2.0 and above by @qgallouedec in https://github.com/huggingface/trl/pull/4981\r\n* Fix CI FutureWarning: ref_model_init_kwargs is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5009\r\n* Fix typo in DPO max_prompt_length deprecation warning message by @albertvillanova in https://github.com/huggingface/trl/pull/5020\r\n* Fix vision model prompt truncation bug in DPOTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/5023\r\n* Pin transformers < 5 in judges extra due to incompatibility by @albertvillanova in https://github.com/huggingface/trl/pull/5024\r\n* Fix CI FutureWarning: generate_during_eval is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5017\r\n* Fix typo in xfail test reason by @albertvillanova in https://github.com/huggingface/trl/pull/5028\r\n* Fix CI FutureWarning: rpo_alpha is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5011\r\n* Fix CI FutureWarning: use_logits_to_keep is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5013\r\n* Mark Qwen3VL tests as xfail for transformers 5.0.x by @albertvillanova in https://github.com/huggingface/trl/pull/5029\r\n* [CI] Silence PyTorch JIT and DataLoader deprecation warnings by @qgallouedec in https://github.com/huggingface/trl/pull/4999\r\n* Add length-unbiased GRPO loss (LUSPO) by @Haseebasif7 in https://github.com/huggingface/trl/pull/4988\r\n* Fix CI FutureWarning: tools is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5015\r\n* Filter max_prompt_length UserWarning in all test cases by @albertvillanova in https://github.com/huggingface/trl/pull/5035\r\n* Fix CI FutureWarning: max_prompt_length is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5019\r\n* Allow testing with transformers 5.1.0 via xfail marks by @albertvillanova in https://github.com/huggingface/trl/pull/5034\r\n* Rename AOT loss type 'aot_pair' to 'aot_unpaired' in DPO by @qgallouedec in https://github.com/huggingface/trl/pull/5038\r\n* Deprecate string usage for `ref_model` in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5040\r\n* Deprecate FDivergenceType in DPOConfig; update f_divergence_type to use string values by @qgallouedec in https://github.com/huggingface/trl/pull/5039\r\n* Fix multiprocessing start method to 'spawn' for test compatibility with Python 3.12+ by @qgallouedec in https://github.com/huggingface/trl/pull/5036\r\n* Add Online Direct Preference Optimization section to paper index by @qgallouedec in https://github.com/huggingface/trl/pull/5037\r\n* Release: 0.28 by @albertvillanova in https://github.com/huggingface/trl/pull/5043\r\n\r\n## New Contributors\r\n* @kansalaman made their first contribution in https://github.com/huggingface/trl/pull/4860\r\n* @javadtaghia made their first contribution in https://github.com/huggingface/trl/pull/4849\r\n* @Michellehbn made their first contribution in https://github.com/huggingface/trl/pull/4877\r\n* @Datta0 made their first contribution in https://github.com/huggingface/trl/pull/4868\r\n* @kdubovikov made their first contribution in https://github.com/huggingface/trl/pull/4873\r\n* @liyc-ai made their first contribution in https://github.com/huggingface/trl/pull/4887\r\n* @141forever made their first contribution in https://github.com/huggingface/trl/pull/4888\r\n* @adityachallapally made their first contribution in https://github.com/huggingface/trl/pull/4907\r\n* @billycrapediem made their first contribution in https://github.com/huggingface/trl/pull/4892\r\n* @mel3c made their first contribution in https://github.com/huggingface/trl/pull/4945\r\n* @cmunley1 made their first contribution in https://github.com/huggingface/trl/pull/4848\r\n* @akshayballal95 made their first contribution in https://github.com/huggingface/trl/pull/4890\r\n* @nabin2004 made their first contribution in https://github.com/huggingface/trl/pull/5002\r\n* @Haseebasif7 made their first contribution in https://github.com/huggingface/trl/pull/4988\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v0.27.0...v0.28.0","publishedAt":"2026-02-10T13:28:21.000Z","url":"https://github.com/huggingface/trl/releases/tag/v0.28.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_XFESFhBPm2oPozZ-NVNhp","version":"v0.27.2","type":"feature","title":"v0.27.2","summary":"## What's Changed\r\n\r\n* Remove access to `warnings_issued` by @qgallouedec in #4960\r\n* Fix SFTTrainer init logic: remove TrainingArguments.push_to_hub_...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## What's Changed\r\n\r\n* Remove access to `warnings_issued` by @qgallouedec in #4960\r\n* Fix SFTTrainer init logic: remove TrainingArguments.push_to_hub_token only for transformers < v5 by @albertvillanova in #4942\r\n* Fix extra EOS appended in DPO preprocessing for conversational data by @qgallouedec in https://github.com/huggingface/trl/pull/4908\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v0.27.1...v0.27.2","publishedAt":"2026-02-03T18:10:01.000Z","url":"https://github.com/huggingface/trl/releases/tag/v0.27.2","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_rYcthIea281SdvkQcUhJE","version":"v0.27.1","type":"feature","title":"v0.27.1","summary":"## What's Changed\r\n\r\n* Fix: undefined `current_gradient_accumulation_steps` by @qgallouedec in https://github.com/huggingface/trl/pull/4852\r\n* fix(Dee...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## What's Changed\r\n\r\n* Fix: undefined `current_gradient_accumulation_steps` by @qgallouedec in https://github.com/huggingface/trl/pull/4852\r\n* fix(DeepSeek OPSM): passing correct (vLLM) logprobs by @casinca in https://github.com/huggingface/trl/pull/4857\r\n* Fix SFT training for prompt-completion type and transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4880\r\n* Bugfix: Logprob drift in vLLM serving mode (compared to colocate mode) by @kdubovikov in https://github.com/huggingface/trl/pull/4873\r\n* Fix RewardTrainer's results not reproducible by @liyc-ai in https://github.com/huggingface/trl/pull/4887\r\n\r\n## New Contributors\r\n\r\n* @kdubovikov made their first contribution in https://github.com/huggingface/trl/pull/4873\r\n* @liyc-ai made their first contribution in https://github.com/huggingface/trl/pull/4887\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v0.27.0...v0.27.1","publishedAt":"2026-01-24T03:42:17.000Z","url":"https://github.com/huggingface/trl/releases/tag/v0.27.1","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null},{"id":"rel_S-yTeQ59mlshRUtbUNwVe","version":"v0.27.0","type":"feature","title":"v0.27.0","summary":"## Features\r\n\r\n* Add `vllm_group_port` argument to GRPO, RLOO and OnlineDPO configuration by @pointerhacker in https://github.com/huggingface/trl/pull...","titleGenerated":null,"titleShort":null,"breaking":"unknown","importance":null,"content":"## Features\r\n\r\n* Add `vllm_group_port` argument to GRPO, RLOO and OnlineDPO configuration by @pointerhacker in https://github.com/huggingface/trl/pull/4545\r\n* Preserve truncated tokens in BFD packing by @qgallouedec in https://github.com/huggingface/trl/pull/4632\r\n* Support async reward functions and parallelize call to reward functions. by @pramodith in https://github.com/huggingface/trl/pull/4567\r\n* RLOO supports async rewards. by @pramodith in https://github.com/huggingface/trl/pull/4718\r\n* Support vLLM 0.12.0 by @jiqing-feng in https://github.com/huggingface/trl/pull/4117\r\n* feat: DeepSeek V3.2 Off-policy sequence masking by @casinca in https://github.com/huggingface/trl/pull/4689\r\n* 🎭 Up to 50% less VRAM during forward with `forward_masked_logits` function by @qgallouedec in https://github.com/huggingface/trl/pull/4729\r\n* [GRPO] Add a config to limit the number of tool calling iterations by @pramodith in https://github.com/huggingface/trl/pull/4761\r\n* Switch gradient checkpointing default to use_reentrant=False (PyTorch recommended) by @qgallouedec in https://github.com/huggingface/trl/pull/4811\r\n* Add support for GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization by @nbasyl in https://github.com/huggingface/trl/pull/4785\r\n\r\n## Experimental\r\n\r\n* Move `AutoModelForCausalLMWithValueHead` and `AutoModelForSeq2SeqLMWithValueHead` to experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4654\r\n* Move DPODataCollatorWithPadding to `experimental.utils`  by @qgallouedec in https://github.com/huggingface/trl/pull/4667\r\n* Move `DataCollatorForChatML` to `experimental.utils` by @qgallouedec in https://github.com/huggingface/trl/pull/4668\r\n* Move `add_bos_token_if_needed` and `add_eos_token_if_needed` to `experimental.utils` by @qgallouedec in https://github.com/huggingface/trl/pull/4674\r\n* Move `truncate_right` and `SIMPLE_CHAT_TEMPLATE` to `experimental.utils` by @qgallouedec in https://github.com/huggingface/trl/pull/4677\r\n* Move `prepare_model_for_kbit_training`, `enable_gradient_checkpointing`, `prepare_peft_model` to `experimental.utils` by @qgallouedec in https://github.com/huggingface/trl/pull/4704\r\n* Move `get_reward` function to `experimental.utils` by @qgallouedec in https://github.com/huggingface/trl/pull/4683\r\n* Remove experimental imports from testing_utils by @albertvillanova in https://github.com/huggingface/trl/pull/4727\r\n* ORPO: Avoid catastrophic cancellation in loss function by @hartmans in https://github.com/huggingface/trl/pull/4763\r\n* Refactor KTO [1/N]: Modernize model initialization by @albertvillanova in https://github.com/huggingface/trl/pull/4783\r\n* [GOLD] add probability merging fix to implement chain rule by @kashif in https://github.com/huggingface/trl/pull/4765\r\n* Refactor KTO coordinated with DPO [a/N]: Remove encoder-decoder support by @albertvillanova in https://github.com/huggingface/trl/pull/4792\r\n* Refactor KTO coordinated with DPO [b/N]: Simplify truncation logic by @albertvillanova in https://github.com/huggingface/trl/pull/4808\r\n\r\n## Fixes\r\n\r\n* Accounting for case `num_generations_eval=1` in the calculation of the advantage  by @qgallouedec in https://github.com/huggingface/trl/pull/4662\r\n* Fix vLLM error for tools usage not supported when running GRPO training by @apalmas-saifh in https://github.com/huggingface/trl/pull/4663\r\n* Fix GRPO config validation in case `num_generations_eval` is specified and different than `num_generations` by @apalmas-saifh in https://github.com/huggingface/trl/pull/4682\r\n* Fix top_k default value to 0 for disabling top-k filtering by @albertvillanova in https://github.com/huggingface/trl/pull/4695\r\n* Include `generation_config` for tiny model uploads by @qgallouedec in https://github.com/huggingface/trl/pull/4643\r\n* Fix KeyError with transformers 5.0.0+ where push_to_hub_token is removed by @Manodeepray in https://github.com/huggingface/trl/pull/4691\r\n* Overwrite model default generation config used by model.generate by @albertvillanova in https://github.com/huggingface/trl/pull/4647\r\n* Fix: handle multiple tool calls in `qwen3_schema` by @mattbui in https://github.com/huggingface/trl/pull/4709\r\n* Fix bugs when using multi-gpu: dataset streaming for offline trainers + dtype initialization by @kaixuanliu in https://github.com/huggingface/trl/pull/3950\r\n* Ensure llm-blender is importable with transformers >= v5 by @albertvillanova in https://github.com/huggingface/trl/pull/4781\r\n* Monkey patch for `HybridCache` in Liger-Kernel with transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4798\r\n* [fix] GRPOTrainer: proper access `args` by @carlyou in https://github.com/huggingface/trl/pull/4801\r\n* Fix vllm compat patches to be applied only to affected versions by @albertvillanova in https://github.com/huggingface/trl/pull/4815\r\n* fix bug when sft calc outputs.token_accuracy by @kaixuanliu in https://github.com/huggingface/trl/pull/4814\r\n* fix xpu vllm client server by @jiqing-feng in https://github.com/huggingface/trl/pull/4780\r\n\r\n## Documentation and Examples\r\n\r\n* docs: add RapidFire AI integration section to SFT Trainer by @kamran-rapidfireAI in https://github.com/huggingface/trl/pull/4661\r\n* Fix environment image name for BrowserGym example script by @sergiopaniego in https://github.com/huggingface/trl/pull/4680\r\n* Docs(`grpo_trainer.md`): Added Qwen SAPO details under `Loss Types` by @casinca in https://github.com/huggingface/trl/pull/4681\r\n* [docs] Adds GRPO, RSO and LoRA to Paper Index by @SSusantAchary in https://github.com/huggingface/trl/pull/4441\r\n* Enable zero3 init and 16-bit model saving for ds ulysses config by @edbeeching in https://github.com/huggingface/trl/pull/4701\r\n* Set version to packaged one in notebooks by @sergiopaniego in https://github.com/huggingface/trl/pull/4648\r\n* BrowserGym example for LLMs (no vision) by @sergiopaniego in https://github.com/huggingface/trl/pull/4696\r\n* docs: Add RapidFire AI cross-references to DPO and GRPO trainer docs by @kamran-rapidfireAI in https://github.com/huggingface/trl/pull/4705\r\n* [docs] Fix RapidFire AI position in documentation by @qgallouedec in https://github.com/huggingface/trl/pull/4715\r\n* Add inference example to GRPO agent training notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4710\r\n* Upload FunctionGemma notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4721\r\n* Update agents notebook dependencies by @sergiopaniego in https://github.com/huggingface/trl/pull/4724\r\n* Add uv/hf jobs support to OpenEnv scripts  by @sergiopaniego in https://github.com/huggingface/trl/pull/4720\r\n* Add GRPO QLoRA free notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4660\r\n* Hotfix for browsergym openenv notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4740\r\n* docs: fix \"Good Second Issue\" redirection link by @casinca in https://github.com/huggingface/trl/pull/4749\r\n* [Docs] Add SRL (Supervised Reinforcement Learning) to Community Tutorials by @s23deepak in https://github.com/huggingface/trl/pull/4758\r\n* Add LFM2.5 to GRPO notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4793\r\n* Sudoku GRPO example script using TextArena by @sergiopaniego in https://github.com/huggingface/trl/pull/4762\r\n* [EXAMPLES] Update wordle to new openenv release by @burtenshaw in https://github.com/huggingface/trl/pull/4791\r\n* Update the typos in docs/source/grpo_trainer.md by @Tianyi-Billy-Ma in https://github.com/huggingface/trl/pull/4804\r\n* Updat examples to new OpenEnv version by @sergiopaniego in https://github.com/huggingface/trl/pull/4796\r\n* Update GRPO example to use Qwen2.5 instead of Qwen2 by @BurnyCoder in https://github.com/huggingface/trl/pull/4803\r\n\r\n## Deprecations\r\n\r\n* Remove deprecated functions and parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4651\r\n* Remove `MergeModelCallback` from import structure by @qgallouedec in https://github.com/huggingface/trl/pull/4664\r\n* Remove `ChatMlSpecialTokens` by @qgallouedec in https://github.com/huggingface/trl/pull/4666\r\n* Remove unused `_win_rate_completions_df` function from callbacks by @qgallouedec in https://github.com/huggingface/trl/pull/4672\r\n* Deprecate max_prompt_length in RLOOTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/4703\r\n* Small fix on contributing docs by @murilo-cunha in https://github.com/huggingface/trl/pull/4753\r\n* Remove `DbrxForCausalLM` support by @qgallouedec in https://github.com/huggingface/trl/pull/4799\r\n\r\n## CI Improvements\r\n\r\n* Hotfix CI due to generation config by setting tests as xfail by @albertvillanova in https://github.com/huggingface/trl/pull/4657\r\n* Upgrade GitHub Actions to latest versions by @salmanmkc in https://github.com/huggingface/trl/pull/4734\r\n* Upgrade GitHub Actions for Node 24 compatibility by @salmanmkc in https://github.com/huggingface/trl/pull/4733\r\n* Include data type for tiny models and update tests by @qgallouedec in https://github.com/huggingface/trl/pull/4728\r\n* Change tiny model dtype from float16 to bfloat16 to fix CUDA error by @albertvillanova in https://github.com/huggingface/trl/pull/4745\r\n* Add revision override mechanism for testing tiny models by @albertvillanova in https://github.com/huggingface/trl/pull/4769\r\n* Hotfix: Set float32 as default dtype for testing tiny models by @albertvillanova in https://github.com/huggingface/trl/pull/4770\r\n* Hotfix CI with dev dependencies: xfail test_training_vlm_and_liger by @albertvillanova in https://github.com/huggingface/trl/pull/4777\r\n* Add initial multi-GPU CI tests for distributed training by @qgallouedec in https://github.com/huggingface/trl/pull/4784\r\n* Set dtype default to float32 by @albertvillanova in https://github.com/huggingface/trl/pull/4778\r\n* Test FSDP2 by @qgallouedec in https://github.com/huggingface/trl/pull/4813\r\n* Test ZeRO Stage 3 by @qgallouedec in https://github.com/huggingface/trl/pull/4821\r\n* Hotfix CI main tests: Pin transformers 4.57.4 by @albertvillanova in https://github.com/huggingface/trl/pull/4830\r\n* Hotfix CI distributed smoke tests: xfail test_sft_peft[zero3] by @albertvillanova in https://github.com/huggingface/trl/pull/4831\r\n* Test ZeRO Stage 2 by @qgallouedec in https://github.com/huggingface/trl/pull/4822\r\n\r\n## Miscellaneous\r\n\r\n* Move `compute_accuracy` to PRM Trainer file by @qgallouedec in https://github.com/huggingface/trl/pull/4656\r\n* Move `clone_chat_template` to `chat_template_utils` by @qgallouedec in https://github.com/huggingface/trl/pull/4653\r\n* Move `GeometricMixtureWrapper` to `nash_md_trainer.py` by @qgallouedec in https://github.com/huggingface/trl/pull/4670\r\n* Move `exact_div`, `print_rich_table`, `truncate_response`, `forward` to `ppo_trainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4676\r\n* Merge `OnPolicyConfig` and `PPOConfig` and move `OnlineTrainerState` by @qgallouedec in https://github.com/huggingface/trl/pull/4671\r\n* Move PEFT tests for `AutoModelForCausalLMWithValueHead` to `test_ppo_trainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4678\r\n* Move `generate` and `batch_generation` to `ppo_trainer.py` by @qgallouedec in https://github.com/huggingface/trl/pull/4675\r\n* Import `TrainerCallback` from top-level transformers by @qgallouedec in https://github.com/huggingface/trl/pull/4694\r\n* Fix typos by @qgallouedec in https://github.com/huggingface/trl/pull/4690\r\n* Align import utils with transformers by @qgallouedec in https://github.com/huggingface/trl/pull/4684\r\n* Align stable trainers by @qgallouedec in https://github.com/huggingface/trl/pull/4687\r\n* Align GRPO and RLOO initialization by @qgallouedec in https://github.com/huggingface/trl/pull/4685\r\n* Align use of vllm_max_model_length in RLOOTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/4702\r\n* Align RLOO with GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/4706\r\n* Fix test assertion for `top_k` parameter in `OnlineDPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4714\r\n* Disallow `PeftModel` + `peft_config` in trainers by @qgallouedec in https://github.com/huggingface/trl/pull/4713\r\n* Fix deprecation version for RLOO max_prompt_length by @albertvillanova in https://github.com/huggingface/trl/pull/4726\r\n* Refactor vLLM generation [3/N]: Decouple profiling from trainer by @albertvillanova in https://github.com/huggingface/trl/pull/4717\r\n* Avoid docstyle formatting for `TestParseResponse` by @qgallouedec in https://github.com/huggingface/trl/pull/4736\r\n* 🥂 Happy New Year by @qgallouedec in https://github.com/huggingface/trl/pull/4775\r\n* Update import structure by @qgallouedec in https://github.com/huggingface/trl/pull/4665\r\n* Improve PEFT integration by @qgallouedec in https://github.com/huggingface/trl/pull/4723\r\n* Replace `GuidedDecodingParams` with `StructuredOutputsParams` in sampling parameter configuration by @qgallouedec in https://github.com/huggingface/trl/pull/4797\r\n* Move compatibility shims to dedicated module _compat by @albertvillanova in https://github.com/huggingface/trl/pull/4807\r\n* Refactor _compat module by @albertvillanova in https://github.com/huggingface/trl/pull/4809\r\n* Revised comments explaining the higher learning rate choice given tiny gradients by @qgallouedec in https://github.com/huggingface/trl/pull/4810\r\n* Simplify version checks in compat patches by @albertvillanova in https://github.com/huggingface/trl/pull/4817\r\n* Set packaging as explicit dependency and standardize version comparison by @albertvillanova in https://github.com/huggingface/trl/pull/4819\r\n* Fix _patch_transformers_hybrid_cache also for peft by @albertvillanova in https://github.com/huggingface/trl/pull/4820\r\n* Fix _patch_vllm_cached_tokenizer to only apply if transformers >= v5 by @albertvillanova in https://github.com/huggingface/trl/pull/4827\r\n* Fix code quality in SFTTrainer file by @albertvillanova in https://github.com/huggingface/trl/pull/4832\r\n\r\n\r\n--\r\n\r\n## New Contributors\r\n* @pointerhacker made their first contribution in https://github.com/huggingface/trl/pull/4545\r\n* @apalmas-saifh made their first contribution in https://github.com/huggingface/trl/pull/4663\r\n* @Manodeepray made their first contribution in https://github.com/huggingface/trl/pull/4691\r\n* @salmanmkc made their first contribution in https://github.com/huggingface/trl/pull/4734\r\n* @mattbui made their first contribution in https://github.com/huggingface/trl/pull/4709\r\n* @murilo-cunha made their first contribution in https://github.com/huggingface/trl/pull/4753\r\n* @hartmans made their first contribution in https://github.com/huggingface/trl/pull/4763\r\n* @s23deepak made their first contribution in https://github.com/huggingface/trl/pull/4758\r\n* @Tianyi-Billy-Ma made their first contribution in https://github.com/huggingface/trl/pull/4804\r\n* @carlyou made their first contribution in https://github.com/huggingface/trl/pull/4801\r\n* @BurnyCoder made their first contribution in https://github.com/huggingface/trl/pull/4803\r\n\r\n**Full Changelog**: https://github.com/huggingface/trl/compare/v0.26.0...v0.27.0","publishedAt":"2026-01-16T02:34:32.000Z","url":"https://github.com/huggingface/trl/releases/tag/v0.27.0","media":[],"prerelease":false,"source":{"slug":"trl","name":"TRL","type":"github"},"product":{"slug":"fine-tuning","name":"Fine-tuning"},"groupSlug":"fine-tuning","groupName":"Fine-tuning","coverageCount":0,"contentChars":null,"contentTokens":null,"composition":null}],"pagination":{"nextCursor":"2026-01-16T02:34:32.000Z|2026-04-07T17:28:22.697Z|rel_S-yTeQ59mlshRUtbUNwVe","limit":20}}