v0.27.0 — TRL — releases.sh

Features

Add vllm_group_port argument to GRPO, RLOO and OnlineDPO configuration by @pointerhacker in https://github.com/huggingface/trl/pull/4545
Preserve truncated tokens in BFD packing by @qgallouedec in https://github.com/huggingface/trl/pull/4632
Support async reward functions and parallelize call to reward functions. by @pramodith in https://github.com/huggingface/trl/pull/4567
RLOO supports async rewards. by @pramodith in https://github.com/huggingface/trl/pull/4718
Support vLLM 0.12.0 by @jiqing-feng in https://github.com/huggingface/trl/pull/4117
feat: DeepSeek V3.2 Off-policy sequence masking by @casinca in https://github.com/huggingface/trl/pull/4689
🎭 Up to 50% less VRAM during forward with forward_masked_logits function by @qgallouedec in https://github.com/huggingface/trl/pull/4729
[GRPO] Add a config to limit the number of tool calling iterations by @pramodith in https://github.com/huggingface/trl/pull/4761
Switch gradient checkpointing default to use_reentrant=False (PyTorch recommended) by @qgallouedec in https://github.com/huggingface/trl/pull/4811
Add support for GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization by @nbasyl in https://github.com/huggingface/trl/pull/4785

Experimental

Move AutoModelForCausalLMWithValueHead and AutoModelForSeq2SeqLMWithValueHead to experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4654
Move DPODataCollatorWithPadding to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4667
Move DataCollatorForChatML to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4668
Move add_bos_token_if_needed and add_eos_token_if_needed to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4674
Move truncate_right and SIMPLE_CHAT_TEMPLATE to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4677
Move prepare_model_for_kbit_training, enable_gradient_checkpointing, prepare_peft_model to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4704
Move get_reward function to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4683
Remove experimental imports from testing_utils by @albertvillanova in https://github.com/huggingface/trl/pull/4727
ORPO: Avoid catastrophic cancellation in loss function by @hartmans in https://github.com/huggingface/trl/pull/4763
Refactor KTO [1/N]: Modernize model initialization by @albertvillanova in https://github.com/huggingface/trl/pull/4783
[GOLD] add probability merging fix to implement chain rule by @kashif in https://github.com/huggingface/trl/pull/4765
Refactor KTO coordinated with DPO [a/N]: Remove encoder-decoder support by @albertvillanova in https://github.com/huggingface/trl/pull/4792
Refactor KTO coordinated with DPO [b/N]: Simplify truncation logic by @albertvillanova in https://github.com/huggingface/trl/pull/4808

Fixes

Accounting for case num_generations_eval=1 in the calculation of the advantage by @qgallouedec in https://github.com/huggingface/trl/pull/4662
Fix vLLM error for tools usage not supported when running GRPO training by @apalmas-saifh in https://github.com/huggingface/trl/pull/4663
Fix GRPO config validation in case num_generations_eval is specified and different than num_generations by @apalmas-saifh in https://github.com/huggingface/trl/pull/4682
Fix top_k default value to 0 for disabling top-k filtering by @albertvillanova in https://github.com/huggingface/trl/pull/4695
Include generation_config for tiny model uploads by @qgallouedec in https://github.com/huggingface/trl/pull/4643
Fix KeyError with transformers 5.0.0+ where push_to_hub_token is removed by @Manodeepray in https://github.com/huggingface/trl/pull/4691
Overwrite model default generation config used by model.generate by @albertvillanova in https://github.com/huggingface/trl/pull/4647
Fix: handle multiple tool calls in qwen3_schema by @mattbui in https://github.com/huggingface/trl/pull/4709
Fix bugs when using multi-gpu: dataset streaming for offline trainers + dtype initialization by @kaixuanliu in https://github.com/huggingface/trl/pull/3950
Ensure llm-blender is importable with transformers >= v5 by @albertvillanova in https://github.com/huggingface/trl/pull/4781
Monkey patch for HybridCache in Liger-Kernel with transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4798
[fix] GRPOTrainer: proper access args by @carlyou in https://github.com/huggingface/trl/pull/4801
Fix vllm compat patches to be applied only to affected versions by @albertvillanova in https://github.com/huggingface/trl/pull/4815
fix bug when sft calc outputs.token_accuracy by @kaixuanliu in https://github.com/huggingface/trl/pull/4814
fix xpu vllm client server by @jiqing-feng in https://github.com/huggingface/trl/pull/4780

Documentation and Examples

docs: add RapidFire AI integration section to SFT Trainer by @kamran-rapidfireAI in https://github.com/huggingface/trl/pull/4661
Fix environment image name for BrowserGym example script by @sergiopaniego in https://github.com/huggingface/trl/pull/4680
Docs(grpo_trainer.md): Added Qwen SAPO details under Loss Types by @casinca in https://github.com/huggingface/trl/pull/4681
[docs] Adds GRPO, RSO and LoRA to Paper Index by @SSusantAchary in https://github.com/huggingface/trl/pull/4441
Enable zero3 init and 16-bit model saving for ds ulysses config by @edbeeching in https://github.com/huggingface/trl/pull/4701
Set version to packaged one in notebooks by @sergiopaniego in https://github.com/huggingface/trl/pull/4648
BrowserGym example for LLMs (no vision) by @sergiopaniego in https://github.com/huggingface/trl/pull/4696
docs: Add RapidFire AI cross-references to DPO and GRPO trainer docs by @kamran-rapidfireAI in https://github.com/huggingface/trl/pull/4705
[docs] Fix RapidFire AI position in documentation by @qgallouedec in https://github.com/huggingface/trl/pull/4715
Add inference example to GRPO agent training notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4710
Upload FunctionGemma notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4721
Update agents notebook dependencies by @sergiopaniego in https://github.com/huggingface/trl/pull/4724
Add uv/hf jobs support to OpenEnv scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/4720
Add GRPO QLoRA free notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4660
Hotfix for browsergym openenv notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4740
docs: fix "Good Second Issue" redirection link by @casinca in https://github.com/huggingface/trl/pull/4749
[Docs] Add SRL (Supervised Reinforcement Learning) to Community Tutorials by @s23deepak in https://github.com/huggingface/trl/pull/4758
Add LFM2.5 to GRPO notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4793
Sudoku GRPO example script using TextArena by @sergiopaniego in https://github.com/huggingface/trl/pull/4762
[EXAMPLES] Update wordle to new openenv release by @burtenshaw in https://github.com/huggingface/trl/pull/4791
Update the typos in docs/source/grpo_trainer.md by @Tianyi-Billy-Ma in https://github.com/huggingface/trl/pull/4804
Updat examples to new OpenEnv version by @sergiopaniego in https://github.com/huggingface/trl/pull/4796
Update GRPO example to use Qwen2.5 instead of Qwen2 by @BurnyCoder in https://github.com/huggingface/trl/pull/4803

Deprecations

Remove deprecated functions and parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4651
Remove MergeModelCallback from import structure by @qgallouedec in https://github.com/huggingface/trl/pull/4664
Remove ChatMlSpecialTokens by @qgallouedec in https://github.com/huggingface/trl/pull/4666
Remove unused _win_rate_completions_df function from callbacks by @qgallouedec in https://github.com/huggingface/trl/pull/4672
Deprecate max_prompt_length in RLOOTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/4703
Small fix on contributing docs by @murilo-cunha in https://github.com/huggingface/trl/pull/4753
Remove DbrxForCausalLM support by @qgallouedec in https://github.com/huggingface/trl/pull/4799

CI Improvements

Hotfix CI due to generation config by setting tests as xfail by @albertvillanova in https://github.com/huggingface/trl/pull/4657
Upgrade GitHub Actions to latest versions by @salmanmkc in https://github.com/huggingface/trl/pull/4734
Upgrade GitHub Actions for Node 24 compatibility by @salmanmkc in https://github.com/huggingface/trl/pull/4733
Include data type for tiny models and update tests by @qgallouedec in https://github.com/huggingface/trl/pull/4728
Change tiny model dtype from float16 to bfloat16 to fix CUDA error by @albertvillanova in https://github.com/huggingface/trl/pull/4745
Add revision override mechanism for testing tiny models by @albertvillanova in https://github.com/huggingface/trl/pull/4769
Hotfix: Set float32 as default dtype for testing tiny models by @albertvillanova in https://github.com/huggingface/trl/pull/4770
Hotfix CI with dev dependencies: xfail test_training_vlm_and_liger by @albertvillanova in https://github.com/huggingface/trl/pull/4777
Add initial multi-GPU CI tests for distributed training by @qgallouedec in https://github.com/huggingface/trl/pull/4784
Set dtype default to float32 by @albertvillanova in https://github.com/huggingface/trl/pull/4778
Test FSDP2 by @qgallouedec in https://github.com/huggingface/trl/pull/4813
Test ZeRO Stage 3 by @qgallouedec in https://github.com/huggingface/trl/pull/4821
Hotfix CI main tests: Pin transformers 4.57.4 by @albertvillanova in https://github.com/huggingface/trl/pull/4830
Hotfix CI distributed smoke tests: xfail test_sft_peft[zero3] by @albertvillanova in https://github.com/huggingface/trl/pull/4831
Test ZeRO Stage 2 by @qgallouedec in https://github.com/huggingface/trl/pull/4822

Miscellaneous

Move compute_accuracy to PRM Trainer file by @qgallouedec in https://github.com/huggingface/trl/pull/4656
Move clone_chat_template to chat_template_utils by @qgallouedec in https://github.com/huggingface/trl/pull/4653
Move GeometricMixtureWrapper to nash_md_trainer.py by @qgallouedec in https://github.com/huggingface/trl/pull/4670
Move exact_div, print_rich_table, truncate_response, forward to ppo_trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4676
Merge OnPolicyConfig and PPOConfig and move OnlineTrainerState by @qgallouedec in https://github.com/huggingface/trl/pull/4671
Move PEFT tests for AutoModelForCausalLMWithValueHead to test_ppo_trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4678
Move generate and batch_generation to ppo_trainer.py by @qgallouedec in https://github.com/huggingface/trl/pull/4675
Import TrainerCallback from top-level transformers by @qgallouedec in https://github.com/huggingface/trl/pull/4694
Fix typos by @qgallouedec in https://github.com/huggingface/trl/pull/4690
Align import utils with transformers by @qgallouedec in https://github.com/huggingface/trl/pull/4684
Align stable trainers by @qgallouedec in https://github.com/huggingface/trl/pull/4687
Align GRPO and RLOO initialization by @qgallouedec in https://github.com/huggingface/trl/pull/4685
Align use of vllm_max_model_length in RLOOTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/4702
Align RLOO with GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/4706
Fix test assertion for top_k parameter in OnlineDPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4714
Disallow PeftModel + peft_config in trainers by @qgallouedec in https://github.com/huggingface/trl/pull/4713
Fix deprecation version for RLOO max_prompt_length by @albertvillanova in https://github.com/huggingface/trl/pull/4726
Refactor vLLM generation [3/N]: Decouple profiling from trainer by @albertvillanova in https://github.com/huggingface/trl/pull/4717
Avoid docstyle formatting for TestParseResponse by @qgallouedec in https://github.com/huggingface/trl/pull/4736
🥂 Happy New Year by @qgallouedec in https://github.com/huggingface/trl/pull/4775
Update import structure by @qgallouedec in https://github.com/huggingface/trl/pull/4665
Improve PEFT integration by @qgallouedec in https://github.com/huggingface/trl/pull/4723
Replace GuidedDecodingParams with StructuredOutputsParams in sampling parameter configuration by @qgallouedec in https://github.com/huggingface/trl/pull/4797
Move compatibility shims to dedicated module _compat by @albertvillanova in https://github.com/huggingface/trl/pull/4807
Refactor _compat module by @albertvillanova in https://github.com/huggingface/trl/pull/4809
Revised comments explaining the higher learning rate choice given tiny gradients by @qgallouedec in https://github.com/huggingface/trl/pull/4810
Simplify version checks in compat patches by @albertvillanova in https://github.com/huggingface/trl/pull/4817
Set packaging as explicit dependency and standardize version comparison by @albertvillanova in https://github.com/huggingface/trl/pull/4819
Fix _patch_transformers_hybrid_cache also for peft by @albertvillanova in https://github.com/huggingface/trl/pull/4820
Fix _patch_vllm_cached_tokenizer to only apply if transformers >= v5 by @albertvillanova in https://github.com/huggingface/trl/pull/4827
Fix code quality in SFTTrainer file by @albertvillanova in https://github.com/huggingface/trl/pull/4832

New Contributors

@pointerhacker made their first contribution in https://github.com/huggingface/trl/pull/4545
@apalmas-saifh made their first contribution in https://github.com/huggingface/trl/pull/4663
@Manodeepray made their first contribution in https://github.com/huggingface/trl/pull/4691
@salmanmkc made their first contribution in https://github.com/huggingface/trl/pull/4734
@mattbui made their first contribution in https://github.com/huggingface/trl/pull/4709
@murilo-cunha made their first contribution in https://github.com/huggingface/trl/pull/4753
@hartmans made their first contribution in https://github.com/huggingface/trl/pull/4763
@s23deepak made their first contribution in https://github.com/huggingface/trl/pull/4758
@Tianyi-Billy-Ma made their first contribution in https://github.com/huggingface/trl/pull/4804
@carlyou made their first contribution in https://github.com/huggingface/trl/pull/4801
@BurnyCoder made their first contribution in https://github.com/huggingface/trl/pull/4803

Full Changelog: https://github.com/huggingface/trl/compare/v0.26.0...v0.27.0