v0.21.0

Major and breaking

🌺 OpenAI GPT OSS & Harmony support

<img width="4544" height="2344" alt="Group 293-2" src="https://github.com/user-attachments/assets/17241da2-1b1d-41bc-a5f8-0983ea46606f" />

Open AI GPT OSS models are here! Check out the OpenAI Cookbook to see an example of how to SFT these models.

by @qgallouedec in https://github.com/huggingface/trl/pull/3848

Add vLLM transformers backend to online methods

You can now pass vllm_model_impl to the TRL vLLM server. Example, for transformers backend:

trl vllm-serve ... --vllm_model_impl transformers

by @merveenoyan in https://github.com/huggingface/trl/pull/3773

What's Changed

• ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/3793
• Fix broken PEFT+TRL docs link in using_llama_models.md by @bwook00 in https://github.com/huggingface/trl/pull/3794
• 🐙 Add MPO VLM example script by @sergiopaniego in https://github.com/huggingface/trl/pull/3799
• Examples list updated in docs by @sergiopaniego in https://github.com/huggingface/trl/pull/3806
• Add vLLM transformers backend to online methods by @merveenoyan in https://github.com/huggingface/trl/pull/3773
• Correction parameter description by @1787648106 in https://github.com/huggingface/trl/pull/3803
• Add GSPO script examples (VLM/LLM) by @sergiopaniego in https://github.com/huggingface/trl/pull/3810
• add xpu support for mergekit by @yao-matrix in https://github.com/huggingface/trl/pull/3800
• GSPO parameters update from v2 by @BounharAbdelaziz in https://github.com/huggingface/trl/pull/3798
• fix CI docs and grpo slow test by @kashif in https://github.com/huggingface/trl/pull/3814
• Performance optimization: Replace list comprehensions with tensor operations in BCO and KTO trainers by @chi2liu in https://github.com/huggingface/trl/pull/3813
• Improve trainer doc by @qgallouedec in https://github.com/huggingface/trl/pull/3818
• Add 'Post training a VLM for reasoning with GRPO using TRL' recipe to Community tutorials by @sergiopaniego in https://github.com/huggingface/trl/pull/3843
• [GRPO]: Fix Entropy Mask Threshold Calculation when using Multi-GPU training by @pramodith in https://github.com/huggingface/trl/pull/3833
• 🪦 Remove deprecated by @qgallouedec in https://github.com/huggingface/trl/pull/3817
• 🌺 OpenAI GPT OSS & Harmony support by @qgallouedec in https://github.com/huggingface/trl/pull/3848
• Release: v0.21 by @qgallouedec in https://github.com/huggingface/trl/pull/3849

New Contributors

• @bwook00 made their first contribution in https://github.com/huggingface/trl/pull/3794
• @merveenoyan made their first contribution in https://github.com/huggingface/trl/pull/3773
• @1787648106 made their first contribution in https://github.com/huggingface/trl/pull/3803
• @BounharAbdelaziz made their first contribution in https://github.com/huggingface/trl/pull/3798
• @chi2liu made their first contribution in https://github.com/huggingface/trl/pull/3813

Full Changelog: https://github.com/huggingface/trl/compare/v0.20.0...v0.21.0