v0.14.0

Major and breaking changes

👨‍👨‍👧‍👧 GRPO

by @qgallouedec in https://github.com/huggingface/trl/pull/2565

What's Changed

• ⚰️ Remove deprecated by @qgallouedec in https://github.com/huggingface/trl/pull/2485
• 🗣️ Improve prose for smol course by @burtenshaw in https://github.com/huggingface/trl/pull/2487
• 🤩 Add SmolVLM tutorials to Community Tutorials page by @sergiopaniego in https://github.com/huggingface/trl/pull/2498
• 🏞️ Proper dataset for documentation images by @qgallouedec in https://github.com/huggingface/trl/pull/2499
• 🗂️ Reorganize documentation by @qgallouedec in https://github.com/huggingface/trl/pull/2483
• [ORPO] fix orpo chosen-nll loss by @kashif in https://github.com/huggingface/trl/pull/2502
• 🏚 Remove unused components by @qgallouedec in https://github.com/huggingface/trl/pull/2480
• Update community_tutorials.md by @qgallouedec in https://github.com/huggingface/trl/pull/2509
• ❎ Remove RLOO example test by @qgallouedec in https://github.com/huggingface/trl/pull/2513
• 👨‍🍳 Clarify DPO data preparation by @qgallouedec in https://github.com/huggingface/trl/pull/2512
• 💧 Generalize disable_dropout by @qgallouedec in https://github.com/huggingface/trl/pull/2511
• 👬 Rename collator PreferenceCollator to DataCollatorForPreference by @qgallouedec in https://github.com/huggingface/trl/pull/2510
• 📦 Packing documentation by @qgallouedec in https://github.com/huggingface/trl/pull/2503
• ☄️ Update Comet integration to include LogCompletionsCallback and Trainer.evaluation_loop() by @yaricom in https://github.com/huggingface/trl/pull/2501
• Remove graph breaks for torch.compile() in padding free branch in DataCollatorForCompletionOnlyLM by @Abhishek-TAMU in https://github.com/huggingface/trl/pull/2158
• 🚜 Use field in dataclasses by @qgallouedec in https://github.com/huggingface/trl/pull/2494
• ©️ Update copyrights year by @qgallouedec in https://github.com/huggingface/trl/pull/2547
• 🧑‍🤝‍🧑 Proper metrics gathering across ranks before logging by @zhc7 in https://github.com/huggingface/trl/pull/2474
• ✒️ Fix typo in formatting_func's documentation in ConstantLengthDataset by @SamuelLarkin in https://github.com/huggingface/trl/pull/2549
• 🕊️ DPO padding free by @qgallouedec in https://github.com/huggingface/trl/pull/2520
• ℹ️ XPU support for DPO by @faaany in https://github.com/huggingface/trl/pull/2533
• 🔠 Fix SFT truncation documentation by @umbilnm in https://github.com/huggingface/trl/pull/2521
• ↩️ Revert ORPO loss changes by @kashif in https://github.com/huggingface/trl/pull/2527
• 🎴 Add readme for datasets by @August-murr in https://github.com/huggingface/trl/pull/2491
• 💔 Fix dataset type unpair conversion docs by @claralp in https://github.com/huggingface/trl/pull/2550
• [RLOO] Reinforce++ by @kashif in https://github.com/huggingface/trl/pull/2552
• 🏛️ Improve DPO configuration documentation structure by @qgallouedec in https://github.com/huggingface/trl/pull/2561
• ✨ Refine model card method docstring by @qgallouedec in https://github.com/huggingface/trl/pull/2566
• 🪄 Minor comment style modif by @qgallouedec in https://github.com/huggingface/trl/pull/2582
• 🏎️ vllm for Online DPO by @qgallouedec in https://github.com/huggingface/trl/pull/2558
• 🔖 Issues Auto-Labeller by @August-murr in https://github.com/huggingface/trl/pull/2542
• 🐛 Simplify bug report template by @qgallouedec in https://github.com/huggingface/trl/pull/2585
• [RLOO] fix token_level_kl by @kashif in https://github.com/huggingface/trl/pull/2575
• ✂️ Truncate by default by @qgallouedec in https://github.com/huggingface/trl/pull/2587
• 🫢 Add max_prompt_length parameter in tests by @qgallouedec in https://github.com/huggingface/trl/pull/2588
• 🎞️ Fix documentation SFT -max_seq_length instead of max_length by @skandermoalla in https://github.com/huggingface/trl/pull/2590
• 👨‍👨‍👧‍👧 GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/2565
• 🫣 Ignore CLI test for Python 3.9 by @qgallouedec in https://github.com/huggingface/trl/pull/2592
• Fix merge error by @qgallouedec in https://github.com/huggingface/trl/pull/2595
• 🧰 Tool fine-tuning support DPO by @August-murr in https://github.com/huggingface/trl/pull/2479
• 💾 Reduce memory peak in GRPO by adding max_prompt_length and loop usage in logp computation by @qgallouedec in https://github.com/huggingface/trl/pull/2598
• ⚡ Add uv installation instructions by @stevhliu in https://github.com/huggingface/trl/pull/2601
• 🧩 PPO/RLOO/OnlineDPO sequence generation: make deepsped 3 weight gathering optional by @dawidm in https://github.com/huggingface/trl/pull/2557
• 🫷 Include stop token in policy model's generation_config by @dawidm in https://github.com/huggingface/trl/pull/2528
• ✂️ Reintroduce truncation_mode in DPOTrainer by @anakin87 in https://github.com/huggingface/trl/pull/2551
• 👋 Drop MDX by @qgallouedec in https://github.com/huggingface/trl/pull/2611
• 💎 Rename an inner var in GRPO to improve clarity by @qgallouedec in https://github.com/huggingface/trl/pull/2616
• 🏆 Custom reward function for GRPO and shiny doc by @qgallouedec in https://github.com/huggingface/trl/pull/2606
• 🥞 Fix DPO gradient accumulation loss scaling by @winglian in https://github.com/huggingface/trl/pull/2615
• 🥞 Fix BCO gradient accumulation loss scaling by @qgallouedec in https://github.com/huggingface/trl/pull/2638
• 🍭 Custom reward function for RLOO by @August-murr in https://github.com/huggingface/trl/pull/2612
• 🌯 Fix context manager runtime error when gather is disabled by @Superskyyy in https://github.com/huggingface/trl/pull/2639
• 🥞 Fix CPO gradient accumulation loss scaling by @qgallouedec in https://github.com/huggingface/trl/pull/2645
• 🥞 Fix GRPO gradient accumulation loss scaling by @qgallouedec in https://github.com/huggingface/trl/pull/2647
• 🥞 Fix KTO gradient accumulation loss scaling by @qgallouedec in https://github.com/huggingface/trl/pull/2648
• 🚛 Provide all columns of the dataset to the reward function by @qgallouedec in https://github.com/huggingface/trl/pull/2650
• 👐 DeepSpeed integration for GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/2652
• 🔎 Finegrained reward logging for GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/2651
• 📍 Disable caching when grad checkpointing enable in GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/2653
• 📏 Log completion length in GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/2659
• 🌀 Fix GRPO default completion length doc by @andyl98 in https://github.com/huggingface/trl/pull/2662
• 🏷️ Add model tags to model trained with GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/2663
• 🖊 Fix typos by @omahs in https://github.com/huggingface/trl/pull/2673
• ⚡ vLLM for fast generation in GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/2600
• 📉 Use num_logits_to_keep to reduce memory usage in GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/2683

New Contributors

• @Abhishek-TAMU made their first contribution in https://github.com/huggingface/trl/pull/2158
• @zhc7 made their first contribution in https://github.com/huggingface/trl/pull/2474
• @SamuelLarkin made their first contribution in https://github.com/huggingface/trl/pull/2549
• @umbilnm made their first contribution in https://github.com/huggingface/trl/pull/2521
• @stevhliu made their first contribution in https://github.com/huggingface/trl/pull/2601
• @dawidm made their first contribution in https://github.com/huggingface/trl/pull/2557
• @Superskyyy made their first contribution in https://github.com/huggingface/trl/pull/2639
• @andyl98 made their first contribution in https://github.com/huggingface/trl/pull/2662
• @omahs made their first contribution in https://github.com/huggingface/trl/pull/2673

Full Changelog: https://github.com/huggingface/trl/compare/v0.13.0...v0.14.0