PEFT

A small patch release containing these fixes:

• #3161
• #3165

Full Changelog: https://github.com/huggingface/peft/compare/v0.19.0...v0.19.1

Highlights

This PEFT release contains no less than nine new PEFT methods, described below. It also contains numerous enhancements that should make PEFT more useful to many users.

<img width="1248" height="560" alt="peft-v0 19 0"…

Small patch release containing the following changes:

• #2934: Small fixes required for some special cases to work with the upcoming transformers v5 release
• #2963: Fix to enable PEFT to run with AMD ROCm thanks to @vladmandic
• #2976: Fix a regression that inadvertently…

Highlights

FIXME update list of all changes, so some more commits were added

New Methods

RoAd

@ppetrushkov added…

This patch release contains a few fixes (via #2710) for the newly introduced target_parameters feature, which allows LoRA to target nn.Parameters directly (useful for mixture of…

Highlights

New Methods

SHiRA

@kkb-code contributed [Sparse High Rank…

Highlights

New Methods

LoRA-FA

In #2468, @AaronZLT added the [LoRA-FA…

This patch fixes a bug that resulted in prompt learning methods like P-tuning not to work (#2477).

This patch includes a fix for #2450. In this bug modules_to_save was not handled correctly when used in conjunction with DeepSpeed ZeRO stage 3 which resulted in those modules being placeholder values in the saved checkpoints.

Full Changelog:…

Highlights

New Methods

CorDA: Context-Oriented Decomposition Adaptation

@iboing and @5eqn contributed [CorDA: Context-Oriented Decomposition Adaptation of Large…

Highlights

New Methods

Context-aware Prompt Tuning

@tsachiblau added a new soft prompt method called [Context-aware Prompt Tuning…

This patch release contains a small bug fix for an issue that prevented some LoRA checkpoints to be loaded correctly (mostly concerning stable diffusion checkpoints not trained with PEFT when loaded in diffusers, #2144).

Full Changelog:…

This patch release contains a small bug fix for the low_cpu_mem_usage=True option (#2113).

Full Changelog: https://github.com/huggingface/peft/compare/v0.13.0...v0.13.1

Highlights

New methods

LoRA+

@kallewoof added LoRA+ to PEFT (#1915). This is a function that allows to [initialize an…

Highlights

New methods

OLoRA

@tokenizer-decode added support for a new LoRA initialization strategy called OLoRA (#1828).…

Patch release v0.11.1

Fix a bug that could lead to C++ compilation errors after importing PEFT (#1738 #1739).

Full Changelog: https://github.com/huggingface/peft/compare/v0.11.0...v0.11.1

Highlights

New methods

BOFT

Thanks to @yfeng95, @Zeju1997, and @YuliangXiu, PEFT was extended with BOFT: Parameter-Efficient Orthogonal Finetuning via…

Highlights

Support for QLoRA with DeepSpeed ZeRO3 and FSDP

We added a couple of changes to allow QLoRA to work with DeepSpeed ZeRO3 and Fully Sharded Data Parallel…

Highlights

New methods for merging LoRA weights together

With PR #1364, we added new methods for merging LoRA weights together. This is not about merging…

What's Changed

• Release v0.8.2.dev0 by @pacman100 in https://github.com/huggingface/peft/pull/1416
• Add IA3 Modules for Phi by @arnavgarg1 in https://github.com/huggingface/peft/pull/1407
• Update custom_models.md by @boyufan in…