Highlights

Integration with diffusers

<img src="https://github.com/huggingface/peft/assets/13534540/57eea400-ce5f-4bf8-a499-711fd83f590b" width="600" height="600">

🧨 Diffusers now leverage PEFT as a backend for LoRA inference for Stable Diffusion models (#873, #993, #961). Relevant PRs on 🧨 Diffusers are https://github.com/huggingface/diffusers/pull/5058, https://github.com/huggingface/diffusers/pull/5147, https://github.com/huggingface/diffusers/pull/5151 and https://github.com/huggingface/diffusers/pull/5359. This helps in unlocking a vast number of practically demanding use cases around adapter-based inference 🚀. Now you can do the following with easy-to-use APIs and it supports different checkpoint formats (Diffusers format, Kohya format ...):

1. use multiple LoRAs
2. switch between them instantaneously
3. scale and combine them
4. merge/unmerge
5. enable/disable

For details, refer to the documentation at Inference with PEFT.

New tuning methods

• Multitask Prompt Tuning: Thanks @mayank31398 for implementing this method from https://arxiv.org/abs/2303.02861 (#400)
• LoHa (low-rank Hadamard product): @kovalexal did a great job adding LoHa from https://arxiv.org/abs/2108.06098 (#956)
• LoKr (Kronecker Adapter): Not happy with just one new adapter, @kovalexal also added LoKr from https://arxiv.org/abs/2212.10650 to PEFT (#978)

Other notable additions

• Allow merging of LoRA weights when using 4bit and 8bit quantization (bitsandbytes), thanks to @jiqing-feng (#851, #875)
• IA³ now supports 4bit quantization thanks to @His-Wardship (#864)
• We increased the speed of adapter layer initialization: This should be most notable when creating a PEFT LoRA model on top of a large base model (#887, #915, #994)
• More fine-grained control when configuring LoRA: It is now possible to have different ranks and alpha values for different layers (#873)

Experimental features

• For some adapters like LoRA, it is now possible to activate multiple adapters at the same time (#873)

Breaking changes

• It is no longer allowed to create a LoRA adapter with rank 0 (r=0). This used to be possible, in which case the adapter was ignored.

What's Changed

As always, a bunch of small improvements, bug fixes and doc improvements were added. We thank all the external contributors, both new and recurring. Below is the list of all changes since the last release.

• Fixed typos in custom_models.mdx by @Psancs05 in https://github.com/huggingface/peft/pull/847
• Release version 0.6.0.dev0 by @pacman100 in https://github.com/huggingface/peft/pull/849
• DOC: Add a contribution guide by @BenjaminBossan in https://github.com/huggingface/peft/pull/848
• clarify the new model size by @stas00 in https://github.com/huggingface/peft/pull/839
• DOC: Remove backlog section from README.md by @BenjaminBossan in https://github.com/huggingface/peft/pull/853
• MNT: Refactor tuner forward methods for simplicity by @BenjaminBossan in https://github.com/huggingface/peft/pull/833
• 🎉 Add Multitask Prompt Tuning by @mayank31398 in https://github.com/huggingface/peft/pull/400
• Fix typos in ia3.py by @metaprotium in https://github.com/huggingface/peft/pull/844
• Support merge lora module for 4bit and 8bit linear by @jiqing-feng in https://github.com/huggingface/peft/pull/851
• Fix seq2seq prompt tuning (#439) by @glerzing in https://github.com/huggingface/peft/pull/809
• MNT: Move tuners to subpackages by @BenjaminBossan in https://github.com/huggingface/peft/pull/807
• FIX: Error in forward of 4bit linear lora layer by @BenjaminBossan in https://github.com/huggingface/peft/pull/878
• MNT: Run tests that were skipped previously by @BenjaminBossan in https://github.com/huggingface/peft/pull/884
• FIX: PeftModel save_pretrained Doc (#881) by @houx15 in https://github.com/huggingface/peft/pull/888
• Upgrade docker actions to higher versions by @younesbelkada in https://github.com/huggingface/peft/pull/889
• Fix error using deepspeed zero2 + load_in_8bit + lora by @tmm1 in https://github.com/huggingface/peft/pull/874
• Fix doc for semantic_segmentation_lora by @raghavanone in https://github.com/huggingface/peft/pull/891
• fix_gradient_accumulation_steps_in_examples by @zspo in https://github.com/huggingface/peft/pull/898
• FIX: linting issue in example by @BenjaminBossan in https://github.com/huggingface/peft/pull/908
• ENH Remove redundant initialization layer calls by @BenjaminBossan in https://github.com/huggingface/peft/pull/887
• [docs] Remove duplicate section by @stevhliu in https://github.com/huggingface/peft/pull/911
• support prefix tuning for starcoder models by @pacman100 in https://github.com/huggingface/peft/pull/913
• Merge lora module to 8bit model by @jiqing-feng in https://github.com/huggingface/peft/pull/875
• DOC: Section on common issues encountered with PEFT by @BenjaminBossan in https://github.com/huggingface/peft/pull/909
• Enh speed up init emb conv2d by @BenjaminBossan in https://github.com/huggingface/peft/pull/915
• Make base_model.peft_config single source of truth by @BenjaminBossan in https://github.com/huggingface/peft/pull/921
• Update accelerate dependency version by @rohithkrn in https://github.com/huggingface/peft/pull/892
• fix lora layer init by @SunMarc in https://github.com/huggingface/peft/pull/928
• Fixed LoRA conversion for kohya_ss by @kovalexal in https://github.com/huggingface/peft/pull/916
• [CI] Pin diffusers by @younesbelkada in https://github.com/huggingface/peft/pull/936
• [LoRA] Add scale_layer / unscale_layer by @younesbelkada in https://github.com/huggingface/peft/pull/935
• TST: Add GH action to run unit tests with torch.compile by @BenjaminBossan in https://github.com/huggingface/peft/pull/943
• FIX: torch compile gh action installs pytest by @BenjaminBossan in https://github.com/huggingface/peft/pull/944
• Fix NotImplementedError for no bias. by @Datta0 in https://github.com/huggingface/peft/pull/946
• TST: Fix some tests that would fail with torch.compile by @BenjaminBossan in https://github.com/huggingface/peft/pull/949
• ENH Allow compile GH action to run on torch nightly by @BenjaminBossan in https://github.com/huggingface/peft/pull/952
• Install correct PyTorch nightly in GH action by @BenjaminBossan in https://github.com/huggingface/peft/pull/954
• support multiple ranks and alphas for LoRA by @pacman100 in https://github.com/huggingface/peft/pull/873
• feat: add type hints by @SauravMaheshkar in https://github.com/huggingface/peft/pull/858
• FIX: setting requires_grad on adapter layers by @BenjaminBossan in https://github.com/huggingface/peft/pull/905
• [tests] add transformers & diffusers integration tests by @younesbelkada in https://github.com/huggingface/peft/pull/962
• Fix integrations_tests.yml by @younesbelkada in https://github.com/huggingface/peft/pull/965
• Add 4-bit support to IA3 - Outperforms QLoRA in both speed and memory consumption by @His-Wardship in https://github.com/huggingface/peft/pull/864
• Update integrations_tests.yml by @younesbelkada in https://github.com/huggingface/peft/pull/966
• add the lora target modules for Mistral Models by @pacman100 in https://github.com/huggingface/peft/pull/974
• TST: Fix broken save_pretrained tests by @BenjaminBossan in https://github.com/huggingface/peft/pull/969
• [tests] add multiple active adapters tests by @pacman100 in https://github.com/huggingface/peft/pull/961
• Fix missing tokenizer attribute in test by @BenjaminBossan in https://github.com/huggingface/peft/pull/977
• Add implementation of LyCORIS LoHa (FedPara-like adapter) for SD&SDXL models by @kovalexal in https://github.com/huggingface/peft/pull/956
• update BibTeX by @pacman100 in https://github.com/huggingface/peft/pull/989
• FIX: issues with (un)merging multiple LoRA and IA³ adapters by @BenjaminBossan in https://github.com/huggingface/peft/pull/976
• add lora target modules for stablelm models by @kbulutozler in https://github.com/huggingface/peft/pull/982
• Correct minor errors in example notebooks for causal language modelling by @SumanthRH in https://github.com/huggingface/peft/pull/926
• Fix typo in custom_models.mdx by @Pairshoe in https://github.com/huggingface/peft/pull/964
• Add base model metadata to model card by @BenjaminBossan in https://github.com/huggingface/peft/pull/975
• MNT Make .merged a property by @BenjaminBossan in https://github.com/huggingface/peft/pull/979
• Fix lora creation by @pacman100 in https://github.com/huggingface/peft/pull/993
• TST: Comment out flaky LoHA test by @BenjaminBossan in https://github.com/huggingface/peft/pull/1002
• ENH Support Conv2d layers for IA³ by @BenjaminBossan in https://github.com/huggingface/peft/pull/972
• Fix word_embeddings match for deepspeed wrapped model by @mayank31398 in https://github.com/huggingface/peft/pull/1000
• FEAT: Add safe_merge option in merge by @younesbelkada in https://github.com/huggingface/peft/pull/1001
• [core / LoRA] Add safe_merge to bnb layers by @younesbelkada in https://github.com/huggingface/peft/pull/1009
• ENH: Refactor LoRA bnb layers for faster initialization by @BenjaminBossan in https://github.com/huggingface/peft/pull/994
• FIX Don't assume model_config contains the key model_type by @BenjaminBossan in https://github.com/huggingface/peft/pull/1012
• FIX stale.py uses timezone-aware datetime by @BenjaminBossan in https://github.com/huggingface/peft/pull/1016
• FEAT: Add fp16 + cpu merge support by @younesbelkada in https://github.com/huggingface/peft/pull/1017
• fix lora scaling and unscaling by @pacman100 in https://github.com/huggingface/peft/pull/1027
• [LoRA] Revert original behavior for scale / unscale by @younesbelkada in https://github.com/huggingface/peft/pull/1029
• [LoRA] Raise error when adapter name not found in set_scale by @younesbelkada in https://github.com/huggingface/peft/pull/1034
• Fix target_modules type in config.from_pretrained by @BenjaminBossan in https://github.com/huggingface/peft/pull/1046
• docs(README): bit misspell current path link StackLLaMa by @guspan-tanadi in https://github.com/huggingface/peft/pull/1047
• Fixed wrong construction of LoHa weights, updated adapters conversion script by @kovalexal in https://github.com/huggingface/peft/pull/1021
• Fix P-tuning for sequence classification docs by @ehcalabres in https://github.com/huggingface/peft/pull/1049
• FIX: Setting active adapter correctly by @BenjaminBossan in https://github.com/huggingface/peft/pull/1051
• Fix Conv1D merge error for IA3 by @SumanthRH in https://github.com/huggingface/peft/pull/1014
• Add implementation of LyCORIS LoKr (KronA-like adapter) for SD&SDXL models by @kovalexal in https://github.com/huggingface/peft/pull/978
• [core] Fix use_reentrant issues by @younesbelkada in https://github.com/huggingface/peft/pull/1036
• [tests] Update Dockerfile to use cuda 12.2 by @younesbelkada in https://github.com/huggingface/peft/pull/1050
• Add testing for regex matching and other custom kwargs by @SumanthRH in https://github.com/huggingface/peft/pull/1031
• Fix Slack bot not displaying error messages by @younesbelkada in https://github.com/huggingface/peft/pull/1068
• Fix slow tests not running by @younesbelkada in https://github.com/huggingface/peft/pull/1071
• Release version 0.6.0 by @BenjaminBossan in https://github.com/huggingface/peft/pull/1072

New Contributors

• @Psancs05 made their first contribution in https://github.com/huggingface/peft/pull/847
• @metaprotium made their first contribution in https://github.com/huggingface/peft/pull/844
• @jiqing-feng made their first contribution in https://github.com/huggingface/peft/pull/851
• @houx15 made their first contribution in https://github.com/huggingface/peft/pull/888
• @tmm1 made their first contribution in https://github.com/huggingface/peft/pull/874
• @raghavanone made their first contribution in https://github.com/huggingface/peft/pull/891
• @zspo made their first contribution in https://github.com/huggingface/peft/pull/898
• @rohithkrn made their first contribution in https://github.com/huggingface/peft/pull/892
• @Datta0 made their first contribution in https://github.com/huggingface/peft/pull/946
• @kbulutozler made their first contribution in https://github.com/huggingface/peft/pull/982
• @Pairshoe made their first contribution in https://github.com/huggingface/peft/pull/964
• @ehcalabres made their first contribution in https://github.com/huggingface/peft/pull/1049

Full Changelog: https://github.com/huggingface/peft/compare/v0.5.0...v0.6.0