releases.shpreview
Hugging Face/PEFT/v0.4.0

v0.4.0

QLoRA, IA3 PEFT method, support for QA and Feature Extraction tasks, AutoPeftModelForxxx for simplified UX , LoRA for custom models with new added utils

July 18, 2023PEFTView original ↗
$npx -y @buildinternet/releases show rel_Uk57mBVEsS0VLRr_JndvS

QLoRA Support:

QLoRA uses 4-bit quantization to compress a pretrained language model. The LM parameters are then frozen and a relatively small number of trainable parameters are added to the model in the form of Low-Rank Adapters. During finetuning, QLoRA backpropagates gradients through the frozen 4-bit quantized pretrained language model into the Low-Rank Adapters. The LoRA layers are the only parameters being updated during training. For more details read the blog Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA

New PEFT methods: IA3 from T-Few paper

To make fine-tuning more efficient, IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations) rescales inner activations with learned vectors. These learned vectors are injected into the attention and feedforward modules in a typical transformer-based architecture. These learned vectors are the only trainable parameters during fine-tuning, and thus the original weights remain frozen. Dealing with learned vectors (as opposed to learned low-rank updates to a weight matrix like LoRA) keeps the number of trainable parameters much smaller. For more details, read the paper Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

Support for new tasks: QA and Feature Extraction

Addition of PeftModelForQuestionAnswering and PeftModelForFeatureExtraction classes to support QA and Feature Extraction tasks, respectively. This enables exciting new use-cases with PEFT, e.g., LoRA for semantic similarity tasks.

AutoPeftModelForxxx for better and Simplified UX

Introduces a new paradigm, AutoPeftModelForxxx intended for users that want to rapidly load and run peft models.

from peft import AutoPeftModelForCausalLM

peft_model = AutoPeftModelForCausalLM.from_pretrained("ybelkada/opt-350m-lora")

LoRA for custom models

Not a transformer model, no problem, we have got you covered. PEFT now enables the usage of LoRA with custom models.

New LoRA utilities

Improvements to add_weighted_adapter method to support SVD for combining multiple LoRAs when creating new LoRA. New utils such as unload and delete_adapter providing users much better control about how they deal with the adapters.

PEFT and Stable Diffusion

PEFT is very extensible and easy to use for performing DreamBooth of Stable Diffusion. Community has added conversion scripts to be able to use PEFT models with Civitai/webui format and vice-versa.

What's Changed

New Contributors

Full Changelog: https://github.com/huggingface/peft/compare/v0.3.0...v0.4.0

Significant community contributions

The following contributors have made significant changes to the library over the last release:

@TimDettmers

@SumanthRH

@kovalexal

@sywangyi

@aarnphm

@martin-liu

@thomas-schillaci

Fetched April 7, 2026