LCM LoRA, LCM SDXL, Consistency Decoder

LCM LoRA

Latent Consistency Models (LCM) made quite the mark in the Stable Diffusion community by enabling ultra-fast inference. LCM author @luosiallen, alongside @patil-suraj and @dg845, managed to extend the LCM support for Stable Diffusion XL (SDXL) and pack everything into a LoRA.

The approach is called LCM LoRA.

Below is an example of using LCM LoRA, taking just 4 inference steps:

from diffusers import DiffusionPipeline, LCMScheduler
import torch

model_id = "stabilityai/stable-diffusion-xl-base-1.0"
lcm_lora_id = "latent-consistency/lcm-lora-sdxl"

pipe = DiffusionPipeline.from_pretrained(model_id, variant="fp16", torch_dtype=torch.float16).to("cuda")

pipe.load_lora_weights(lcm_lora_id)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

prompt = "close-up photography of old man standing in the rain at night, in a street lit by lamps, leica 35mm summilux"
image = pipe(
    prompt=prompt,
    num_inference_steps=4,
    guidance_scale=1,
).images[0]

You can combine the LoRA with Img2Img, Inpaint, ControlNet, ...

as well as with other LoRAs 🤯

👉 Checkpoints 📜 Docs

If you want to learn more about the approach, please have a look at the following:

• Paper
• Blog

LCM SDXL

Continuing the work of Latent Consistency Models (LCM), we've applied the approach to SDXL as well and give you SSD-1B and SDXL fine-tuned checkpoints.

from diffusers import DiffusionPipeline, UNet2DConditionModel, LCMScheduler
import torch

unet = UNet2DConditionModel.from_pretrained(
    "latent-consistency/lcm-sdxl",
    torch_dtype=torch.float16,
    variant="fp16",
)
pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", unet=unet, torch_dtype=torch.float16
).to("cuda")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

generator = torch.manual_seed(0)
image = pipe(
    prompt=prompt, num_inference_steps=4, generator=generator, guidance_scale=1.0
).images[0]

👉 Checkpoints 📜 Docs

Consistency Decoder

OpenAI open-sourced the consistency decoder used in DALL-E 3. It improves the decoding part in the Stable Diffusion v1 family of models.

import torch
from diffusers import DiffusionPipeline, ConsistencyDecoderVAE

vae = ConsistencyDecoderVAE.from_pretrained("openai/consistency-decoder", torch_dtype=pipe.torch_dtype)
pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", vae=vae, torch_dtype=torch.float16
).to("cuda")

pipe("horse", generator=torch.manual_seed(0)).images

Find the documentation here to learn more.

All commits

• [Custom Pipelines] Make sure that community pipelines can use repo revision by @patrickvonplaten in #5659
• post release (v0.22.0) by @sayakpaul in #5658
• Add Pixart to AUTO_TEXT2IMAGE_PIPELINES_MAPPING by @Beinsezii in #5664
• Update custom diffusion attn processor by @DN6 in #5663
• Model tests xformers fixes by @DN6 in #5679
• Update free model hooks by @DN6 in #5680
• Fix Basic Transformer Block by @DN6 in #5683
• Explicit torch/flax dependency check by @DN6 in #5673
• [PixArt-Alpha] fix mask_feature so that precomputed embeddings work with a batch size > 1 by @sayakpaul in #5677
• Make sure DDPM and diffusers can be used without Transformers by @sayakpaul in #5668
• [PixArt-Alpha] Support non-square images by @sayakpaul in #5672
• Improve LCMScheduler by @dg845 in #5681
• [Docs] Fix typos, improve, update at Using Diffusers' Task page by @standardAI in #5611
• Replacing the nn.Mish activation function with a get_activation function. by @hi-sushanta in #5651
• speed up Shap-E fast test by @yiyixuxu in #5686
• Fix the misaligned pipeline usage in dreamshaper docstrings by @kirill-fedyanin in #5700
• Fixed is_safetensors_compatible() handling of windows path separators by @PhilLab in #5650
• [LCM] Fix img2img by @patrickvonplaten in #5698
• [PixArt-Alpha] fix mask feature condition. by @sayakpaul in #5695
• Fix styling issues by @patrickvonplaten in #5699
• Add adapter fusing + PEFT to the docs by @apolinario in #5662
• Fix prompt bug in AnimateDiff by @DN6 in #5702
• [Bugfix] fix error of peft lora when xformers enabled by @okotaku in #5697
• Install accelerate from PyPI in PR test runner by @DN6 in #5721
• consistency decoder by @williamberman in #5694
• Correct consist dec by @patrickvonplaten in #5722
• LCM Add Tests by @patrickvonplaten in #5707
• [LCM] add: locm docs. by @sayakpaul in #5723
• Add LCM Scripts by @patil-suraj in #5727