Accelerate

Activation checkpointing now wraps the matched transformer layer instead of each child, cutting saved activations and memory at long sequence lengths; a new FSDP2-only activation_checkpointing_offload moves layer inputs to pinned CPU memory. Also added tied-embedding support on torch >= 2.13 and fixed gradient clipping and prepare_model for DTensor-sharded models.

Accelerate now works end-to-end on AMD ROCm devices. This release also includes numerous FSDP2 fixes and quality-of-life improvements, such as correct dtype handling on load, sharding of embeddings/norms, and QLoRA crash prevention.

AWS Neuron support

We now have support for AWS Neuron (Trainium/Inferentia) devices. Thanks @michaelbenayoun for adding this.

• Neuron integration by @michaelbenayoun in https://github.com/huggingface/accelerate/pull/3935

XPU Improvements

We've removed IPEX…

Deepspeed Ulysses/ALST integration

Deepspeed Ulysses/ALST is an efficient way of training on long sequences by employing sequence parallelism and attention head parallelism. You can learn more about this technology in this paper https://arxiv.org/abs/2506.13996 or this…

TE MXFP8 support

We've added support for MXFP8 in our TransformerEngine integration. To use that, you need to set use_mxfp8_block_scaling in fp8_config. See nvidia docs [here].…

• Feat: add to_json by @S1ro1 in https://github.com/huggingface/accelerate/pull/3743
• Protect import for device_mesh by @SunMarc in https://github.com/huggingface/accelerate/pull/3742.

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.10.0...v1.10.1

N-D Parallelism

Training large models across multiple GPUs can be complex, especially when combining different parallelism strategies (e.g TP, CP, DP). To simplify this process, we've collaborated with…

Trackio tracker support

We've added support for a trackio, lightweight, 💯 free experiment tracking Python library built on top of 🤗 Datasets and Spaces.

![Screen Recording 2025-06-11 at 5 39…

• Add support for e5e2 and default to hybrid when launcher is used by @IlyasMoutawwakil in https://github.com/huggingface/accelerate/pull/3640
• shards by @SunMarc in https://github.com/huggingface/accelerate/pull/3645

Full Changelog:…

FSDPv2 refactor + FP8 support

We've simplified how to prepare FSDPv2 models, as there were too many ways to compose FSDP2 with other features (e.g., FP8, torch.compile, activation checkpointing, etc.). Although the setup is now more restrictive, it leads to fewer errors and…

Regional compilation

Instead of compiling the entire model at once, regional compilation targets repeated blocks (such as decoder layers) first. This allows the compiler to cache and reuse optimized code for subsequent blocks, significantly reducing the cold start…

FSDPv2 support

This release introduces the support for FSDPv2 thanks to @S1ro1.

If you are using python code, you need to set fsdp_version=2 in FullyShardedDataParallelPlugin:

from…

Bug Fixes:

• Fixed an issue with torch.get_default_device() requiring a higher version than what we support
• Fixed a broken pytest import in prod

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.5.0...v1.5.2

HPU Support

• Adds in HPU accelerator support for 🤗 Accelerate

What's Changed

• [bug] fix device index bug for model training loaded with bitsandbytes by @faaany in https://github.com/huggingface/accelerate/pull/3408
• [docs] add the missing import torch by…

torchao FP8, initial Tensor Parallel support, and memory leak fixes

torchao FP8

This release introduces a new FP8 API and brings in a new backend: torchao. To use, pass in AORecipeKwargs to the…

Torch 2.0

As it's been ~2 years since torch 2.0 was first released, we are now requiring this as the minimum version for Accelerate, which similarly was done in transformers as of its last release.

Core

• [docs] no hard-coding cuda by @faaany in…

• fix: add max_memory to _init_infer_auto_device_map's return statement in https://github.com/huggingface/accelerate/pull/3279 by @Nech-C
• fix load_state_dict for npu in https://github.com/huggingface/accelerate/pull/3211 by @statelesshz

Full Changelog:…

Core

• enable find_executable_batch_size on XPU by @faaany in https://github.com/huggingface/accelerate/pull/3236
• Use numpy._core instead of numpy.core by @qgallouedec in https://github.com/huggingface/accelerate/pull/3247
• Add warnings and fallback for unassigned…

Internals:

• Allow for a data_seed argument in https://github.com/huggingface/accelerate/pull/3150
• Trigger weights_only=True by default for all compatible objects when checkpointing and saving with torch.save in https://github.com/huggingface/accelerate/pull/3036 *…

Bugfixes

• Fixes an issue where the auto values were no longer being parsed when using deepspeed
• Fixes a broken test in the deepspeed tests related to the [auto…