We now have support for AWS Neuron (Trainium/Inferentia) devices. Thanks @michaelbenayoun for adding this.
We've removed IPEX dependency and improved device-agnostic code for XPU.
We've added a bunch of important fixes for FSDP2 users: upcasting only grad-requiring params, better tied embedding errors, DCP optimizer loading, bf16 optimizer step crash fix, and torch < 2.7.0 compatibility.
We've added several fixes to the DeepSpeed + Sequence Parallelism integration introduced in v1.12.0, including evaluation support during SP training and proper process group handling.
We've enhanced FP8 training. Thanks @shimizust for fixing torchao support.
Accelerate now imports faster by deferring heavy dependencies, and torch.compile hooks are disabled lazily.
Deepspeed Ulysses/ALST is an efficient way of training on long sequences by employing sequence parallelism and attention head parallelism. You can learn more about this technology in this paper https://arxiv.org/abs/2506.13996 or this deepspeed tutorial https://www.deepspeed.ai/tutorials/ulysses-alst-sequence-parallelism/.
<img width="2368" height="1250" alt="0d8bd9e0" src="https://github.com/user-attachments/assets/b94e90c9-4368-4711-ad57-58de3c714ebc" />To enable Deepspeed Ulysses, you first need to create ParallelismConfig and setting sp related args:
parallelism_config = ParallelismConfig(
sp_backend="deepspeed",
sp_size=2,
sp_handler=DeepSpeedSequenceParallelConfig(...),
)
Then, you need to make sure to compute the correct loss as described on our docs
...
losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)
good_tokens = (shift_labels != -100).view(-1).sum()
good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)
total_loss = sum(
losses_per_rank[rank] * good_tokens_per_rank[rank]
for rank in range(sp_world_size)
if good_tokens_per_rank[rank] > 0
)
total_good_tokens = sum(good_tokens_per_rank)
loss = total_loss / max(total_good_tokens, 1)
Thanks @S1ro1 for starting this work and for @stas00 for finishing this work. Also thanks @kashif for adding docs and reviewing/testing this PR !
This feature will also be available in HF Trainer thanks for this PR from @stas00: https://github.com/huggingface/transformers/pull/41832
cpu_ram_efficient_loading by @SunMarc in https://github.com/huggingface/accelerate/pull/3816Full Changelog: https://github.com/huggingface/accelerate/compare/v1.11.0...v1.12.0
We've added support for MXFP8 in our TransformerEngine integration. To use that, you need to set use_mxfp8_block_scaling in fp8_config. See nvidia docs [here]. (https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#MXFP8-and-block-scaling)
BF16 and FP16 support for MPS devices is finally here. You can now pass mixed_precision = "fp16" or "bf16" when training on a mac (fp16 requires torch 2.8 and bf16 requires torch 2.6)
The following PRs add respectively support to ignored_params and no_sync() for FSDPv2:
Mixed precision can now be passed as a dtype string from accelerate cli flag or fsdp_config in accelerate config file:
Some minor updates concerning nd-parallelism.
We've dropped support for python 3.9 as it reached EOL in October.
cpu and offloaded to meta by @Qubitium in https://github.com/huggingface/accelerate/pull/3796with in Accelerator.autocast()instead of __enter__() and __exit__() for more elegant style. by @EquationWalker in https://github.com/huggingface/accelerate/pull/3767SWANLAB_MODE by @SunMarc in https://github.com/huggingface/accelerate/pull/3808Full Changelog: https://github.com/huggingface/accelerate/compare/v1.10.1...v1.11.0
Full Changelog: https://github.com/huggingface/accelerate/compare/v1.10.0...v1.10.1
Training large models across multiple GPUs can be complex, especially when combining different parallelism strategies (e.g TP, CP, DP). To simplify this process, we've collaborated with Axolotl to introduce an easy-to-use integration that allows you to apply any combination of parallelism strategies directly in your training script. Just pass a ParallelismConfig specifying the size of each parallelism type—it's that simple.
Learn more about how it works in our latest blogpost.
parallelism_config = ParallelismConfig(
dp_shard_size=2,
dp_replicate_size=2,
cp_size=2,
tp_size=2,
)
accelerator = Accelerator(
parallelism_config=parallelism_config,
...
)
model = AutoModelForCausalLM.from_pretrained("your-model-name", device_mesh=accelerator.torch_device_mesh)
model = accelerator.prepare(model)
ParallelismConfig from PartialState by @SunMarc in https://github.com/huggingface/accelerate/pull/3720We've fixed ignored modules attribute. With this, it is now possible to train PEFT model that moe layers that contrains q_proj and v_proj parameters. This is especially important for fine-tuning gpt-oss model.
Full Changelog: https://github.com/huggingface/accelerate/compare/v1.9.0...v1.10.0
We've added support for a trackio, lightweight, 💯 free experiment tracking Python library built on top of 🤗 Datasets and Spaces.
Main features are:
space_id.To use it with accelerate, you need to set log_with and initialize the trackers
accelerator = Accelerator(log_with="trackio")
config={"learning_rate": 0.001, "batch_size": 32}
# init_kwargs in order to host the dashboard on spaces
init_kwargs = {"trackio": {"space_id": "hf_username/space_name"}
accelerator.init_trackers("example_project", config=config, init_kwargs=init_kwargs})
Thanks @pcuenca for the integration !
set_module_tensor_to_device Setting tensor while clearing cache is very slow, so we added clear_device option to disable it.
Another small optimization is using non_blocking everywhere and syncing just before returning control to the user. This makes the loading slightly faster.
Accelerator() configuring by @pstjohn in https://github.com/huggingface/accelerate/pull/3677find_executable_batch_size() will no longer halves the batch after every OOM. Instead, we will multiply the batch size by 0.9. This should help user not waste gpu capacity.
Full Changelog: https://github.com/huggingface/accelerate/compare/v1.8.1...v1.9.0
Full Changelog: https://github.com/huggingface/accelerate/compare/v1.8.0...v1.8.1
We've simplified how to prepare FSDPv2 models, as there were too many ways to compose FSDP2 with other features (e.g., FP8, torch.compile, activation checkpointing, etc.). Although the setup is now more restrictive, it leads to fewer errors and a more performant user experience. We’ve also added support for FP8. You can read about the results here. Thanks to @S1ro1 for this contribution!
We updated the CCL_WORKER_COUNT variable and added KMP parameters for Intel CPU users. This significantly improves distributed training performance (e.g., Tensor Parallelism), with up to a 40% speed-up on Intel 4th Gen Xeon when training transformer TP models.
We added support for regional compilation with the DeepSpeed engine. DeepSpeed’s .compile() modifies models in-place using torch.nn.Module.compile(...), rather than the out-of-place torch.compile(...), so we had to account for that. Thanks @IlyasMoutawwakil for this feature!
ipex.optimize is being deprecated. Most optimizations have been upstreamed to PyTorch, and future improvements will land there directly. For users without PyTorch 2.8, we’ll continue to rely on IPEX for now.
We've greatly expanded and stabilized support for Intel XPUs:
We've added support for SwanLab as an experiment tracking backend. Huge thanks to @ShaohonChen for this contribution ! We also deferred all tracker initializations to prevent premature setup of distributed environments.
accelerator().load_state() by @luiz0992 in https://github.com/huggingface/accelerate/pull/3540dtype_byte_size by @SunMarc in https://github.com/huggingface/accelerate/pull/3625Full Changelog: https://github.com/huggingface/accelerate/compare/v1.7.0...v1.8.0
Instead of compiling the entire model at once, regional compilation targets repeated blocks (such as decoder layers) first. This allows the compiler to cache and reuse optimized code for subsequent blocks, significantly reducing the cold start compilation time typically seen during the first inference. Thanks @IlyasMoutawwakil for the feature ! You can view the full benchmark here, and check out our updated compilation guide for more details!
To enable this feature, set use_regional_compilation=True in the TorchDynamoPlugin configuration.
# Configure the compilation backend
dynamo_plugin = TorchDynamoPlugin(
use_regional_compilation=True,
... # other parameters
)
# Initialize accelerator with the plugin
accelerator = Accelerator(dynamo_plugin=dynamo_plugin)
# This will apply compile_regions to your model
model = accelerator.prepare(model)
We've introduced a new hook that enables per-layer upcasting and downcasting (e.g., for Linear layers) during inference. This allows users to run models with separate storage and compute dtypes, resulting in memory savings. The concept was first implemented in diffusers, where downcasting models to FP8 proved effective without major quality degradation. Contributed by @sayakpaul in https://github.com/huggingface/accelerate/pull/3427
model = ....
storage_dtype = torch.float8_e4m3fn
compute_dtype = torch.bfloat16
attach_layerwise_casting_hooks(
model,
storage_dtype=storage_dtype,
compute_dtype=compute_dtype,
)
This release includes numerous new features and bug fixes. Notably, we’ve added support for FULL_STATE_DICT, a widely used option in FSDP, now enabling .save_pretrained() in transformers to work with FSDP2 wrapped models. QLoRA training is now supported as well but more testing is needed. We have also resolved a backend issue related to parameter offloading to CPU. Additionally, a significant memory spike that occurred when cpu_ram_efficient_loading=True was enabled has been fixed. Several other minor improvements and fixes are also included—see the What’s Changed section for full details.
FULL_STATE_DICT have been enabled by @S1ro1 in https://github.com/huggingface/accelerate/pull/3527cpu_ram_efficient_loading=True by @S1ro1 in https://github.com/huggingface/accelerate/pull/3482We have added a documentation for Intel Gaudi hardware ! The support is already available since v1.5.0 through this PR.
dynamic argumentWe've updated the logic for setting self.dynamic to explicitly preserve None rather than defaulting to False when the USE_DYNAMIC environment variable is unset. This change aligns the behavior with the PyTorch documentation for torch.compile. Thanks to @yafshar for contributing this improvement in #3567.
low_precision_training guide by @sadra-barikbin in https://github.com/huggingface/accelerate/pull/3488torch.distributed.checkpoint.state_dict.set_model_state_dict in load_checkpoint_in_model by @ringohoffman in https://github.com/huggingface/accelerate/pull/3432weights_only=True by @bzhong-solink in https://github.com/huggingface/accelerate/pull/3497cpu_ram_efficient_loading=True by @S1ro1 in https://github.com/huggingface/accelerate/pull/3482accelerator.prepare + IPEX for 2+ nn.Models and/or optim.Optimizers by @mariusarvinte in https://github.com/huggingface/accelerate/pull/3517set_epoch does not take effect. by @hongjx175 in https://github.com/huggingface/accelerate/pull/3556_cast_and_contiguous by @dlvp in https://github.com/huggingface/accelerate/pull/3559cpu_ram_efficient_loading by @SumanthRH in https://github.com/huggingface/accelerate/pull/3307synchronize call for xpu in _gpu_gather by @faaany in https://github.com/huggingface/accelerate/pull/3563Full Changelog: https://github.com/huggingface/accelerate/compare/v1.6.0...v1.7.0
This release introduces the support for FSDPv2 thanks to @S1ro1.
If you are using python code, you need to set fsdp_version=2 in FullyShardedDataParallelPlugin:
from accelerate import FullyShardedDataParallelPlugin, Accelerator
fsdp_plugin = FullyShardedDataParallelPlugin(
fsdp_version=2
# other options...
)
accelerator = Accelerator(fsdp_plugin=fsdp_plugin)
If want to convert a YAML config that contains the FSDPv1 config to FSDPv2 one , use our conversion tool:
accelerate to-fsdp2 --config_file config.yaml --output_file new_config.yaml`
To learn more about the difference between FSDPv1 and FSDPv2, read the following documentation.
We have added initial support for DeepSpeed + TP. Not many changes were required as the DeepSpeed APIs was already compatible. We only needed to make sure that the dataloader was compatible with TP and that we were able to save the TP weights. Thanks @inkcherry for the work ! https://github.com/huggingface/accelerate/pull/3390.
To use TP with deepspeed, you need to update the setting in the deepspeed config file by including tensor_parallel key:
....
"tensor_parallel":{
"autotp_size": ${autotp_size}
},
...
More details in this deepspeed PR.
We've added support for XCCL which is an Intel distributed backend which can be used with XPU devices. More details in this torch PR. Thanks @dvrogozh for the integration !
log_artifact, log_artifacts and log_figure capabilities to the MLflowTracker. by @luiz0992 in https://github.com/huggingface/accelerate/pull/3419Full Changelog: https://github.com/huggingface/accelerate/compare/v1.5.2...v1.6.0
Bug Fixes:
torch.get_default_device() requiring a higher version than what we supportpytest import in prodFull Changelog: https://github.com/huggingface/accelerate/compare/v1.5.0...v1.5.2
import torch by @faaany in https://github.com/huggingface/accelerate/pull/3396device=torch.get_default_device() in torch.Generators by @saforem2 in https://github.com/huggingface/accelerate/pull/3420Full Changelog: https://github.com/huggingface/accelerate/compare/v1.4.0...v1.5.0
torchao FP8, initial Tensor Parallel support, and memory leak fixestorchao FP8This release introduces a new FP8 API and brings in a new backend: torchao. To use, pass in AORecipeKwargs to the Accelerator while setting mixed_precision="fp8". This is initial support, as it matures we will incorporate more into it (such as accelerate config/yaml) in future releases. See our benchmark examples here
We have intial support for an in-house solution to TP when working with accelerate dataloaders. check out the PR here
memory leak] Replace GradientState -> DataLoader reference with weakrefs by @tomaarsen in https://github.com/huggingface/accelerate/pull/3391tests/test_quantization.py on XPU by @faaany in https://github.com/huggingface/accelerate/pull/3349require_non_xpu test markers by @faaany in https://github.com/huggingface/accelerate/pull/3301memory leak] Replace GradientState -> DataLoader reference with weakrefs by @tomaarsen in https://github.com/huggingface/accelerate/pull/3391get_quantized_model_device_map by @faaany in https://github.com/huggingface/accelerate/pull/3397Full Changelog: https://github.com/huggingface/accelerate/compare/v1.3.0...v1.4.0
As it's been ~2 years since torch 2.0 was first released, we are now requiring this as the minimum version for Accelerate, which similarly was done in transformers as of its last release.
keep_torch_compile param to unwrap_model and extract_model_from_parallel for distributed compiled model. by @ggoggam in https://github.com/huggingface/accelerate/pull/3282keep_torch_compile param to unwrap_model and extract_model_from_parallel for distributed compiled model. by @ggoggam in https://github.com/huggingface/accelerate/pull/3282Full Changelog: https://github.com/huggingface/accelerate/compare/v1.2.1...v1.3.0
Full Changelog: https://github.com/huggingface/accelerate/compare/v1.2.0...v1.2.1
find_executable_batch_size on XPU by @faaany in https://github.com/huggingface/accelerate/pull/3236numpy._core instead of numpy.core by @qgallouedec in https://github.com/huggingface/accelerate/pull/3247data_loader] Optionally also propagate set_epoch to batch sampler by @tomaarsen in https://github.com/huggingface/accelerate/pull/3246accelerate config prompt text by @faaany in https://github.com/huggingface/accelerate/pull/3268align_module_device, ensure only cpu tensors for get_state_dict_offloaded_model by @kylesayrs in https://github.com/huggingface/accelerate/pull/3217get_state_dict_from_offload by @kylesayrs in https://github.com/huggingface/accelerate/pull/3253preload_module_classes is lost for nested modules by @wejoncy in https://github.com/huggingface/accelerate/pull/3248Update code in tracking documentation by @faaany in https://github.com/huggingface/accelerate/pull/3235
Replaced set/check breakpoint with set/check trigger in the troubleshooting documentation by @relh in https://github.com/huggingface/accelerate/pull/3259
Update set-seed by @faaany in https://github.com/huggingface/accelerate/pull/3228
Fix typo by @faaany in https://github.com/huggingface/accelerate/pull/3221
Use real path for checkpoint by @faaany in https://github.com/huggingface/accelerate/pull/3220
Fixed multiple typos for Tutorials and Guides docs by @henryhmko in https://github.com/huggingface/accelerate/pull/3274
align_module_device, ensure only cpu tensors for get_state_dict_offloaded_model by @kylesayrs in https://github.com/huggingface/accelerate/pull/3217find_executable_batch_size on XPU by @faaany in https://github.com/huggingface/accelerate/pull/3236data_loader] Optionally also propagate set_epoch to batch sampler by @tomaarsen in https://github.com/huggingface/accelerate/pull/3246numpy._core instead of numpy.core by @qgallouedec in https://github.com/huggingface/accelerate/pull/3247accelerate config prompt text by @faaany in https://github.com/huggingface/accelerate/pull/3268get_state_dict_from_offload by @kylesayrs in https://github.com/huggingface/accelerate/pull/3253preload_module_classes is lost for nested modules by @wejoncy in https://github.com/huggingface/accelerate/pull/3248checkpoint by @faaany in https://github.com/huggingface/accelerate/pull/3220Release diff: https://github.com/huggingface/accelerate/compare/v1.1.1...v1.2.0
data_seed argument in https://github.com/huggingface/accelerate/pull/3150weights_only=True by default for all compatible objects when checkpointing and saving with torch.save in https://github.com/huggingface/accelerate/pull/3036dim input in pad_across_processes in https://github.com/huggingface/accelerate/pull/3114has_offloaded_params utility added in https://github.com/huggingface/accelerate/pull/3188dim input in pad_across_processes by @mariusarvinte in https://github.com/huggingface/accelerate/pull/3114data_seed by @muellerzr in https://github.com/huggingface/accelerate/pull/3150save_model by @muellerzr in https://github.com/huggingface/accelerate/pull/3146weights_only=True by default for all compatible objects by @muellerzr in https://github.com/huggingface/accelerate/pull/3036get_xpu_available_memory by @faaany in https://github.com/huggingface/accelerate/pull/3165has_offloaded_params by @kylesayrs in https://github.com/huggingface/accelerate/pull/3188torch.nn.Module model into account when moving to device by @faaany in https://github.com/huggingface/accelerate/pull/3167torchrun by @faaany in https://github.com/huggingface/accelerate/pull/3166align_module_device by @kylesayrs in https://github.com/huggingface/accelerate/pull/3204Full Changelog: https://github.com/huggingface/accelerate/compare/v1.0.1...v1.1.0
auto values were no longer being parsed when using deepspeedFull Changelog: https://github.com/huggingface/accelerate/compare/v1.0.0...v1.0.1
With accelerate 1.0, we are officially stating that the core parts of the API are now "stable" and ready for the future of what the world of distributed training and PyTorch has to handle. With these release notes, we will focus first on the major breaking changes to get your code fixed, followed by what is new specifically between 0.34.0 and 1.0.
To read more, check out our official blog here
dispatch_batches, split_batches, even_batches, and use_seedable_sampler to the Accelerator() should now be handled by creating an accelerate.utils.DataLoaderConfiguration() and passing this to the Accelerator() instead (Accelerator(dataloader_config=DataLoaderConfiguration(...)))Accelerator().use_fp16 and AcceleratorState().use_fp16 have been removed; this should be replaced by checking accelerator.mixed_precision == "fp16"Accelerator().autocast() no longer accepts a cache_enabled argument. Instead, an AutocastKwargs() instance should be used which handles this flag (among others) passing it to the Accelerator (Accelerator(kwargs_handlers=[AutocastKwargs(cache_enabled=True)]))accelerate.utils.is_tpu_available should be replaced with accelerate.utils.is_torch_xla_availableaccelerate.utils.modeling.shard_checkpoint should be replaced with split_torch_state_dict_into_shards from the huggingface_hub libraryaccelerate.tqdm.tqdm() no longer accepts True/False as the first argument, and instead, main_process_only should be passed in as a named argumentAfter long request, we finally have multiple model DeepSpeed support in Accelerate! (though it is quite early still). Read the full tutorial here, however essentially:
When using multiple models, a DeepSpeed plugin should be created for each model (and as a result, a separate config). a few examples are below:
(Where we train only one model, zero3, and another is used for inference, zero2)
from accelerate import Accelerator
from accelerate.utils import DeepSpeedPlugin
zero2_plugin = DeepSpeedPlugin(hf_ds_config="zero2_config.json")
zero3_plugin = DeepSpeedPlugin(hf_ds_config="zero3_config.json")
deepspeed_plugins = {"student": zero2_plugin, "teacher": zero3_plugin}
accelerator = Accelerator(deepspeed_plugins=deepspeed_plugins)
To then select which plugin to be used at a certain time (aka when calling prepare), we call `accelerator.state.select_deepspeed_plugin("name"), where the first plugin is active by default:
accelerator.state.select_deepspeed_plugin("student")
student_model, optimizer, scheduler = ...
student_model, optimizer, scheduler, train_dataloader = accelerator.prepare(student_model, optimizer, scheduler, train_dataloader)
accelerator.state.select_deepspeed_plugin("teacher") # This will automatically enable zero init
teacher_model = AutoModel.from_pretrained(...)
teacher_model = accelerator.prepare(teacher_model)
For disjoint models, separate accelerators should be used for each model, and their own .backward() should be called later:
for batch in dl:
outputs1 = first_model(**batch)
first_accelerator.backward(outputs1.loss)
first_optimizer.step()
first_scheduler.step()
first_optimizer.zero_grad()
outputs2 = model2(**batch)
second_accelerator.backward(outputs2.loss)
second_optimizer.step()
second_scheduler.step()
second_optimizer.zero_grad()
We've enabled MS-AMP support up to FSDP. At this time we are not going forward with implementing FSDP support with MS-AMP, due to design issues between both libraries that don't make them inter-op easily.
__reduce__ by @byi8220 in https://github.com/huggingface/accelerate/pull/3074_get_named_modules by @faaany in https://github.com/huggingface/accelerate/pull/3052skip_keys usage in forward hooks by @152334H in https://github.com/huggingface/accelerate/pull/3088torch.cuda.amp.GradScaler FutureWarning for pytorch 2.4+ by @Mon-ius in https://github.com/huggingface/accelerate/pull/3132Full Changelog: https://github.com/huggingface/accelerate/compare/v0.34.2...v1.0.0
DataLoaders could no longer be pickled in #3074 thanks to @byi8220default_transformers_cls_names_to_wrap would separate _no_split_modules by characters instead of keeping it as a list of layer names in #3075Full Changelog: https://github.com/huggingface/accelerate/compare/v0.34.0...v0.34.1