v0.32.0 — Accelerate

Core

Utilize shard saving from the huggingface_hub rather than our own implementation (https://github.com/huggingface/accelerate/pull/2795)
Refactor logging to use logger in dispatch_model (https://github.com/huggingface/accelerate/pull/2855)
The Accelerator.step number is now restored when using save_state and load_state (https://github.com/huggingface/accelerate/pull/2765)
A new profiler has been added allowing users to collect performance metrics during model training and inference, including detailed analysis of execution time and memory consumption. These can then be generated in Chrome's tracing tool. Read more about it here (https://github.com/huggingface/accelerate/pull/2883)
Reduced import times for doing import accelerate and any other major core import by 68%, now should be only slightly longer than doing import torch (https://github.com/huggingface/accelerate/pull/2845)
Fixed a bug in get_backend and added a clear_device_cache utility (https://github.com/huggingface/accelerate/pull/2857)

Distributed Data Parallelism

Introduce DDP communication hooks to have more flexibility in how gradients are communicated across workers, overriding the standard allreduce. (https://github.com/huggingface/accelerate/pull/2841)
Make log_line_prefix_template optional the notebook_launcher (https://github.com/huggingface/accelerate/pull/2888)

FSDP

If the output directory doesn't exist when using accelerate merge-weights, one will be automatically created (https://github.com/huggingface/accelerate/pull/2854)
When merging weights, the default is now .safetensors (https://github.com/huggingface/accelerate/pull/2853)

XPU

Migrate to pytorch's native XPU backend on torch>=2.4 (https://github.com/huggingface/accelerate/pull/2825)
Add @require_triton test decorator and enable test_dynamo work on xpu (https://github.com/huggingface/accelerate/pull/2878)
Fixed load_state_dict not working on xpu and refine xpu safetensors version check (https://github.com/huggingface/accelerate/pull/2879)

XLA

Added support for XLA Dynamo backends for both training and inference (https://github.com/huggingface/accelerate/pull/2892)

Examples

Added a new multi-cpu SLURM example using accelerate launch (https://github.com/huggingface/accelerate/pull/2902)

Full Changelog

Use shard saving from huggingface_hub by @SunMarc in https://github.com/huggingface/accelerate/pull/2795
doc: fix link by @imba-tjd in https://github.com/huggingface/accelerate/pull/2844
Revert "Slight rename" by @SunMarc in https://github.com/huggingface/accelerate/pull/2850
remove warning hook addede during dispatch_model by @SunMarc in https://github.com/huggingface/accelerate/pull/2843
Remove underlines between badges by @novialriptide in https://github.com/huggingface/accelerate/pull/2851
Auto create dir when merging FSDP weights by @helloworld1 in https://github.com/huggingface/accelerate/pull/2854
Add DDP Communication Hooks by @yhna940 in https://github.com/huggingface/accelerate/pull/2841
Refactor logging to use logger in dispatch_model by @panjd123 in https://github.com/huggingface/accelerate/pull/2855
xpu: support xpu backend from stock pytorch (>=2.4) by @dvrogozh in https://github.com/huggingface/accelerate/pull/2825
Drop torch re-imports in npu and mlu paths by @dvrogozh in https://github.com/huggingface/accelerate/pull/2856
Default FSDP weights merge to safetensors by @helloworld1 in https://github.com/huggingface/accelerate/pull/2853
[tests] fix bug in test_tracking.ClearMLTest by @faaany in https://github.com/huggingface/accelerate/pull/2863
[tests] use torch_device instead of 0 for device check by @faaany in https://github.com/huggingface/accelerate/pull/2861
[tests] skip bnb-related tests instead of failing on xpu by @faaany in https://github.com/huggingface/accelerate/pull/2860
Potentially fix tests by @muellerzr in https://github.com/huggingface/accelerate/pull/2862
[tests] enable XPU backend for test_zero3_integration by @faaany in https://github.com/huggingface/accelerate/pull/2864
Support saving and loading of step while saving and loading state by @bipinKrishnan in https://github.com/huggingface/accelerate/pull/2765
Add Profiler Support for Performance Analysis by @yhna940 in https://github.com/huggingface/accelerate/pull/2883
Speed up imports and add a CI by @muellerzr in https://github.com/huggingface/accelerate/pull/2845
Make log_line_prefix_template Optional in Elastic Launcher for Backward Compatibility by @yhna940 in https://github.com/huggingface/accelerate/pull/2888
Add XLA Dynamo backends for training and inference by @johnsutor in https://github.com/huggingface/accelerate/pull/2892
Added a MultiCPU SLURM example using Accelerate Launch and MPIRun by @okhleif-IL in https://github.com/huggingface/accelerate/pull/2902
make more cuda-only tests device-agnostic by @faaany in https://github.com/huggingface/accelerate/pull/2876
fix mlu device longTensor bugs by @huismiling in https://github.com/huggingface/accelerate/pull/2887
add require_triton and enable test_dynamo work on xpu by @faaany in https://github.com/huggingface/accelerate/pull/2878
fix load_state_dict for xpu and refine xpu safetensor version check by @faaany in https://github.com/huggingface/accelerate/pull/2879
Fix get_backend bug and add clear_device_cache function by @NurmaU in https://github.com/huggingface/accelerate/pull/2857

New Contributors

@McPatate made their first contribution in https://github.com/huggingface/accelerate/pull/2836
@imba-tjd made their first contribution in https://github.com/huggingface/accelerate/pull/2844
@novialriptide made their first contribution in https://github.com/huggingface/accelerate/pull/2851
@panjd123 made their first contribution in https://github.com/huggingface/accelerate/pull/2855
@dvrogozh made their first contribution in https://github.com/huggingface/accelerate/pull/2825
@johnsutor made their first contribution in https://github.com/huggingface/accelerate/pull/2892
@okhleif-IL made their first contribution in https://github.com/huggingface/accelerate/pull/2902
@NurmaU made their first contribution in https://github.com/huggingface/accelerate/pull/2857

Full Changelog: https://github.com/huggingface/accelerate/compare/v0.31.0...v0.32.0