v0.28.0

Core

• Introduce a DataLoaderConfiguration and begin deprecation of arguments in the Accelerator

+from accelerate import DataLoaderConfiguration
+dl_config = DataLoaderConfiguration(split_batches=True, dispatch_batches=True)
-accelerator = Accelerator(split_batches=True, dispatch_batches=True)
+accelerator = Accelerator(dataloader_config=dl_config)

• Allow gradients to be synced each data batch while performing gradient accumulation, useful when training in FSDP by @fabianlim in https://github.com/huggingface/accelerate/pull/2531

from accelerate import GradientAccumulationPlugin
plugin = GradientAccumulationPlugin(
+    num_steps=2, 
    sync_each_batch=sync_each_batch
)
accelerator = Accelerator(gradient_accumulation_plugin=plugin)

Torch XLA

• Support for XLA on the GPU by @anw90 in https://github.com/huggingface/accelerate/pull/2176
• Enable gradient accumulation on TPU in https://github.com/huggingface/accelerate/pull/2453

FSDP

• Support downstream FSDP + QLORA support through tweaks by allowing configuration of buffer precision by @pacman100 in https://github.com/huggingface/accelerate/pull/2544

launch changes

• Support mpirun for multi-cpu training by @dmsuehir in https://github.com/huggingface/accelerate/pull/2493

What's Changed

• Fix model metadata issue check by @muellerzr in https://github.com/huggingface/accelerate/pull/2435
• Use py 3.9 by @muellerzr in https://github.com/huggingface/accelerate/pull/2436
• Fix seedable sampler logic and expound docs by @muellerzr in https://github.com/huggingface/accelerate/pull/2434
• Fix tied_pointers_to_remove type by @fxmarty in https://github.com/huggingface/accelerate/pull/2439
• Make test assertions more idiomatic by @akx in https://github.com/huggingface/accelerate/pull/2420
• Prefer is_torch_tensor over hasattr for torch.compile. by @PhilJd in https://github.com/huggingface/accelerate/pull/2387
• Enable more Ruff lints & fix issues by @akx in https://github.com/huggingface/accelerate/pull/2419
• Fix warning when dispatching model by @SunMarc in https://github.com/huggingface/accelerate/pull/2442
• Make torch xla available on GPU by @anw90 in https://github.com/huggingface/accelerate/pull/2176
• Include pippy_file_path by @muellerzr in https://github.com/huggingface/accelerate/pull/2444
• [Big deprecation] Introduces a DataLoaderConfig by @muellerzr in https://github.com/huggingface/accelerate/pull/2441
• Check for None by @muellerzr in https://github.com/huggingface/accelerate/pull/2452
• Fix the pytest version to be less than 8.0.1 by @BenjaminBossan in https://github.com/huggingface/accelerate/pull/2461
• Fix wrong is_namedtuple implementation by @fxmarty in https://github.com/huggingface/accelerate/pull/2475
• Use grad-accum on TPU by @muellerzr in https://github.com/huggingface/accelerate/pull/2453
• Add pre-commit configuration by @akx in https://github.com/huggingface/accelerate/pull/2451
• Replace os.path.sep.join path manipulations with a helper by @akx in https://github.com/huggingface/accelerate/pull/2446
• DOC: Fixes to Accelerator docstring by @BenjaminBossan in https://github.com/huggingface/accelerate/pull/2443
• Context manager fixes by @akx in https://github.com/huggingface/accelerate/pull/2450
• Fix TPU with new XLA device type by @will-cromar in https://github.com/huggingface/accelerate/pull/2467
• Free mps memory by @SunMarc in https://github.com/huggingface/accelerate/pull/2483
• [FIX] allow Accelerator to detect distributed type from the "LOCAL_RANK" env variable for XPU by @faaany in https://github.com/huggingface/accelerate/pull/2473
• Fix CI tests due to pathlib issues by @muellerzr in https://github.com/huggingface/accelerate/pull/2491
• Remove all cases of torchrun in tests and centralize as accelerate launch by @muellerzr in https://github.com/huggingface/accelerate/pull/2498
• Fix link typo by @SunMarc in https://github.com/huggingface/accelerate/pull/2503
• [docs] Accelerator API by @stevhliu in https://github.com/huggingface/accelerate/pull/2465
• Docstring fixup by @muellerzr in https://github.com/huggingface/accelerate/pull/2504
• [docs] Divide training and inference by @stevhliu in https://github.com/huggingface/accelerate/pull/2466
• add custom dtype INT2 by @SunMarc in https://github.com/huggingface/accelerate/pull/2505
• quanto compatibility for cpu/disk offload by @SunMarc in https://github.com/huggingface/accelerate/pull/2481
• [docs] Quicktour by @stevhliu in https://github.com/huggingface/accelerate/pull/2456
• Check if hub down by @muellerzr in https://github.com/huggingface/accelerate/pull/2506
• Remove offline stuff by @muellerzr in https://github.com/huggingface/accelerate/pull/2509
• Fixed 0MiB bug in convert_file_size_to_int by @StoyanStAtanasov in https://github.com/huggingface/accelerate/pull/2507
• Fix edge case in infer_auto_device_map when dealing with buffers by @SunMarc in https://github.com/huggingface/accelerate/pull/2511
• [docs] Fix typos by @omahs in https://github.com/huggingface/accelerate/pull/2490
• fix typo in launch.py (----main_process_port to --main_process_port) by @DerrickWang005 in https://github.com/huggingface/accelerate/pull/2516
• Add copyright + some ruff lint things by @muellerzr in https://github.com/huggingface/accelerate/pull/2523
• Don't manage PYTORCH_NVML_BASED_CUDA_CHECK when calling accelerate.utils.imports.is_cuda_available() by @luiscape in https://github.com/huggingface/accelerate/pull/2524
• Quanto compatibility with QBitsTensor by @SunMarc in https://github.com/huggingface/accelerate/pull/2526
• Remove unnecessary env=os.environ.copy()s by @akx in https://github.com/huggingface/accelerate/pull/2449
• Launch mpirun from accelerate launch for multi-CPU training by @dmsuehir in https://github.com/huggingface/accelerate/pull/2493
• Enable using dash or underscore for CLI args by @muellerzr in https://github.com/huggingface/accelerate/pull/2527
• Update the default behavior of zero_grad(set_to_none=None) to align with PyTorch by @yongchanghao in https://github.com/huggingface/accelerate/pull/2472
• Update link to dynamo/compile doc by @WarmongeringBeaver in https://github.com/huggingface/accelerate/pull/2533
• Check if the buffers fit GPU memory after device map auto inferred by @notsyncing in https://github.com/huggingface/accelerate/pull/2412
• [Refactor] Refactor send_to_device to treat tensor-like first by @vmoens in https://github.com/huggingface/accelerate/pull/2438
• Overdue email change... by @muellerzr in https://github.com/huggingface/accelerate/pull/2534
• [docs] Troubleshoot by @stevhliu in https://github.com/huggingface/accelerate/pull/2538
• Remove extra double-dash in error message by @drscotthawley in https://github.com/huggingface/accelerate/pull/2541
• Allow Gradients to be Synced Each Data Batch While Performing Gradient Accumulation by @fabianlim in https://github.com/huggingface/accelerate/pull/2531
• Update FSDP mixed precision setter to enable fsdp+qlora by @pacman100 in https://github.com/huggingface/accelerate/pull/2544
• Use uv instead of pip install for github CI by @muellerzr in https://github.com/huggingface/accelerate/pull/2546

New Contributors

• @anw90 made their first contribution in https://github.com/huggingface/accelerate/pull/2176
• @StoyanStAtanasov made their first contribution in https://github.com/huggingface/accelerate/pull/2507
• @omahs made their first contribution in https://github.com/huggingface/accelerate/pull/2490
• @DerrickWang005 made their first contribution in https://github.com/huggingface/accelerate/pull/2516
• @luiscape made their first contribution in https://github.com/huggingface/accelerate/pull/2524
• @dmsuehir made their first contribution in https://github.com/huggingface/accelerate/pull/2493
• @yongchanghao made their first contribution in https://github.com/huggingface/accelerate/pull/2472
• @WarmongeringBeaver made their first contribution in https://github.com/huggingface/accelerate/pull/2533
• @vmoens made their first contribution in https://github.com/huggingface/accelerate/pull/2438
• @drscotthawley made their first contribution in https://github.com/huggingface/accelerate/pull/2541
• @fabianlim made their first contribution in https://github.com/huggingface/accelerate/pull/2531

Full Changelog: https://github.com/huggingface/accelerate/compare/v0.27.2...v0.28.0