PyTorch 2.0 stack support
We are very excited by the newly announced PyTorch 2.0 stack and you can try it using Accelerate on any model by using the dynamo_backend argument of the Accelerator, or when filling your config with accelerate config.
Note that to get the best performance, we recommend:
- using an Ampere GPU (or more recent)
- sticking to fixed shaped for now
- Add support for torch dynamo by @sgugger in #829
New CLI commands
- Added two new commands,
accelerate config update and accelerate config default. The first will update a config file to have the latest keys added from latter releases of Accelerate, and the second will create a default configuration file automatically mimicking write_default_config() introduced in #851 and #853 by @muellerzr
- Also introduced a filterable help for
accelerate launch which will show options relevant to the choices shown, such as accelerate launch --multi_gpu will show launch parameters relevant to multi-gpu training.
What's new?
- fix 🐛 by @pacman100 in #836
- Deepspeed example should use gather_for_metrics by @HammadB in #821
- Highlight selection with pretty colors by @muellerzr in #839
- Add
join_uneven_inputs context manager to Accelerator by @Chris-hughes10 in #820
- Introduce
default-config command by @muellerzr in #840
- Fix log error and add log level to get_logger by @muellerzr in #842
- Fix if/else by @muellerzr in #849
- Fix complete_cv example by @muellerzr in #848
- Refactor Accelerate config and introduce a multi-argument CLI interface by @muellerzr in #851
- Clean up, add update command by @muellerzr in #853
- Revert "Update pr docs actions by @mishig25 in #827)"
- Switch default log to warn by @muellerzr in #859
- Remove mixed precision hook as part of the unwrap_model by @muellerzr in #860
- update deepspeed error message wrt
batch_size by @pacman100 in #861
- fix failing deepspeed test by @pacman100 in #868
- Even more log level refined, leave alone if not explicitly set by @muellerzr in #871
- Solve pickling issues by @muellerzr in #872
- Spring cleaning by @muellerzr in #865
- fixing lr_scheduler prepare issue when using pytorch nightly by @pacman100 in #878
- fix fsdp state_dict_config because of PyTorch changes by @pacman100 in #877
- Update deprecated logging warn by @SHi-ON in #881
- fix a bug by @xiaohu2015 in #887
- Allow safetensors offload by @sgugger in #873
- fixing lr scheduler for pytorch nightly by @pacman100 in #884
- Prefix all accelerate env vars with ACCELERATE by @muellerzr in #890
- fix prefix issues in tests by @pacman100 in #891
- Fix windows cli selector by @muellerzr in #893
- Better description for improper kwargs by @muellerzr in #894
- Support bfloat16 in load_offloaded_weight by @sgugger in #892
Significant community contributions
The following contributors have made significant changes to the library over the last release:
- @Chris-hughes10
- Add
join_uneven_inputs context manager to Accelerator (#820)