v0.20.2
Release v0.20.2
Thanks a MILE to @diliop we now have support for python 3.13! 🥳
What's Changed
- Bump cookie and express in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1648
- Fix off-by-one error in tokenizer::normalizer::Range::len by @rlanday in https://github.com/huggingface/tokenizers/pull/1638
- Arg name correction: auth_token -> token by @rravenel in https://github.com/huggingface/tokenizers/pull/1621
- Unsound call of
set_varby @sftse in https://github.com/huggingface/tokenizers/pull/1664 - Add safety comments by @Manishearth in https://github.com/huggingface/tokenizers/pull/1651
- Bump actions/checkout to v4 by @tinyboxvk in https://github.com/huggingface/tokenizers/pull/1667
- PyO3 0.22 by @diliop in https://github.com/huggingface/tokenizers/pull/1665
- Bump actions versions by @tinyboxvk in https://github.com/huggingface/tokenizers/pull/1669
New Contributors
- @rlanday made their first contribution in https://github.com/huggingface/tokenizers/pull/1638
- @rravenel made their first contribution in https://github.com/huggingface/tokenizers/pull/1621
- @sftse made their first contribution in https://github.com/huggingface/tokenizers/pull/1664
- @Manishearth made their first contribution in https://github.com/huggingface/tokenizers/pull/1651
- @tinyboxvk made their first contribution in https://github.com/huggingface/tokenizers/pull/1667
- @diliop made their first contribution in https://github.com/huggingface/tokenizers/pull/1665
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.20.1...v0.20.2
Fetched April 7, 2026
