v0.15.2
What's Changed
Big shoutout to @rlrs for the fast replace normalizers PR. This boosts the performances of the tokenizers:
- chore: Update dependencies to latest supported versions by @bryantbiggs in https://github.com/huggingface/tokenizers/pull/1441
- Convert word counts to u64 by @stephenroller in https://github.com/huggingface/tokenizers/pull/1433
- Efficient Replace normalizer by @rlrs in https://github.com/huggingface/tokenizers/pull/1413
New Contributors
- @bryantbiggs made their first contribution in https://github.com/huggingface/tokenizers/pull/1441
- @stephenroller made their first contribution in https://github.com/huggingface/tokenizers/pull/1433
- @rlrs made their first contribution in https://github.com/huggingface/tokenizers/pull/1413
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.15.1...v0.15.2rc1
Fetched April 7, 2026
