Release v0.20.4 v0.21.0
- More cache options. by @Narsil in https://github.com/huggingface/tokenizers/pull/1675
- Disable caching for long strings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1676
- Testing ABI3 wheels to reduce number of wheels by @Narsil in https://github.com/huggingface/tokenizers/pull/1674
- Adding an API for decode streaming. by @Narsil in https://github.com/huggingface/tokenizers/pull/1677
- Decode stream python by @Narsil in https://github.com/huggingface/tokenizers/pull/1678
- Fix encode_batch and encode_batch_fast to accept ndarrays again by @diliop in https://github.com/huggingface/tokenizers/pull/1679
We also no longer support python 3.7 or 3.8 (similar to transformers) as they are deprecated.
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.20.3...v0.21.0
Fetched April 7, 2026
