v0.21.1

What's Changed

• Update dev version and pyproject.toml by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1693
• Add feature flag hint to README.md, fixes #1633 by @sftse in https://github.com/huggingface/tokenizers/pull/1709
• Upgrade to PyO3 0.23 by @Narsil in https://github.com/huggingface/tokenizers/pull/1708
• Fixing the README. by @Narsil in https://github.com/huggingface/tokenizers/pull/1714
• Fix typo in Split docstrings by @Dylan-Harden3 in https://github.com/huggingface/tokenizers/pull/1701
• Fix typos by @tinyboxvk in https://github.com/huggingface/tokenizers/pull/1715
• Update documentation of Rust feature by @sondalex in https://github.com/huggingface/tokenizers/pull/1711
• Fix panic in DecodeStream::step due to incorrect index usage by @n0gu-furiosa in https://github.com/huggingface/tokenizers/pull/1699
• Fixing the stream by removing the read_index altogether. by @Narsil in https://github.com/huggingface/tokenizers/pull/1716
• Fixing NormalizedString append when normalized is empty. by @Narsil in https://github.com/huggingface/tokenizers/pull/1717
• 🚨 Support updating template processors by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1652. Removed in this release to keep backware compatibility temporarily.
• Update metadata as Python3.7 and Python3.8 support was dropped by @earlytobed in https://github.com/huggingface/tokenizers/pull/1724
• Add rustls-tls feature by @torymur in https://github.com/huggingface/tokenizers/pull/1732

New Contributors

• @Dylan-Harden3 made their first contribution in https://github.com/huggingface/tokenizers/pull/1701
• @sondalex made their first contribution in https://github.com/huggingface/tokenizers/pull/1711
• @n0gu-furiosa made their first contribution in https://github.com/huggingface/tokenizers/pull/1699
• @earlytobed made their first contribution in https://github.com/huggingface/tokenizers/pull/1724
• @torymur made their first contribution in https://github.com/huggingface/tokenizers/pull/1732

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.0...v0.21.1