Tokenizers

Fixes a breaking truncation and overflowing regression introduced in v0.23.1, and loosens the huggingface_hub requirement to allow v2, unblocking transformers installs. Also fixes type hints not resolving for pyright.

The tokenizers library has been refactored so its stack is faster than before, with a crate size 6x smaller and reduced peak memory, while keeping the same API, same token IDs, and same standards. The work focused on reducing the UTF-8 tax so improvements span across all languages.

The final v0 release adds a Parity-aware BPE implementation with an example and trainer API docs entry, and avoids a full vocab clone in get_vocab_size(). Maintainers note this is the last v0 release before moving to v1.

TL;DR

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without…

What's Changed

Okay mostly doing the release for these PR:

• Update deserialize of added tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1891
• update stub for typing by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1896 *…

Release v0.22.1

Main change:

• Bump huggingface_hub upper version (#1866) from @Wauplin
• chore(trainer): add and improve trainer signature (#1838) from @shenxiangzhuang
• Some doc updates: c91d76ae558ca2dc1aa725959e65dc21bf1fed7e,…

What's Changed

• Bump on-headers and compression in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in https://github.com/huggingface/tokenizers/pull/1827
• Implement from_bytes and read_bytes Methods in WordPiece Tokenizer for WebAssembly Compatibility by…

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.3...v0.21.4

No change, the 0.21.3 release failed, this is just a re-release.

https://github.com/huggingface/tokenizers/releases/tag/v0.21.3

What's Changed

• Clippy fixes. by @Narsil in https://github.com/huggingface/tokenizers/pull/1818
• Fixed an introduced backward breaking change in our Rust APIs.

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.2...v0.21.3

What's Changed

This release if focused around some performance optimization, enabling broader python no gil support, and fixing some onig issues!

• Update the release builds following 0.21.1. by @Narsil in https://github.com/huggingface/tokenizers/pull/1746 *…

What's Changed

• Update dev version and pyproject.toml by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1693
• Add feature flag hint to README.md, fixes #1633 by @sftse in https://github.com/huggingface/tokenizers/pull/1709
• Upgrade to PyO3 0.23 by…

What's Changed

• Update dev version and pyproject.toml by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1693
• Add feature flag hint to README.md, fixes #1633 by @sftse in https://github.com/huggingface/tokenizers/pull/1709
• Upgrade to PyO3 0.23 by…

Release v0.20.4 v0.21.0

• More cache options. by @Narsil in https://github.com/huggingface/tokenizers/pull/1675
• Disable caching for long strings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1676
• Testing ABI3 wheels to reduce number of wheels by…

What's Changed

There was a breaking change in 0.20.3 for tuple inputs of encode_batch!

• fix pylist by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1673
• [MINOR:TYPO] Fix docstrings by @cakiki in…

Release v0.20.2

Thanks a MILE to @diliop we now have support for python 3.13! 🥳

What's Changed

• Bump cookie and express in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1648
• Fix off-by-one error in…

What's Changed

The most awaited offset issue with Llama is fixed 🥳

• Update README.md by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1608
• fix benchmark file link by @152334H in https://github.com/huggingface/tokenizers/pull/1610
• Bump…

Release v0.20.0

This release is focused on performances and user experience.

Performances:

First off, we did a bit of benchmarking, and found some place for improvement for us! With a few minor changes (mostly #1587) here is what we get on Llama3 running…

What's Changed

• add serialization for ignore_merges by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1504

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.19.0...v0.19.1

What's Changed

• chore: Remove CLI - this was originally intended for local development by @bryantbiggs in https://github.com/huggingface/tokenizers/pull/1442
• [remove black] And use ruff by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1436
• Bump ip…

Bumping 3 versions because of this: https://github.com/huggingface/transformers/blob/60dea593edd0b94ee15dc3917900b26e3acfbbee/setup.py#L177

What's Changed

• chore: Remove CLI - this was originally intended for local development by @bryantbiggs in…