Tokenizers

TL;DR

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeli...

What's Changed

Okay mostly doing the release for these PR:

• Update deserialize of added tokens by @ArthurZucker in https://github.com/hugging...

Release v0.22.1

Main change:

• Bump huggingface_hub upper version (#1866) from @Wauplin
• chore(trainer): add and improve trainer signature (#...

What's Changed

• Bump on-headers and compression in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in https://github.com/huggingface/t...

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.3...v0.21.4

No change, the 0.21.3 release failed, this is just a re-r...

What's Changed

• Clippy fixes. by @Narsil in https://github.com/huggingface/tokenizers/pull/1818
• Fixed an introduced backward breaking change i...

What's Changed

This release if focused around some performance optimization, enabling broader python no gil support, and fixing some onig issues...

What's Changed

• Update dev version and pyproject.toml by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1693
• Add feature flag...

What's Changed

• Update dev version and pyproject.toml by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1693
• Add feature flag...

Release v0.20.4 v0.21.0

• More cache options. by @Narsil in https://github.com/huggingface/tokenizers/pull/1675
• Disable caching for long str...

What's Changed

There was a breaking change in 0.20.3 for tuple inputs of encode_batch!

• fix pylist by @ArthurZucker in https://github.com/h...

Release v0.20.2

Thanks a MILE to @diliop we now have support for python 3.13! 🥳

What's Changed

• Bump cookie and express in /tokenizers/...

What's Changed

The most awaited offset issue with Llama is fixed 🥳

• Update README.md by @ArthurZucker in https://github.com/huggingface/...

Release v0.20.0

This release is focused on performances and user experience.

Performances:

First off, we did a bit of benchmarkin...

What's Changed

• add serialization for ignore_merges by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1504

**Full Changel...

What's Changed

• chore: Remove CLI - this was originally intended for local development by @bryantbiggs in https://github.com/huggingface/tokenize...

Bumping 3 versions because of this: https://github.com/huggingface/transformers/blob/60dea593edd0b94ee15dc3917900b26e3acfbbee/setup.py#L177

What...

What's Changed

Big shoutout to @rlrs for the fast replace normalizers PR. This boosts the p...

What's Changed

• udpate to version = "0.15.1-dev0" by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1390
• Derive Clone on `T...

What's Changed

• fix a clerical error in the comment by @tiandiweizun in https://github.com/huggingface/tokenizers/pull/1356
• fix: remove usele...