npx @buildinternet/releases get tokenizersThe tokenizers library has been refactored so its stack is faster than before, with a crate size 6x smaller and reduced peak memory, while keeping the same API, same token IDs, and same standards. The work focused on reducing the UTF-8 tax so improvements span across all languages.
The final v0 release adds a Parity-aware BPE implementation with an example and trainer API docs entry, and avoids a full vocab clone in get_vocab_size(). Maintainers note this is the last v0 release before moving to v1.
tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without…
Okay mostly doing the release for these PR:
Main change:
from_bytes and read_bytes Methods in WordPiece Tokenizer for WebAssembly Compatibility by…Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.3...v0.21.4
No change, the 0.21.3 release failed, this is just a re-release.
https://github.com/huggingface/tokenizers/releases/tag/v0.21.3
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.2...v0.21.3
This release if focused around some performance optimization, enabling broader python no gil support, and fixing some onig issues!
There was a breaking change in 0.20.3 for tuple inputs of encode_batch!
Thanks a MILE to @diliop we now have support for python 3.13! 🥳
The most awaited offset issue with Llama is fixed 🥳
This release is focused on performances and user experience.
First off, we did a bit of benchmarking, and found some place for improvement for us!
With a few minor changes (mostly #1587) here is what we get on Llama3 running…
ignore_merges by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1504Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.19.0...v0.19.1
remove black] And use ruff by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1436Bumping 3 versions because of this: https://github.com/huggingface/transformers/blob/60dea593edd0b94ee15dc3917900b26e3acfbbee/setup.py#L177
Big shoutout to @rlrs for the fast replace normalizers PR. This boosts the performances of the tokenizers:…
