npx @buildinternet/releases get tokenizersNew WebAssembly bindings are published on npm as tokenizers-wasm under the next dist-tag, and Python can now encode pairs of text. Fixed load failures for byte-level BPE vocabularies missing impossible UTF-8 bytes and for WordPiece vocabularies with an empty-string token, plus bindings rejecting tokenizer.json files with regex Split pre-tokenizers or Replace normalizers.
Fixes a breaking truncation and overflowing regression introduced in v0.23.1, and loosens the huggingface_hub requirement to allow v2, unblocking transformers installs. Also fixes type hints not resolving for pyright.
The final v0 release adds a Parity-aware BPE implementation with an example and trainer API docs entry, and avoids a full vocab clone in get_vocab_size(). Maintainers note this is the last v0 release before moving to v1.
tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without…
Okay mostly doing the release for these PR:
Main change:
from_bytes and read_bytes Methods in WordPiece Tokenizer for WebAssembly Compatibility by…Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.3...v0.21.4
No change, the 0.21.3 release failed, this is just a re-release.
https://github.com/huggingface/tokenizers/releases/tag/v0.21.3
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.2...v0.21.3
This release if focused around some performance optimization, enabling broader python no gil support, and fixing some onig issues!
There was a breaking change in 0.20.3 for tuple inputs of encode_batch!
Thanks a MILE to @diliop we now have support for python 3.13! 🥳
The most awaited offset issue with Llama is fixed 🥳
This release is focused on performances and user experience.
First off, we did a bit of benchmarking, and found some place for improvement for us!
With a few minor changes (mostly #1587) here is what we get on Llama3 running…
ignore_merges by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1504Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.19.0...v0.19.1
remove black] And use ruff by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1436Bumping 3 versions because of this: https://github.com/huggingface/transformers/blob/60dea593edd0b94ee15dc3917900b26e3acfbbee/setup.py#L177
