What's Changed
The most awaited offset issue with Llama is fixed 🥳
- Update README.md by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1608
- fix benchmark file link by @152334H in https://github.com/huggingface/tokenizers/pull/1610
- Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows by @dependabot in https://github.com/huggingface/tokenizers/pull/1626
- [
ignore_merges] Fix offsets by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1640 - Bump body-parser and express in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1629
- Bump serve-static and express in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1630
- Bump send and express in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1631
- Bump webpack from 5.76.0 to 5.95.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1641
- Fix documentation build by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1642
- style: simplify string formatting for readability by @hamirmahal in https://github.com/huggingface/tokenizers/pull/1632
New Contributors
- @152334H made their first contribution in https://github.com/huggingface/tokenizers/pull/1610
- @hamirmahal made their first contribution in https://github.com/huggingface/tokenizers/pull/1632
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.20.0...v0.20.1
Fetched April 7, 2026
