v0.20.1

What's Changed

The most awaited offset issue with Llama is fixed 🥳

• Update README.md by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1608
• fix benchmark file link by @152334H in https://github.com/huggingface/tokenizers/pull/1610
• Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows by @dependabot in https://github.com/huggingface/tokenizers/pull/1626
• [ignore_merges] Fix offsets by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1640
• Bump body-parser and express in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1629
• Bump serve-static and express in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1630
• Bump send and express in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1631
• Bump webpack from 5.76.0 to 5.95.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1641
• Fix documentation build by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1642
• style: simplify string formatting for readability by @hamirmahal in https://github.com/huggingface/tokenizers/pull/1632

New Contributors

• @152334H made their first contribution in https://github.com/huggingface/tokenizers/pull/1610
• @hamirmahal made their first contribution in https://github.com/huggingface/tokenizers/pull/1632

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.20.0...v0.20.1