v0.15.0

What's Changed

• fix a clerical error in the comment by @tiandiweizun in https://github.com/huggingface/tokenizers/pull/1356
• fix: remove useless token by @rtrompier in https://github.com/huggingface/tokenizers/pull/1371
• Bump @babel/traverse from 7.22.11 to 7.23.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1370
• Allow hf_hub 0.18 by @mariosasko in https://github.com/huggingface/tokenizers/pull/1383
• Allow huggingface_hub<1.0 by @Wauplin in https://github.com/huggingface/tokenizers/pull/1385
• [pre_tokenizers] Fix sentencepiece based Metaspace by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1357

New Contributors

• @tiandiweizun made their first contribution in https://github.com/huggingface/tokenizers/pull/1356
• @rtrompier made their first contribution in https://github.com/huggingface/tokenizers/pull/1371
• @mariosasko made their first contribution in https://github.com/huggingface/tokenizers/pull/1383
• @Wauplin made their first contribution in https://github.com/huggingface/tokenizers/pull/1385

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.14.1...v0.15.0