v0.15.1.rc0

What's Changed

• pyo3: update to 0.19 by @mikelui in https://github.com/huggingface/tokenizers/pull/1322
• Add expect() for disabling truncation by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1316
• Re-using scritpts from safetensors. by @Narsil in https://github.com/huggingface/tokenizers/pull/1328
• Reduce number of different revisions by 1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1329
• Python 38 arm by @Narsil in https://github.com/huggingface/tokenizers/pull/1330
• Move to maturing mimicking move for safetensors. + Rewritten node bindings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1331
• Updating the docs with the new command. by @Narsil in https://github.com/huggingface/tokenizers/pull/1333
• Update added tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1335
• update package version for dev by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1339
• Added ability to inspect a 'Sequence' pre-tokenizer. by @eaplatanios in https://github.com/huggingface/tokenizers/pull/1341
• Let's allow hf_hub < 1.0 by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1344
• Fixing the progressbar. by @Narsil in https://github.com/huggingface/tokenizers/pull/1353
• Preparing release. by @Narsil in https://github.com/huggingface/tokenizers/pull/1355
• fix a clerical error in the comment by @tiandiweizun in https://github.com/huggingface/tokenizers/pull/1356
• fix: remove useless token by @rtrompier in https://github.com/huggingface/tokenizers/pull/1371
• Bump @babel/traverse from 7.22.11 to 7.23.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1370
• Allow hf_hub 0.18 by @mariosasko in https://github.com/huggingface/tokenizers/pull/1383
• Allow huggingface_hub<1.0 by @Wauplin in https://github.com/huggingface/tokenizers/pull/1385
• [pre_tokenizers] Fix sentencepiece based Metaspace by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1357
• udpate to version = "0.15.1-dev0" by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1390
• Derive Clone on Tokenizer, add Encoding.into_tokens() method by @epwalsh in https://github.com/huggingface/tokenizers/pull/1381
• Stale bot. by @Narsil in https://github.com/huggingface/tokenizers/pull/1404
• Fix doc links in readme by @Pierrci in https://github.com/huggingface/tokenizers/pull/1367
• Faster HF dataset iteration in docs by @mariosasko in https://github.com/huggingface/tokenizers/pull/1414
• Add quick doc to byte_level.rs by @steventrouble in https://github.com/huggingface/tokenizers/pull/1420
• Fix make bench. by @Narsil in https://github.com/huggingface/tokenizers/pull/1428
• Bump follow-redirects from 1.15.1 to 1.15.4 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1430
• pyo3: update to 0.20 by @mikelui in https://github.com/huggingface/tokenizers/pull/1386

New Contributors

• @mikelui made their first contribution in https://github.com/huggingface/tokenizers/pull/1322
• @eaplatanios made their first contribution in https://github.com/huggingface/tokenizers/pull/1341
• @tiandiweizun made their first contribution in https://github.com/huggingface/tokenizers/pull/1356
• @rtrompier made their first contribution in https://github.com/huggingface/tokenizers/pull/1371
• @mariosasko made their first contribution in https://github.com/huggingface/tokenizers/pull/1383
• @Wauplin made their first contribution in https://github.com/huggingface/tokenizers/pull/1385
• @steventrouble made their first contribution in https://github.com/huggingface/tokenizers/pull/1420

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.4.rc2...v0.15.1.rc0