v0.15.1

What's Changed

• udpate to version = "0.15.1-dev0" by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1390
• Derive Clone on Tokenizer, add Encoding.into_tokens() method by @epwalsh in https://github.com/huggingface/tokenizers/pull/1381
• Stale bot. by @Narsil in https://github.com/huggingface/tokenizers/pull/1404
• Fix doc links in readme by @Pierrci in https://github.com/huggingface/tokenizers/pull/1367
• Faster HF dataset iteration in docs by @mariosasko in https://github.com/huggingface/tokenizers/pull/1414
• Add quick doc to byte_level.rs by @steventrouble in https://github.com/huggingface/tokenizers/pull/1420
• Fix make bench. by @Narsil in https://github.com/huggingface/tokenizers/pull/1428
• Bump follow-redirects from 1.15.1 to 1.15.4 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1430
• pyo3: update to 0.20 by @mikelui in https://github.com/huggingface/tokenizers/pull/1386
• Encode special tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1437
• Update release for python3.12 windows by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1438

New Contributors

• @steventrouble made their first contribution in https://github.com/huggingface/tokenizers/pull/1420

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.15.0...v0.15.1