v0.21.2 — Tokenizers

What's Changed

This release if focused around some performance optimization, enabling broader python no gil support, and fixing some onig issues!

Update the release builds following 0.21.1. by @Narsil in https://github.com/huggingface/tokenizers/pull/1746
replace lazy_static with stabilized std::sync::LazyLock in 1.80 by @sftse in https://github.com/huggingface/tokenizers/pull/1739
Fix no-onig no-wasm builds by @414owen in https://github.com/huggingface/tokenizers/pull/1772
Fix typos in strings and comments by @co63oc in https://github.com/huggingface/tokenizers/pull/1770
Fix type notation of merges in BPE Python binding by @Coqueue in https://github.com/huggingface/tokenizers/pull/1766
Bump http-proxy-middleware from 2.0.6 to 2.0.9 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1762
Fix data path in test_continuing_prefix_trainer_mismatch by @GaetanLepage in https://github.com/huggingface/tokenizers/pull/1747
clippy by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1781
Update pyo3 and rust-numpy depends for no-gil/free-threading compat by @Qubitium in https://github.com/huggingface/tokenizers/pull/1774
Use ApiBuilder::from_env() in from_pretrained function by @BenLocal in https://github.com/huggingface/tokenizers/pull/1737
Upgrade onig, to get it compiling with GCC 15 by @414owen in https://github.com/huggingface/tokenizers/pull/1771
Itertools upgrade by @sftse in https://github.com/huggingface/tokenizers/pull/1756
Bump webpack-dev-server from 4.10.0 to 5.2.1 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1792
Bump brace-expansion from 1.1.11 to 1.1.12 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1796
Fix features blending into a paragraph by @bionicles in https://github.com/huggingface/tokenizers/pull/1798
Adding throughput to benches to have a more consistent measure across by @Narsil in https://github.com/huggingface/tokenizers/pull/1800
Upgrading dependencies. by @Narsil in https://github.com/huggingface/tokenizers/pull/1801
[docs] Whitespace by @stevhliu in https://github.com/huggingface/tokenizers/pull/1785
Hotfixing the stub. by @Narsil in https://github.com/huggingface/tokenizers/pull/1802
Bpe clones by @sftse in https://github.com/huggingface/tokenizers/pull/1707
Fixed Length Pre-Tokenizer by @jonvet in https://github.com/huggingface/tokenizers/pull/1713
Consolidated optimization ahash dary compact str by @Narsil in https://github.com/huggingface/tokenizers/pull/1799
🚨 breaking: Fix training with special tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1617

New Contributors

@414owen made their first contribution in https://github.com/huggingface/tokenizers/pull/1772
@co63oc made their first contribution in https://github.com/huggingface/tokenizers/pull/1770
@Coqueue made their first contribution in https://github.com/huggingface/tokenizers/pull/1766
@GaetanLepage made their first contribution in https://github.com/huggingface/tokenizers/pull/1747
@Qubitium made their first contribution in https://github.com/huggingface/tokenizers/pull/1774
@BenLocal made their first contribution in https://github.com/huggingface/tokenizers/pull/1737
@bionicles made their first contribution in https://github.com/huggingface/tokenizers/pull/1798
@stevhliu made their first contribution in https://github.com/huggingface/tokenizers/pull/1785
@jonvet made their first contribution in https://github.com/huggingface/tokenizers/pull/1713

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.1...v0.21.2rc0