Hugging Face/Tokenizers

Tokenizers

$npx -y @buildinternet/releases show tokenizers

Sun

Mon

Tue

Wed

Thu

Fri

Sat

AprMayJunJulAugSepOctNovDecJanFebMarApr

Less

Releases2Avg0/wkVersionsv0.22.1 → v0.22.2

Oct 6, 2023

What's Changed

Fix conda release by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1211
Fix node release by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1212
Printing warning to stderr. by @Narsil in https://github.com/huggingface/tokenizers/pull/1222
Fixing padding_left sequence_ids. by @Narsil in https://github.com/huggingface/tokenizers/pull/1233
Use LTO for release and benchmark builds by @csko in https://github.com/huggingface/tokenizers/pull/1157
fix unigram.rs test_sample() by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1244
implement a simple max_sentencepiece_length into BPE by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1228
Makes decode and decode_batch work on borrowed content. by @mfuntowicz in https://github.com/huggingface/tokenizers/pull/1251
Update all GH Actions with dependency on actions/checkout by @mfuntowicz in https://github.com/huggingface/tokenizers/pull/1256
Parallelize unigram trainer by @mishig25 in https://github.com/huggingface/tokenizers/pull/976
Update unigram/trainer.rs by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1257
Fixing broken link. by @Narsil in https://github.com/huggingface/tokenizers/pull/1268
fix documentation regarding regex by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1264
Update Cargo.toml by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1266
Update README.md - Broken link by @sbhavani in https://github.com/huggingface/tokenizers/pull/1272
[doc build] Use secrets by @mishig25 in https://github.com/huggingface/tokenizers/pull/1273
Improve error for truncation with too high stride by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1275
Add unigram bytefallback by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1217
revise type specification by @hiroshi-matsuda-rit in https://github.com/huggingface/tokenizers/pull/1289
Bump tough-cookie from 4.0.0 to 4.1.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1291
Update path name: master -> main by @bact in https://github.com/huggingface/tokenizers/pull/1292
import Tuple from typing by @kellymarchisio in https://github.com/huggingface/tokenizers/pull/1295
Fixing clippy warnings on 1.71. by @Narsil in https://github.com/huggingface/tokenizers/pull/1296
Bump word-wrap from 1.2.3 to 1.2.4 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1299
feat: Added CITATION.cff. by @SamuelLarkin in https://github.com/huggingface/tokenizers/pull/1302
Single warning for holes. by @Narsil in https://github.com/huggingface/tokenizers/pull/1303
Give error when initializing tokenizer with too high stride by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1306
Handle when precompiled charsmap is empty by @kellymarchisio in https://github.com/huggingface/tokenizers/pull/1308
Derive clone for TrainerWrapper by @jonatanklosko in https://github.com/huggingface/tokenizers/pull/1317
CD backports by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1318
0.13.4.rc1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1319
Release all at once for simplicity. by @Narsil in https://github.com/huggingface/tokenizers/pull/1320
Fix stride condition. by @Narsil in https://github.com/huggingface/tokenizers/pull/1321
pyo3: update to 0.19 by @mikelui in https://github.com/huggingface/tokenizers/pull/1322
Add expect() for disabling truncation by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1316
Re-using scritpts from safetensors. by @Narsil in https://github.com/huggingface/tokenizers/pull/1328
Reduce number of different revisions by 1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1329
Python 38 arm by @Narsil in https://github.com/huggingface/tokenizers/pull/1330
Move to maturing mimicking move for safetensors. + Rewritten node bindings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1331
Updating the docs with the new command. by @Narsil in https://github.com/huggingface/tokenizers/pull/1333
Update added tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1335
update package version for dev by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1339
Added ability to inspect a 'Sequence' pre-tokenizer. by @eaplatanios in https://github.com/huggingface/tokenizers/pull/1341
Let's allow hf_hub < 1.0 by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1344
Fixing the progressbar. by @Narsil in https://github.com/huggingface/tokenizers/pull/1353
Preparing release. by @Narsil in https://github.com/huggingface/tokenizers/pull/1355

New Contributors

@csko made their first contribution in https://github.com/huggingface/tokenizers/pull/1157
@chris-ha458 made their first contribution in https://github.com/huggingface/tokenizers/pull/1244
@sbhavani made their first contribution in https://github.com/huggingface/tokenizers/pull/1272
@boyleconnor made their first contribution in https://github.com/huggingface/tokenizers/pull/1275
@hiroshi-matsuda-rit made their first contribution in https://github.com/huggingface/tokenizers/pull/1289
@bact made their first contribution in https://github.com/huggingface/tokenizers/pull/1292
@kellymarchisio made their first contribution in https://github.com/huggingface/tokenizers/pull/1295
@SamuelLarkin made their first contribution in https://github.com/huggingface/tokenizers/pull/1302
@jonatanklosko made their first contribution in https://github.com/huggingface/tokenizers/pull/1317
@mikelui made their first contribution in https://github.com/huggingface/tokenizers/pull/1322
@eaplatanios made their first contribution in https://github.com/huggingface/tokenizers/pull/1341

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.3...v0.14.1

Oct 5, 2023

What's Changed

pyo3: update to 0.19 by @mikelui in https://github.com/huggingface/tokenizers/pull/1322
Add expect() for disabling truncation by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1316
Re-using scritpts from safetensors. by @Narsil in https://github.com/huggingface/tokenizers/pull/1328
Reduce number of different revisions by 1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1329
Python 38 arm by @Narsil in https://github.com/huggingface/tokenizers/pull/1330
Move to maturing mimicking move for safetensors. + Rewritten node bindings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1331
Updating the docs with the new command. by @Narsil in https://github.com/huggingface/tokenizers/pull/1333
Update added tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1335
update package version for dev by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1339
Added ability to inspect a 'Sequence' pre-tokenizer. by @eaplatanios in https://github.com/huggingface/tokenizers/pull/1341
Let's allow hf_hub < 1.0 by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1344
Fixing the progressbar. by @Narsil in https://github.com/huggingface/tokenizers/pull/1353

New Contributors

@mikelui made their first contribution in https://github.com/huggingface/tokenizers/pull/1322
@eaplatanios made their first contribution in https://github.com/huggingface/tokenizers/pull/1341

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.4.rc2...v0.14.1rc1

Sep 7, 2023

⚠️ Reworks the release pipeline. Other breaking changes ⚠️ :

#1335, AddedToken is reworked, is_special_token rename to special for consistency
feature http is now OFF by default, and depends on hf-hub instead of cached_path (updated cache directory, better sync implementation)
Removed SSL link on the python package, calling huggingface_hub directly instead.
New dependency : huggingface_hub (while we deprecate Tokenizer.from_pretrained(...) to Tokenizer.from_file(hugginngface_hub.hf_hub_download(MODEL_ID, "tokenizer.json")

What's Changed

Fix conda release by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1211
Fix node release by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1212
Printing warning to stderr. by @Narsil in https://github.com/huggingface/tokenizers/pull/1222
Fixing padding_left sequence_ids. by @Narsil in https://github.com/huggingface/tokenizers/pull/1233
Use LTO for release and benchmark builds by @csko in https://github.com/huggingface/tokenizers/pull/1157
fix unigram.rs test_sample() by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1244
implement a simple max_sentencepiece_length into BPE by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1228
Makes decode and decode_batch work on borrowed content. by @mfuntowicz in https://github.com/huggingface/tokenizers/pull/1251
Update all GH Actions with dependency on actions/checkout by @mfuntowicz in https://github.com/huggingface/tokenizers/pull/1256
Parallelize unigram trainer by @mishig25 in https://github.com/huggingface/tokenizers/pull/976
Update unigram/trainer.rs by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1257
Fixing broken link. by @Narsil in https://github.com/huggingface/tokenizers/pull/1268
fix documentation regarding regex by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1264
Update Cargo.toml by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1266
Update README.md - Broken link by @sbhavani in https://github.com/huggingface/tokenizers/pull/1272
[doc build] Use secrets by @mishig25 in https://github.com/huggingface/tokenizers/pull/1273
Improve error for truncation with too high stride by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1275
Add unigram bytefallback by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1217
revise type specification by @hiroshi-matsuda-rit in https://github.com/huggingface/tokenizers/pull/1289
Bump tough-cookie from 4.0.0 to 4.1.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1291
Update path name: master -> main by @bact in https://github.com/huggingface/tokenizers/pull/1292
import Tuple from typing by @kellymarchisio in https://github.com/huggingface/tokenizers/pull/1295
Fixing clippy warnings on 1.71. by @Narsil in https://github.com/huggingface/tokenizers/pull/1296
Bump word-wrap from 1.2.3 to 1.2.4 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1299
feat: Added CITATION.cff. by @SamuelLarkin in https://github.com/huggingface/tokenizers/pull/1302
Single warning for holes. by @Narsil in https://github.com/huggingface/tokenizers/pull/1303
Give error when initializing tokenizer with too high stride by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1306
Handle when precompiled charsmap is empty by @kellymarchisio in https://github.com/huggingface/tokenizers/pull/1308
Derive clone for TrainerWrapper by @jonatanklosko in https://github.com/huggingface/tokenizers/pull/1317
CD backports by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1318
0.13.4.rc1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1319
Release all at once for simplicity. by @Narsil in https://github.com/huggingface/tokenizers/pull/1320
Fix stride condition. by @Narsil in https://github.com/huggingface/tokenizers/pull/1321
pyo3: update to 0.19 by @mikelui in https://github.com/huggingface/tokenizers/pull/1322
Add expect() for disabling truncation by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1316
Re-using scritpts from safetensors. by @Narsil in https://github.com/huggingface/tokenizers/pull/1328
Reduce number of different revisions by 1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1329
Python 38 arm by @Narsil in https://github.com/huggingface/tokenizers/pull/1330
Move to maturing mimicking move for safetensors. + Rewritten node bindings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1331
Updating the docs with the new command. by @Narsil in https://github.com/huggingface/tokenizers/pull/1333
Update added tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1335

New Contributors

@csko made their first contribution in https://github.com/huggingface/tokenizers/pull/1157
@chris-ha458 made their first contribution in https://github.com/huggingface/tokenizers/pull/1244
@sbhavani made their first contribution in https://github.com/huggingface/tokenizers/pull/1272
@boyleconnor made their first contribution in https://github.com/huggingface/tokenizers/pull/1275
@hiroshi-matsuda-rit made their first contribution in https://github.com/huggingface/tokenizers/pull/1289
@bact made their first contribution in https://github.com/huggingface/tokenizers/pull/1292
@kellymarchisio made their first contribution in https://github.com/huggingface/tokenizers/pull/1295
@SamuelLarkin made their first contribution in https://github.com/huggingface/tokenizers/pull/1302
@jonatanklosko made their first contribution in https://github.com/huggingface/tokenizers/pull/1317
@mikelui made their first contribution in https://github.com/huggingface/tokenizers/pull/1322

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.3...v0.14.0

Reworks the release pipeline. Other breaking changes are mostly related to https://github.com/huggingface/tokenizers/pull/1335, where AddedToken is reworked

What's Changed

pyo3: update to 0.19 by @mikelui in https://github.com/huggingface/tokenizers/pull/1322
Add expect() for disabling truncation by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1316
Re-using scritpts from safetensors. by @Narsil in https://github.com/huggingface/tokenizers/pull/1328
Reduce number of different revisions by 1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1329
Python 38 arm by @Narsil in https://github.com/huggingface/tokenizers/pull/1330
Move to maturing mimicking move for safetensors. + Rewritten node bindings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1331
Updating the docs with the new command. by @Narsil in https://github.com/huggingface/tokenizers/pull/1333
Update added tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1335

New Contributors

@mikelui made their first contribution in https://github.com/huggingface/tokenizers/pull/1322

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.4.rc2...v0.14.0.rc1

Aug 23, 2023

Mostly checking the new release scripts actually work.

What's Changed

pyo3: update to 0.19 by @mikelui in https://github.com/huggingface/tokenizers/pull/1322
Add expect() for disabling truncation by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1316
Re-using scritpts from safetensors. by @Narsil in https://github.com/huggingface/tokenizers/pull/1328

New Contributors

@mikelui made their first contribution in https://github.com/huggingface/tokenizers/pull/1322

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.4.rc2...v0.13.4.rc3

Aug 14, 2023

What's Changed

Fix stride condition. by @Narsil in https://github.com/huggingface/tokenizers/pull/1321

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.4.rc1...v0.13.4.rc2

Python v0.13.4.rc1

What's Changed

Update all GH Actions with dependency on actions/checkout by @mfuntowicz in https://github.com/huggingface/tokenizers/pull/1256
Parallelize unigram trainer by @mishig25 in https://github.com/huggingface/tokenizers/pull/976
Update unigram/trainer.rs by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1257
Fixing broken link. by @Narsil in https://github.com/huggingface/tokenizers/pull/1268
fix documentation regarding regex by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1264
Update Cargo.toml by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1266
Update README.md - Broken link by @sbhavani in https://github.com/huggingface/tokenizers/pull/1272
[doc build] Use secrets by @mishig25 in https://github.com/huggingface/tokenizers/pull/1273
Improve error for truncation with too high stride by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1275
Add unigram bytefallback by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1217
revise type specification by @hiroshi-matsuda-rit in https://github.com/huggingface/tokenizers/pull/1289
Bump tough-cookie from 4.0.0 to 4.1.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1291
Update path name: master -> main by @bact in https://github.com/huggingface/tokenizers/pull/1292
import Tuple from typing by @kellymarchisio in https://github.com/huggingface/tokenizers/pull/1295
Fixing clippy warnings on 1.71. by @Narsil in https://github.com/huggingface/tokenizers/pull/1296
Bump word-wrap from 1.2.3 to 1.2.4 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1299
feat: Added CITATION.cff. by @SamuelLarkin in https://github.com/huggingface/tokenizers/pull/1302
Single warning for holes. by @Narsil in https://github.com/huggingface/tokenizers/pull/1303
Give error when initializing tokenizer with too high stride by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1306
Handle when precompiled charsmap is empty by @kellymarchisio in https://github.com/huggingface/tokenizers/pull/1308
Derive clone for TrainerWrapper by @jonatanklosko in https://github.com/huggingface/tokenizers/pull/1317
CD backports by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1318
0.13.4.rc1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1319

New Contributors

@sbhavani made their first contribution in https://github.com/huggingface/tokenizers/pull/1272
@boyleconnor made their first contribution in https://github.com/huggingface/tokenizers/pull/1275
@hiroshi-matsuda-rit made their first contribution in https://github.com/huggingface/tokenizers/pull/1289
@bact made their first contribution in https://github.com/huggingface/tokenizers/pull/1292
@kellymarchisio made their first contribution in https://github.com/huggingface/tokenizers/pull/1295
@SamuelLarkin made their first contribution in https://github.com/huggingface/tokenizers/pull/1302
@jonatanklosko made their first contribution in https://github.com/huggingface/tokenizers/pull/1317

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.4-rc2...v0.13.4.rc1

May 17, 2023

May 15, 2023

Apr 5, 2023

Node v0.13.3

What's Changed

Update pr docs actions by @mishig25 in https://github.com/huggingface/tokenizers/pull/1101
Adding rust audit. by @Narsil in https://github.com/huggingface/tokenizers/pull/1099
Revert "Update pr docs actions" by @mishig25 in https://github.com/huggingface/tokenizers/pull/1107
Bump loader-utils from 1.4.0 to 1.4.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1108
Include license file in Rust crate by @ankane in https://github.com/huggingface/tokenizers/pull/1115
Bump decode-uri-component from 0.2.0 to 0.2.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1116
[FIX] In SentencePieceBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1120
Fixing conda ssl location by @Narsil in https://github.com/huggingface/tokenizers/pull/1124
Adding stale bot ? by @Narsil in https://github.com/huggingface/tokenizers/pull/1123
Bump minimatch from 3.0.4 to 3.1.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1126
Bump decode-uri-component from 0.2.0 to 0.2.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1125
Bump cached-path from 0.5 to 0.6 by @hvaara in https://github.com/huggingface/tokenizers/pull/1127
Wrap rustdoc html entity in code block by @hvaara in https://github.com/huggingface/tokenizers/pull/1130
Fix broken links in docs by @hvaara in https://github.com/huggingface/tokenizers/pull/1133
Bump derive_builder from 0.9 to 0.12 by @hvaara in https://github.com/huggingface/tokenizers/pull/1129
Ignore Cargo.lock for subfolders by @hvaara in https://github.com/huggingface/tokenizers/pull/1131
Fix one char super tiny typo by @fzyzcjy in https://github.com/huggingface/tokenizers/pull/1137
[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1136
Bump json5, copy-webpack-plugin, webpack and webpack-cli in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1139
Bump json5 from 2.2.0 to 2.2.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1140
Add missing build targets by @Narsil in https://github.com/huggingface/tokenizers/pull/1145
Adding python 3.8 for M1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1147
Made dirs optional by @ankane in https://github.com/huggingface/tokenizers/pull/1148
Update info on environment variable for threading by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1150
Making Tokenizer clone. by @Narsil in https://github.com/huggingface/tokenizers/pull/1152
Prevent using from_pretrained on invalid ids (better error message). by @Narsil in https://github.com/huggingface/tokenizers/pull/1153
Improved version. by @Narsil in https://github.com/huggingface/tokenizers/pull/1154
Update model.rs by @thomasw21 in https://github.com/huggingface/tokenizers/pull/1166
Using clippy 1.67 by @Narsil in https://github.com/huggingface/tokenizers/pull/1167
pyo3 v0.18 migration by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1173
Bump webpack from 5.75.0 to 5.76.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1181
Fixing infinite loop in UnigramTrainer. by @Narsil in https://github.com/huggingface/tokenizers/pull/1182
Bump dirs from 3.0 to 4.0 by @hvaara in https://github.com/huggingface/tokenizers/pull/1142
Adding ByteFallback support for tokenizers. by @Narsil in https://github.com/huggingface/tokenizers/pull/1183
Faster datasets train example by @lhoestq in https://github.com/huggingface/tokenizers/pull/1192
Adding Replace to decoder (to undo the Replace Normalizer for Metaspace split). by @Narsil in https://github.com/huggingface/tokenizers/pull/1195
Creating normalizers.Prepend (To be used instead of Metaspace). by @Narsil in https://github.com/huggingface/tokenizers/pull/1194
Adding 2 new decoders: by @Narsil in https://github.com/huggingface/tokenizers/pull/1196
Fixing decoder strip because of char boundaries. by @Narsil in https://github.com/huggingface/tokenizers/pull/1197
Add content to Strip decoder to allow decoding mid tokens. by @Narsil in https://github.com/huggingface/tokenizers/pull/1199

New Contributors

@ankane made their first contribution in https://github.com/huggingface/tokenizers/pull/1115
@SeongBeomLEE made their first contribution in https://github.com/huggingface/tokenizers/pull/1120
@hvaara made their first contribution in https://github.com/huggingface/tokenizers/pull/1127
@fzyzcjy made their first contribution in https://github.com/huggingface/tokenizers/pull/1137
@mert-kurttutan made their first contribution in https://github.com/huggingface/tokenizers/pull/1150
@lhoestq made their first contribution in https://github.com/huggingface/tokenizers/pull/1192

Full Changelog: https://github.com/huggingface/tokenizers/compare/node-v0.13.2...python-v0.13.3rc1

What's Changed

Update pr docs actions by @mishig25 in https://github.com/huggingface/tokenizers/pull/1101
Adding rust audit. by @Narsil in https://github.com/huggingface/tokenizers/pull/1099
Revert "Update pr docs actions" by @mishig25 in https://github.com/huggingface/tokenizers/pull/1107
Bump loader-utils from 1.4.0 to 1.4.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1108
Include license file in Rust crate by @ankane in https://github.com/huggingface/tokenizers/pull/1115
Bump decode-uri-component from 0.2.0 to 0.2.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1116
[FIX] In SentencePieceBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1120
Fixing conda ssl location by @Narsil in https://github.com/huggingface/tokenizers/pull/1124
Adding stale bot ? by @Narsil in https://github.com/huggingface/tokenizers/pull/1123
Bump minimatch from 3.0.4 to 3.1.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1126
Bump decode-uri-component from 0.2.0 to 0.2.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1125
Bump cached-path from 0.5 to 0.6 by @hvaara in https://github.com/huggingface/tokenizers/pull/1127
Wrap rustdoc html entity in code block by @hvaara in https://github.com/huggingface/tokenizers/pull/1130
Fix broken links in docs by @hvaara in https://github.com/huggingface/tokenizers/pull/1133
Bump derive_builder from 0.9 to 0.12 by @hvaara in https://github.com/huggingface/tokenizers/pull/1129
Ignore Cargo.lock for subfolders by @hvaara in https://github.com/huggingface/tokenizers/pull/1131
Fix one char super tiny typo by @fzyzcjy in https://github.com/huggingface/tokenizers/pull/1137
[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1136
Bump json5, copy-webpack-plugin, webpack and webpack-cli in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1139
Bump json5 from 2.2.0 to 2.2.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1140
Add missing build targets by @Narsil in https://github.com/huggingface/tokenizers/pull/1145
Adding python 3.8 for M1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1147
Made dirs optional by @ankane in https://github.com/huggingface/tokenizers/pull/1148
Update info on environment variable for threading by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1150
Making Tokenizer clone. by @Narsil in https://github.com/huggingface/tokenizers/pull/1152
Prevent using from_pretrained on invalid ids (better error message). by @Narsil in https://github.com/huggingface/tokenizers/pull/1153
Improved version. by @Narsil in https://github.com/huggingface/tokenizers/pull/1154
Update model.rs by @thomasw21 in https://github.com/huggingface/tokenizers/pull/1166
Using clippy 1.67 by @Narsil in https://github.com/huggingface/tokenizers/pull/1167
pyo3 v0.18 migration by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1173
Bump webpack from 5.75.0 to 5.76.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1181
Fixing infinite loop in UnigramTrainer. by @Narsil in https://github.com/huggingface/tokenizers/pull/1182
Bump dirs from 3.0 to 4.0 by @hvaara in https://github.com/huggingface/tokenizers/pull/1142
Adding ByteFallback support for tokenizers. by @Narsil in https://github.com/huggingface/tokenizers/pull/1183
Faster datasets train example by @lhoestq in https://github.com/huggingface/tokenizers/pull/1192
Adding Replace to decoder (to undo the Replace Normalizer for Metaspace split). by @Narsil in https://github.com/huggingface/tokenizers/pull/1195
Creating normalizers.Prepend (To be used instead of Metaspace). by @Narsil in https://github.com/huggingface/tokenizers/pull/1194
Adding 2 new decoders: by @Narsil in https://github.com/huggingface/tokenizers/pull/1196
Fixing decoder strip because of char boundaries. by @Narsil in https://github.com/huggingface/tokenizers/pull/1197
Add content to Strip decoder to allow decoding mid tokens. by @Narsil in https://github.com/huggingface/tokenizers/pull/1199
New version 0.13.3 by @Narsil in https://github.com/huggingface/tokenizers/pull/1205

New Contributors

@ankane made their first contribution in https://github.com/huggingface/tokenizers/pull/1115
@SeongBeomLEE made their first contribution in https://github.com/huggingface/tokenizers/pull/1120
@hvaara made their first contribution in https://github.com/huggingface/tokenizers/pull/1127
@fzyzcjy made their first contribution in https://github.com/huggingface/tokenizers/pull/1137
@mert-kurttutan made their first contribution in https://github.com/huggingface/tokenizers/pull/1150
@lhoestq made their first contribution in https://github.com/huggingface/tokenizers/pull/1192

Full Changelog: https://github.com/huggingface/tokenizers/compare/node-v0.13.2...python-v0.13.3rc1

Rust v0.13.3

What's Changed

Update pr docs actions by @mishig25 in https://github.com/huggingface/tokenizers/pull/1101
Adding rust audit. by @Narsil in https://github.com/huggingface/tokenizers/pull/1099
Revert "Update pr docs actions" by @mishig25 in https://github.com/huggingface/tokenizers/pull/1107
Bump loader-utils from 1.4.0 to 1.4.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1108
Include license file in Rust crate by @ankane in https://github.com/huggingface/tokenizers/pull/1115
Bump decode-uri-component from 0.2.0 to 0.2.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1116
[FIX] In SentencePieceBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1120
Fixing conda ssl location by @Narsil in https://github.com/huggingface/tokenizers/pull/1124
Adding stale bot ? by @Narsil in https://github.com/huggingface/tokenizers/pull/1123
Bump minimatch from 3.0.4 to 3.1.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1126
Bump decode-uri-component from 0.2.0 to 0.2.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1125
Bump cached-path from 0.5 to 0.6 by @hvaara in https://github.com/huggingface/tokenizers/pull/1127
Wrap rustdoc html entity in code block by @hvaara in https://github.com/huggingface/tokenizers/pull/1130
Fix broken links in docs by @hvaara in https://github.com/huggingface/tokenizers/pull/1133
Bump derive_builder from 0.9 to 0.12 by @hvaara in https://github.com/huggingface/tokenizers/pull/1129
Ignore Cargo.lock for subfolders by @hvaara in https://github.com/huggingface/tokenizers/pull/1131
Fix one char super tiny typo by @fzyzcjy in https://github.com/huggingface/tokenizers/pull/1137
[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1136
Bump json5, copy-webpack-plugin, webpack and webpack-cli in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1139
Bump json5 from 2.2.0 to 2.2.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1140
Add missing build targets by @Narsil in https://github.com/huggingface/tokenizers/pull/1145
Adding python 3.8 for M1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1147
Made dirs optional by @ankane in https://github.com/huggingface/tokenizers/pull/1148
Update info on environment variable for threading by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1150
Making Tokenizer clone. by @Narsil in https://github.com/huggingface/tokenizers/pull/1152
Prevent using from_pretrained on invalid ids (better error message). by @Narsil in https://github.com/huggingface/tokenizers/pull/1153
Improved version. by @Narsil in https://github.com/huggingface/tokenizers/pull/1154
Update model.rs by @thomasw21 in https://github.com/huggingface/tokenizers/pull/1166
Using clippy 1.67 by @Narsil in https://github.com/huggingface/tokenizers/pull/1167
pyo3 v0.18 migration by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1173
Bump webpack from 5.75.0 to 5.76.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1181
Fixing infinite loop in UnigramTrainer. by @Narsil in https://github.com/huggingface/tokenizers/pull/1182
Bump dirs from 3.0 to 4.0 by @hvaara in https://github.com/huggingface/tokenizers/pull/1142
Adding ByteFallback support for tokenizers. by @Narsil in https://github.com/huggingface/tokenizers/pull/1183
Faster datasets train example by @lhoestq in https://github.com/huggingface/tokenizers/pull/1192
Adding Replace to decoder (to undo the Replace Normalizer for Metaspace split). by @Narsil in https://github.com/huggingface/tokenizers/pull/1195
Creating normalizers.Prepend (To be used instead of Metaspace). by @Narsil in https://github.com/huggingface/tokenizers/pull/1194
Adding 2 new decoders: by @Narsil in https://github.com/huggingface/tokenizers/pull/1196
Fixing decoder strip because of char boundaries. by @Narsil in https://github.com/huggingface/tokenizers/pull/1197
Add content to Strip decoder to allow decoding mid tokens. by @Narsil in https://github.com/huggingface/tokenizers/pull/1199
New version 0.13.3 by @Narsil in https://github.com/huggingface/tokenizers/pull/1205
New release by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1207

New Contributors

@ankane made their first contribution in https://github.com/huggingface/tokenizers/pull/1115
@SeongBeomLEE made their first contribution in https://github.com/huggingface/tokenizers/pull/1120
@hvaara made their first contribution in https://github.com/huggingface/tokenizers/pull/1127
@fzyzcjy made their first contribution in https://github.com/huggingface/tokenizers/pull/1137
@mert-kurttutan made their first contribution in https://github.com/huggingface/tokenizers/pull/1150
@lhoestq made their first contribution in https://github.com/huggingface/tokenizers/pull/1192

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.2...v0.13.3

Python v0.13.3

What's Changed

Update pr docs actions by @mishig25 in https://github.com/huggingface/tokenizers/pull/1101
Adding rust audit. by @Narsil in https://github.com/huggingface/tokenizers/pull/1099
Revert "Update pr docs actions" by @mishig25 in https://github.com/huggingface/tokenizers/pull/1107
Bump loader-utils from 1.4.0 to 1.4.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1108
Include license file in Rust crate by @ankane in https://github.com/huggingface/tokenizers/pull/1115
Bump decode-uri-component from 0.2.0 to 0.2.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1116
[FIX] In SentencePieceBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1120
Fixing conda ssl location by @Narsil in https://github.com/huggingface/tokenizers/pull/1124
Adding stale bot ? by @Narsil in https://github.com/huggingface/tokenizers/pull/1123
Bump minimatch from 3.0.4 to 3.1.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1126
Bump decode-uri-component from 0.2.0 to 0.2.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1125
Bump cached-path from 0.5 to 0.6 by @hvaara in https://github.com/huggingface/tokenizers/pull/1127
Wrap rustdoc html entity in code block by @hvaara in https://github.com/huggingface/tokenizers/pull/1130
Fix broken links in docs by @hvaara in https://github.com/huggingface/tokenizers/pull/1133
Bump derive_builder from 0.9 to 0.12 by @hvaara in https://github.com/huggingface/tokenizers/pull/1129
Ignore Cargo.lock for subfolders by @hvaara in https://github.com/huggingface/tokenizers/pull/1131
Fix one char super tiny typo by @fzyzcjy in https://github.com/huggingface/tokenizers/pull/1137
[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1136
Bump json5, copy-webpack-plugin, webpack and webpack-cli in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1139
Bump json5 from 2.2.0 to 2.2.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1140
Add missing build targets by @Narsil in https://github.com/huggingface/tokenizers/pull/1145
Adding python 3.8 for M1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1147
Made dirs optional by @ankane in https://github.com/huggingface/tokenizers/pull/1148
Update info on environment variable for threading by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1150
Making Tokenizer clone. by @Narsil in https://github.com/huggingface/tokenizers/pull/1152
Prevent using from_pretrained on invalid ids (better error message). by @Narsil in https://github.com/huggingface/tokenizers/pull/1153
Improved version. by @Narsil in https://github.com/huggingface/tokenizers/pull/1154
Update model.rs by @thomasw21 in https://github.com/huggingface/tokenizers/pull/1166
Using clippy 1.67 by @Narsil in https://github.com/huggingface/tokenizers/pull/1167
pyo3 v0.18 migration by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1173
Bump webpack from 5.75.0 to 5.76.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1181
Fixing infinite loop in UnigramTrainer. by @Narsil in https://github.com/huggingface/tokenizers/pull/1182
Bump dirs from 3.0 to 4.0 by @hvaara in https://github.com/huggingface/tokenizers/pull/1142
Adding ByteFallback support for tokenizers. by @Narsil in https://github.com/huggingface/tokenizers/pull/1183
Faster datasets train example by @lhoestq in https://github.com/huggingface/tokenizers/pull/1192
Adding Replace to decoder (to undo the Replace Normalizer for Metaspace split). by @Narsil in https://github.com/huggingface/tokenizers/pull/1195
Creating normalizers.Prepend (To be used instead of Metaspace). by @Narsil in https://github.com/huggingface/tokenizers/pull/1194
Adding 2 new decoders: by @Narsil in https://github.com/huggingface/tokenizers/pull/1196
Fixing decoder strip because of char boundaries. by @Narsil in https://github.com/huggingface/tokenizers/pull/1197
Add content to Strip decoder to allow decoding mid tokens. by @Narsil in https://github.com/huggingface/tokenizers/pull/1199

New Contributors

@ankane made their first contribution in https://github.com/huggingface/tokenizers/pull/1115
@SeongBeomLEE made their first contribution in https://github.com/huggingface/tokenizers/pull/1120
@hvaara made their first contribution in https://github.com/huggingface/tokenizers/pull/1127
@fzyzcjy made their first contribution in https://github.com/huggingface/tokenizers/pull/1137
@mert-kurttutan made their first contribution in https://github.com/huggingface/tokenizers/pull/1150
@lhoestq made their first contribution in https://github.com/huggingface/tokenizers/pull/1192

Full Changelog: https://github.com/huggingface/tokenizers/compare/node-v0.13.2...python-v0.13.3rc1

What's Changed

Update pr docs actions by @mishig25 in https://github.com/huggingface/tokenizers/pull/1101
Adding rust audit. by @Narsil in https://github.com/huggingface/tokenizers/pull/1099
Revert "Update pr docs actions" by @mishig25 in https://github.com/huggingface/tokenizers/pull/1107
Bump loader-utils from 1.4.0 to 1.4.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1108
Include license file in Rust crate by @ankane in https://github.com/huggingface/tokenizers/pull/1115
Bump decode-uri-component from 0.2.0 to 0.2.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1116
[FIX] In SentencePieceBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1120
Fixing conda ssl location by @Narsil in https://github.com/huggingface/tokenizers/pull/1124
Adding stale bot ? by @Narsil in https://github.com/huggingface/tokenizers/pull/1123
Bump minimatch from 3.0.4 to 3.1.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1126
Bump decode-uri-component from 0.2.0 to 0.2.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1125
Bump cached-path from 0.5 to 0.6 by @hvaara in https://github.com/huggingface/tokenizers/pull/1127
Wrap rustdoc html entity in code block by @hvaara in https://github.com/huggingface/tokenizers/pull/1130
Fix broken links in docs by @hvaara in https://github.com/huggingface/tokenizers/pull/1133
Bump derive_builder from 0.9 to 0.12 by @hvaara in https://github.com/huggingface/tokenizers/pull/1129
Ignore Cargo.lock for subfolders by @hvaara in https://github.com/huggingface/tokenizers/pull/1131
Fix one char super tiny typo by @fzyzcjy in https://github.com/huggingface/tokenizers/pull/1137
[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1136
Bump json5, copy-webpack-plugin, webpack and webpack-cli in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1139
Bump json5 from 2.2.0 to 2.2.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1140
Add missing build targets by @Narsil in https://github.com/huggingface/tokenizers/pull/1145
Adding python 3.8 for M1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1147
Made dirs optional by @ankane in https://github.com/huggingface/tokenizers/pull/1148
Update info on environment variable for threading by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1150
Making Tokenizer clone. by @Narsil in https://github.com/huggingface/tokenizers/pull/1152
Prevent using from_pretrained on invalid ids (better error message). by @Narsil in https://github.com/huggingface/tokenizers/pull/1153
Improved version. by @Narsil in https://github.com/huggingface/tokenizers/pull/1154
Update model.rs by @thomasw21 in https://github.com/huggingface/tokenizers/pull/1166
Using clippy 1.67 by @Narsil in https://github.com/huggingface/tokenizers/pull/1167
pyo3 v0.18 migration by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1173
Bump webpack from 5.75.0 to 5.76.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1181
Fixing infinite loop in UnigramTrainer. by @Narsil in https://github.com/huggingface/tokenizers/pull/1182
Bump dirs from 3.0 to 4.0 by @hvaara in https://github.com/huggingface/tokenizers/pull/1142
Adding ByteFallback support for tokenizers. by @Narsil in https://github.com/huggingface/tokenizers/pull/1183
Faster datasets train example by @lhoestq in https://github.com/huggingface/tokenizers/pull/1192
Adding Replace to decoder (to undo the Replace Normalizer for Metaspace split). by @Narsil in https://github.com/huggingface/tokenizers/pull/1195
Creating normalizers.Prepend (To be used instead of Metaspace). by @Narsil in https://github.com/huggingface/tokenizers/pull/1194
Adding 2 new decoders: by @Narsil in https://github.com/huggingface/tokenizers/pull/1196
Fixing decoder strip because of char boundaries. by @Narsil in https://github.com/huggingface/tokenizers/pull/1197
Add content to Strip decoder to allow decoding mid tokens. by @Narsil in https://github.com/huggingface/tokenizers/pull/1199
New version 0.13.3 by @Narsil in https://github.com/huggingface/tokenizers/pull/1205

New Contributors

@ankane made their first contribution in https://github.com/huggingface/tokenizers/pull/1115
@SeongBeomLEE made their first contribution in https://github.com/huggingface/tokenizers/pull/1120
@hvaara made their first contribution in https://github.com/huggingface/tokenizers/pull/1127
@fzyzcjy made their first contribution in https://github.com/huggingface/tokenizers/pull/1137
@mert-kurttutan made their first contribution in https://github.com/huggingface/tokenizers/pull/1150
@lhoestq made their first contribution in https://github.com/huggingface/tokenizers/pull/1192

Full Changelog: https://github.com/huggingface/tokenizers/compare/node-v0.13.2...python-v0.13.3rc1

Apr 4, 2023

Python v0.13.3rc1

What's Changed

Update pr docs actions by @mishig25 in https://github.com/huggingface/tokenizers/pull/1101
Adding rust audit. by @Narsil in https://github.com/huggingface/tokenizers/pull/1099
Revert "Update pr docs actions" by @mishig25 in https://github.com/huggingface/tokenizers/pull/1107
Bump loader-utils from 1.4.0 to 1.4.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1108
Include license file in Rust crate by @ankane in https://github.com/huggingface/tokenizers/pull/1115
Bump decode-uri-component from 0.2.0 to 0.2.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1116
[FIX] In SentencePieceBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1120
Fixing conda ssl location by @Narsil in https://github.com/huggingface/tokenizers/pull/1124
Adding stale bot ? by @Narsil in https://github.com/huggingface/tokenizers/pull/1123
Bump minimatch from 3.0.4 to 3.1.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1126
Bump decode-uri-component from 0.2.0 to 0.2.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1125
Bump cached-path from 0.5 to 0.6 by @hvaara in https://github.com/huggingface/tokenizers/pull/1127
Wrap rustdoc html entity in code block by @hvaara in https://github.com/huggingface/tokenizers/pull/1130
Fix broken links in docs by @hvaara in https://github.com/huggingface/tokenizers/pull/1133
Bump derive_builder from 0.9 to 0.12 by @hvaara in https://github.com/huggingface/tokenizers/pull/1129
Ignore Cargo.lock for subfolders by @hvaara in https://github.com/huggingface/tokenizers/pull/1131
Fix one char super tiny typo by @fzyzcjy in https://github.com/huggingface/tokenizers/pull/1137
[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1136
Bump json5, copy-webpack-plugin, webpack and webpack-cli in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1139
Bump json5 from 2.2.0 to 2.2.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1140
Add missing build targets by @Narsil in https://github.com/huggingface/tokenizers/pull/1145
Adding python 3.8 for M1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1147
Made dirs optional by @ankane in https://github.com/huggingface/tokenizers/pull/1148
Update info on environment variable for threading by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1150
Making Tokenizer clone. by @Narsil in https://github.com/huggingface/tokenizers/pull/1152
Prevent using from_pretrained on invalid ids (better error message). by @Narsil in https://github.com/huggingface/tokenizers/pull/1153
Improved version. by @Narsil in https://github.com/huggingface/tokenizers/pull/1154
Update model.rs by @thomasw21 in https://github.com/huggingface/tokenizers/pull/1166
Using clippy 1.67 by @Narsil in https://github.com/huggingface/tokenizers/pull/1167
pyo3 v0.18 migration by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1173
Bump webpack from 5.75.0 to 5.76.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1181
Fixing infinite loop in UnigramTrainer. by @Narsil in https://github.com/huggingface/tokenizers/pull/1182
Bump dirs from 3.0 to 4.0 by @hvaara in https://github.com/huggingface/tokenizers/pull/1142
Adding ByteFallback support for tokenizers. by @Narsil in https://github.com/huggingface/tokenizers/pull/1183
Faster datasets train example by @lhoestq in https://github.com/huggingface/tokenizers/pull/1192
Adding Replace to decoder (to undo the Replace Normalizer for Metaspace split). by @Narsil in https://github.com/huggingface/tokenizers/pull/1195
Creating normalizers.Prepend (To be used instead of Metaspace). by @Narsil in https://github.com/huggingface/tokenizers/pull/1194
Adding 2 new decoders: by @Narsil in https://github.com/huggingface/tokenizers/pull/1196
Fixing decoder strip because of char boundaries. by @Narsil in https://github.com/huggingface/tokenizers/pull/1197
Add content to Strip decoder to allow decoding mid tokens. by @Narsil in https://github.com/huggingface/tokenizers/pull/1199

New Contributors

@ankane made their first contribution in https://github.com/huggingface/tokenizers/pull/1115
@SeongBeomLEE made their first contribution in https://github.com/huggingface/tokenizers/pull/1120
@hvaara made their first contribution in https://github.com/huggingface/tokenizers/pull/1127
@fzyzcjy made their first contribution in https://github.com/huggingface/tokenizers/pull/1137
@mert-kurttutan made their first contribution in https://github.com/huggingface/tokenizers/pull/1150
@lhoestq made their first contribution in https://github.com/huggingface/tokenizers/pull/1192

Full Changelog: https://github.com/huggingface/tokenizers/compare/node-v0.13.2...python-v0.13.3rc1

What's Changed

Update pr docs actions by @mishig25 in https://github.com/huggingface/tokenizers/pull/1101
Adding rust audit. by @Narsil in https://github.com/huggingface/tokenizers/pull/1099
Revert "Update pr docs actions" by @mishig25 in https://github.com/huggingface/tokenizers/pull/1107
Bump loader-utils from 1.4.0 to 1.4.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1108
Include license file in Rust crate by @ankane in https://github.com/huggingface/tokenizers/pull/1115
Bump decode-uri-component from 0.2.0 to 0.2.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1116
[FIX] In SentencePieceBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1120
Fixing conda ssl location by @Narsil in https://github.com/huggingface/tokenizers/pull/1124
Adding stale bot ? by @Narsil in https://github.com/huggingface/tokenizers/pull/1123
Bump minimatch from 3.0.4 to 3.1.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1126
Bump decode-uri-component from 0.2.0 to 0.2.2 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1125
Bump cached-path from 0.5 to 0.6 by @hvaara in https://github.com/huggingface/tokenizers/pull/1127
Wrap rustdoc html entity in code block by @hvaara in https://github.com/huggingface/tokenizers/pull/1130
Fix broken links in docs by @hvaara in https://github.com/huggingface/tokenizers/pull/1133
Bump derive_builder from 0.9 to 0.12 by @hvaara in https://github.com/huggingface/tokenizers/pull/1129
Ignore Cargo.lock for subfolders by @hvaara in https://github.com/huggingface/tokenizers/pull/1131
Fix one char super tiny typo by @fzyzcjy in https://github.com/huggingface/tokenizers/pull/1137
[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. by @SeongBeomLEE in https://github.com/huggingface/tokenizers/pull/1136
Bump json5, copy-webpack-plugin, webpack and webpack-cli in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1139
Bump json5 from 2.2.0 to 2.2.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1140
Add missing build targets by @Narsil in https://github.com/huggingface/tokenizers/pull/1145
Adding python 3.8 for M1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1147
Made dirs optional by @ankane in https://github.com/huggingface/tokenizers/pull/1148
Update info on environment variable for threading by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1150
Making Tokenizer clone. by @Narsil in https://github.com/huggingface/tokenizers/pull/1152
Prevent using from_pretrained on invalid ids (better error message). by @Narsil in https://github.com/huggingface/tokenizers/pull/1153
Improved version. by @Narsil in https://github.com/huggingface/tokenizers/pull/1154
Update model.rs by @thomasw21 in https://github.com/huggingface/tokenizers/pull/1166
Using clippy 1.67 by @Narsil in https://github.com/huggingface/tokenizers/pull/1167
pyo3 v0.18 migration by @mert-kurttutan in https://github.com/huggingface/tokenizers/pull/1173
Bump webpack from 5.75.0 to 5.76.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in https://github.com/huggingface/tokenizers/pull/1181
Fixing infinite loop in UnigramTrainer. by @Narsil in https://github.com/huggingface/tokenizers/pull/1182
Bump dirs from 3.0 to 4.0 by @hvaara in https://github.com/huggingface/tokenizers/pull/1142
Adding ByteFallback support for tokenizers. by @Narsil in https://github.com/huggingface/tokenizers/pull/1183
Faster datasets train example by @lhoestq in https://github.com/huggingface/tokenizers/pull/1192
Adding Replace to decoder (to undo the Replace Normalizer for Metaspace split). by @Narsil in https://github.com/huggingface/tokenizers/pull/1195
Creating normalizers.Prepend (To be used instead of Metaspace). by @Narsil in https://github.com/huggingface/tokenizers/pull/1194
Adding 2 new decoders: by @Narsil in https://github.com/huggingface/tokenizers/pull/1196
Fixing decoder strip because of char boundaries. by @Narsil in https://github.com/huggingface/tokenizers/pull/1197
Add content to Strip decoder to allow decoding mid tokens. by @Narsil in https://github.com/huggingface/tokenizers/pull/1199
New version 0.13.3 by @Narsil in https://github.com/huggingface/tokenizers/pull/1205

New Contributors

@ankane made their first contribution in https://github.com/huggingface/tokenizers/pull/1115
@SeongBeomLEE made their first contribution in https://github.com/huggingface/tokenizers/pull/1120
@hvaara made their first contribution in https://github.com/huggingface/tokenizers/pull/1127
@fzyzcjy made their first contribution in https://github.com/huggingface/tokenizers/pull/1137
@mert-kurttutan made their first contribution in https://github.com/huggingface/tokenizers/pull/1150
@lhoestq made their first contribution in https://github.com/huggingface/tokenizers/pull/1192

Full Changelog: https://github.com/huggingface/tokenizers/compare/node-v0.13.2...python-v0.13.3rc1

Nov 7, 2022

Node 0.13.2

Python 3.11 support (Python only modification)

Rust 0.13.2

Python 3.11 support (Python only modification)

Python 0.13.2

[0.13.2]

[#1096] Python 3.11 support

Oct 6, 2022

Node 0.13.1

[0.13.1]

[#1072] Fixing Roberta type ids.

Rust 0.13.1

[0.13.1]

[#1072] Fixing Roberta type ids.

Python v0.13.1

[0.13.1]

[#1072] Fixing Roberta type ids.

Sep 21, 2022

Python v0.13.0

[0.13.0]

[#956] PyO3 version upgrade
[#1055] M1 automated builds
[#1008] Decoder is now a composable trait, but without being backward incompatible
[#1047, #1051, #1052] Processor is now a composable trait, but without being backward incompatible

Both trait changes warrant a "major" number since, despite best efforts to not break backward compatibility, the code is different enough that we cannot be exactly sure.

Previous 1 2 3 4 Next

Similar releases

Other sources from this team

Similar sources

Latest

v0.22.2

Source

@huggingface/tokenizers

Tracking Since

Dec 3, 2019

Last checked Apr 19, 2026

.json·.md·.atom

Tokenizers

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

[0.13.2]

[0.13.1]

[0.13.1]

[0.13.1]

[0.13.0]

More from this team

Similar releases

Other sources from this team

Similar sources

More from this team

Similar releases

Other sources from this team

Similar sources