releases.shpreview

Hugging Face/Tokenizers/v0.20.3

v0.20.3

November 5, 2024TokenizersView original ↗

$npx -y @buildinternet/releases show rel_yXIiIa7QUOF38kQdyBXA7

What's Changed

There was a breaking change in 0.20.3 for tuple inputs of encode_batch!

fix pylist by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1673
[MINOR:TYPO] Fix docstrings by @cakiki in https://github.com/huggingface/tokenizers/pull/1653

New Contributors

@cakiki made their first contribution in https://github.com/huggingface/tokenizers/pull/1653

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.20.2...v0.20.3

Fetched April 7, 2026