releases.shpreview

Hugging Face/Tokenizers/v0.19.1

v0.19.1

April 17, 2024TokenizersView original ↗

$npx -y @buildinternet/releases show rel_vrGbQrXZRSfsuzllX66NF

What's Changed

add serialization for ignore_merges by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1504

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.19.0...v0.19.1

Fetched April 7, 2026