v0.19.1
What's Changed
- add serialization for
ignore_mergesby @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1504
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.19.0...v0.19.1
Fetched April 7, 2026
ignore_merges by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1504Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.19.0...v0.19.1
Fetched April 7, 2026