releases.shpreview
Hugging Face/Tokenizers/python-v0.9.3

python-v0.9.3

Python v0.9.3

$npx -y @buildinternet/releases show rel_42pmQxE4aqOLoYboMmxUg

Fixed

  • [#470]: Fix hanging error when training with custom component
  • [#476]: TemplateProcessing serialization is now deterministic
  • [#481]: Fix SentencePieceBPETokenizer.from_files

Added

  • [#477]: UnicodeScripts PreTokenizer to avoid merges between various scripts
  • [#480]: Unigram now accepts an initial_alphabet and handles special_tokens correctly

Fetched April 7, 2026