releases.shpreview
Hugging Face/Tokenizers/python-v0.8.1

python-v0.8.1

Python 0.8.1

$npx -y @buildinternet/releases show rel_6ypGbyj26EM1h6WlgVv80

Fixed

  • [#333]: Fix deserialization of AddedToken, where the content was not restored properly

Changed

  • [#329]: Improved warning and behavior when we detect a fork
  • [#330]: BertNormalizer now keeps the same behavior than the original implementation when strip_accents is not specified.

Fetched April 7, 2026