releases.shpreview
Hugging Face/Tokenizers/python-v0.10.1

python-v0.10.1

Python v0.10.1

$npx -y @buildinternet/releases show rel_K0IzKIOnc2sDcacaQznrJ

Fixed

  • [#616]: Fix SentencePiece tokenizers conversion
  • [#617]: Fix offsets produced by Precompiled Normalizer (used by tokenizers converted from SPM)
  • [#618]: Fix Normalizer.normalize with PyNormalizedStringRefMut
  • [#620]: Fix serialization/deserialization for overlapping models
  • [#621]: Fix ByteLevel instantiation from a previously saved state (using __getstate__())

Fetched April 7, 2026