releases.shpreview

rust-v0.8.0

Rust v0.8.0

$npx -y @buildinternet/releases show rel_c5naMRxvZZgEUtbLeNU31

Changes:

  • Big improvements in speed for BPE (Both training and tokenization) (#165)

Fixes:

  • Do not open all files directly while training (#163)
  • There was a bug in ByteLevel PreTokenizer that caused offsets to be wrong if a char got split up in multiple bytes. (cf #156)
  • The LongestFirst truncation strategy had a bug (#174)

Fetched April 7, 2026