rust-v0.8.0
Rust v0.8.0
$npx -y @buildinternet/releases show rel_c5naMRxvZZgEUtbLeNU31 Changes:
- Big improvements in speed for BPE (Both training and tokenization) (#165)
Fixes:
- Do not open all files directly while training (#163)
- There was a bug in ByteLevel PreTokenizer that caused offsets to be wrong if a char got split up
in multiple bytes. (cf #156)
- The
LongestFirst truncation strategy had a bug (#174)
Fetched April 7, 2026