decode and decode_batch work on borrowed content. by @mfuntowicz in https://github.com/huggingface/tokenizers/pull/1251expect() for disabling truncation by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1316safetensors. + Rewritten node bindings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1331Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.3...v0.14.1
Fetched April 7, 2026