TL;DR
tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeli...
tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeli...
Okay mostly doing the release for these PR:
Main change:
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.3...v0.21.4
No change, the 0.21.3 release failed, this is just a re-r...
This release if focused around some performance optimization, enabling broader python no gil support, and fixing some onig issues...
There was a breaking change in 0.20.3 for tuple inputs of encode_batch!
Thanks a MILE to @diliop we now have support for python 3.13! 🥳
The most awaited offset issue with Llama is fixed 🥳
This release is focused on performances and user experience.
First off, we did a bit of benchmarkin...
ignore_merges by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1504**Full Changel...
Bumping 3 versions because of this: https://github.com/huggingface/transformers/blob/60dea593edd0b94ee15dc3917900b26e3acfbbee/setup.py#L177
Big shoutout to @rlrs for the fast replace normalizers PR. This boosts the p...
Clone on `T...What's Changed Okay mostly doing the release for these PR: Update deserialize of added tokens by @ArthurZucker in https://github.com/huggin…
Hugging Face · TransformersWhat's Changed Bump on-headers and compression in /tokenizers/examples/unstablewasm/www by @dependabot[bot] in https://github.com/huggingfa…
Hugging Face · TransformersGemma 4 MTP (Multi-token Processing) for the MLX runner Gemma 4 MTP speculative decoding is now supported on Macs. This can give over a 2x…
OllamaChanges since langchain-perplexity==1.1.0 release(perplexity): 1.2.0 (37091) feat(perplexity): add (37082) chore(docs): update x handle ref…
LangChain · LangChain (Python)