Optimized inference servers for text and embeddings
npx @buildinternet/releases get inferenceAdds minimal AMD ROCm GPU support with a ROCm flash-attn varlen build, triton layer norm, and an AMD Dockerfile, plus ARM64 and multi-arch CUDA Docker images including sm_121 for DGX Spark GB10. Also adds harrier-oss-v1 model support and fixes unbounded buffer allocation in PaddedBatch.from_pb, DebertaV2 large-batch runs, max_position_embeddings handling in NomicBertConfig, and single predict request metrics.
rust-toolchain.toml before rustup on Dockerfile-{cuda,cuda-all} by @alvarobartt in https://github.com/huggingface/text-embeddings-inference/pull/842pad_token_id as nullable & add support for rope_parameters by @alvarobartt in…max_input_length is bigger than max-batch-tokens by @kozistr in…Since Text Embeddings Inference (TEI) v1.7.0, Intel MKL support had been broken due to changes in the candle dependency. Neither static-linking nor dynamic-linking worked correctly, which caused models using Intel MKL on CPU to fail with…
Today, Google releases EmbeddingGemma, a state-of-the-art multilingual embedding model perfect for…
Qwen3 was not working fine on CPU / MPS when sending batched requests on FP16 precision, due to the FP32 minimum value downcast (now manually set to FP16 minimum value instead) leading to null values, as well as a missing to_dtype call on the…
Qwen3 support included for Intel HPU, and fixed for CPU / Metal / CUDA.
README.md and docs/ examples by @alvarobartt in https://github.com/huggingface/text-embeddings-inference/pull/641Fix for Neuron models exported with batch_size 1.
Neuron backend update.
Qwen3Model by @alvarobartt in…README.md and supported_models.md by @alvarobartt in https://github.com/huggingface/text-embeddings-inference/pull/572Gaudi improvements.
This release updates TGI to Torch 2.7 and CUDA 12.8.
round_up_seq logic to align…