Text Generation Inference

What's Changed

• misc(gha): expose action cache url and runtime as secrets by @mfuntowicz in https://github.com/huggingface/text-generation-inference/pull/2964
• feat: support max_image_fetch_size to limit by @drbh in…

What's Changed

• Add missing backslash by @philsupertramp in https://github.com/huggingface/text-generation-inference/pull/3311
• Revert "feat: bump flake including transformers and huggingface_hub versions" by @drbh in…

What's Changed

• [gaudi] Refine rope memory, do not need to keep sin/cos cache per layer by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3274
• Gaudi: add CI by @baptistecolle in…

Fix for Neuron models exported with batch_size 1.

What's Changed

• [gaudi] gemma3 text and vlm model intial support. need to add sliding window … by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3270
• Neuron backend fix by @dacorvo in…

Neuron backend update.

What's Changed

• Remove useless packages by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/3253
• Bump neuron SDK version by @dacorvo in https://github.com/huggingface/text-generation-inference/pull/3260
• Perf opt by…

Gaudi improvements.

What's Changed

• upgrade to new vllm extension ops(fix issue in exponential bucketing) by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3239
• Nix: switch to hf-nix by @danieldk in…

This release updates TGI to Torch 2.7 and CUDA 12.8.

What's Changed

• change HPU warmup logic: seq length should be with exponential growth by @kaixuanliu in https://github.com/huggingface/text-generation-inference/pull/3217
• adjust the round_up_seq logic to align…

Notable changes

• Prefill chunking for VLMs.

What's Changed

• Fixing Qwen 2.5 VL (32B). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3157
• Fixing tokenization like https://github.com/huggingface/text-embeddin… by @Narsil in…

Main changes

• Patching Llama 4

What's Changed

• Use ROCM 6.3.1 by @mht-sharma in https://github.com/huggingface/text-generation-inference/pull/3141
• Update transformers to 4.51 by @mht-sharma in…

What's Changed

• Minor fixes. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3125
• configurable termination timeout by @ErikKaum in https://github.com/huggingface/text-generation-inference/pull/3126
• CI: enable server tests for backends by…

What's Changed

• Update to kernels 0.2.1 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3084
• Router: add gemma3-text model type by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3107
• We need gcc during…

Important changes

• BREAKING CHANGE: Lots of modifications around tool calling. Tool calling now respects fully OpenAI return results (arguments return type is a string instead of a real JSON object). Lots of improvements around the tool calling and side effects…

What's Changed

• Back on nix main. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2979
• hotfix: fix trtllm CI build on release by @Hugoch in https://github.com/huggingface/text-generation-inference/pull/2981
• Add strftime_now callable…

Important changes

Deepseek R1 is fully supported on both AMD and Nvidia !

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \
    ghcr.io/huggingface/text-generation-inference:3.1.0 --model-id deepseek-ai/DeepSeek-R1

What's Changed

*…

Tl;dr

New transformers backend supporting flashattention at roughly same performance as pure TGI for all non officially supported models directly in TGI. Congrats @Cyrilvallez

New models unlocked: Cohere2, olmo, olmo2, helium.

What's Changed

• docs(README):…

Summary

Patch release to handle a few older models and corner cases.

What's Changed

• Hotfix link2 by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2812
• Small update to docs by @Narsil in…

TL;DR

Big new release

Details: https://huggingface.co/docs/text-generation-inference/conceptual/chunking

What's…

Notable changes

• Choose input/total tokens automatically based on available VRAM
• Support Qwen2 VL
• Decrease latency of very large batches (> 128)

What's Changed

• feat: add triton kernels to decrease latency of large batches by @OlivierDehaene in…

Notable changes

• Experimental prefill chunking (PREFILL_CHUNKING=1)
• Experimental FP8 KV cache support
• Greatly decrease latency for large batches (> 128 requests)
• Faster MoE kernels and support for GPTQ-quantized MoE
• Faster implementation of MLLama

##…

Important changes

• Added support for Mllama (3.2, vision models). Flashed, unpadded.
• FP8 performance improvements
• Moe performance improvements
• BREAKING CHANGE - When using tools, models could answer with a tool call notify_error with the content error, it will…