Important changes

• Renamed HUGGINGFACE_HUB_CACHE to use HF_HOME. This is done to harmonize environment variables across HF ecosystem. So locations of data moved from /data/models-.... to /data/hub/models-.... on the Docker.

• Prefix caching by default ! To help with long running queries TGI will use prefix caching a reuse pre-existing queries in the kv-cache in order to speed up TTFT. This should be totally transparent for most users, however this has required a instense rewrite of internals and therefore bugs can potentially exist. Also we changed kernels from paged_attention to flashinfer (and flashdecoding as a fallback for some specific models that aren't supported by flashinfer).

• Lots of performance improvements with Marlin and quantization.

What's Changed

• chore: update to torch 2.4 by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2259
• fix crash in multi-modal by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2245
• fix of use of unquantized weights in cohere GQA loading, also enable … by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2291
• Split up layers.marlin into several files by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2292
• fix: refactor adapter weight loading and mapping by @drbh in https://github.com/huggingface/text-generation-inference/pull/2193
• Using g6 instead of g5. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2281
• Some small fixes for the Torch 2.4.0 update by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2304
• Fixing idefics on g6 tests. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2306
• Fix registry name by @XciD in https://github.com/huggingface/text-generation-inference/pull/2307
• Support tied embeddings in 0.5B and 1.5B Qwen2 models by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2313
• feat: add ruff and resolve issue by @drbh in https://github.com/huggingface/text-generation-inference/pull/2262
• Run ci api key by @ErikKaum in https://github.com/huggingface/text-generation-inference/pull/2315
• Install Marlin from standalone package by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2320
• fix: reject grammars without properties by @drbh in https://github.com/huggingface/text-generation-inference/pull/2309
• patch-error-on-invalid-grammar by @ErikKaum in https://github.com/huggingface/text-generation-inference/pull/2282
• fix: adjust test snapshots and small refactors by @drbh in https://github.com/huggingface/text-generation-inference/pull/2323
• server quantize: store quantizer config in standard format by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2299
• Rebase TRT-llm by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2331
• Handle GPTQ-Marlin loading in GPTQMarlinWeightLoader by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2300
• Pr 2290 ci run by @drbh in https://github.com/huggingface/text-generation-inference/pull/2329
• refactor usage stats by @ErikKaum in https://github.com/huggingface/text-generation-inference/pull/2339
• enable HuggingFaceM4/idefics-9b in intel gpu by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2338
• Fix cache block size for flash decoding by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2351
• Unify attention output handling by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2343
• fix: attempt forward on flash attn2 to check hardware support by @drbh in https://github.com/huggingface/text-generation-inference/pull/2335
• feat: include local lora adapter loading docs by @drbh in https://github.com/huggingface/text-generation-inference/pull/2359
• fix: return the out tensor rather then the functions return value by @drbh in https://github.com/huggingface/text-generation-inference/pull/2361
• feat: implement a templated endpoint for visibility into chat requests by @drbh in https://github.com/huggingface/text-generation-inference/pull/2333
• feat: prefer stop over eos_token to align with openai finish_reason by @drbh in https://github.com/huggingface/text-generation-inference/pull/2344
• feat: return the generated text when parsing fails by @drbh in https://github.com/huggingface/text-generation-inference/pull/2353
• fix: default num_ln_in_parallel_attn to one if not supplied by @drbh in https://github.com/huggingface/text-generation-inference/pull/2364
• fix: prefer original layernorm names for 180B by @drbh in https://github.com/huggingface/text-generation-inference/pull/2365
• fix: fix num_ln_in_parallel_attn attribute name typo in RWConfig by @almersawi in https://github.com/huggingface/text-generation-inference/pull/2350
• add gptj modeling in TGI #2366 (CI RUN) by @drbh in https://github.com/huggingface/text-generation-inference/pull/2372
• Fix the prefix for OPT model in opt_modelling.py #2370 (CI RUN) by @drbh in https://github.com/huggingface/text-generation-inference/pull/2371
• Pr 2374 ci branch by @drbh in https://github.com/huggingface/text-generation-inference/pull/2378
• fix EleutherAI/gpt-neox-20b does not work in tgi by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2346
• Pr 2337 ci branch by @drbh in https://github.com/huggingface/text-generation-inference/pull/2379
• fix: prefer hidden_activation over hidden_act in gemma2 by @drbh in https://github.com/huggingface/text-generation-inference/pull/2381
• Update Quantization docs and minor doc fix. by @Vaibhavs10 in https://github.com/huggingface/text-generation-inference/pull/2368
• Pr 2352 ci branch by @drbh in https://github.com/huggingface/text-generation-inference/pull/2382
• Add FlashInfer support by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2354
• Add experimental flake by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2384
• Using HF_HOME instead of CACHE to get token read in addition to models. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2288
• flake: add fmt and clippy by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2389
• Update documentation for Supported models by @Vaibhavs10 in https://github.com/huggingface/text-generation-inference/pull/2386
• flake: use rust-overlay by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2390
• Using an enum for flash backens (paged/flashdecoding/flashinfer) by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2385
• feat: add guideline to chat request and template by @drbh in https://github.com/huggingface/text-generation-inference/pull/2391
• Update flake for 9.0a capability in Torch by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2394
• nix: add router to the devshell by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2396
• Upgrade fbgemm by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2398
• Adding launcher to build. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2397
• Fixing import exl2 by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2399
• Cpu dockerimage by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2367
• Add support for prefix caching to the v3 router by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2392
• Keeping the benchmark somewhere by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2401
• feat: validate template variables before apply and improve sliding wi… by @drbh in https://github.com/huggingface/text-generation-inference/pull/2403
• fix: allocate tmp based on sgmv kernel if available by @drbh in https://github.com/huggingface/text-generation-inference/pull/2345
• fix: improve completions to send a final chunk with usage details by @drbh in https://github.com/huggingface/text-generation-inference/pull/2336
• Updating the flake. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2404
• Pr 2395 ci run by @drbh in https://github.com/huggingface/text-generation-inference/pull/2406
• fix: include create_exllama_buffers and set_device for exllama by @drbh in https://github.com/huggingface/text-generation-inference/pull/2407
• nix: incremental build of the launcher by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2410
• Adding more kernels to flake. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2411
• add numa to improve cpu inference perf by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2330
• fix: adds causal to attention params by @drbh in https://github.com/huggingface/text-generation-inference/pull/2408
• nix: partial incremental build of the router by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2416
• Upgrading exl2. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2415
• More fixes trtllm by @mfuntowicz in https://github.com/huggingface/text-generation-inference/pull/2342
• nix: build router incrementally by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2422
• Fixing exl2 and other quanize tests again. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2419
• Upgrading the tests to match the current workings. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2423
• nix: try to reduce the number of Rust rebuilds by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2424
• Improve the Consuming TGI + Streaming docs. by @Vaibhavs10 in https://github.com/huggingface/text-generation-inference/pull/2412
• Further fixes. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2426
• doc: Add metrics documentation and add a 'Reference' section by @Hugoch in https://github.com/huggingface/text-generation-inference/pull/2230
• All integration tests back everywhere (too many failed CI). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2428
• nix: update to CUDA 12.4 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2429
• Prefix caching by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2402
• nix: add pure server to flake, add both pure and impure devshells by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2430
• nix: add text-generation-benchmark to pure devshell by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2431
• Adding eetq to flake. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2438
• nix: add awq-inference-engine as server dependency by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2442
• nix: add default package by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2453
• Fix: don't apply post layernorm in SiglipVisionTransformer by @drbh in https://github.com/huggingface/text-generation-inference/pull/2459
• Pr 2451 ci branch by @drbh in https://github.com/huggingface/text-generation-inference/pull/2454
• Fixing CI. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2462
• fix: bump minijinja version and add test for llama 3.1 tools by @drbh in https://github.com/huggingface/text-generation-inference/pull/2463
• fix: improve regex expression by @drbh in https://github.com/huggingface/text-generation-inference/pull/2468
• nix: build Torch against MKL and various other improvements by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2469
• Lots of improvements (Still 2 allocators) by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2449
• feat: add /v1/models endpoint by @drbh in https://github.com/huggingface/text-generation-inference/pull/2433
• update doc with intel cpu part by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2420
• Tied embeddings in MLP speculator. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2473
• nix: improve impure devshell by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2478
• nix: add punica-kernels by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2477
• fix: enable chat requests in vertex endpoint by @drbh in https://github.com/huggingface/text-generation-inference/pull/2481
• feat: support lora revisions and qkv_proj weights by @drbh in https://github.com/huggingface/text-generation-inference/pull/2482
• hotfix: avoid non-prefilled block use when using prefix caching by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2489
• Adding links to Adyen blogpost. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2492
• Add two handy gitignores for Nix environments by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2484
• hotfix: fix regression of attention api change in intel platform by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2439
• nix: add pyright/ruff for proper LSP in the impure devshell by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2496
• Fix incompatibility with latest syrupy and update in Poetry by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2497
• radix trie: add assertions by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2491
• hotfix: add syrupy to the right subproject by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2499
• Add links to Adyen blogpost by @martinigoyanes in https://github.com/huggingface/text-generation-inference/pull/2500
• Fixing more correctly the invalid drop of the batch. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2498
• Add Directory Check to Prevent Redundant Cloning in Build Process by @vamsivallepu in https://github.com/huggingface/text-generation-inference/pull/2486
• Prefix test - Different kind of load test to trigger prefix test bugs. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2490
• Fix tokenization yi by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2507
• Fix truffle by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2514
• nix: support Python tokenizer conversion in the router by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2515
• Add nix test. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2513
• fix: pass missing revision arg for lora adapter when loading multiple… by @drbh in https://github.com/huggingface/text-generation-inference/pull/2510
• hotfix : enable intel ipex cpu and xpu in python3.11 by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2517
• Use ratatui not (deprecated) tui by @strickvl in https://github.com/huggingface/text-generation-inference/pull/2521
• Add tests for Mixtral by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2520
• Adding a test for FD. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2516
• nix: pure Rust check/fmt/clippy/test by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2525
• fix: metrics unbounded memory by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2528
• Move to moe-kernels package and switch to common MoE layer by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2511
• Stream options. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2533
• Update to moe-kenels 0.3.1 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2535
• doc: clarify that --quantize is not needed for pre-quantized models by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2536
• hotfix: ipex fails since cuda moe kernel is not supported by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2532
• fix: wrap python basic logs in debug assertion in launcher by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2539
• Preparing for release. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2540

New Contributors

• @almersawi made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2350
• @Vaibhavs10 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2368
• @mfuntowicz made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2342
• @vamsivallepu made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2486
• @strickvl made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2521

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v2.2.0...v2.3.0

Notable changes

• Llama 3.1 support (including 405B, FP8 support in a lot of mixed configurations, FP8, AWQ, GPTQ, FP8+FP16).
• Gemma2 softcap support
• Deepseek v2 support.
• Lots of internal reworks/cleanup (allowing for cool features)
• Lots of AWQ/GPTQ work with marlin kernels (everything should be faster by default)
• Flash decoding support (FLASH_DECODING=1 environment variables which will probably enable some nice improvements in the future)

What's Changed

• Preparing patch release. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2186
• Adding "longrope" for Phi-3 (#2172) by @amihalik in https://github.com/huggingface/text-generation-inference/pull/2179
• Refactor dead code - Removing all flash_xxx.py files. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2166
• Fix Starcoder2 after refactor by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2189
• GPTQ CI improvements by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2151
• Consistently take prefix in model constructors by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2191
• fix dbrx & opt model prefix bug by @icyxp in https://github.com/huggingface/text-generation-inference/pull/2201
• hotfix: Fix number of KV heads by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2202
• Fix incorrect cache allocation with multi-query by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2203
• Falcon/DBRX: get correct number of key-value heads by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2205
• add doc for intel gpus by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2181
• fix: python deserialization by @jaluma in https://github.com/huggingface/text-generation-inference/pull/2178
• update to metrics 0.23.0 or could work with metrics-exporter-promethe… by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2190
• feat: use model name as adapter id in chat endpoints by @drbh in https://github.com/huggingface/text-generation-inference/pull/2128
• Fix nccl regression on PyTorch 2.3 upgrade by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/2099
• Fix buildx cache + change runner type by @glegendre01 in https://github.com/huggingface/text-generation-inference/pull/2176
• Fixed README ToC by @vinkamath in https://github.com/huggingface/text-generation-inference/pull/2196
• Updating the self check by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2209
• Move quantized weight handling out of the Weights class by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2194
• Add support for FP8 on compute capability >=8.0, <8.9 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2213
• fix: append DONE message to chat stream by @drbh in https://github.com/huggingface/text-generation-inference/pull/2221
• [fix] Modifying base in yarn embedding by @SeongBeomLEE in https://github.com/huggingface/text-generation-inference/pull/2212
• Use symmetric quantization in the quantize subcommand by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2120
• feat: simple mistral lora integration tests by @drbh in https://github.com/huggingface/text-generation-inference/pull/2180
• fix custom cache dir by @ErikKaum in https://github.com/huggingface/text-generation-inference/pull/2226
• fix: Remove bitsandbytes installation when running cpu-only install by @Hugoch in https://github.com/huggingface/text-generation-inference/pull/2216
• Add support for AWQ-quantized Idefics2 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2233
• server quantize: expose groupsize option by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2225
• Remove stray quantize argument in get_weights_col_packed_qkv by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2237
• fix(server): fix cohere by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2249
• Improve the handling of quantized weights by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2250
• Hotfix: fix of use of unquantized weights in Gemma GQA loading by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2255
• Hotfix: various GPT-based model fixes by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2256
• Hotfix: fix MPT after recent refactor by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2257
• Hotfix: pass through model revision in VlmCausalLM by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2258
• usage stats and crash reports by @ErikKaum in https://github.com/huggingface/text-generation-inference/pull/2220
• add usage stats to toctree by @ErikKaum in https://github.com/huggingface/text-generation-inference/pull/2260
• fix: adjust default tool choice by @drbh in https://github.com/huggingface/text-generation-inference/pull/2244
• Add support for Deepseek V2 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2224
• re-push to internal registry by @XciD in https://github.com/huggingface/text-generation-inference/pull/2242
• Add FP8 release test by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2261
• feat(fp8): use fbgemm kernels and load fp8 weights directly by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2248
• fix(server): fix deepseekv2 loading by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2266
• Hotfix: fix of use of unquantized weights in Mixtral GQA loading by @icyxp in https://github.com/huggingface/text-generation-inference/pull/2269
• legacy warning on text_generation client by @ErikKaum in https://github.com/huggingface/text-generation-inference/pull/2271
• fix(ci): test new instances by @XciD in https://github.com/huggingface/text-generation-inference/pull/2272
• fix(server): fix fp8 weight loading by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2268
• Softcapping for gemma2. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2273
• use proper name for ci by @XciD in https://github.com/huggingface/text-generation-inference/pull/2274
• Fixing mistral nemo. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2276
• fix(l4): fix fp8 logic on l4 by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2277
• Add support for repacking AWQ weights for GPTQ-Marlin by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2278
• [WIP] Add support for Mistral-Nemo by supporting head_dim through config by @shaltielshmid in https://github.com/huggingface/text-generation-inference/pull/2254
• Preparing for release. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2285
• Add support for Llama 3 rotary embeddings by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2286
• hotfix: pin numpy by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2289

New Contributors

• @jaluma made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2178
• @vinkamath made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2196
• @ErikKaum made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2226
• @Hugoch made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2216
• @XciD made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2242
• @shaltielshmid made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2254

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v2.1.1...v2.2.0

Main changes

• Bugfixes
• Added FlashDecoding support (Beta) use FLASH_DECODING=1 to use TGI with flash decoding (large speedups on long queries). https://github.com/huggingface/text-generation-inference/pull/1940
• Use Marlin over GPTQ kernels for faster GPTQ inference https://github.com/huggingface/text-generation-inference/pull/2111

What's Changed

• Fixing the CI to also run in release when it's a tag ? by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2138
• fix microsoft/Phi-3-mini-4k-instruct crash in batch.slots[batch.slot_… by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2148
• Fixing clippy. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2149
• fix: use weights from base_layer by @drbh in https://github.com/huggingface/text-generation-inference/pull/2141
• feat: download lora adapter weights from launcher by @drbh in https://github.com/huggingface/text-generation-inference/pull/2140
• Use GPTQ-Marlin for supported GPTQ configurations by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2111
• fix AttributeError: 'MixtralLayer' object has no attribute 'mlp' by @icyxp in https://github.com/huggingface/text-generation-inference/pull/2123
• refine get xpu free memory/enable Qwen2/gemma2/gemma/phi in intel platform by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2132
• fix: prefer serde structs over custom functions by @drbh in https://github.com/huggingface/text-generation-inference/pull/2127
• Fixing test. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2152
• GH router. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2153
• Fixing baichuan override. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2158
• [Major Change][Undecided yet] Move to FlashDecoding instead of PagedAttention kernel. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1940
• Fixing graph capture for flash decoding. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2163
• fix FlashDecoding change's regression in intel platform by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2161
• fix: use the base layers weight in mistral rocm by @drbh in https://github.com/huggingface/text-generation-inference/pull/2155
• Fixing rocm. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2164
• Ci test by @glegendre01 in https://github.com/huggingface/text-generation-inference/pull/2124
• Hotfixing qwen2 and starcoder2 (which also get clamping). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2167
• feat: improve update_docs for openapi schema by @drbh in https://github.com/huggingface/text-generation-inference/pull/2169
• Fixing the dockerfile warnings. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2173
• Fixing missing object field for regular completions. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2175

New Contributors

• @icyxp made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2123

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v2.1.0...v2.1.1

Notable changes

• New models : gemma2

• Multi lora adapters. You can now run multiple loras on the same TGI deployment https://github.com/huggingface/text-generation-inference/pull/2010

• Faster GPTQ inference and Marlin support (up to 2x speedup).

• Reworked the entire scheduling logic (better block allocations, and allowing further speedups in new releases)

• Lots of Rocm support and bugfixes,

• Lots of new contributors ! Thanks a lot for these contributions

What's Changed

• OpenAI function calling compatible support by @phangiabao98 in https://github.com/huggingface/text-generation-inference/pull/1888
• Fixing types. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1906
• Types. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1909
• Fixing signals. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1910
• Removing some unused code. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1915
• MI300 compatibility by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1764
• Add TGI monitoring guide through Grafana and Prometheus by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1908
• Update grafana template by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1918
• Fix TunableOp bug by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1920
• Fix TGI issues with ROCm by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1921
• Fixing the download strategy for ibm-fms by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1917
• ROCm: make CK FA2 default instead of Triton by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1924
• docs: Fix grafana dashboard url by @edwardzjl in https://github.com/huggingface/text-generation-inference/pull/1925
• feat: include token in client test like server tests by @drbh in https://github.com/huggingface/text-generation-inference/pull/1932
• Creating doc automatically for supported models. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1929
• fix: use path inside of speculator config by @drbh in https://github.com/huggingface/text-generation-inference/pull/1935
• feat: add train medusa head tutorial by @drbh in https://github.com/huggingface/text-generation-inference/pull/1934
• reenable xpu for tgi by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/1939
• Fixing some legacy behavior (big swapout of serverless on legacy stuff). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1937
• Add completion route to client and add stop parameter where it's missing by @thomas-schillaci in https://github.com/huggingface/text-generation-inference/pull/1869
• Improving the logging system. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1938
• Fixing codellama loads by using purely AutoTokenizer. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1947
• Fix seeded output. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1949
• Fix (flash) Gemma prefix and enable tests by @danieldk in https://github.com/huggingface/text-generation-inference/pull/1950
• Fix GPTQ for models which do not have float16 at the default dtype (simpler) by @danieldk in https://github.com/huggingface/text-generation-inference/pull/1953
• Processor config chat template by @drbh in https://github.com/huggingface/text-generation-inference/pull/1954
• fix small typo and broken link by @MoritzLaurer in https://github.com/huggingface/text-generation-inference/pull/1958
• Upgrade to Axum 0.7 and Hyper 1.0 (Breaking change: disabled ngrok tunneling). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1959
• Fix (non-container) pytest stdout buffering-related lock-up by @danieldk in https://github.com/huggingface/text-generation-inference/pull/1963
• Fixing the text part from tokenizer endpoint. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1967
• feat: adjust attn weight loading logic by @drbh in https://github.com/huggingface/text-generation-inference/pull/1975
• Add support for exl2-quantized models by @danieldk in https://github.com/huggingface/text-generation-inference/pull/1965
• Update documentation version to 2.0.4 by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1980
• Purely refactors paged/attention into layers/attention and make hardware differences more obvious with 1 file per hardware. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1986
• Fixing exl2 scratch buffer. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1990
• single char ` addition for docs by @nbroad1881 in https://github.com/huggingface/text-generation-inference/pull/1989
• Fixing GPTQ imports. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1994
• reable xpu, broken by gptq and setuptool upgrade by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/1988
• router: send the input as chunks to the backend by @danieldk in https://github.com/huggingface/text-generation-inference/pull/1981
• Fix Phi-2 with tp>1 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2003
• fix: update triton implementation reference by @emmanuel-ferdman in https://github.com/huggingface/text-generation-inference/pull/2002
• feat: add SchedulerV3 by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1996
• Support GPTQ models with column-packed up/gate tensor by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2006
• Making make install work better by default. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2004
• Hotfixing make install. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2008
• Do not initialize scratch space when there are no ExLlamaV2 layers by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2015
• feat: move allocation logic to rust by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1835
• Fixing rocm. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2021
• Fix GPTQWeight import by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2020
• Update version on init.py to 0.7.0 by @andimarafioti in https://github.com/huggingface/text-generation-inference/pull/2017
• Add support for Marlin-quantized models by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2014
• marlin: support tp>1 when group_size==-1 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2032
• marlin: improve build by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2031
• Internal runner ? by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2023
• Xpu gqa by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2013
• server: use chunked inputs by @danieldk in https://github.com/huggingface/text-generation-inference/pull/1985
• ROCm and sliding windows fixes by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/2033
• Add Phi-3 medium support by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2039
• feat(ci): add trufflehog secrets detection by @McPatate in https://github.com/huggingface/text-generation-inference/pull/2038
• fix(ci): remove unnecessary permissions by @McPatate in https://github.com/huggingface/text-generation-inference/pull/2045
• Update LLMM1 bound by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/2050
• Support chat response format by @drbh in https://github.com/huggingface/text-generation-inference/pull/2046
• fix(server): fix OPT implementation by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2061
• fix(layers): fix SuRotaryEmbedding by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/2060
• PR #2049 CI run by @drbh in https://github.com/huggingface/text-generation-inference/pull/2054
• implement Open Inference Protocol endpoints by @drbh in https://github.com/huggingface/text-generation-inference/pull/1942
• Add support for GPTQ Marlin by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2052
• Update the link for qwen2 by @xianbaoqian in https://github.com/huggingface/text-generation-inference/pull/2068
• Adding architecture document by @tengomucho in https://github.com/huggingface/text-generation-inference/pull/2044
• Support different image sizes in prefill in VLMs by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2065
• Contributing guide & Code of Conduct by @LysandreJik in https://github.com/huggingface/text-generation-inference/pull/2074
• fix build.rs watch files by @zirconium-n in https://github.com/huggingface/text-generation-inference/pull/2072
• Set maximum grpc message receive size to 2GiB by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2075
• CI: Tailscale improvements by @glegendre01 in https://github.com/huggingface/text-generation-inference/pull/2079
• CI: pass pre-commit hooks again by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2084
• feat: rotate tests ci token by @drbh in https://github.com/huggingface/text-generation-inference/pull/2091
• Support exl2-quantized Qwen2 models by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2085
• Factor out sharding of packed tensors by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2059
• Fix text-generation-server quantize by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2103
• feat: sort cuda graphs in descending order by @drbh in https://github.com/huggingface/text-generation-inference/pull/2104
• New runner. Manual squash. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2110
• Fix cargo-chef prepare by @ur4t in https://github.com/huggingface/text-generation-inference/pull/2101
• Support HF_TOKEN environment variable by @Wauplin in https://github.com/huggingface/text-generation-inference/pull/2066
• Add OTLP Service Name Environment Variable by @KevinDuffy94 in https://github.com/huggingface/text-generation-inference/pull/2076
• corrected Pydantic warning. by @yukiman76 in https://github.com/huggingface/text-generation-inference/pull/2095
• use xpu-smi to dump used memory by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2047
• fix ChatCompletion and ChatCompletionChunk object string not compatible with standard openai api by @sunxichen in https://github.com/huggingface/text-generation-inference/pull/2089
• Cpu tgi by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/1936
• feat: add simple tests for weights by @drbh in https://github.com/huggingface/text-generation-inference/pull/2092
• Removing IPEX_AVAIL. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2115
• fix cpu and xpu issue by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/2116
• Add pytest release marker by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2114
• Fix CI . by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2118
• Enable multiple LoRa adapters by @drbh in https://github.com/huggingface/text-generation-inference/pull/2010
• Support AWQ quantization with bias by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2117
• Add support for Marlin 2:4 sparsity by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2102
• fix: simplify kserve endpoint and fix imports by @drbh in https://github.com/huggingface/text-generation-inference/pull/2119
• Fixing prom leak by upgrading. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2129
• Bumping to 2.1 by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2131
• Idefics2: sync added image tokens with transformers by @danieldk in https://github.com/huggingface/text-generation-inference/pull/2080
• Fixing malformed rust tokenizers by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2134
• Fixing gemma2. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/2135
• fix: refactor post_processor logic and add test by @drbh in https://github.com/huggingface/text-generation-inference/pull/2137

New Contributors

• @phangiabao98 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1888
• @edwardzjl made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1925
• @thomas-schillaci made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1869
• @nbroad1881 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1989
• @emmanuel-ferdman made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2002
• @andimarafioti made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2017
• @McPatate made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2038
• @xianbaoqian made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2068
• @tengomucho made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2044
• @LysandreJik made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2074
• @zirconium-n made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2072
• @glegendre01 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2079
• @ur4t made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2101
• @KevinDuffy94 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2076
• @yukiman76 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2095
• @sunxichen made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2089

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v2.0.3...v2.1.0

Main changes

• AMD MI300 compatibility by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1764
• Many bugfixes.

What's Changed

• OpenAI function calling compatible support by @phangiabao98 in https://github.com/huggingface/text-generation-inference/pull/1888
• Fixing types. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1906
• Types. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1909
• Fixing signals. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1910
• Removing some unused code. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1915
• MI300 compatibility by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1764
• Add TGI monitoring guide through Grafana and Prometheus by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1908
• Update grafana template by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1918
• Fix TunableOp bug by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1920
• Fix TGI issues with ROCm by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1921
• Fixing the download strategy for ibm-fms by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1917
• ROCm: make CK FA2 default instead of Triton by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1924
• docs: Fix grafana dashboard url by @edwardzjl in https://github.com/huggingface/text-generation-inference/pull/1925
• feat: include token in client test like server tests by @drbh in https://github.com/huggingface/text-generation-inference/pull/1932
• Creating doc automatically for supported models. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1929
• fix: use path inside of speculator config by @drbh in https://github.com/huggingface/text-generation-inference/pull/1935
• feat: add train medusa head tutorial by @drbh in https://github.com/huggingface/text-generation-inference/pull/1934
• reenable xpu for tgi by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/1939
• Fixing some legacy behavior (big swapout of serverless on legacy stuff). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1937
• Add completion route to client and add stop parameter where it's missing by @thomas-schillaci in https://github.com/huggingface/text-generation-inference/pull/1869
• Improving the logging system. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1938
• Fixing codellama loads by using purely AutoTokenizer. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1947

New Contributors

• @phangiabao98 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1888
• @edwardzjl made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1925
• @thomas-schillaci made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1869

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v2.0.3...v2.0.4

Important changes

• Add: Support for the Falcon2 by @Nilabhra in https://github.com/huggingface/text-generation-inference/pull/1886
• New speculation method MLPSpeculator. by @JRosenkranz in https://github.com/huggingface/text-generation-inference/pull/1865
• Pali gemma modeling by @drbh in https://github.com/huggingface/text-generation-inference/pull/1895

What's Changed

• Fix: "Fixing" double BOS for mistral too. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1843
• Adding scripts to prepare load data. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1841
• Remove misleading warning (not that important nowadays anyway). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1848
• feat: prefer huggingface_hub in docs and show image api by @drbh in https://github.com/huggingface/text-generation-inference/pull/1844
• Updating Phi3 (long context). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1849
• Add router name to /info endpoint by @Wauplin in https://github.com/huggingface/text-generation-inference/pull/1854
• Upgrading to rust 1.78. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1851
• update xpu docker image and use public ipex whel by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/1860
• Refactor layers. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1866
• Granite support? by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1882
• Add: Support for the Falcon2 11B architecture by @Nilabhra in https://github.com/huggingface/text-generation-inference/pull/1886
• MLPSpeculator. by @JRosenkranz in https://github.com/huggingface/text-generation-inference/pull/1865
• Fixing truncation. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1890
• Correct 'using guidance' link by @brandon-lockaby in https://github.com/huggingface/text-generation-inference/pull/1892
• Add GPT-2 with flash attention by @danieldk in https://github.com/huggingface/text-generation-inference/pull/1889
• Removing accepted ids in the regular info logs, downgrade to debug. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1898
• feat: add deprecation warning to clients by @drbh in https://github.com/huggingface/text-generation-inference/pull/1855
• [Bug Fix] Update torch import reference in bnb quantization by @DhruvSrikanth in https://github.com/huggingface/text-generation-inference/pull/1902
• Pali gemma modeling by @drbh in https://github.com/huggingface/text-generation-inference/pull/1895

New Contributors

• @Nilabhra made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1886
• @brandon-lockaby made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1892
• @danieldk made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1889
• @DhruvSrikanth made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1902

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v2.0.2...v2.0.3

Tl;dr

• New models (idefics2, phi3)
• Cleaner VLM support in the openai layer
• Upgraded to pytorch 2.3.0

What's Changed

• Make --cuda-graphs 0 work as expected (bis) by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1768
• fix typos in docs and add small clarifications by @MoritzLaurer in https://github.com/huggingface/text-generation-inference/pull/1790
• Add attribute descriptions for GenerateParameters by @Wauplin in https://github.com/huggingface/text-generation-inference/pull/1798
• feat: allow null eos and bos tokens in config by @drbh in https://github.com/huggingface/text-generation-inference/pull/1791
• Phi3 support by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1797
• Idefics2. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1756
• fix: avoid frequency and repetition penalty on padding tokens by @drbh in https://github.com/huggingface/text-generation-inference/pull/1765
• Adding support for HF_HUB_OFFLINE support in the router. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1789
• feat: improve temperature logic in chat by @drbh in https://github.com/huggingface/text-generation-inference/pull/1749
• Updating the benchmarks so everyone uses openai compat layer. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1800
• Update guidance docs to reflect grammar support in API by @dr3s in https://github.com/huggingface/text-generation-inference/pull/1775
• Use the generation config. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1808
• 2nd round of benchmark modifications (tiny adjustements to avoid overloading the host). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1816
• Adding new env variables for TPU backends. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1755
• add intel xpu support for TGI by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/1475
• Blunder by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1815
• Fixing qwen2. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1818
• Dummy CI run. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1817
• Changing the waiting_served_ratio default (stack more aggressively by default). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1820
• Better graceful shutdown. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1827
• Add the missing tool_prompt parameter to Python client by @maziyarpanahi in https://github.com/huggingface/text-generation-inference/pull/1825
• Small CI cleanup. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1801
• Add reference to TPU support by @brandonroyal in https://github.com/huggingface/text-generation-inference/pull/1760
• fix: use get_speculate to the number of layers by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1737
• feat: add how it works section by @drbh in https://github.com/huggingface/text-generation-inference/pull/1773
• Fixing frequency penalty by @martinigoyanes in https://github.com/huggingface/text-generation-inference/pull/1811
• feat: add vlm docs and simple examples by @drbh in https://github.com/huggingface/text-generation-inference/pull/1812
• Handle images in chat api by @drbh in https://github.com/huggingface/text-generation-inference/pull/1828
• chore: update torch by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1730
• (chore): torch 2.3.0 by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1833

New Contributors

• @MoritzLaurer made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1790
• @dr3s made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1775
• @maziyarpanahi made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1825
• @brandonroyal made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1760
• @martinigoyanes made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1811

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v2.0.1...v2.0.2

What's Changed

• feat: improve tools to include name and add tests by @drbh in https://github.com/huggingface/text-generation-inference/pull/1693
• Update response type for /v1/chat/completions and /v1/completions by @Wauplin in https://github.com/huggingface/text-generation-inference/pull/1747
• accept list as prompt for OpenAI API by @drbh in https://github.com/huggingface/text-generation-inference/pull/1702
• fix ROCm docker image

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v2.0.0...v2.0.1

TGI is back to Apache 2.0!

Highlights

• License was reverted to Apache 2.0
• Cuda graphs are now used by default. They improve latency substancially on high end nodes.
• Llava-next was added. It is the second multimodal model available on TGI after Idefics.
• Cohere Command R+ support. TGI is the fastest open source backend for Command R+
• FP8 support.
• We now share the vocabulary for all medusa heads, greatly improving latency and memory use.

Try out Command R+ with Medusa heads on 4xA100s with:

model=text-generation-inference/commandrplus-medusa
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --speculate 3 --num-shard 4

What's Changed

• Add cuda graphs sizes and make it default. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1703
• Pickle conversion now requires --trust-remote-code. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1704
• Push users to streaming in the readme. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1698
• Fixing cohere tokenizer. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1697
• Force weights_only (before fully breaking pickle files anyway). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1710
• Regenerate ld.so.cache by @oOraph in https://github.com/huggingface/text-generation-inference/pull/1708
• Revert license to Apache 2.0 by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1714
• Automatic quantization config. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1719
• Adding Llava-Next (Llava 1.6) with full support. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1709
• fix: fix CohereForAI/c4ai-command-r-plus by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1707
• Update libraries by @abhishekkrthakur in https://github.com/huggingface/text-generation-inference/pull/1713
• Dev/mask ldconfig output v2 by @oOraph in https://github.com/huggingface/text-generation-inference/pull/1716
• Fp8 Support by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1726
• Upgrade EETQ (Fixes the cuda graphs). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1729
• fix(router): fix a possible deadlock in next_batch by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1731
• chore(cargo-toml): apply lto fat and codegen-units of one by @somehowchris in https://github.com/huggingface/text-generation-inference/pull/1651
• Improve the defaults for the launcher by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1727
• feat: medusa shared by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1734
• Fix typo in guidance.md by @eltociear in https://github.com/huggingface/text-generation-inference/pull/1735

New Contributors

• @somehowchris made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1651

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.4.5...v2.0.0

Highlights

• DBRX support https://github.com/huggingface/text-generation-inference/pull/1685. See #1679 on how to prompt the model.

What's Changed

• fix: adjust logprob response logic by @drbh in https://github.com/huggingface/text-generation-inference/pull/1682
• fix: handle batches with and without grammars by @drbh in https://github.com/huggingface/text-generation-inference/pull/1676
• feat: Add dbrx support by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1685

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.4.4...v1.4.5

Highlights

• CohereForAI/c4ai-command-r-v01 model support

What's Changed

• Handle concurrent grammar requests by @drbh in https://github.com/huggingface/text-generation-inference/pull/1610
• Fix idefics default. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1614
• Fix async client timeout by @hugoabonizio in https://github.com/huggingface/text-generation-inference/pull/1617
• accept legacy request format and response by @drbh in https://github.com/huggingface/text-generation-inference/pull/1527
• add missing stop parameter for chat request by @drbh in https://github.com/huggingface/text-generation-inference/pull/1619
• correctly index into mask when applying grammar by @drbh in https://github.com/huggingface/text-generation-inference/pull/1618
• Use a better model for the quick tour by @lewtun in https://github.com/huggingface/text-generation-inference/pull/1639
• Upgrade nix version from 0.27.1 to 0.28.0 by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/1638
• Update peft + transformers + accelerate + bnb + safetensors by @abhishekkrthakur in https://github.com/huggingface/text-generation-inference/pull/1646
• Fix index in ChatCompletionChunk by @Wauplin in https://github.com/huggingface/text-generation-inference/pull/1648
• Fixing minor typo in documentation: supported hardware section by @SachinVarghese in https://github.com/huggingface/text-generation-inference/pull/1632
• bump minijina and add test for core templates by @drbh in https://github.com/huggingface/text-generation-inference/pull/1626
• support force downcast after FastRMSNorm multiply for Gemma by @drbh in https://github.com/huggingface/text-generation-inference/pull/1658
• prefer spaces url over temp url by @drbh in https://github.com/huggingface/text-generation-inference/pull/1662
• improve tool type, bump pydantic and outlines by @drbh in https://github.com/huggingface/text-generation-inference/pull/1650
• Remove unecessary cuda graph. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1664
• Repair idefics integration tests. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1663
• fix: LlamaTokenizerFast to AutoTokenizer at flash_mistral.py by @SeongBeomLEE in https://github.com/huggingface/text-generation-inference/pull/1637
• Inline images for multimodal models. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1666

New Contributors

• @hugoabonizio made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1617
• @yuanwu2017 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1638
• @abhishekkrthakur made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1646
• @Wauplin made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1648
• @SachinVarghese made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1632
• @SeongBeomLEE made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1637

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.4.3...v1.4.4

Highlights

• Add support for Starcoder 2
• Add support for Qwen2

What's Changed

• fix openapi schema by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1586
• avoid default message by @drbh in https://github.com/huggingface/text-generation-inference/pull/1579
• Revamp medusa implementation so that every model can benefit. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1588
• Support tools by @drbh in https://github.com/huggingface/text-generation-inference/pull/1587
• Fixing x-compute-time. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1606
• Fixing guidance docs. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1607
• starcoder2 by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1605
• Qwen2 by @Jason-CKY in https://github.com/huggingface/text-generation-inference/pull/1608

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.4.2...v1.4.3

Highlights

• Add support for Google Gemma models

What's Changed

• Fix mistral with length > window_size for long prefills (rotary doesn't create long enough cos, sin). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1571
• improve endpoint support by @drbh in https://github.com/huggingface/text-generation-inference/pull/1577
• refactor syntax to correctly include structs by @drbh in https://github.com/huggingface/text-generation-inference/pull/1580
• fix openapi and add jsonschema validation by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1578
• add support for Gemma by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1583

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.4.1...v1.4.2

Highlights

• Mamba support by @drbh in https://github.com/huggingface/text-generation-inference/pull/1480 and by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1552
• Experimental support for cuda graphs by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1428
• Outlines guided generation by @drbh in https://github.com/huggingface/text-generation-inference/pull/1539
• Added name field to OpenAI compatible API Messages by @amihalik in https://github.com/huggingface/text-generation-inference/pull/1563

What's Changed

• Fixing top_n_tokens. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1497
• Sending compute type from the environment instead of hardcoded string by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1504
• Create the compute type at launch time (if not provided in the env). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1505
• Modify default for max_new_tokens in python client by @freitng in https://github.com/huggingface/text-generation-inference/pull/1336
• feat: eetq gemv optimization when batch_size <= 4 by @dtlzhuangz in https://github.com/huggingface/text-generation-inference/pull/1502
• fix: improve messages api docs content and formatting by @drbh in https://github.com/huggingface/text-generation-inference/pull/1506
• GPTNeoX: Use static rotary embedding by @dwyatte in https://github.com/huggingface/text-generation-inference/pull/1498
• Hotfix the / health - route. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1515
• fix: tokenizer config should use local model path when possible by @drbh in https://github.com/huggingface/text-generation-inference/pull/1518
• Updating tokenizers. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1517
• [docs] Fix link to Install CLI by @pcuenca in https://github.com/huggingface/text-generation-inference/pull/1526
• feat: add ie update to message docs by @drbh in https://github.com/huggingface/text-generation-inference/pull/1523
• feat: use existing add_generation_prompt variable from config in temp… by @drbh in https://github.com/huggingface/text-generation-inference/pull/1533
• Update to peft 0.8.2 by @Stillerman in https://github.com/huggingface/text-generation-inference/pull/1537
• feat(server): add frequency penalty by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1541
• chore: bump ci rust version by @drbh in https://github.com/huggingface/text-generation-inference/pull/1543
• ROCm AWQ support by @IlyasMoutawwakil in https://github.com/huggingface/text-generation-inference/pull/1514
• feat(router): add max_batch_size by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1542
• feat: add deserialize_with that handles strings or objects with content by @drbh in https://github.com/huggingface/text-generation-inference/pull/1550
• Fixing glibc version in the runtime. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1556
• Upgrade intermediary layer for nvidia too. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1557
• Improving mamba runtime by using updates by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1552
• Small cleanup. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1560
• Bugfix: eos and bos tokens positions are inconsistent by @amihalik in https://github.com/huggingface/text-generation-inference/pull/1567
• chore: add pre-commit by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1569
• feat: add chat template struct to avoid tuple ordering errors by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1570
• v1.4.1 by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1568

New Contributors

• @freitng made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1336
• @dtlzhuangz made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1502
• @dwyatte made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1498
• @pcuenca made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1526
• @Stillerman made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1537
• @IlyasMoutawwakil made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1514
• @amihalik made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1563

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.4.0...v1.4.1

Highlights

• OpenAI compatible API #1427
• exllama v2 Tensor Parallel #1490
• GPTQ support for AMD GPUs #1489
• Phi support #1442

What's Changed

• fix: fix local loading for .bin models by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1419
• Fix missing make target platform for local install: 'install-flash-attention-v2' by @deepily in https://github.com/huggingface/text-generation-inference/pull/1414
• fix: follow base model for tokenizer in router by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1424
• Fix local load for Medusa by @PYNing in https://github.com/huggingface/text-generation-inference/pull/1420
• Return prompt vs generated tokens. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1436
• feat: supports openai chat completions API by @drbh in https://github.com/huggingface/text-generation-inference/pull/1427
• feat: support raise_exception, bos and eos tokens by @drbh in https://github.com/huggingface/text-generation-inference/pull/1450
• chore: bump rust version and annotate/fix all clippy warnings by @drbh in https://github.com/huggingface/text-generation-inference/pull/1455
• feat: conditionally toggle chat on invocations route by @drbh in https://github.com/huggingface/text-generation-inference/pull/1454
• Disable decoder_input_details on OpenAI-compatible chat streaming, pass temp and top-k from API by @EndlessReform in https://github.com/huggingface/text-generation-inference/pull/1470
• Fixing non divisible embeddings. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1476
• Add messages api compatibility docs by @drbh in https://github.com/huggingface/text-generation-inference/pull/1478
• Add a new /tokenize route to get the tokenized input by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1471
• feat: adds phi model by @drbh in https://github.com/huggingface/text-generation-inference/pull/1442
• fix: read stderr in download by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1486
• fix: show warning with tokenizer config parsing error by @drbh in https://github.com/huggingface/text-generation-inference/pull/1488
• fix: launcher doc typos by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1473
• Reinstate exl2 with tp by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1490
• Add sealion mpt support by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1477
• Trying to fix that flaky test. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1491
• fix: launcher doc typos by @thelinuxkid in https://github.com/huggingface/text-generation-inference/pull/1462
• Update the docs to include newer models. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1492
• GPTQ support on ROCm by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1489
• feat: add tokenizer-config-path to launcher args by @drbh in https://github.com/huggingface/text-generation-inference/pull/1495

New Contributors

• @deepily made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1414
• @PYNing made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1420
• @drbh made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1427
• @EndlessReform made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1470
• @thelinuxkid made their first contribution in https://github.com/huggingface/text-generation-inference/pull/1462

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.3.4...v1.4.0

What's Changed

• feat: relax mistral requirements by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1351
• fix: fix logic if sliding window key is not present in config by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1352
• fix: fix offline (#1341) by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1347
• fix: fix gpt-q with groupsize = -1 by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1358
• Peft safetensors. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1364
• Change URL for Habana Gaudi support in doc by @regisss in https://github.com/huggingface/text-generation-inference/pull/1343
• feat: update exllamav2 kernels by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1370
• Fix local load for peft by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1373

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.3.3...v1.3.4

What's Changed

• fix gptq params loading
• improve decode latency for long sequences two fold
• feat: add more latency metrics in forward by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1346
• fix: max_past default value must be -1, not 0 by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1348

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.3.2...v1.3.3

What's Changed

• fix: support null sliding window for mistral models by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1337
• feat: add quant to mixtral by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1337

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.3.1...v1.3.2

Hotfix Mixtral implementation

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.3.0...v1.3.1

What's Changed

• Fix AMD documentation by @fxmarty in https://github.com/huggingface/text-generation-inference/pull/1307
• Medusa and N-Gram Speculative decoding by @Narsil in https://github.com/huggingface/text-generation-inference/pull/1308
• Mixtral support by @OlivierDehaene in https://github.com/huggingface/text-generation-inference/pull/1328

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v1.2.0...v1.3.0