Features

• server: auto max_batch_total_tokens for flash att models https://github.com/huggingface/text-generation-inference/pull/630
• router: ngrok edge https://github.com/huggingface/text-generation-inference/pull/642
• server: Add trust_remote_code to quantize script by @ChristophRaab https://github.com/huggingface/text-generation-inference/pull/647
• server: Add exllama GPTQ CUDA kernel support #553 https://github.com/huggingface/text-generation-inference/pull/666
• server: Directly load GPTBigCode to specified device by @Atry in https://github.com/huggingface/text-generation-inference/pull/618
• server: add cuda memory fraction https://github.com/huggingface/text-generation-inference/pull/659
• server: Using quantize_config.json instead of GPTQ_BITS env variables https://github.com/huggingface/text-generation-inference/pull/671
• server: support new falcon config https://github.com/huggingface/text-generation-inference/pull/712

Fix

• server: llama v2 GPTQ https://github.com/huggingface/text-generation-inference/pull/648
• server: Fixing non parameters in quantize script bigcode/starcoder was an example https://github.com/huggingface/text-generation-inference/pull/661
• server: use mem_get_info to get kv cache size https://github.com/huggingface/text-generation-inference/pull/664
• server: fix exllama buffers https://github.com/huggingface/text-generation-inference/pull/689
• server: fix quantization python requirements https://github.com/huggingface/text-generation-inference/pull/708

New Contributors

• @ChristophRaab made their first contribution in https://github.com/huggingface/text-generation-inference/pull/647
• @fxmarty made their first contribution in https://github.com/huggingface/text-generation-inference/pull/648
• @Atry made their first contribution in https://github.com/huggingface/text-generation-inference/pull/618

Full Changelog: https://github.com/huggingface/text-generation-inference/compare/v0.9.3...v0.9.4