Features

• server: New faster GPTNeoX implementation based on flash attention

Fix

• server: fix input-length discrepancy between Rust and Python tokenizers

Features

• router: support best_of sampling
• router: support left truncation
• server: support typical sampling
• launcher: allow local models
• clients: add text-generation Python client
• launcher: allow parsing num_shard from CUDA_VISIBLE_DEVICES

Fix

• server: do not warp prefill logits
• server: fix formatting issues in generate_stream tokens
• server: fix galactica batch
• server: fix index out of range issue with watermarking

Features

• router: add support for huggingface api-inference
• server: add logits watermark with "A Watermark for Large Language Models"
• server: use a fixed transformers commit

Fix

• launcher: add missing parameters to launcher
• server: update to hf_transfer==0.1.2 to fix corrupted files issue

Features

• server: allocate full attention mask to decrease latency
• server: enable hf-transfer for insane download speeds
• router: add CORS options

Fix

• server: remove position_ids from galactica forward

Features

• server: support t5 models
• router: add max_total_tokens and empty_input validation
• launcher: add the possibility to disable custom CUDA kernels
• server: add automatic safetensors conversion
• router: add prometheus scrape endpoint
• server, router: add distributed tracing

Fix

• launcher: copy current env vars to subprocesses
• docker: add note around shared memory

Fix

• server: fix bug with repetition penalty when using GPUs and inference mode

Features

• router: support Token streaming using Server Side Events
• router: support seeding
• server: support gpt-neox
• server: support santacoder
• server: support repetition penalty
• server: allow the server to use a local weight cache

Breaking changes

• router: refactor Token API
• router: modify /generate API to only return generated text

Misc

• router: use background task to manage request queue
• ci: docker build/push on update