Transformers

Fixed assisted decoding for models using EncoderDecoderCache (e.g., OlmoHybrid) and a SDPA prefill issue with position_bias during StaticCache. Also bumps FP8 kernels and fixes DeepGEMM on multiple devices.

Added Inkling, a 975B-parameter multimodal model, and TIPSv2. GPTNeoX now remaps embed_out to lm_head and GPTBigCode has attention backend changes for vLLM compatibility. Multi-Token Prediction decoding support, SDPA prefill with FlashAttention for StaticCache (up to 260% faster), and numerous bug fixes across MoE, cache, and generation.

Fixes custom model compatibility with the latest vllm release by being more defensive with remap_legacy_layer_types and handling cases where custom code doesn't know about the new linear layer type names. Also fixed a key type assertion in _LazyAutoMapping.register.

This release adds support for seven new model architectures: Kimi 2.5–2.7 (multimodal agentic coding), MiMo-V2-Flash (256K context MoE model), Nemotron 3.5 ASR and Nemotron ASR Streaming (multilingual speech recognition with configurable latency-accuracy tradeoffs), Qwen3 ASR with forced aligner, ZAYA1 (MoE language model), VideoPrism (video understanding encoder), and RADIO (vision foundation model family).

This release introduces the MiniMax-M3-VL vision-language model, the PP-OCRv6 OCR system, and the Parakeet-RNNT model for speech processing. Several bug fixes and improvements were also made, including changes to CI, stop string matching, and model documentation.

Accelerate now works end-to-end on AMD ROCm devices. This release also includes numerous FSDP2 fixes and quality-of-life improvements, such as correct dtype handling on load, sharding of embeddings/norms, and QLoRA crash prevention.

New models DiffusionGemma and DeepSeek-V3.2 have been added, featuring optimizations for inference speed and efficient long-context handling. The Kernels API was extended for module fusion and parameter transformation, with added support for fp8/fp4 Triton kernels. Model parallel beam search bugs in Qwen2-VL model families were fixed.

Agent traces from Claude Code, Pi, Codex and others can now be loaded with load_dataset and parsed to messages for training with TRL. Streaming shuffle now uses multiple input shards by default for better randomization, though this is a breaking change—use max_buffer_input_shards=1 to restore the prior behavior. Also added batch(by_column=...) for robotics datasets, support for Apache Iceberg, TsFile (IoTDB), 3D mesh, and CoNLL formats, plus fixes for Parquet streaming hangs, Lance file handling, and JSON null encoding.

Fixed a conversion bug for CLIP models that affected downstream models like SAM3.

Added support for Cohere2Moe (a Mixture-of-Experts model with sliding window and full attention), HRM-Text (hierarchical reasoning model with two transformer stacks), and Parakeet tdt speech model. SAM3, EdgeTAM, and SAM3-Lite-Text now expect full text embeddings instead of pooler outputs, requiring input updates. Fixed generation issues including inputs_embeds handling for Gemma4, an AttributeError in RAG's generate() caused by missing config fields, memory leaks from lru decorators in vision models, and improved audio/vision encoder compilability.

Fixed Deepseek V4 integration issues including CSA mask collapse and WeightConverter regex incorrectly matching shared_experts as experts. Also added fatal_error to ContinuousBatchingManager for serving operations.

Release v5.8.0

New Model additions

DeepSeek-V4

DeepSeek-V4 is the next-generation MoE (Mixture of Experts) language model…

New Model additions

Laguna

Laguna is Poolside's mixture-of-experts language model family that extends standard SwiGLU MoE…