Transformers

Fixed DFlash candidate token device mismatch with device_map='auto', aligned logit distributions for sampling-based candidate generators, fixed MTP config when mlp_layer_types is absent, and added a fallback from Lanczos to bicubic filtering on CUDA so images process on accelerator. Also fixed gemma4 video device placement.

Adds support for Meta's Muse Glimmer 30B multimodal model plus GraniteSWA, GraniteMoeSWA, A.X-K1/K2, and Cosmos3 Edge. Kernels for linear attention models (Mamba, GDN, Conv-only) are now opt-in rather than mandatory, cache cropping only accepts negative offsets, and T5 family gains SDPA support with a possible default attention change.

Fixed a symlink-following arbitrary file write in archive extraction and a path traversal via metadata file_name in folder-based builders. Also fixed batch(by_column=...) crashing after shard/shuffle/split, CSV loader issues on pandas 2.0-2.2, and more than 20 other bugs across Arrow, JSON, and dataset card handling.

Fixed assisted decoding for models using EncoderDecoderCache (e.g., OlmoHybrid) and a SDPA prefill issue with position_bias during StaticCache. Also bumps FP8 kernels and fixes DeepGEMM on multiple devices.

Added Inkling, a 975B-parameter multimodal model, and TIPSv2. GPTNeoX now remaps embed_out to lm_head and GPTBigCode has attention backend changes for vLLM compatibility. Multi-Token Prediction decoding support, SDPA prefill with FlashAttention for StaticCache (up to 260% faster), and numerous bug fixes across MoE, cache, and generation.

Fixes custom model compatibility with the latest vllm release by being more defensive with remap_legacy_layer_types and handling cases where custom code doesn't know about the new linear layer type names. Also fixed a key type assertion in _LazyAutoMapping.register.

This release adds support for seven new model architectures: Kimi 2.5–2.7 (multimodal agentic coding), MiMo-V2-Flash (256K context MoE model), Nemotron 3.5 ASR and Nemotron ASR Streaming (multilingual speech recognition with configurable latency-accuracy tradeoffs), Qwen3 ASR with forced aligner, ZAYA1 (MoE language model), VideoPrism (video understanding encoder), and RADIO (vision foundation model family).

This release introduces the MiniMax-M3-VL vision-language model, the PP-OCRv6 OCR system, and the Parakeet-RNNT model for speech processing. Several bug fixes and improvements were also made, including changes to CI, stop string matching, and model documentation.

Accelerate now works end-to-end on AMD ROCm devices. This release also includes numerous FSDP2 fixes and quality-of-life improvements, such as correct dtype handling on load, sharding of embeddings/norms, and QLoRA crash prevention.

New models DiffusionGemma and DeepSeek-V3.2 have been added, featuring optimizations for inference speed and efficient long-context handling. The Kernels API was extended for module fusion and parameter transformation, with added support for fp8/fp4 Triton kernels. Model parallel beam search bugs in Qwen2-VL model families were fixed.

Agent traces from Claude Code, Pi, Codex and others can now be loaded with load_dataset and parsed to messages for training with TRL. Streaming shuffle now uses multiple input shards by default for better randomization, though this is a breaking change—use max_buffer_input_shards=1 to restore the prior behavior. Also added batch(by_column=...) for robotics datasets, support for Apache Iceberg, TsFile (IoTDB), 3D mesh, and CoNLL formats, plus fixes for Parquet streaming hangs, Lance file handling, and JSON null encoding.

Fixed a conversion bug for CLIP models that affected downstream models like SAM3.

Added support for Cohere2Moe (a Mixture-of-Experts model with sliding window and full attention), HRM-Text (hierarchical reasoning model with two transformer stacks), and Parakeet tdt speech model. SAM3, EdgeTAM, and SAM3-Lite-Text now expect full text embeddings instead of pooler outputs, requiring input updates. Fixed generation issues including inputs_embeds handling for Gemma4, an AttributeError in RAG's generate() caused by missing config fields, memory leaks from lru decorators in vision models, and improved audio/vision encoder compilability.

Fixed Deepseek V4 integration issues including CSA mask collapse and WeightConverter regex incorrectly matching shared_experts as experts. Also added fatal_error to ContinuousBatchingManager for serving operations.

Release v5.8.0

New Model additions

DeepSeek-V4

DeepSeek-V4 is the next-generation MoE (Mixture of Experts) language model…