--- name: Transformers.js slug: transformers-js type: github source_url: https://github.com/huggingface/transformers.js organization: Hugging Face organization_slug: hugging-face total_releases: 84 latest_version: 4.0.0 latest_date: 2026-03-30 last_updated: 2026-04-18 tracking_since: 2023-05-15 canonical: https://releases.sh/hugging-face/transformers-js organization_url: https://releases.sh/hugging-face --- Transformers.js shipped v4 with a ground-up rewrite of its WebGPU backend in C++, developed in partnership with the ONNX Runtime team. The new runtime expands operator support across roughly 200 existing model architectures plus new v4-exclusive designs, and enables the same transformers.js code to run across diverse JavaScript environments including browsers and beyond. Transformers.js reached v4.0.0 with a complete rewrite of its WebGPU runtime in C++, built in collaboration with the ONNX Runtime team. The new backend expanded operator support across ~200 existing model architectures plus new v4-exclusive ones, enabling the same code to run across diverse JavaScript environments. # πŸš€ Transformers.js v4 We're excited to announce that Transformers.js v4 is now available on NPM! After a year of development (we started in March 2025 🀯), we're finally ready for you to use it. ```sh npm i @huggingface/transformers ``` Links: [YouTube Video](https://www.youtube.com/watch?v=KnhppkY4gHs), [Blog Post](https://huggingface.co/blog/transformersjs-v4), [Demo Collection](https://huggingface.co/collections/webml-community/transformersjs-v4-demos) ## New WebGPU backend The biggest change is undoubtedly the adoption of a new WebGPU Runtime, completely rewritten in C++. We've worked closely with the ONNX Runtime team to thoroughly test this runtime across our ~200 supported model architectures, as well as many new v4-exclusive architectures. In addition to better operator support (for performance, accuracy, and coverage), this new WebGPU runtime allows the same transformers.js code to be used across a wide variety of JavaScript environments, including browsers, server-side runtimes, and desktop applications. That's right, you can now run WebGPU-accelerated models directly in Node, Bun, and Deno! WebGPU Overview We've proven that it's possible to run state-of-the-art AI models 100% locally in the browser, and now we're focused on performance: making these models run as fast as possible, even in resource-constrained environments. This required completely rethinking our export strategy, especially for large language models. We achieve this by re-implementing new models operation by operation, leveraging specialized ONNX Runtime [Contrib Operators](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md) like [com.microsoft.GroupQueryAttention](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#com.microsoft.GroupQueryAttention), [com.microsoft.MatMulNBits](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#com.microsoft.MatMulNBits), or [com.microsoft.QMoE](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#com.microsoft.QMoE) to maximize performance. For example, adopting the [com.microsoft.MultiHeadAttention](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#com.microsoft.MultiHeadAttention) operator, we were able to achieve a ~4x speedup for BERT-based embedding models. Optimized ONNX Exports * ONNX Runtime improvements by @xenova in https://github.com/huggingface/transformers.js/pull/1306 * Transformers.js V4: Native WebGPU EP, repo restructuring, and more! by @xenova in https://github.com/huggingface/transformers.js/pull/1382 ## New models Thanks to our new export strategy and ONNX Runtime's expanding support for custom operators, we've been able to add many new models and architectures to Transformers.js v4. These include popular models like GPT-OSS, Chatterbox, GraniteMoeHybrid, LFM2-MoE, HunYuanDenseV1, Apertus, Olmo3, FalconH1, and Youtu-LLM. Many of these required us to implement support for advanced architectural patterns, including Mamba (state-space models), Multi-head Latent Attention (MLA), and Mixture of Experts (MoE). Perhaps most importantly, these models are all compatible with WebGPU, allowing users to run them directly in the browser or server-side JavaScript environments with hardware acceleration. We've released several [Transformers.js v4 demos](https://huggingface.co/collections/webml-community/transformersjs-v4-demos) so far... and we'll continue to release more! Additionally, we've added support for larger models exceeding 8B parameters. In our tests, we've been able to run GPT-OSS 20B (q4f16) at ~60 tokens per second on an M4 Pro Max. * Add support for Apertus by @nico-martin in https://github.com/huggingface/transformers.js/pull/1465 * Add support for FalconH1 by @xenova in https://github.com/huggingface/transformers.js/pull/1502 * Add support for Cohere's Tiny Aya models by @xenova in https://github.com/huggingface/transformers.js/pull/1529 * Add support for AFMoE by @xenova in https://github.com/huggingface/transformers.js/pull/1542 * Add support for new Qwen VL models (Qwen2.5-VL, Qwen3-VL, Qwen3.5, and Qwen3.5 MoE) by @xenova in https://github.com/huggingface/transformers.js/pull/1551 * Add support for Qwen2 MoE, Qwen3 MoE, Qwen3 Next, Qwen3-VL MoE, and Olmo Hybrid by @xenova in https://github.com/huggingface/transformers.js/pull/1562 * Add support for EuroBERT by @xenova in https://github.com/huggingface/transformers.js/pull/1583 * Add support for LightOnOCR and GLM-OCR by @xenova in https://github.com/huggingface/transformers.js/pull/1582 * Add support for Nemotron-H by @xenova in https://github.com/huggingface/transformers.js/pull/1585 * Add support for DeepSeek-v3 by @xenova in https://github.com/huggingface/transformers.js/pull/1586 * Add support for mistral4 by @xenova in https://github.com/huggingface/transformers.js/pull/1587 * Add support for GLM-MoE-DSA by @xenova in https://github.com/huggingface/transformers.js/pull/1588 * Add support for Chatterbox @xenova in https://github.com/huggingface/transformers.js/pull/1592 * Add support for Cohere ASR by @xenova in https://github.com/huggingface/transformers.js/pull/1610 * Add support for SolarOpen and CHMv2 models by @xenova in https://github.com/huggingface/transformers.js/pull/1593 * Voxtral Realtime, LFM2-VL, Granite Speech, and modeling type refactoring by @xenova in https://github.com/huggingface/transformers.js/pull/1569 * Add support for Gemma3 VLM architecture by @xenova in https://github.com/huggingface/transformers.js/pull/1601 ## New features ### ModelRegistry The new `ModelRegistry` API is designed for production workflows. It provides explicit visibility into pipeline assets before loading anything: list required files with `get_pipeline_files`, inspect per-file metadata with `get_file_metadata` (quite useful to calculate total download size), check cache status with `is_pipeline_cached`, and clear cached artifacts with `clear_pipeline_cache`. You can also query available precision types for a model with `get_available_dtypes`. Based on this new API, `progress_callback` now includes a `progress_total` event, making it easy to render end-to-end loading progress without manually aggregating per-file updates.
See `ModelRegistry` examples ```javascript import { ModelRegistry, pipeline } from "@huggingface/transformers"; const modelId = "onnx-community/all-MiniLM-L6-v2-ONNX"; const modelOptions = { dtype: "fp32" }; const files = await ModelRegistry.get_pipeline_files( "feature-extraction", modelId, modelOptions ); // ['config.json', 'onnx/model.onnx', ..., 'tokenizer_config.json'] const metadata = await Promise.all( files.map(file => ModelRegistry.get_file_metadata(modelId, file)) ); const downloadSize = metadata.reduce((total, item) => total + item.size, 0); const cached = await ModelRegistry.is_pipeline_cached( "feature-extraction", modelId, modelOptions ); const dtypes = await ModelRegistry.get_available_dtypes(modelId); // ['fp32', 'fp16', 'q4', 'q4f16'] if (cached) { await ModelRegistry.clear_pipeline_cache( "feature-extraction", modelId, modelOptions ); } const pipe = await pipeline( "feature-extraction", modelId, { progress_callback: e => { if (e.status === "progress_total") { console.log(`${Math.round(e.progress)}%`); } }, } ); ```
### New Environment Settings We also added new environment controls for model loading. `env.useWasmCache` enables caching of WASM runtime files (when cache storage is available), allowing applications to work fully offline after the initial load. `env.fetch` lets you provide a custom fetch implementation for use cases such as authenticated model access, custom headers, and abortable requests.
See env examples ```javascript import { env } from "@huggingface/transformers"; env.useWasmCache = true; env.fetch = (url, options) => fetch(url, { ...options, headers: { ...options?.headers, Authorization: `Bearer ${MY_TOKEN}`, }, }); ```
### Improved Logging Controls Finally, logging is easier to manage in real-world deployments. ONNX Runtime WebGPU warnings are now hidden by default, and you can set explicit verbosity levels for both Transformers.js and ONNX Runtime. This update, also driven by community feedback, keeps console output focused on actionable signals rather than low-value noise.
See `logLevel` example ```javascript import { env, LogLevel } from "@huggingface/transformers"; // LogLevel.DEBUG // LogLevel.INFO // LogLevel.WARNING // LogLevel.ERROR // LogLevel.NONE env.logLevel = LogLevel.WARNING; ```
* [v4] added wasm cache by @nico-martin in https://github.com/huggingface/transformers.js/pull/1471 * V4 cache wasm file blob fix by @nico-martin in https://github.com/huggingface/transformers.js/pull/1489 * [v4] suppress console.error while creating InferenceSession by @nico-martin in https://github.com/huggingface/transformers.js/pull/1468 * feat: add configurable log levels via env.logLevel by @taronsung in https://github.com/huggingface/transformers.js/pull/1507 * [v4] Improve download progress tracking (model cache registry and define which files will be loaded for pipelines) by @nico-martin in https://github.com/huggingface/transformers.js/pull/1511 * Add support for seedable random number generation by @xenova in https://github.com/huggingface/transformers.js/pull/1537 * Add customFetch to env for user override by @sroussey in https://github.com/huggingface/transformers.js/pull/1538 and @xenova in https://github.com/huggingface/transformers.js/pull/1543 * [deno] Support both wgpu and dawn webgpu backends by @xenova in https://github.com/huggingface/transformers.js/pull/1546 * feat: experimental Cross-Origin Storage cache backend by @nico-martin in https://github.com/huggingface/transformers.js/pull/1549 * Support PKV cached generation for Qwen-VL models by @xenova in https://github.com/huggingface/transformers.js/pull/1557 * Use ModelRegistry for pipeline file loading by @xenova in https://github.com/huggingface/transformers.js/pull/1555 * feat: add fast boolean `is_cached` / `is_pipeline_cached`, closes #1554 by @nico-martin in https://github.com/huggingface/transformers.js/pull/1559 * feat: add ModelRegistry.get_available_dtypes() to detect quantization by @sroussey in https://github.com/huggingface/transformers.js/pull/1580 * Add support for qwen3_5_text <-> Qwen3_5ForCausalLM by @xenova in https://github.com/huggingface/transformers.js/pull/1602 * Fix ModelRegistry calls when loading multimodal models with text-generation pipeline by @xenova in https://github.com/huggingface/transformers.js/pull/1614 * Emit `progress_total` events from `PreTrainedModel.from_pretrained()` by @xenova in https://github.com/huggingface/transformers.js/pull/1615 ## Repository Restructuring Developing a new major version gave us the opportunity to invest in the codebase and tackle long-overdue refactoring efforts. ### PNPM Workspaces Until now, the GitHub repository served as our npm package. This worked well as long as the repository only exposed a single library. However, looking to the future, we saw the need for various sub-packages that depend heavily on the Transformers.js core while addressing different use cases, like library-specific implementations, or smaller utilities that most users don't need but are essential for some. That's why we converted the repository to a monorepo using pnpm workspaces. This allows us to ship smaller packages that depend on `@huggingface/transformers` without the overhead of maintaining separate repositories. ### Modular Class Structure Another major refactoring effort targeted the ever-growing models.js file. In v3, all available models were defined in a single file spanning over 8,000 lines, becoming increasingly difficult to maintain. For v4, we split this into smaller, focused modules with a clear distinction between utility functions, core logic, and model-specific implementations. This new structure improves readability and makes it much easier to add new models. Developers can now focus on model-specific logic without navigating through thousands of lines of unrelated code. ### Examples Repository In v3, many Transformers.js example projects lived directly in the main repository. For v4, we've moved them to a [dedicated repository](https://github.com/huggingface/transformers.js-examples), allowing us to maintain a cleaner codebase focused on the core library. This also makes it easier for users to find and contribute to examples without sifting through the main repository. ### Prettier We updated the Prettier configuration and reformatted all files in the repository. This ensures consistent formatting throughout the codebase, with all future PRs automatically following the same style. No more debates about formatting... Prettier handles it all, keeping the code clean and readable for everyone. ### Standalone Tokenizers.js Library A frequent request from users was to extract the tokenization logic into a separate library, and with v4, that's exactly what we've done. [@huggingface/tokenizers](https://www.npmjs.com/package/@huggingface/tokenizers) is a complete refactor of the tokenization logic, designed to work seamlessly across browsers and server-side runtimes. At just 8.8kB (gzipped) with zero dependencies, it's incredibly lightweight while remaining fully type-safe.
See example code ```javascript import { Tokenizer } from "@huggingface/tokenizers"; // Load from Hugging Face Hub const modelId = "HuggingFaceTB/SmolLM3-3B"; const tokenizerJson = await fetch( `https://huggingface.co/${modelId}/resolve/main/tokenizer.json` ).then(res => res.json()); const tokenizerConfig = await fetch( `https://huggingface.co/${modelId}/resolve/main/tokenizer_config.json` ).then(res => res.json()); // Create tokenizer const tokenizer = new Tokenizer(tokenizerJson, tokenizerConfig); // Tokenize text const tokens = tokenizer.tokenize("Hello World"); // ['Hello', 'Δ World'] const encoded = tokenizer.encode("Hello World"); // { ids: [9906, 4435], tokens: ['Hello', 'Δ World'], ... } ```
This separation keeps the core of Transformers.js focused and lean while offering a versatile, standalone tool that any WebML project can use independently. * Move examples by @xenova in https://github.com/huggingface/transformers.js/pull/1362 * [v4] Refactor models.js into separate per-model modeling files by @nico-martin in https://github.com/huggingface/transformers.js/pull/1498 * V4 move to pnpm workspaces by @nico-martin in https://github.com/huggingface/transformers.js/pull/1504 * [v4] Tokenizers.js migration by @xenova in https://github.com/huggingface/transformers.js/pull/1501 ## New build system We've migrated our build system from Webpack to esbuild, and the results have been incredible. Build times dropped from 2 seconds to just 200 milliseconds, a 10x improvement that makes development iteration significantly faster. Speed isn't the only benefit, though: bundle sizes also decreased by an average of 10% across all builds. The most notable improvement is in transformers.web.js, our default export, which is now 53% smaller, meaning faster downloads and quicker startup times for users. * [v4] Switch build system to use esbuild by @nico-martin in https://github.com/huggingface/transformers.js/pull/1466 ## Improved types We've made several quality-of-life improvements across the library. The type system has been enhanced with dynamic pipeline types that adapt based on inputs, providing better developer experience and type safety. Type Improvements * [v4] Finalize pipeline types by @xenova in https://github.com/huggingface/transformers.js/pull/1505 * Add type definitions for various pipeline outputs in transformers.js by @sroussey in https://github.com/huggingface/transformers.js/pull/1535 * fix: improves typedef of Message by @philnash in https://github.com/huggingface/transformers.js/pull/1525 * added ProgressCallback and ProgressInfo as type exports by @nico-martin in https://github.com/huggingface/transformers.js/pull/1528 ## Bug fixes * fixed backend loading path by @nico-martin in https://github.com/huggingface/transformers.js/pull/1464 * fixed v4 ts errors by @nico-martin in https://github.com/huggingface/transformers.js/pull/1477 * Hot-fix Float16Array usage in node by @xenova in https://github.com/huggingface/transformers.js/pull/1482 * [v4@next] Fix WebGPU validation failed by @xenova in https://github.com/huggingface/transformers.js/pull/1519 * Fix `stopping_criteria` missing from generation pipelines by @xenova in https://github.com/huggingface/transformers.js/pull/1523 * Overdue Whisper fixes by @xenova in https://github.com/huggingface/transformers.js/pull/1594 * Fix custom cache in Node.js when cache returns non-paths (e.g., Response) by @xenova in https://github.com/huggingface/transformers.js/pull/1617 * Include `content-length` header in COS `Response` by @tomayac in https://github.com/huggingface/transformers.js/pull/1572 * Fix: file size missing for cached files by @nico-martin in https://github.com/huggingface/transformers.js/pull/1575 * [deno via CDN] Fix simultaneous multi-session loading (e.g., VLMs) and d support usage for image loading by @xenova in https://github.com/huggingface/transformers.js/pull/1556 * fix: prevent partial file reads during concurrent downloads by @nico-martin in https://github.com/huggingface/transformers.js/pull/1548 * Fix WASM factory blob URL loading (#1527, #1532) by @nico-martin in https://github.com/huggingface/transformers.js/pull/1558 ## Documentation improvements * Fix broken source code links in the Tutorials sections. by @SeanGallen in https://github.com/huggingface/transformers.js/pull/1526 * V4 (docs): add Vercel AI SDK integration with browser-ai/transformers.js package by @xenova in https://github.com/huggingface/transformers.js/pull/1566 * docs: add step-by-step guide for contributing a new model by @nico-martin in https://github.com/huggingface/transformers.js/pull/1560 * Add background-removal pipeline to JSDoc by @xenova in https://github.com/huggingface/transformers.js/pull/1595 * fix(docs): Fix typo in WebGPU documentation by @mattsoulanille in https://github.com/huggingface/transformers.js/pull/1607 ## Miscellaneous improvements * Improve text streamer when `skip_special_tokens: false` by @xenova in https://github.com/huggingface/transformers.js/pull/1520 * Upgrade GitHub Actions for Node 24 compatibility (follow-up) by @xenova in https://github.com/huggingface/transformers.js/pull/1531 * Align audio_spectrogram_transformer name with python library by @xenova in https://github.com/huggingface/transformers.js/pull/1591 * Memoize network calls in model registry by @nico-martin in https://github.com/huggingface/transformers.js/pull/1552 * added no-store to range request #1561 by @nico-martin in https://github.com/huggingface/transformers.js/pull/1564 ## New Contributors * @taronsung made their first contribution in https://github.com/huggingface/transformers.js/pull/1507 * @philnash made their first contribution in https://github.com/huggingface/transformers.js/pull/1525 * @tomayac made their first contribution in https://github.com/huggingface/transformers.js/pull/1572 * @mattsoulanille made their first contribution in https://github.com/huggingface/transformers.js/pull/1607 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.8.1...4.0.0 ## What's new? * Add support for Ministral 3 in https://github.com/huggingface/transformers.js/pull/1474 * Fix Ernie 4.5 naming in https://github.com/huggingface/transformers.js/pull/1473 * Update Supertonic TTS paper + authors in https://github.com/huggingface/transformers.js/pull/1463 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.8.0...3.8.1 # πŸš€ Transformers.js v3.8 β€” SAM2, SAM3, EdgeTAM, Supertonic TTS * Add support for EdgeTAM in https://github.com/huggingface/transformers.js/pull/1454 * Add support for Supertonic TTS in https://github.com/huggingface/transformers.js/pull/1459 **Example:** ```js import { pipeline } from '@huggingface/transformers'; const tts = await pipeline('text-to-speech', 'onnx-community/Supertonic-TTS-ONNX'); const input_text = 'This is really cool!'; const audio = await tts(input_text, { speaker_embeddings: 'https://huggingface.co/onnx-community/Supertonic-TTS-ONNX/resolve/main/voices/F1.bin', }); await audio.save('output.wav'); ``` * Add support for SAM2 and SAM3 (Tracker) in https://github.com/huggingface/transformers.js/pull/1461 * Remove Metaspace add_prefix_space logic in https://github.com/huggingface/transformers.js/pull/1451 * ImageProcessor preprocess uses image_std for fill value by @NathanKolbas in https://github.com/huggingface/transformers.js/pull/1455 ## New Contributors * @NathanKolbas made their first contribution in https://github.com/huggingface/transformers.js/pull/1455 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.7.6...3.8.0 ## What's new? * Fix issue when `temperature=0` and `do_sample=true` by @nico-martin in https://github.com/huggingface/transformers.js/pull/1431 * Fix type errors by @nico-martin in https://github.com/huggingface/transformers.js/pull/1436 * Add support for NanoChat in https://github.com/huggingface/transformers.js/pull/1441 * Add support for Parakeet CTC in https://github.com/huggingface/transformers.js/pull/1440 ## New Contributors * @nico-martin made their first contribution in https://github.com/huggingface/transformers.js/pull/1431 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.7.5...3.7.6 ## What's new? * Add support for GraniteMoeHybrid in https://github.com/huggingface/transformers.js/pull/1426 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.7.4...3.7.5 ## What's new? * Correctly assign logits warpers in `_get_logits_processor` in https://github.com/huggingface/transformers.js/pull/1422 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.7.3...3.7.4 ## What's new? * Unify inference chains in https://github.com/huggingface/transformers.js/pull/1399 * Fix progress tracking bug by @kukudixiaoming in https://github.com/huggingface/transformers.js/pull/1405 * Add support for MobileLLM-R1 (llama4_text) in https://github.com/huggingface/transformers.js/pull/1412 * Add support for VaultGemma in https://github.com/huggingface/transformers.js/pull/1413 ## New Contributors * @kukudixiaoming made their first contribution in https://github.com/huggingface/transformers.js/pull/1405 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.7.2...3.7.3 ## What's new? * Add support for DINOv3 in https://github.com/huggingface/transformers.js/pull/1390 See [here](https://huggingface.co/models?library=transformers.js&other=dinov3&sort=trending) for the full list of supported models. **Example**: Compute image embeddings ```js import { pipeline } from '@huggingface/transformers'; const image_feature_extractor = await pipeline( 'image-feature-extraction', 'onnx-community/dinov3-vits16-pretrain-lvd1689m-ONNX', ); const url = 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png'; const features = await image_feature_extractor(url); console.log(features); ``` Try it out using our [online demo](https://huggingface.co/spaces/webml-community/dinov3-web): https://github.com/user-attachments/assets/7b46f225-1699-467a-9d62-87c0ef497029 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.7.1...3.7.2 ## What's new? * Add support for Arcee in https://github.com/huggingface/transformers.js/pull/1377 * Optimize tensor.slice() by @Honry in https://github.com/huggingface/transformers.js/pull/1381 ## New Contributors * @Honry made their first contribution in https://github.com/huggingface/transformers.js/pull/1381 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.7.0...3.7.1 # πŸš€ Transformers.js v3.7 β€” Voxtral, LFM2, ModernBERT Decoder ## πŸ€– New models This update adds support for 3 new architectures: - [Voxtral](#new-models-voxtral) - [LFM2](#new-models-lfm2) - [ModernBERT Decoder](#new-models-modernbert-decoder)

Voxtral

Voxtral Mini is an enhancement of [Ministral 3B](https://mistral.ai/news/ministraux), incorporating state-of-the-art audio input capabilities while retaining best-in-class text performance. It excels at speech transcription, translation and audio understanding. ONNX weights for Voxtral-Mini-3B-2507 can be found [here](https://huggingface.co/onnx-community/Voxtral-Mini-3B-2507-ONNX). Learn more about Voxtral in the release [blog post](https://mistral.ai/news/voxtral). Try it out with our [online demo](https://huggingface.co/spaces/webml-community/Voxtral-WebGPU): https://github.com/user-attachments/assets/e1b95fe1-461d-4cb9-8fe8-5ec17e6c93f0 **Example**: Audio transcription ```js import { VoxtralForConditionalGeneration, VoxtralProcessor, TextStreamer, read_audio } from "@huggingface/transformers"; // Load the processor and model const model_id = "onnx-community/Voxtral-Mini-3B-2507-ONNX"; const processor = await VoxtralProcessor.from_pretrained(model_id); const model = await VoxtralForConditionalGeneration.from_pretrained( model_id, { dtype: { embed_tokens: "fp16", // "fp32", "fp16", "q8", "q4" audio_encoder: "q4", // "fp32", "fp16", "q8", "q4", "q4f16" decoder_model_merged: "q4", // "q4", "q4f16" }, device: "webgpu", }, ); // Prepare the conversation const conversation = [ { "role": "user", "content": [ { "type": "audio" }, { "type": "text", "text": "lang:en [TRANSCRIBE]" }, ], } ]; const text = processor.apply_chat_template(conversation, { tokenize: false }); const audio = await read_audio("http://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/mlk.wav", 16000); const inputs = await processor(text, audio); // Generate the response const generated_ids = await model.generate({ ...inputs, max_new_tokens: 256, streamer: new TextStreamer(processor.tokenizer, { skip_special_tokens: true, skip_prompt: true }), }); // Decode the generated tokens const new_tokens = generated_ids.slice(null, [inputs.input_ids.dims.at(-1), null]); const generated_texts = processor.batch_decode( new_tokens, { skip_special_tokens: true }, ); console.log(generated_texts[0]); // I have a dream that one day this nation will rise up and live out the true meaning of its creed. ``` Added in https://github.com/huggingface/transformers.js/pull/1373 and https://github.com/huggingface/transformers.js/pull/1375.

LFM2

LFM2 is a new generation of hybrid models developed by [Liquid AI](https://www.liquid.ai/), specifically designed for edge AI and on-device deployment. It sets a new standard in terms of quality, speed, and memory efficiency. The models, which we have converted to ONNX, come in three different sizes: [350M](https://huggingface.co/onnx-community/LFM2-350M-ONNX), [700M](https://huggingface.co/onnx-community/LFM2-700M-ONNX), and [1.2B](https://huggingface.co/onnx-community/LFM2-1.2B-ONNX) parameters. **Example**: Text-generation with LFM2-350M: ```js import { pipeline, TextStreamer } from "@huggingface/transformers"; // Create a text generation pipeline const generator = await pipeline( "text-generation", "onnx-community/LFM2-350M-ONNX", { dtype: "q4" }, ); // Define the list of messages const messages = [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "What is the capital of France?" }, ]; // Generate a response const output = await generator(messages, { max_new_tokens: 512, do_sample: false, streamer: new TextStreamer(generator.tokenizer, { skip_prompt: true, skip_special_tokens: true }), }); console.log(output[0].generated_text.at(-1).content); // The capital of France is Paris. It is a vibrant city known for its historical landmarks, art, fashion, and gastronomy. ``` Added in https://github.com/huggingface/transformers.js/pull/1367 and https://github.com/huggingface/transformers.js/pull/1369.

ModernBERT Decoder

These models form part of the Ettin suite: the first collection of paired encoder-only and decoder-only models trained with identical data, architecture, and training recipes. Ettin enables fair comparisons between encoder and decoder architectures across multiple scales, providing state-of-the-art performance for open-data models in their respective size categories. The list of supported models can be found [here](https://huggingface.co/models?library=transformers.js&other=modernbert-decoder&sort=trending). ```js import { pipeline, TextStreamer } from "@huggingface/transformers"; // Create a text generation pipeline const generator = await pipeline( "text-generation", "onnx-community/ettin-decoder-150m-ONNX", { dtype: "fp32" }, ); // Generate a response const text = "Q: What is the capital of France?\nA:"; const output = await generator(text, { max_new_tokens: 128, streamer: new TextStreamer(generator.tokenizer, { skip_prompt: true, skip_special_tokens: true }), }); console.log(output[0].generated_text); ``` Added in https://github.com/huggingface/transformers.js/pull/1371. ## πŸ› οΈ Other improvements * Add special tokens in text-generation pipeline if tokenizer requires in https://github.com/huggingface/transformers.js/pull/1370 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.6.3...3.7.0 ## What's new? * Bump `@huggingface/jinja` to version 0.5.1 for new chat template functionality in https://github.com/huggingface/transformers.js/pull/1364 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.6.2...3.6.3 ## What's new? * Add support for SmolLM3 in https://github.com/huggingface/transformers.js/pull/1359 SmolLM3 is a 3B parameter language model designed to push the boundaries of small models. It supports 6 languages, advanced reasoning and long context. SmolLM3 is a fully open model that offers strong performance at the 3B–4B scale. **Example:** ```js import { pipeline, TextStreamer } from "@huggingface/transformers"; // Create a text generation pipeline const generator = await pipeline( "text-generation", "HuggingFaceTB/SmolLM3-3B-ONNX", { dtype: "q4f16" }, ); // Define the list of messages const messages = [ { role: "system", content: "You are SmolLM, a language model created by Hugging Face. If asked by the user, here is some information about you: SmolLM has 3 billion parameters and can converse in 6 languages: English, Spanish, German, French, Italian, and Portuguese. SmolLM is a fully open model and was trained on a diverse mix of public datasets./think" }, { role: "user", content: "Solve the equation x^2 - 3x + 2 = 0" }, ]; // Generate a response const output = await generator(messages, { max_new_tokens: 1024, do_sample: false, streamer: new TextStreamer(generator.tokenizer, { skip_prompt: true, skip_special_tokens: true }), }); console.log(output[0].generated_text.at(-1).content); ``` * Add support for ERNIE-4.5 in https://github.com/huggingface/transformers.js/pull/1354 **Example:** ```js import { pipeline, TextStreamer } from "@huggingface/transformers"; // Create a text generation pipeline const generator = await pipeline( "text-generation", "onnx-community/ERNIE-4.5-0.3B-ONNX", { dtype: "fp32" }, // Options: "fp32", "fp16", "q8", "q4", "q4f16" ); // Define the list of messages const messages = [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "What is the capital of France?" }, ]; // Generate a response const output = await generator(messages, { max_new_tokens: 512, do_sample: false, streamer: new TextStreamer(generator.tokenizer, { skip_prompt: true, skip_special_tokens: true }), }); console.log(output[0].generated_text.at(-1).content); // The capital of France is Paris. ``` **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.6.1...3.6.2 ## What's new? * Add support for NeoBERT in https://github.com/huggingface/transformers.js/pull/1350 ```js import { pipeline } from "@huggingface/transformers"; // Create feature extraction pipeline const extractor = await pipeline("feature-extraction", "onnx-community/NeoBERT-ONNX"); // Compute embeddings const text = "NeoBERT is the most efficient model of its kind!"; const embedding = await extractor(text, { pooling: "cls" }); console.log(embedding.dims); // [1, 768] ``` * Improve webworker detection to support ServiceWorker and SharedWorker by @aungKhantPaing in https://github.com/huggingface/transformers.js/pull/1346 * Pin numpy version for scripts by @fidoriel in https://github.com/huggingface/transformers.js/pull/1351 * Fix optional `from_pretrained` types in https://github.com/huggingface/transformers.js/pull/1352 ## New Contributors * @aungKhantPaing made their first contribution in https://github.com/huggingface/transformers.js/pull/1346 * @fidoriel made their first contribution in https://github.com/huggingface/transformers.js/pull/1351 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.6.0...3.6.1 # πŸš€ Transformers.js v3.6 β€” Gemma 3n, Qwen3-Embedding, Llava-Qwen2 - [πŸ€– New models](#new-models) - [Gemma 3n](#new-models-gemma3n) - [Qwen3-Embedding](#new-models-qwen3-embedding) - [Llava-Qwen2](#new-models-llava-qwen2) - [πŸ› οΈ Other improvements](#other-improvements)

πŸ€– New models

Gemma 3n

Gemma 3n, which was announced as a _preview_ during Google I/O, is a model that is designed from the ground up to **run locally** on your hardware. On top of that, it's natively **multimodal**, supporting image, text, audio, and video inputs 🀯 Gemma 3n models have multiple architecture innovations: * They are available in two sizes based on [effective parameters](https://ai.google.dev/gemma/docs/gemma-3n#parameters). While the raw parameter count of this model is 6B, the architecture design allows the model to be run with a memory footprint comparable to a traditional 2B model by offloading low-utilization matrices from the accelerator. * They use a MatFormer architecture that allows nesting sub-models within the [E4B model](https://huggingface.co/google/gemma-3n-E4B-it). We provide one sub-model (this model repository), or you can access a spectrum of custom-sized models using the [Mix-and-Match method](https://goo.gle/gemma3n-matformer-lab). Learn more about these techniques in the [technical blog post](https://developers.googleblog.com/en/introducing-gemma-3n-developer-guide) and the [Gemma documentation](https://ai.google.dev/gemma/docs/gemma-3n). As part of the release, we are releasing ONNX weights for the `gemma-3n-E2B-it` variant ([link](https://huggingface.co/onnx-community/gemma-3n-E2B-it-ONNX)), making it compatible with Transformers.js: > [!WARNING] > Due to the model's large size, we currently only support Node.js, Deno, and Bun execution. > In-browser WebGPU support is actively being worked on, so stay tuned for an update! **Example:** Caption an image ```js import { AutoProcessor, AutoModelForImageTextToText, load_image, TextStreamer, } from "@huggingface/transformers"; // Load processor and model const model_id = "onnx-community/gemma-3n-E2B-it-ONNX"; const processor = await AutoProcessor.from_pretrained(model_id); const model = await AutoModelForImageTextToText.from_pretrained(model_id, { dtype: { embed_tokens: "q8", audio_encoder: "q8", vision_encoder: "fp16", decoder_model_merged: "q4", }, device: "cpu", // NOTE: WebGPU support coming soon! }); // Prepare prompt const messages = [ { role: "user", content: [ { type: "image" }, { type: "text", text: "Describe this image in detail." }, ], }, ]; const prompt = processor.apply_chat_template(messages, { add_generation_prompt: true, }); // Prepare inputs const url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"; const image = await load_image(url); const audio = null; const inputs = await processor(prompt, image, audio, { add_special_tokens: false, }); // Generate output const outputs = await model.generate({ ...inputs, max_new_tokens: 512, do_sample: false, streamer: new TextStreamer(processor.tokenizer, { skip_prompt: true, skip_special_tokens: false, // callback_function: (text) => { /* Do something with the streamed output */ }, }), }); // Decode output const decoded = processor.batch_decode( outputs.slice(null, [inputs.input_ids.dims.at(-1), null]), { skip_special_tokens: true }, ); console.log(decoded[0]); ```
See example output ``` The image is a close-up, slightly macro shot of a cluster of vibrant pink cosmos flowers in full bloom. The flowers are the focal point, with their delicate, slightly ruffled petals radiating outwards. They have a soft, almost pastel pink hue, and their edges are subtly veined. A small, dark-colored bee is actively visiting one of the pink flowers, its body positioned near the center of the bloom. The bee appears to be collecting pollen or nectar. The flowers are attached to slender, brownish-green stems, and some of the surrounding foliage is visible in a blurred background, suggesting a natural outdoor setting. There are also hints of other flowers in the background, including some red ones, adding a touch of contrast to the pink. The lighting in the image seems to be natural daylight, casting soft shadows and highlighting the textures of the petals and the bee. The overall impression is one of delicate beauty and the gentle activity of nature. ```
**Example:** Transcribe audio ```js import { AutoProcessor, AutoModelForImageTextToText, TextStreamer, } from "@huggingface/transformers"; import wavefile from "wavefile"; // Load processor and model const model_id = "onnx-community/gemma-3n-E2B-it-ONNX"; const processor = await AutoProcessor.from_pretrained(model_id); const model = await AutoModelForImageTextToText.from_pretrained(model_id, { dtype: { embed_tokens: "q8", audio_encoder: "q4", vision_encoder: "fp16", decoder_model_merged: "q4", }, device: "cpu", // NOTE: WebGPU support coming soon! }); // Prepare prompt const messages = [ { role: "user", content: [ { type: "audio" }, { type: "text", text: "Transcribe this audio verbatim." }, ], }, ]; const prompt = processor.apply_chat_template(messages, { add_generation_prompt: true, }); // Prepare inputs const url = "https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav"; const buffer = Buffer.from(await fetch(url).then((x) => x.arrayBuffer())); const wav = new wavefile.WaveFile(buffer); wav.toBitDepth("32f"); // Pipeline expects input as a Float32Array wav.toSampleRate(processor.feature_extractor.config.sampling_rate); let audioData = wav.getSamples(); if (Array.isArray(audioData)) { if (audioData.length > 1) { for (let i = 0; i < audioData[0].length; ++i) { audioData[0][i] = (Math.sqrt(2) * (audioData[0][i] + audioData[1][i])) / 2; } } audioData = audioData[0]; } const image = null; const audio = audioData; const inputs = await processor(prompt, image, audio, { add_special_tokens: false, }); // Generate output const outputs = await model.generate({ ...inputs, max_new_tokens: 512, do_sample: false, streamer: new TextStreamer(processor.tokenizer, { skip_prompt: true, skip_special_tokens: false, // callback_function: (text) => { /* Do something with the streamed output */ }, }), }); // Decode output const decoded = processor.batch_decode( outputs.slice(null, [inputs.input_ids.dims.at(-1), null]), { skip_special_tokens: true }, ); console.log(decoded[0]); ```
See example output ``` And so, my fellow Americans, ask not what your country can do for you. Ask what you can do for your country. ```

Qwen3-Embedding

The Qwen3 Embedding model series is the latest proprietary model of the Qwen family, specifically designed for text embedding and ranking tasks. Building upon the dense foundational models of the Qwen3 series, it provides a comprehensive range of text embeddings and reranking models in various sizes (0.6B, 4B, and 8B). This series inherits the exceptional multilingual capabilities, long-text understanding, and reasoning skills of its foundational model. You can run it with Transformers.js as follows: ```js import { pipeline, matmul } from "@huggingface/transformers"; // Create a feature extraction pipeline const extractor = await pipeline( "feature-extraction", "onnx-community/Qwen3-Embedding-0.6B-ONNX", { dtype: "fp32", // Options: "fp32", "fp16", "q8" // device: "webgpu", }, ); function get_detailed_instruct(task_description, query) { return `Instruct: ${task_description}\nQuery:${query}`; } // Each query must come with a one-sentence instruction that describes the task const task = "Given a web search query, retrieve relevant passages that answer the query"; const queries = [ get_detailed_instruct(task, "What is the capital of China?"), get_detailed_instruct(task, "Explain gravity"), ]; // No need to add instruction for retrieval documents const documents = [ "The capital of China is Beijing.", "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.", ]; const input_texts = [...queries, ...documents]; // Extract embeddings for queries and documents const output = await extractor(input_texts, { pooling: "last_token", normalize: true, }); const scores = await matmul( output.slice([0, queries.length]), // Query embeddings output.slice([queries.length, null]).transpose(1, 0), // Document embeddings ); console.log(scores.tolist()); // [ // [ 0.7645590305328369, 0.14142560958862305 ], // [ 0.13549776375293732, 0.599955141544342 ] // ] ```

Llava-Qwen2

Finally, we also added support for Llava models with a Qwen2 text backbone: ```js import { AutoProcessor, AutoModelForImageTextToText, load_image, TextStreamer, } from "@huggingface/transformers"; // Load processor and model const model_id = "onnx-community/FastVLM-0.5B-ONNX"; const processor = await AutoProcessor.from_pretrained(model_id); const model = await AutoModelForImageTextToText.from_pretrained(model_id, { dtype: { embed_tokens: "fp16", vision_encoder: "q4", decoder_model_merged: "q4", }, }); // Prepare prompt const messages = [ { role: "user", content: "Describe this image in detail.", }, ]; const prompt = processor.apply_chat_template(messages, { add_generation_prompt: true, }); // Prepare inputs const url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"; const image = await load_image(url); const inputs = await processor(image, prompt, { add_special_tokens: false, }); // Generate output const outputs = await model.generate({ ...inputs, max_new_tokens: 512, do_sample: false, streamer: new TextStreamer(processor.tokenizer, { skip_prompt: true, skip_special_tokens: false, // callback_function: (text) => { /* Do something with the streamed output */ }, }), }); // Decode output const decoded = processor.batch_decode( outputs.slice(null, [inputs.input_ids.dims.at(-1), null]), { skip_special_tokens: true }, ); console.log(decoded[0]); ```
See here for example output ``` The image depicts a vibrant and colorful scene featuring a variety of flowers and plants. The main focus is on a striking pink flower with a dark center, which appears to be a type of petunia. The petals are a rich, deep pink, and the flower has a classic, slightly ruffled appearance. The dark center of the flower is a contrasting color, likely a deep purple or black, which adds to the flower's visual appeal. In the background, there are several other flowers and plants, each with their unique colors and shapes. To the left, there is a red flower with a bright, vivid hue, which stands out against the pink flower. The red flower has a more rounded shape and a lighter center, with petals that are a lighter shade of red compared to the pink flower. To the right of the pink flower, there is a plant with red flowers, which are smaller and more densely packed. The red flowers are a deep, rich red color, and they have a more compact shape compared to the pink flower. In the foreground, there is a green plant with a few leaves and a few small flowers. The leaves are a bright green color, and the flowers are a lighter shade of green, with a few petals that are slightly open. Overall, the image is a beautiful representation of a garden or natural setting, with a variety of flowers and plants that are in full bloom. The colors are vibrant and the composition is well-balanced, with the pink flower in the center drawing the viewer's attention. ```

πŸ› οΈ Other improvements

* Improve detection & usage of deno/bun in https://github.com/huggingface/transformers.js/pull/1333 * Add eos/last_token pooling in https://github.com/huggingface/transformers.js/pull/1335 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.5.2...3.6.0 ## What's new? * Update paper links to HF papers by @qgallouedec in https://github.com/huggingface/transformers.js/pull/1318 * Allow older (legacy) BPE models to be detected even when the type is not specified in https://github.com/huggingface/transformers.js/pull/1314 * Fix WhisperTextStreamer when `return_timestamps` is true (correctly ignore printing of timestamp tokens) in https://github.com/huggingface/transformers.js/pull/1327 * Improve typescript exports and expose common types in https://github.com/huggingface/transformers.js/pull/1325 ## New Contributors * @qgallouedec made their first contribution in https://github.com/huggingface/transformers.js/pull/1318 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.5.1...3.5.2 ## What's new? * Add support for Qwen3 in https://github.com/huggingface/transformers.js/pull/1300. Example usage: ```js import { pipeline, TextStreamer } from "@huggingface/transformers"; // Create a text generation pipeline const generator = await pipeline( "text-generation", "onnx-community/Qwen3-0.6B-ONNX", { dtype: "q4f16", device: "webgpu" }, ); // Define the list of messages const messages = [ { role: "user", content: "If 5 brog 5 is 1, and 4 brog 2 is 2, what is 3 brog 1?" }, ]; // Generate a response const output = await generator(messages, { max_new_tokens: 1024, do_sample: true, top_k: 20, temperature: 0.7, streamer: new TextStreamer(generator.tokenizer, { skip_prompt: true, skip_special_tokens: true}), }); console.log(output[0].generated_text.at(-1).content); ``` Try out the [online demo](https://huggingface.co/spaces/webml-community/qwen3-webgpu): https://github.com/user-attachments/assets/d5262390-b70b-4310-b4f3-98be0af79cca * Add support for D-FINE in https://github.com/huggingface/transformers.js/pull/1303 Example usage: ```js import { pipeline } from "@huggingface/transformers"; const detector = await pipeline("object-detection", "onnx-community/dfine_s_coco-ONNX"); const image = "https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/cats.jpg"; const output = await detector(image, { threshold: 0.5 }); console.log(output); ``` See list of supported models: https://huggingface.co/models?library=transformers.js&other=d_fine&sort=trending * Introduce global inference chain (+ other WebGPU fixes) in https://github.com/huggingface/transformers.js/pull/1293 * fix: `RawImage.fromURL` error when input file url by @himself65 in https://github.com/huggingface/transformers.js/pull/1288 * [bugfix] tokenizers respect padding: true with non-null max_length by @dwisdom0 in https://github.com/huggingface/transformers.js/pull/1284 ## New Contributors * @himself65 made their first contribution in https://github.com/huggingface/transformers.js/pull/1288 * @dwisdom0 made their first contribution in https://github.com/huggingface/transformers.js/pull/1284 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.5.0...3.5.1 # πŸ”₯ Transformers.js v3.5

πŸ› οΈ Improvements

* Fix error when dtype in config is unset by @hans00 in https://github.com/huggingface/transformers.js/pull/1271 * [audio utils] fix fft_bin_width computation in https://github.com/huggingface/transformers.js/pull/1274 * Fix bad words logits processor in https://github.com/huggingface/transformers.js/pull/1278 * Implement LRU cache for BPE tokenizer in https://github.com/huggingface/transformers.js/pull/1283 * Return buffer instead of file_path if cache unavailable for model loading by @PrafulB in https://github.com/huggingface/transformers.js/pull/1280 * Use custom cache over FSCache if specified by @PrafulB in https://github.com/huggingface/transformers.js/pull/1285 * Support device-level configuration across all devices by @ibelem in https://github.com/huggingface/transformers.js/pull/1276

πŸ€— New contributors

* @PrafulB made their first contribution in https://github.com/huggingface/transformers.js/pull/1280 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.4.2...3.5.0 ## What's new? * Add support for RF-DETR and RT-DETRv2 in https://github.com/huggingface/transformers.js/pull/1260 * Optimize added token split in https://github.com/huggingface/transformers.js/pull/1261 and https://github.com/huggingface/transformers.js/pull/1265 * Support loading local models using relative paths, absolute paths, and model directory in https://github.com/huggingface/transformers.js/pull/1268 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.4.1...3.4.2 ## What's new? * Add support for SNAC (Multi-Scale Neural Audio Codec) in https://github.com/huggingface/transformers.js/pull/1251 * Add support for Metric3D (v1 & v2) in https://github.com/huggingface/transformers.js/pull/1254 * Add support for Gemma 3 text in https://github.com/huggingface/transformers.js/pull/1229. _Note: Only Node.js execution is supported for now._ * Safeguard against background removal pipeline precision issues in https://github.com/huggingface/transformers.js/pull/1255. Thanks to @LuSrodri for reporting the issue! * Allow RawImage to read from all types of supported sources by @BritishWerewolf in https://github.com/huggingface/transformers.js/pull/1244 * Update pipelines.md api docs in https://github.com/huggingface/transformers.js/pull/1256 * Update extension example to use latest version by @fs-eire in https://github.com/huggingface/transformers.js/pull/1213 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.4.0...3.4.1 # πŸš€ Transformers.js v3.4 β€” Background Removal Pipeline, Ultravox DAC, Mimi, SmolVLM2, LiteWhisper. - [πŸ–ΌοΈ Background Removal Pipeline](#new-pipeline) - [πŸ€– New models: Ultravox DAC, Mimi, SmolVLM2, LiteWhisper](#new-models) - [πŸ› οΈ Other improvements](#other-improvements) - [πŸ€— New contributors](#new-contributors)

πŸ–ΌοΈ New Background Removal Pipeline

Removing backgrounds from images is now as easy as: ```js import { pipeline } from "@huggingface/transformers"; const segmenter = await pipeline("background-removal", "onnx-community/BEN2-ONNX"); const output = await segmenter("input.png"); output[0].save("output.png"); // (Optional) Save the image ``` You can find the full list of compatible models [here](https://huggingface.co/models?library=transformers.js&other=background-removal), which will continue to grow in future! πŸ”₯ For more information, check out https://github.com/huggingface/transformers.js/pull/1216.

πŸ€– New models

* Ultravox for audio-text-to-text generation (https://github.com/huggingface/transformers.js/pull/1207). See [here](https://huggingface.co/models?library=transformers.js&other=ultravox) for the list of supported models.
See example usage ```js import { UltravoxProcessor, UltravoxModel, read_audio } from "@huggingface/transformers"; const processor = await UltravoxProcessor.from_pretrained( "onnx-community/ultravox-v0_5-llama-3_2-1b-ONNX", ); const model = await UltravoxModel.from_pretrained( "onnx-community/ultravox-v0_5-llama-3_2-1b-ONNX", { dtype: { embed_tokens: "q8", // "fp32", "fp16", "q8" audio_encoder: "q4", // "fp32", "fp16", "q8", "q4", "q4f16" decoder_model_merged: "q4", // "q8", "q4", "q4f16" }, }, ); const audio = await read_audio("http://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/mlk.wav", 16000); const messages = [ { role: "system", content: "You are a helpful assistant.", }, { role: "user", content: "Transcribe this audio:<|audio|>" }, ]; const text = processor.tokenizer.apply_chat_template(messages, { add_generation_prompt: true, tokenize: false, }); const inputs = await processor(text, audio); const generated_ids = await model.generate({ ...inputs, max_new_tokens: 128, }); const generated_texts = processor.batch_decode( generated_ids.slice(null, [inputs.input_ids.dims.at(-1), null]), { skip_special_tokens: true }, ); console.log(generated_texts[0]); // "I can transcribe the audio for you. Here's the transcription:\n\n\"I have a dream that one day this nation will rise up and live out the true meaning of its creed.\"\n\n- Martin Luther King Jr.\n\nWould you like me to provide the transcription in a specific format (e.g., word-for-word, character-for-character, or a specific font)?" ```
* DAC and Mimi for audio tokenization/neural audio codecs (https://github.com/huggingface/transformers.js/pull/1215). See [here](https://huggingface.co/models?library=transformers.js&other=dac) for the list of supported DAC models and [here](https://huggingface.co/models?library=transformers.js&other=mimi) for the list of supported Mimi models.
See example usage DAC: ```js import { DacModel, AutoFeatureExtractor } from '@huggingface/transformers'; const model_id = "onnx-community/dac_16khz-ONNX"; const model = await DacModel.from_pretrained(model_id); const feature_extractor = await AutoFeatureExtractor.from_pretrained(model_id); const audio_sample = new Float32Array(12000); // pre-process the inputs const inputs = await feature_extractor(audio_sample); { // explicitly encode then decode the audio inputs const encoder_outputs = await model.encode(inputs); const { audio_values } = await model.decode(encoder_outputs); console.log(audio_values); } { // or the equivalent with a forward pass const { audio_values } = await model(inputs); console.log(audio_values); } ``` Mimi: ```js import { MimiModel, AutoFeatureExtractor } from '@huggingface/transformers'; const model_id = "onnx-community/kyutai-mimi-ONNX"; const model = await MimiModel.from_pretrained(model_id); const feature_extractor = await AutoFeatureExtractor.from_pretrained(model_id); const audio_sample = new Float32Array(12000); // pre-process the inputs const inputs = await feature_extractor(audio_sample); { // explicitly encode then decode the audio inputs const encoder_outputs = await model.encode(inputs); const { audio_values } = await model.decode(encoder_outputs); console.log(audio_values); } { // or the equivalent with a forward pass const { audio_values } = await model(inputs); console.log(audio_values); } ```
* SmolVLM2, a lightweight multimodal model designed to analyze image and video content (https://github.com/huggingface/transformers.js/pull/1196). See [here](https://huggingface.co/models?library=onnx&other=smolvlm&sort=trending) for the list of supported models. Usage is identical to SmolVLM. * LiteWhisper for automatic speech recognition (https://github.com/huggingface/transformers.js/pull/1219). See [here](https://huggingface.co/models?library=transformers.js&other=lite-whisper&sort=trending) for the list of supported models. Usage is identical to Whisper.

πŸ› οΈ Other improvements

* Add support for multi-chunk external data files in https://github.com/huggingface/transformers.js/pull/1212 * Fix package export by @fs-eire in https://github.com/huggingface/transformers.js/pull/1161 * Add NFD normalizer in https://github.com/huggingface/transformers.js/pull/1211. Thanks to @adewdev for reporting! * Documentation improvements by @viksit in https://github.com/huggingface/transformers.js/pull/1184 * Optimize conversion script in https://github.com/huggingface/transformers.js/pull/1204 and https://github.com/huggingface/transformers.js/pull/1218 * Use Float16Array instead of Uint16Array for kvcache when available in https://github.com/huggingface/transformers.js/pull/1208

πŸ€— New contributors

* @axrati made their first contribution in https://github.com/huggingface/transformers.js/pull/602 * @viksit made their first contribution in https://github.com/huggingface/transformers.js/pull/1184 * @tangkunyin made their first contribution in https://github.com/huggingface/transformers.js/pull/1203 **Full Changelog**: https://github.com/huggingface/transformers.js/compare/3.3.3...3.4.0