What's new?

🖼️ New task: Image-to-image

This release adds support for image-to-image translation (e.g., super-resolution) with Swin2SR models.

Side-by-side (full)	Animated (zoomed)

As always, you can get started in just a few lines of code!

import { pipeline } from '@xenova/transformers';

let url = 'https://huggingface.co/spaces/jjourney1125/swin2sr/resolve/main/testsets/real-inputs/0855.jpg';
let upscaler = await pipeline('image-to-image', 'Xenova/swin2SR-compressed-sr-x4-48');
let output = await upscaler(url);
// RawImage {
//   data: Uint8Array(12582912) [165, 166, 163, ...],
//   width: 2048,
//   height: 2048,
//   channels: 3
// }

💻 New architectures: TrOCR, Swin2SR, Mistral, and Falcon

We also added support for 4 new architectures, bringing the total up to 57! 🤯

• TrOCR for optical character recognition (OCR).

  import { pipeline } from '@xenova/transformers';
  
  let url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/handwriting.jpg';
  let captioner = await pipeline('image-to-text', 'Xenova/trocr-small-handwritten');
  let output = await captioner(url);
  // [{ generated_text: 'Mr. Brown commented icily.' }]

  

  Added in https://github.com/xenova/transformers.js/pull/375. See here for the list of available models.

• Swin2SR for super-resolution and image restoration.

  import { pipeline } from '@xenova/transformers';
  
  let url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/butterfly.jpg';
  let upscaler = await pipeline('image-to-image', 'Xenova/swin2SR-classical-sr-x2-64');
  let output = await upscaler(url);
  // RawImage {
  //   data: Uint8Array(786432) [ 41, 31, 24,  43, ... ],
  //   width: 512,
  //   height: 512,
  //   channels: 3
  // }

  Added in https://github.com/xenova/transformers.js/pull/381. See here for the list of available models.

• Mistral and Falcon for text-generation. Added in https://github.com/xenova/transformers.js/pull/379. Note: Other than testing models, we haven't yet converted any of the larger (≥7B parameter) models. Stay tuned for more updates on this!

🐛 Bug fixes:

• By default, do not add special tokens at start of text-generation (see commit)
• Fix Firefox bug when displaying progress events while reading file from browser cache in https://github.com/xenova/transformers.js/pull/374. Thanks to @felladrin for reporting this issue!
• Fix text2text-generation pipeline output inconsistency w/ python library in https://github.com/xenova/transformers.js/pull/384

🔨 Minor improvements:

• Upgrade typescript dependency version by @Kit-p in https://github.com/xenova/transformers.js/pull/368
• Improve docs in https://github.com/xenova/transformers.js/pull/385

🤗 New Contributors

• @Kit-p made their first contribution in https://github.com/xenova/transformers.js/pull/368

Full Changelog: https://github.com/xenova/transformers.js/compare/2.7.0...2.8.0