Datasets

Agent traces from Claude Code, Pi, Codex and others can now be loaded with load_dataset and parsed to messages for training with TRL. Streaming shuffle now uses multiple input shards by default for better randomization, though this is a breaking change—use max_buffer_input_shards=1 to restore the prior behavior. Also added batch(by_column=...) for robotics datasets, support for Apache Iceberg, TsFile (IoTDB), 3D mesh, and CoNLL formats, plus fixes for Parquet streaming hangs, Lance file handling, and JSON null encoding.

Main bug fixes

• fix: decode Json() values before calling DataFrame.to_json() (#8116) by @Brianzhengca in https://github.com/huggingface/datasets/pull/8122
• Fix: decode JSON type before to_list or to_dict is called by @ItsTania in…

What's Changed

• Support latest torchvision by @lhoestq in https://github.com/huggingface/datasets/pull/8087
• fix regression when loading JSON with one file = one object by @lhoestq in https://github.com/huggingface/datasets/pull/8086

Full Changelog:…

What's Changed

• Fix split_dataset_by_node step by @lhoestq in https://github.com/huggingface/datasets/pull/8081
• Fix docstring of Json.cast_storage by @albertvillanova in https://github.com/huggingface/datasets/pull/8080

Full Changelog:…

What's Changed

• Json type for empty struct by @lhoestq in https://github.com/huggingface/datasets/pull/8074

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.1...4.8.2

What's Changed

• Fix formatted iter arrow double yield by @HaukurPall in https://github.com/huggingface/datasets/pull/8063

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.0...4.8.1

Dataset Features

• Read (and write) from HF Storage Buckets: load raw data, process and save to Dataset Repos by @lhoestq in https://github.com/huggingface/datasets/pull/8064

  from datasets import load_dataset
  # load raw…

Datasets Features

• Add Json() type by @lhoestq in https://github.com/huggingface/datasets/pull/8027
  • JSON Lines files that contain arbitrary JSON objects like tool calling datasets are now supported. When there is a field or subfield containing mixed types (e.g. mix…

Bug fix

• Remove tmp file in push to hub by @lhoestq in https://github.com/huggingface/datasets/pull/8030

Full Changelog: https://github.com/huggingface/datasets/compare/4.6.0...4.6.1

Dataset Features

• Support Image, Video and Audio types in Lance datasets

  • Infer types from lance blobs by @lhoestq in https://github.com/huggingface/datasets/pull/7966

  >>> from datasets import load_dataset
  >>> ds =…

Dataset Features

• Add lance format support by @eddyxu in https://github.com/huggingface/datasets/pull/7913
  • Support for both Lance dataset (including metadata / manifests) and standalone .lance files
  • e.g. with…

Bug fixes

• Fix embed storage nifti by @CloseChoice in https://github.com/huggingface/datasets/pull/7853
• ArXiv -> HF Papers by @qgallouedec in https://github.com/huggingface/datasets/pull/7855
• fix some broken links by @julien-c in…

Bug fixes and improvements

• Better streaming retries (504 and 429) by @lhoestq in https://github.com/huggingface/datasets/pull/7847
• DOC: remove mode parameter in docstring of pdf and video feature by @CloseChoice in…

Dataset Features

• Add nifti support by @CloseChoice in https://github.com/huggingface/datasets/pull/7815

  • Load medical imaging datasets from Hugging Face:

  ds = load_dataset("username/my_nifti_dataset")
  ds["train"][0]  # {"nifti":…

Dataset Features

Enable large scale distributed dataset streaming:

• Keep hffs cache in workers when streaming by @lhoestq in https://github.com/huggingface/datasets/pull/7820
• Retry open hf file by @lhoestq in…

Dataset Features

• Sample without replacement option when interleaving datasets by @radulescupetru in https://github.com/huggingface/datasets/pull/7786

  ds = interleave_datasets(datasets, stopping_strategy="all_exhausted_without_replacement")

*…

What's Changed

• fix iterate nested field by @lhoestq in https://github.com/huggingface/datasets/pull/7775
• Add support for arrow iterable when concatenating or interleaving by @radulescupetru in https://github.com/huggingface/datasets/pull/7771
• fix empty dataset…

Dataset Features

• feat: use content defined chunking by @kszucs in https://github.com/huggingface/datasets/pull/7589
  • Parquet datasets are now Optimized Parquet ! <img width="462"…

New Features

• Add IterableDataset.push_to_hub() by @lhoestq in https://github.com/huggingface/datasets/pull/7595

  # Build streaming data pipelines in a few lines of code !
  from datasets import load_dataset
  
  ds = load_dataset(..., streaming=True)…

Dataset Features

• Enable xet in push to hub by @lhoestq in https://github.com/huggingface/datasets/pull/7552
  • Faster downloads/uploads with Xet storage
  • more info: https://github.com/huggingface/datasets/issues/7526

Other improvements and bug fixes

• Add…