npx @buildinternet/releases get datasetsDatasets 5.1.0 adds the Vortex columnar format, Harbor RL environments, and biological formats including FASTA, FASTQ, GenBank, PDB, and mmCIF with BioSequence/BioStructure decoding. Fixes include an archive path check that allowed sibling-directory escapes, disallowed HDF5 external links, and an infinite-repetition bug in empty iterable datasets.
Fixed a symlink-following arbitrary file write in archive extraction and a path traversal via metadata file_name in folder-based builders. Also fixed batch(by_column=...) crashing after shard/shuffle/split, CSV loader issues on pandas 2.0-2.2, and more than 20 other bugs across Arrow, JSON, and dataset card handling.
Agent traces from Claude Code, Pi, Codex and others can now be loaded with load_dataset and parsed to messages for training with TRL. Streaming shuffle now uses multiple input shards by default for better randomization, though this is a breaking change—use max_buffer_input_shards=1 to restore the prior behavior. Also added batch(by_column=...) for robotics datasets, support for Apache Iceberg, TsFile (IoTDB), 3D mesh, and CoNLL formats, plus fixes for Parquet streaming hangs, Lance file handling, and JSON null encoding.
Full Changelog:…
Full Changelog:…
Full Changelog: https://github.com/huggingface/datasets/compare/4.8.1...4.8.2
Full Changelog: https://github.com/huggingface/datasets/compare/4.8.0...4.8.1
Read (and write) from HF Storage Buckets: load raw data, process and save to Dataset Repos by @lhoestq in https://github.com/huggingface/datasets/pull/8064
from datasets import load_dataset
# load raw…Json() type by @lhoestq in https://github.com/huggingface/datasets/pull/8027
Full Changelog: https://github.com/huggingface/datasets/compare/4.6.0...4.6.1
Support Image, Video and Audio types in Lance datasets
>>> from datasets import load_dataset
>>> ds =…Add nifti support by @CloseChoice in https://github.com/huggingface/datasets/pull/7815
ds = load_dataset("username/my_nifti_dataset")
ds["train"][0] # {"nifti":…Enable large scale distributed dataset streaming:
Sample without replacement option when interleaving datasets by @radulescupetru in https://github.com/huggingface/datasets/pull/7786
ds = interleave_datasets(datasets, stopping_strategy="all_exhausted_without_replacement")*…
