2.11.0 — Datasets — releases.sh

Important

Use soundfile for mp3 decoding instead of torchaudio by @polinaeterna in https://github.com/huggingface/datasets/pull/5573
- this allows to not have dependencies on pytorch to decode audio files
- this was possible with soundfile 0.12 which bundles libsndfile binaries at a recent version with MP3 support
Deprecated batch_size on Dataset.to_dict()

Datasets Features

Add writer_batch_size for ArrowBasedBuilder by @lhoestq in https://github.com/huggingface/datasets/pull/5565
- allow to specofy the row group / record batch size when you download_and_prepare() a dataset
Experimental support of cloud storage in load_dataset():
- Support cloud storage in load_dataset via fsspec by @dwyatte in https://github.com/huggingface/datasets/pull/5580
- Pass down storage options by @dwyatte in https://github.com/huggingface/datasets/pull/5673
Support PyArrow arrays as column values in from_dict by @mariosasko in https://github.com/huggingface/datasets/pull/5643
Allow direct cast from binary to Audio/Image by @mariosasko in https://github.com/huggingface/datasets/pull/5644
Add column_names to IterableDataset by @patrickloeber in https://github.com/huggingface/datasets/pull/5582
pass the dataset features to the IterableDataset.from_generator function by @Hubert-Bonisseur in https://github.com/huggingface/datasets/pull/5569
add Dataset.to_list by @kyoto7250 in https://github.com/huggingface/datasets/pull/5611

General imrovements and bug fixes

Update csv.py by @XDoubleU in https://github.com/huggingface/datasets/pull/5562
Remove instructions for ffmpeg system package installation on Colab by @polinaeterna in https://github.com/huggingface/datasets/pull/5558
Apply ruff flake8-comprehension checks by @Skylion007 in https://github.com/huggingface/datasets/pull/5549
Fix datasets.load_from_disk, DatasetDict.load_from_disk and Dataset.load_from_disk by @alvarobartt in https://github.com/huggingface/datasets/pull/5529
Add pre-commit config yaml file to enable automatic code formatting by @polinaeterna in https://github.com/huggingface/datasets/pull/5561
Add huggingface_hub version to env cli command by @mariosasko in https://github.com/huggingface/datasets/pull/5578
Do no write index by default when exporting a dataset by @mariosasko in https://github.com/huggingface/datasets/pull/5583
Flatten dataset on the fly in save_to_disk by @mariosasko in https://github.com/huggingface/datasets/pull/5588
Fix sort with indices mapping by @mariosasko in https://github.com/huggingface/datasets/pull/5587
Fix docstring example by @stevhliu in https://github.com/huggingface/datasets/pull/5592
Fix push_to_hub with no dataset_infos by @lhoestq in https://github.com/huggingface/datasets/pull/5598
Don't compute checksums if not necessary in datasets-cli test by @lhoestq in https://github.com/huggingface/datasets/pull/5603
Update README logo by @gary149 in https://github.com/huggingface/datasets/pull/5605
Fix CI by temporarily pinning fsspec < 2023.3.0 by @albertvillanova in https://github.com/huggingface/datasets/pull/5617
Fix archive fs test by @lhoestq in https://github.com/huggingface/datasets/pull/5614
unpin fsspec by @lhoestq in https://github.com/huggingface/datasets/pull/5619
Bump pyarrow to 8.0.0 by @lhoestq in https://github.com/huggingface/datasets/pull/5620
Remove set_access_token usage + fail tests if FutureWarning by @Wauplin in https://github.com/huggingface/datasets/pull/5623
Fix outdated verification_mode values by @polinaeterna in https://github.com/huggingface/datasets/pull/5607
Adding Oracle Cloud to docs by @ahosler in https://github.com/huggingface/datasets/pull/5621
Fix CI: ignore C901 ("some_func" is to complex) in ruff by @polinaeterna in https://github.com/huggingface/datasets/pull/5636
add kwargs to index search by @SaulLu in https://github.com/huggingface/datasets/pull/5628
Less zip false positives by @lhoestq in https://github.com/huggingface/datasets/pull/5640
Allow self as key in Features by @mariosasko in https://github.com/huggingface/datasets/pull/5646
Bump hfh to 0.11.0 by @lhoestq in https://github.com/huggingface/datasets/pull/5642
Support streaming datasets with numpy.load by @albertvillanova in https://github.com/huggingface/datasets/pull/5626
Fix unnecessary dict comprehension by @albertvillanova in https://github.com/huggingface/datasets/pull/5662
Fix CI by temporarily pinning tensorflow < 2.12.0 by @albertvillanova in https://github.com/huggingface/datasets/pull/5664
Copy features by @lhoestq in https://github.com/huggingface/datasets/pull/5652
Improve features decoding in to_iterable_dataset by @lhoestq in https://github.com/huggingface/datasets/pull/5655
Fix fsspec.open when using an HTTP proxy by @bryant1410 in https://github.com/huggingface/datasets/pull/5656
Jax requires jaxlib by @lhoestq in https://github.com/huggingface/datasets/pull/5667
docs: Update num_shards docs to mention num_proc on Dataset and DatasetDict by @connor-henderson in https://github.com/huggingface/datasets/pull/5658
Allow loading/saving of FAISS index using fsspec by @Dref360 in https://github.com/huggingface/datasets/pull/5526
Fix verification_mode when ignore_verifications is passed by @albertvillanova in https://github.com/huggingface/datasets/pull/5683
Release: 2.11.0 by @lhoestq in https://github.com/huggingface/datasets/pull/5684

New Contributors

@XDoubleU made their first contribution in https://github.com/huggingface/datasets/pull/5562
@Skylion007 made their first contribution in https://github.com/huggingface/datasets/pull/5549
@Hubert-Bonisseur made their first contribution in https://github.com/huggingface/datasets/pull/5569
@ahosler made their first contribution in https://github.com/huggingface/datasets/pull/5621
@patrickloeber made their first contribution in https://github.com/huggingface/datasets/pull/5582
@SaulLu made their first contribution in https://github.com/huggingface/datasets/pull/5628
@connor-henderson made their first contribution in https://github.com/huggingface/datasets/pull/5658
@kyoto7250 made their first contribution in https://github.com/huggingface/datasets/pull/5611

Full Changelog: https://github.com/huggingface/datasets/compare/2.10.0...2.11.0