2.8.0

Important

• Removed YAML integer keys from class_label metadata by @albertvillanova in https://github.com/huggingface/datasets/pull/5277
  • From now on, datasets pushed on the Hub and using ClassLabel will use a new YAML model to store the feature types
  • The new model uses strings instead of integers for the ids in label name mapping (e.g. 0 -> "0"). This is due to the Hub limitations. In a few months the Hub may stop allowing users to push the old YAML model.
  • Old versions of datasets are not able to reload datasets pushed with this new model, so we encourage everyone to update.

Datasets Features

• Fix methods using IterableDataset.map that lead to features=None by @alvarobartt in https://github.com/huggingface/datasets/pull/5287
  • Datasets in streaming mode now update their features after column renaming or removal
• Add num_proc to from_csv/generator/json/parquet/text by @lhoestq in https://github.com/huggingface/datasets/pull/5239
  • Use multiprocessing to load multiple files in parallel
• Add features param to IterableDataset.map by @alvarobartt in https://github.com/huggingface/datasets/pull/5311
• Sharded save_to_disk + multiprocessing by @lhoestq in https://github.com/huggingface/datasets/pull/5268
  • Pass num_shards or max_shard_size to ds.save_to_disk() or ds.push_to_hub()
  • Pass num_proc to use multiprocessing.
• Support for decoding Image/Audio types in map when format type is not default one by @mariosasko in https://github.com/huggingface/datasets/pull/5252
• Support torch dataloader without torch formatting for IterableDataset by @lhoestq in https://github.com/huggingface/datasets/pull/5357
  • You can now pass any dataset in streaming mode to a PyTorch DataLoader directly:

  from datasets import load_dataset
  ds = load_dataset("c4", "en", streaming=True, split="train")
  dataloader = DataLoader(ds, batch_size=32, num_workers=4)
  

Docs

• Complete doc migration by @mishig25 in https://github.com/huggingface/datasets/pull/5248

General improvements and bug fixes

• typo by @WrRan in https://github.com/huggingface/datasets/pull/5253
• typo by @WrRan in https://github.com/huggingface/datasets/pull/5254
• remove an unused statement by @WrRan in https://github.com/huggingface/datasets/pull/5257
• fix wrong print by @WrRan in https://github.com/huggingface/datasets/pull/5256
• Fix max_shard_size docs by @lhoestq in https://github.com/huggingface/datasets/pull/5267
• Specify arguments as keywords in librosa.reshape to avoid future errors by @polinaeterna in https://github.com/huggingface/datasets/pull/5266
• Change release procedure to use only pull requests by @albertvillanova in https://github.com/huggingface/datasets/pull/5250
• Warn about checksums by @lhoestq in https://github.com/huggingface/datasets/pull/5279
• Tweak readme by @lhoestq in https://github.com/huggingface/datasets/pull/5210
• Save file name in embed_storage by @lhoestq in https://github.com/huggingface/datasets/pull/5285
• Use correct dataset type in from_generator docs by @mariosasko in https://github.com/huggingface/datasets/pull/5307
• Support streaming datasets with pathlib.Path.with_suffix by @albertvillanova in https://github.com/huggingface/datasets/pull/5294
• Fix xjoin for Windows pathnames by @albertvillanova in https://github.com/huggingface/datasets/pull/5297
• Fix xopen for Windows pathnames by @albertvillanova in https://github.com/huggingface/datasets/pull/5299
• Ci py3.10 by @lhoestq in https://github.com/huggingface/datasets/pull/5065
• Update Overview.ipynb google colab by @lhoestq in https://github.com/huggingface/datasets/pull/5211
• Support xPath for Windows pathnames by @albertvillanova in https://github.com/huggingface/datasets/pull/5310
• Fix description of streaming in the docs by @polinaeterna in https://github.com/huggingface/datasets/pull/5313
• Fix Text sample_by paragraph by @albertvillanova in https://github.com/huggingface/datasets/pull/5319
• [Extract] Place the lock file next to the destination directory by @lhoestq in https://github.com/huggingface/datasets/pull/5320
• Fix loading from HF GCP cache by @lhoestq in https://github.com/huggingface/datasets/pull/5321
  • This was affecting datasets like wikipedia or natural_questions
• Fix docs building for main by @albertvillanova in https://github.com/huggingface/datasets/pull/5328
• Origin/fix missing features error by @eunseojo in https://github.com/huggingface/datasets/pull/5318
• fix: 🐛 pass the token to get the list of config names by @severo in https://github.com/huggingface/datasets/pull/5333
• Clarify imagefolder is for small datasets by @stevhliu in https://github.com/huggingface/datasets/pull/5329
• Close stream in ArrowWriter.finalize before inference error by @mariosasko in https://github.com/huggingface/datasets/pull/5309
• Use same num_proc for dataset download and generation by @mariosasko in https://github.com/huggingface/datasets/pull/5300
• Set IterableDataset.map param batch_size typing as optional by @alvarobartt in https://github.com/huggingface/datasets/pull/5336
• fix: dataset path should be absolute by @vigsterkr in https://github.com/huggingface/datasets/pull/5234
• Clean up DatasetInfo and Dataset docstrings by @stevhliu in https://github.com/huggingface/datasets/pull/5340
• Clean up docstrings by @stevhliu in https://github.com/huggingface/datasets/pull/5334
• Remove tasks.json by @lhoestq in https://github.com/huggingface/datasets/pull/5341
• Support topdown parameter in xwalk by @mariosasko in https://github.com/huggingface/datasets/pull/5308
• Improve use_auth_token docstring and deprecate use_auth_token in download_and_prepare by @mariosasko in https://github.com/huggingface/datasets/pull/5302
• Clean up Loading methods docstrings by @stevhliu in https://github.com/huggingface/datasets/pull/5350
• Clean up remaining Main Classes docstrings by @stevhliu in https://github.com/huggingface/datasets/pull/5349
• Clean up Dataset and DatasetDict by @stevhliu in https://github.com/huggingface/datasets/pull/5344
• Clean up Table class docstrings by @stevhliu in https://github.com/huggingface/datasets/pull/5355
• Raise error for .tar archives in the same way as for .tar.gz and .tgz in _get_extraction_protocol by @polinaeterna in https://github.com/huggingface/datasets/pull/5322
• Clean filesystem and logging docstrings by @stevhliu in https://github.com/huggingface/datasets/pull/5356
• ExamplesIterable fixes by @lhoestq in https://github.com/huggingface/datasets/pull/5366
• Simplify skipping by @Muennighoff in https://github.com/huggingface/datasets/pull/5373
• Release: 2.8.0 by @lhoestq in https://github.com/huggingface/datasets/pull/5375

New Contributors

• @WrRan made their first contribution in https://github.com/huggingface/datasets/pull/5253
• @eunseojo made their first contribution in https://github.com/huggingface/datasets/pull/5318
• @vigsterkr made their first contribution in https://github.com/huggingface/datasets/pull/5234
• @Muennighoff made their first contribution in https://github.com/huggingface/datasets/pull/5373

Full Changelog: https://github.com/huggingface/datasets/compare/2.7.0...2.8.0