🤗 Datasets 2.0.0

We're happy to announce that our new documentation is available at hf.co/docs/datasets !

Dataset Features

Load a folder of images using the imagefolder dataset loader:
- Add imagefolder dataset by @nateraw in https://github.com/huggingface/datasets/pull/2830
- Faster ImageFolder + add option to drop labels by @mariosasko in https://github.com/huggingface/datasets/pull/3887
Push your image and audio datasets on the Hugging Face Hub with push_to_hub:
- Add support for Audio and Image feature in push_to_hub by @mariosasko in https://github.com/huggingface/datasets/pull/3685
New processing methods for streaming datasets:
- Add IterableDataset.filter by @lhoestq in https://github.com/huggingface/datasets/pull/3826
- Manipulate columns on IterableDataset (rename columns, cast, etc.) by @lhoestq in https://github.com/huggingface/datasets/pull/3862
- Add the new methods to IterableDatasetDict by @lhoestq in https://github.com/huggingface/datasets/pull/3923
And more:
- Add more compression types for to_json by @bhavitvyamalik in https://github.com/huggingface/datasets/pull/3551
- Multi-GPU support for FaissIndex by @rentruewang in https://github.com/huggingface/datasets/pull/3721

Breaking changes

API changes for map and shuffle for datasets loaded in streaming mode:
- Align map when streaming: update instead of overwrite + add missing parameters by @lhoestq in https://github.com/huggingface/datasets/pull/3801
- Align IterableDataset.shuffle with Dataset.shuffle by @lhoestq in https://github.com/huggingface/datasets/pull/3842
Rename GenerateMode to DownloadMode by @albertvillanova in https://github.com/huggingface/datasets/pull/3759
Remove deprecated methods/params (preparation for v2.0) by @mariosasko in https://github.com/huggingface/datasets/pull/3803
Remove deprecated remove_columns param in filter by @mariosasko in https://github.com/huggingface/datasets/pull/3827
Module namespace cleanup for v2.0 by @mariosasko in https://github.com/huggingface/datasets/pull/3875

Dataset Changes

New: CFPB Consumer Complaints by @kayvane1 in https://github.com/huggingface/datasets/pull/3617
New: told-br (brazilian hate speech) by @JAugusto97 in https://github.com/huggingface/datasets/pull/3683
New: electricity load diagram by @kashif in https://github.com/huggingface/datasets/pull/3722
New: MIT Scene Parsing Benchmark by @mariosasko in https://github.com/huggingface/datasets/pull/3607
New: ElkarHizketak v1.0 by @antxa in https://github.com/huggingface/datasets/pull/3780
New: wikitablequestions by @SivilTaram in https://github.com/huggingface/datasets/pull/3870
New: ontonotes_conll by @richarddwang in https://github.com/huggingface/datasets/pull/3853
Update: BnL Historical Newspapers - make the dataset streamable by @albertvillanova in https://github.com/huggingface/datasets/pull/3616
Update: Common voice - add validated partition by @shalymin-amzn in https://github.com/huggingface/datasets/pull/3669
Update: Common Voice - add local paths to audio files by @lhoestq in https://github.com/huggingface/datasets/pull/3736
Update: Common Voice - simplify code by @lhoestq in https://github.com/huggingface/datasets/pull/3817
Update: Natural Questions - add dev-only configuration by @albertvillanova in https://github.com/huggingface/datasets/pull/3699
Update: pubmed - update data url by @albertvillanova in https://github.com/huggingface/datasets/pull/3692
Update: pubmed - make the dataset streamable by @abhi-mosaic in https://github.com/huggingface/datasets/pull/3740
Update: RedCaps - make the dataset streamable by @mariosasko in https://github.com/huggingface/datasets/pull/3737
Update: cats_vs_dogs - update metadata by @albertvillanova in https://github.com/huggingface/datasets/pull/3752
Update: newsroom - update manual download url by @albertvillanova in https://github.com/huggingface/datasets/pull/3779
Update: xcopa - update to new version by @albertvillanova in https://github.com/huggingface/datasets/pull/3810
Update: cats_vs_dogs size by @mariosasko in https://github.com/huggingface/datasets/pull/3878
Fix: sem_eval_2018_task_1 - fix download location by @maxpel in https://github.com/huggingface/datasets/pull/3643
Fix: newsqa - fix unique keys by @albertvillanova in https://github.com/huggingface/datasets/pull/3696
Fix: The Pile datasets - fix host urls by @albertvillanova in https://github.com/huggingface/datasets/pull/3627
Fix: Evidence Infer Treatment - fix dataset script by @albertvillanova in https://github.com/huggingface/datasets/pull/3718
Fix: NewsQA - fix dataset script by @albertvillanova in https://github.com/huggingface/datasets/pull/3734
Fix: head_qa - fix data url by @albertvillanova in https://github.com/huggingface/datasets/pull/3766
Fix: msr_sqa - fix unique keys by @albertvillanova in https://github.com/huggingface/datasets/pull/3771
Fix: reddit_tifu - fix data url by @albertvillanova in https://github.com/huggingface/datasets/pull/3774
Fix: wiki_lingua - fix spanish data file url by @albertvillanova in https://github.com/huggingface/datasets/pull/3806
Fix: beans - fix data urls by @mariosasko in https://github.com/huggingface/datasets/pull/3890
Fix: CRD3 - fix NonMatchingChecksumError by @albertvillanova in https://github.com/huggingface/datasets/pull/3921
Fix: MultiWOZ 2.2 - fix NonMatchingChecksumError by @albertvillanova in https://github.com/huggingface/datasets/pull/3922

Dataset cards

Add code example in wikipedia card by @lhoestq in https://github.com/huggingface/datasets/pull/3678
Fix Multi-News dataset metadata and card by @albertvillanova in https://github.com/huggingface/datasets/pull/3731
Reddit dataset card additions by @anna-kay in https://github.com/huggingface/datasets/pull/3781
Update gigaword card and info by @mariosasko in https://github.com/huggingface/datasets/pull/3775
Reddit dataset card contribution by @anna-kay in https://github.com/huggingface/datasets/pull/3797

Metric Changes

New: FrugalScore by @moussaKam in https://github.com/huggingface/datasets/pull/3674
New: Mahalanobis distance by @JoaoLages in https://github.com/huggingface/datasets/pull/3794
New: mIoU by @NielsRogge in https://github.com/huggingface/datasets/pull/3745
New: MSE and MAE - V2 by @dnaveenr in https://github.com/huggingface/datasets/pull/3874
Fix: METEOR - fix bug due to nltk version by @albertvillanova in https://github.com/huggingface/datasets/pull/3884

Metric cards

Add perplexity to metrics by @emibaylor in https://github.com/huggingface/datasets/pull/3757
Create SQuAD metric README.md by @sashavor in https://github.com/huggingface/datasets/pull/3873
SQuAD v2 metric: create README.md by @sashavor in https://github.com/huggingface/datasets/pull/3879
Update README.md for SQuAD v2 metric by @sashavor in https://github.com/huggingface/datasets/pull/3908
Update README.md for SQuAD metric by @sashavor in https://github.com/huggingface/datasets/pull/3907
Create README.md for WER metric by @sashavor in https://github.com/huggingface/datasets/pull/3898
Create README.md for GLUE by @sashavor in https://github.com/huggingface/datasets/pull/3916

New documentation

Update docs to new frontend/UI by @mishig25 in https://github.com/huggingface/datasets/pull/3690
Image process doc by @stevhliu in https://github.com/huggingface/datasets/pull/3882

General improvements and bug fixes

Better TQDM output by @mariosasko in https://github.com/huggingface/datasets/pull/3654
Prioritize module.builder_kwargs over defaults in TestCommand by @lvwerra in https://github.com/huggingface/datasets/pull/3672
Extend support for streaming datasets that use os.path.relpath by @albertvillanova in https://github.com/huggingface/datasets/pull/3623
Add Fon language tag by @albertvillanova in https://github.com/huggingface/datasets/pull/3620
Remove unnecessary 'r' arg in by @bryant1410 in https://github.com/huggingface/datasets/pull/3661
Fix TestCommand to copy dataset_infos to local dir with only data files by @albertvillanova in https://github.com/huggingface/datasets/pull/3680
Upgrade black to version ~=22.0 by @LysandreJik in https://github.com/huggingface/datasets/pull/3691
Fix streaming for servers not supporting HTTP range requests by @albertvillanova in https://github.com/huggingface/datasets/pull/3689
Pin ElasticSearch by @lhoestq in https://github.com/huggingface/datasets/pull/3701
Raise informative error when loading a save_to_disk dataset by @albertvillanova in https://github.com/huggingface/datasets/pull/3705
Fix ClassLabel to/from dict when passed names_file by @albertvillanova in https://github.com/huggingface/datasets/pull/3695
Fix CI code quality issue by @albertvillanova in https://github.com/huggingface/datasets/pull/3710
Check if indices values in Dataset.select are within bounds by @mariosasko in https://github.com/huggingface/datasets/pull/3719
Pin pandas to avoid bug in streaming mode by @albertvillanova in https://github.com/huggingface/datasets/pull/3725
Use config pandas version in CSV dataset builder by @albertvillanova in https://github.com/huggingface/datasets/pull/3726
Set base path to hub url for canonical datasets by @lhoestq in https://github.com/huggingface/datasets/pull/3709
Fix ValueError message formatting in int2str by @akulchik in https://github.com/huggingface/datasets/pull/3742
Patch all module attributes in its namespace by @albertvillanova in https://github.com/huggingface/datasets/pull/3727
Fix typo in train split name by @albertvillanova in https://github.com/huggingface/datasets/pull/3751
feat: 🎸 generate info if dataset_infos.json does not exist by @severo in https://github.com/huggingface/datasets/pull/3670
Support streaming in size estimation function in push_to_hub by @mariosasko in https://github.com/huggingface/datasets/pull/3732
Expose method and fix param by @severo in https://github.com/huggingface/datasets/pull/3767
Fix HfFileSystem docstring by @lhoestq in https://github.com/huggingface/datasets/pull/3768
process .opus files (for Multilingual Spoken Words) by @polinaeterna in https://github.com/huggingface/datasets/pull/3666
Fix: dataset name is stored in keys by @thomasw21 in https://github.com/huggingface/datasets/pull/3772
Use the same seed to shuffle shards and metadata in streaming mode by @lhoestq in https://github.com/huggingface/datasets/pull/3746
Start removing canonical datasets logic by @lhoestq in https://github.com/huggingface/datasets/pull/3777
Support passing str to iter_files by @albertvillanova in https://github.com/huggingface/datasets/pull/3783
Fix Google Drive URL to avoid Virus scan warning by @albertvillanova in https://github.com/huggingface/datasets/pull/3787
Skip checksum computation if ignore_verifications is True by @mariosasko in https://github.com/huggingface/datasets/pull/3796
Fix error message in CSV loader for newer Pandas versions by @mariosasko in https://github.com/huggingface/datasets/pull/3798
Add data_dir to data_files resolution and misc improvements to HfFileSystem by @mariosasko in https://github.com/huggingface/datasets/pull/3791
Error of writing with different schema, due to nonpreservation of nullability by @richarddwang in https://github.com/huggingface/datasets/pull/3782
Handle Nones in PyArrow struct by @mariosasko in https://github.com/huggingface/datasets/pull/3814
Fix iter_archive getting reset by @lhoestq in https://github.com/huggingface/datasets/pull/3815
Added computer vision tasks by @merveenoyan in https://github.com/huggingface/datasets/pull/3800
Fix typo in doc build yml by @mishig25 in https://github.com/huggingface/datasets/pull/3819
Allow not specifying feature cols other than predictions/references in Metric.compute by @mariosasko in https://github.com/huggingface/datasets/pull/3824
Logo float left by @mishig25 in https://github.com/huggingface/datasets/pull/3836
Pin responses to fix CI for Windows by @albertvillanova in https://github.com/huggingface/datasets/pull/3840
Fix dead dataset scripts creation link. by @dnaveenr in https://github.com/huggingface/datasets/pull/3834
Remove decode: true for image feature in head_qa by @craffel in https://github.com/huggingface/datasets/pull/3805
Update faiss device docstring by @lhoestq in https://github.com/huggingface/datasets/pull/3846
Udpate index.mdx margins by @gary149 in https://github.com/huggingface/datasets/pull/3858
Fix push_to_hub with null images by @lhoestq in https://github.com/huggingface/datasets/pull/3856
Redundant add dataset information and dead link. by @dnaveenr in https://github.com/huggingface/datasets/pull/3852
Update image dataset tags by @mariosasko in https://github.com/huggingface/datasets/pull/3864
Bring back imgs so that forsk dont get broken by @mishig25 in https://github.com/huggingface/datasets/pull/3866
Small typos in How-to-train tutorial. by @lkhphuc in https://github.com/huggingface/datasets/pull/3833
Small doc fixes by @mishig25 in https://github.com/huggingface/datasets/pull/3860
add pandas to env command by @patrickvonplaten in https://github.com/huggingface/datasets/pull/3871
Ignore duplicate keys if ignore_verifications=True by @mariosasko in https://github.com/huggingface/datasets/pull/3868
Update code blocks by @lhoestq in https://github.com/huggingface/datasets/pull/3863
Fix download_mode in dataset_module_factory by @albertvillanova in https://github.com/huggingface/datasets/pull/3876
Fix some shuffle docs by @lhoestq in https://github.com/huggingface/datasets/pull/3885
Fix race condition in doc build by @lhoestq in https://github.com/huggingface/datasets/pull/3891
Add default branch for doc building by @sgugger in https://github.com/huggingface/datasets/pull/3893
[docs] make dummy data creation optional by @lhoestq in https://github.com/huggingface/datasets/pull/3894
Fix code examples indentation by @lhoestq in https://github.com/huggingface/datasets/pull/3895
Align tqdm control/cache control with Transformers by @mariosasko in https://github.com/huggingface/datasets/pull/3897
Fix CLI test checksums by @albertvillanova in https://github.com/huggingface/datasets/pull/3892
Fix Google Drive URL to avoid Virus scan warning in streaming mode by @mariosasko in https://github.com/huggingface/datasets/pull/3843
Change the framework switches to the new syntax by @sgugger in https://github.com/huggingface/datasets/pull/3880

New Contributors

@kayvane1 made their first contribution in https://github.com/huggingface/datasets/pull/3617
@JAugusto97 made their first contribution in https://github.com/huggingface/datasets/pull/3683
@shalymin-amzn made their first contribution in https://github.com/huggingface/datasets/pull/3669
@kashif made their first contribution in https://github.com/huggingface/datasets/pull/3722
@akulchik made their first contribution in https://github.com/huggingface/datasets/pull/3742
@abhi-mosaic made their first contribution in https://github.com/huggingface/datasets/pull/3740
@emibaylor made their first contribution in https://github.com/huggingface/datasets/pull/3757
@anna-kay made their first contribution in https://github.com/huggingface/datasets/pull/3781
@JoaoLages made their first contribution in https://github.com/huggingface/datasets/pull/3794
@mishig25 made their first contribution in https://github.com/huggingface/datasets/pull/3690
@antxa made their first contribution in https://github.com/huggingface/datasets/pull/3780
@dnaveenr made their first contribution in https://github.com/huggingface/datasets/pull/3834
@lkhphuc made their first contribution in https://github.com/huggingface/datasets/pull/3833
@rentruewang made their first contribution in https://github.com/huggingface/datasets/pull/3721
@gary149 made their first contribution in https://github.com/huggingface/datasets/pull/3858
@NielsRogge made their first contribution in https://github.com/huggingface/datasets/pull/3745
@sashavor made their first contribution in https://github.com/huggingface/datasets/pull/3873
@SivilTaram made their first contribution in https://github.com/huggingface/datasets/pull/3870
Document cases for github datasets by @lhoestq in https://github.com/huggingface/datasets/pull/3924
Fix text loader to split only on universal newlines by @albertvillanova in https://github.com/huggingface/datasets/pull/3910
Retry HfApi call inside push_to_hub when 504 error by @albertvillanova in https://github.com/huggingface/datasets/pull/3886

Full Changelog: https://github.com/huggingface/datasets/compare/1.18.3...0.0.0