2.18.0

Dataset features

• Make JSON builder support an array of strings by @albertvillanova in https://github.com/huggingface/datasets/pull/6696
• Base parquet batch_size on parquet row group size by @lhoestq in https://github.com/huggingface/datasets/pull/6701
  • Faster cold start for streaming
• Change default compression argument for JsonDatasetWriter by @Rexhaif in https://github.com/huggingface/datasets/pull/6659
• Automatic Conversion for uint16/uint32 to Compatible PyTorch Dtypes by @mohalisad in https://github.com/huggingface/datasets/pull/6660
• fsspec: support fsspec>=2023.12.0 glob changes by @pmrowla in https://github.com/huggingface/datasets/pull/6687
  • Support latest fsspec up to 2024.2.0

General improvements and bug fixes

• Fix for Incorrect ex_iterable used with multi num_worker by @kq-chen in https://github.com/huggingface/datasets/pull/6582
  • Previously using PyTorch DDP and num_workers could lead to incorrect shards assignments to workers and cause errors
• Fix imagefolder dataset url by @mariosasko in https://github.com/huggingface/datasets/pull/6683
• Improve error message for gated datasets on load by @lewtun in https://github.com/huggingface/datasets/pull/6684
• Updated Quickstart Notebook link by @Codeblockz in https://github.com/huggingface/datasets/pull/6685
• Update the print message for chunked_dataset in process.mdx by @gzbfgjf2 in https://github.com/huggingface/datasets/pull/6693
• Faster xlistdir by @mariosasko in https://github.com/huggingface/datasets/pull/6698
• Update GitHub Actions to Node 20 by @albertvillanova in https://github.com/huggingface/datasets/pull/6682
• Update release instructions by @albertvillanova in https://github.com/huggingface/datasets/pull/6681
• Pass through information about location of cache directory. by @stridge-cruxml in https://github.com/huggingface/datasets/pull/6677
• Allow SplitDict setitem to replace existing SplitInfo by @lhoestq in https://github.com/huggingface/datasets/pull/6665
• Update ruff by @lhoestq in https://github.com/huggingface/datasets/pull/6706
• Silence ruff deprecation messages by @mariosasko in https://github.com/huggingface/datasets/pull/6707
• fix: show correct package name to install biopython by @BioGeek in https://github.com/huggingface/datasets/pull/6662
• Fix data_files when passing data_dir by @lhoestq in https://github.com/huggingface/datasets/pull/6705
• Release: 2.18.0 by @lhoestq in https://github.com/huggingface/datasets/pull/6708

New Contributors

• @Codeblockz made their first contribution in https://github.com/huggingface/datasets/pull/6685
• @gzbfgjf2 made their first contribution in https://github.com/huggingface/datasets/pull/6693
• @stridge-cruxml made their first contribution in https://github.com/huggingface/datasets/pull/6677
• @pmrowla made their first contribution in https://github.com/huggingface/datasets/pull/6687
• @BioGeek made their first contribution in https://github.com/huggingface/datasets/pull/6662
• @Rexhaif made their first contribution in https://github.com/huggingface/datasets/pull/6659
• @mohalisad made their first contribution in https://github.com/huggingface/datasets/pull/6660
• @kq-chen made their first contribution in https://github.com/huggingface/datasets/pull/6582

Full Changelog: https://github.com/huggingface/datasets/compare/2.17.1...2.18.0