3.2.0

Dataset Features

• Faster parquet streaming + filters with predicate pushdown by @lhoestq in https://github.com/huggingface/datasets/pull/7309
  • Up to +100% streaming speed
  • Fast filtering via predicate pushdown (skip files/row groups based on predicate instead of downloading the full data), e.g.

    from datasets import load_dataset
    filters = [('date', '>=', '2023')]
    ds = load_dataset("HuggingFaceFW/fineweb-2", "fra_Latn", streaming=True, filters=filters)
    

Other improvements and bug fixes

• fix conda release worlflow by @lhoestq in https://github.com/huggingface/datasets/pull/7272
• Add link to video dataset by @NielsRogge in https://github.com/huggingface/datasets/pull/7277
• Raise error for incorrect JSON serialization by @varadhbhatnagar in https://github.com/huggingface/datasets/pull/7273
• support for custom feature encoding/decoding by @alex-hh in https://github.com/huggingface/datasets/pull/7284
• update load_dataset doctring by @lhoestq in https://github.com/huggingface/datasets/pull/7301
• Let server decide default repo visibility by @Wauplin in https://github.com/huggingface/datasets/pull/7302
• fix: update elasticsearch version by @ruidazeng in https://github.com/huggingface/datasets/pull/7300
• Fix typing in iterable_dataset.py by @lhoestq in https://github.com/huggingface/datasets/pull/7304
• Updated inconsistent output in documentation examples for ClassLabel by @sergiopaniego in https://github.com/huggingface/datasets/pull/7293
• More docs to from_dict to mention that the result lives in RAM by @lhoestq in https://github.com/huggingface/datasets/pull/7316
• Release: 3.2.0 by @lhoestq in https://github.com/huggingface/datasets/pull/7317

New Contributors

• @ruidazeng made their first contribution in https://github.com/huggingface/datasets/pull/7300
• @sergiopaniego made their first contribution in https://github.com/huggingface/datasets/pull/7293

Full Changelog: https://github.com/huggingface/datasets/compare/3.1.0...3.2.0