3.3.0

Dataset Features

• Support async functions in map() by @lhoestq in https://github.com/huggingface/datasets/pull/7384

  • Especially useful to download content like images or call inference APIs

  prompt = "Answer the following question: {question}. You should think step by step."
  async def ask_llm(example):
      return await query_model(prompt.format(question=example["question"]))
  ds = ds.map(ask_llm)
  

• Add repeat method to datasets by @alex-hh in https://github.com/huggingface/datasets/pull/7198

  ds = ds.repeat(10)
  

• Support faster processing using pandas or polars functions in IterableDataset.map() by @lhoestq in https://github.com/huggingface/datasets/pull/7370

  • Add support for "pandas" and "polars" formats in IterableDatasets
  • This enables optimized data processing using pandas or polars functions with zero-copy, e.g.

  ds = load_dataset("ServiceNow-AI/R1-Distill-SFT", "v0", split="train", streaming=True)
  ds = ds.with_format("polars")
  expr = pl.col("solution").str.extract("boxed\\{(.*)\\}").alias("value_solution")
  ds = ds.map(lambda df: df.with_columns(expr), batched=True)
  

• Apply formatting after iter_arrow to speed up format -> map, filter for iterable datasets by @alex-hh in https://github.com/huggingface/datasets/pull/7207

  • IterableDatasets with "numpy" format are now much faster

What's Changed

• don't import soundfile in tests by @lhoestq in https://github.com/huggingface/datasets/pull/7340
• minor video docs on how to install by @lhoestq in https://github.com/huggingface/datasets/pull/7341
• Fix typo in arrow_dataset by @AndreaFrancis in https://github.com/huggingface/datasets/pull/7328
• remove filecheck to enable symlinks by @fschlatt in https://github.com/huggingface/datasets/pull/7133
• Webdataset special columns in last position by @lhoestq in https://github.com/huggingface/datasets/pull/7349
• Bump hfh to 0.24 to fix ci by @lhoestq in https://github.com/huggingface/datasets/pull/7350
• fsspec 2024.12.0 by @lhoestq in https://github.com/huggingface/datasets/pull/7352
• changes to MappedExamplesIterable to resolve #7345 by @vttrifonov in https://github.com/huggingface/datasets/pull/7353
• Catch OSError for arrow by @lhoestq in https://github.com/huggingface/datasets/pull/7348
• Remove .h5 from imagefolder extensions by @lhoestq in https://github.com/huggingface/datasets/pull/7374
• Add Pandas, PyArrow and Polars docs by @lhoestq in https://github.com/huggingface/datasets/pull/7382
• Optimized sequence encoding for scalars by @lukasgd in https://github.com/huggingface/datasets/pull/7393
• Update docs by @lhoestq in https://github.com/huggingface/datasets/pull/7395
• Update README.md by @lhoestq in https://github.com/huggingface/datasets/pull/7396
• Release: 3.3.0 by @lhoestq in https://github.com/huggingface/datasets/pull/7398

New Contributors

• @AndreaFrancis made their first contribution in https://github.com/huggingface/datasets/pull/7328
• @vttrifonov made their first contribution in https://github.com/huggingface/datasets/pull/7353
• @lukasgd made their first contribution in https://github.com/huggingface/datasets/pull/7393

Full Changelog: https://github.com/huggingface/datasets/compare/3.2.0...3.3.0