Pith. sign in

REVIEW 4 cited by

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.03915 v1 pith:EVAP7EY7 submitted 2023-03-07 cs.CL cs.AI

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Hugo Lauren\c{c}on , Lucile Saulnier , Thomas Wang , Christopher Akiki , Albert Villanova del Moral , Teven Le Scao , Leandro Von Werra , Chenghao Mou
show 46 more authors
Eduardo Gonz\'alez Ponferrada Huu Nguyen J\"org Frohberg Mario \v{S}a\v{s}ko Quentin Lhoest Angelina McMillan-Major Gerard Dupont Stella Biderman Anna Rogers Loubna Ben allal Francesco De Toni Giada Pistilli Olivier Nguyen Somaieh Nikpoor Maraim Masoud Pierre Colombo Javier de la Rosa Paulo Villegas Tristan Thrush Shayne Longpre Sebastian Nagel Leon Weber Manuel Mu\~noz Jian Zhu Daniel Van Strien Zaid Alyafeai Khalid Almubarak Minh Chien Vu Itziar Gonzalez-Dios Aitor Soroa Kyle Lo Manan Dey Pedro Ortiz Suarez Aaron Gokaslan Shamik Bose David Adelani Long Phan Hieu Tran Ian Yu Suhas Pai Jenny Chim Violette Lepercq Suzana Ilic Margaret Mitchell Sasha Alexandra Luccioni Yacine Jernite
This is my paper
classification cs.CL cs.AI
keywords multilingualbigsciencecorpuslargelanguagedatadatasetlarge-scale
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs

    cs.SE 2025-09 unverdicted novelty 7.0

    Clotho ranks LLM test inputs by failure likelihood using pre-generation hidden states and GMMs, achieving 0.716 ROC-AUC after labeling 5.4% of inputs on average across eight tasks and three models, with transfer to pr...

  2. Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm

    cs.CR 2025-09 conditional novelty 7.0

    A trigger-tag watermark embedded by fine-tuning lets modified LLMs mark their own phishing outputs for cheap detection.

  3. Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs

    cs.SE 2025-09 conditional novelty 6.0

    Clotho ranks LLM test inputs by failure likelihood using a Gaussian mixture model over pre-generation hidden states of passing inputs, reaching 0.716 ROC-AUC with 5.4% labelling and transferring from open-weight to pr...

  4. Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining

    cs.CL 2025-09 unverdicted novelty 6.0

    Sparse crosscoders on LLM checkpoint triplets track emergence, maintenance, and discontinuation of linguistic features during pretraining via a new RelIE metric.