pith. sign in

arxiv: 2506.14111 · v2 · pith:Z5QZSGALnew · submitted 2025-06-17 · 💻 cs.CL · cs.AI· cs.LG

Essential-Web v1.0: 24T tokens of organized web data

Essential AI: Andrew Hojel , Michael Pust , Tim Romanski , Yash Vanjani , Ritvik Kapila , Mohit Parmar , Adarsh Chaluvaraju , Alok Tripathy
show 15 more authors
Anil Thomas Ashish Tanwer Darsh J Shah Ishaan Shah Karl Stratos Khoi Nguyen Kurt Smith Michael Callahan Peter Rushton Philip Monk Platon Mazarakis Saad Jamal Saurabh Srivastava Somanshu Singla Ashish Vaswani
This is my paper
classification 💻 cs.CL cs.AIcs.LG
keywords datadatasetsessential-webhuggingfacetaxonomyachievesacquireagreement
0
0 comments X
read the original abstract

Data plays the most prominent role in how language models acquire skills and knowledge. The lack of massive, well-organized pre-training datasets results in costly and inaccessible data pipelines. We present Essential-Web v1.0, a 24-trillion-token dataset in which every document is annotated with a twelve-category taxonomy covering topic, format, content complexity, and quality. Taxonomy labels are produced by EAI-Distill-0.5b, a fine-tuned 0.5b-parameter model that achieves an annotator agreement within 3% of Qwen2.5-32B-Instruct. With nothing more than SQL-style filters, we obtain competitive web-curated datasets in math (-8.0% relative to SOTA), web code (+14.3%), STEM (+24.5%) and medical (+8.6%). Essential-Web v1.0 is available on HuggingFace: https://huggingface.co/datasets/EssentialAI/essential-web-v1.0

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Scaling Laws for Mixture Pretraining Under Data Constraints

    cs.LG 2026-05 conditional novelty 7.0

    Repetition-aware scaling laws show scarce target data in pretraining mixtures can be repeated 15-20 times optimally, with the best count depending on data size, compute, and model scale.

  2. Unlocking Latent Value: Taxonomy-Guided Recovery of High-Performing Data from Low-Tier Web Corpora

    cs.CL 2026-06 unverdicted novelty 6.0

    A multi-dimensional taxonomy filtering approach recovers high-performing data from deprioritized web corpora, with filtered low-tier subsets outperforming unfiltered top-tier data on reasoning and coding benchmarks.

  3. Scaling Laws for Mixture Pretraining Under Data Constraints

    cs.LG 2026-05 unverdicted novelty 6.0

    Empirical study shows mixture pretraining tolerates higher target data repetition than single-source training, with a new repetition-aware scaling law enabling principled mixture selection based on data size, compute,...