Zyda-2: a 5 Trillion Token High-Quality Dataset

Beren Millidge; Paolo Glorioso; Quentin Anthony; Yury Tokpanov

Zyda-2: a 5 Trillion Token High-Quality Dataset

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.06068 v1 pith:6NIHJWXX submitted 2024-11-09 cs.CL cs.AI

Zyda-2: a 5 Trillion Token High-Quality Dataset

Yury Tokpanov , Paolo Glorioso , Quentin Anthony , Beren Millidge This is my paper

classification cs.CL cs.AI

keywords zyda-2datasethigh-qualitytokentrillionavailablebuildclass

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

Share X Bluesky LinkedIn Reddit HN

read the original abstract

In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-the-art for their weight class. We build Zyda-2 by collating high-quality open-source tokens such as FineWeb and DCLM, then distilling them to the highest-quality subset via cross-deduplication and model-based quality filtering. Zyda-2 is released under a permissive open license, and is available at https://huggingface.co/datasets/Zyphra/Zyda-2

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
cs.LG 2026-06 unverdicted novelty 6.0

FlexMoE produces nested pruned subnetworks for MoE LLMs across budgets via channel importance ranking and discrete action learning, plus one mid-budget recovery fine-tune, retaining 99.8% performance at 50% expert par...