Pith. sign in

REVIEW 6 major objections 5 minor 54 references

Jointly optimizing quality, redundancy, and diversity in web-scale pretraining data beats quality-only filtering for language models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:35 UTC pith:XBUMV42O

load-bearing objection CuraWeb is a serious, well-engineered curation pipeline, but the headline 1.82% gain rests on a single unseeded run with an unspecified 200B sampling protocol—worth refereeing, not desk-rejecting. the 6 major comments →

arxiv 2607.22662 v1 pith:XBUMV42O submitted 2026-06-29 cs.AI

CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

Peiguang Li , Yongwei Zhou , Juncheng Diao , Yuchun Fan , Jian Yang , Jianxiao Yang , Zhongda Su , Shuguang Jiao
show 7 more authors
Xiao Wei Zhiye Zou Gan Dong Zhizhao Zeng Rongxiang Weng Jingang Wang Xunliang Cai
This is my paper
classification cs.AI
keywords data curationpretraining corpusLLM pretrainingquality-diversity tradeoffdeduplicationopen-web crawlmulti-objective optimizationlong-tail knowledge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the dominant approach to pretraining-data curation—aggressively filtering open-web text with a single quality classifier—sacrifices long-tail knowledge and domain diversity, and that a pipeline which jointly optimizes quality, redundancy, and diversity yields a strictly better training corpus. To support this, the authors construct CuraWeb, a 2-trillion-token English corpus from twelve years of open-web crawls, using domain-adaptive rule filtering, soft semantic deduplication, and a quality-diversity sampler. At a fixed 200-billion-token budget, a 3-billion-parameter model trained on a CuraWeb subset averages 48.07% across ten benchmarks, beating the strongest comprehensive baseline by 1.82 absolute points, with the largest gains on reasoning and knowledge-intensive tasks. If correct, this shifts the design goal of web-scale curation from maximizing per-document quality to balancing quality with coverage.

Core claim

The paper's central discovery is that the three objectives curation pipelines usually treat as separate stages—quality, redundancy removal, and diversity—should be optimized jointly under shared signals, and that doing so recovers data that aggressive quality filters discard. The authors show that uniform heuristic thresholds calibrated on general web text systematically purge STEM documents, and that hard-threshold semantic deduplication over-filters specialized technical clusters that naturally share vocabulary. Their framework annotates every document with nine quality scores and a two-level domain label (26 coarse, 105 fine categories), then uses these signals to relax rules per domain w

What carries the argument

The load-bearing mechanism is a multi-signal data-understanding system: a lightweight multi-task transformer that scores each document on eight quality dimensions plus safety and assigns a domain class from a 26/105 taxonomy. These signals drive every pipeline stage—domain-aware rule thresholds with an automatic bypass for math, STEM, and code documents; a soft semantic deduplication that accumulates weighted similarity-edge penalties across a cluster graph instead of applying a single hard cosine threshold; and a power sampler that fuses a content-value score (linear sum of knowledge, education, helpfulness, reasoning) with a diversity score (product of intra-cluster compactness and inter-c

Load-bearing premise

The entire superiority claim rests on a single 200B-token, 3B-parameter training run per corpus, with no reported error bars, multiple seeds, or benchmark-contamination analysis; if the 200B subset is not representative of the full 2T corpus or the evaluation benchmarks leaked into filter or sampler decisions, the 1.82-point advantage may not generalize.

What would settle it

Compute how much of the 200B-token CuraWeb subset consists of documents that overlap, even approximately, with the ten evaluation benchmarks' test prompts; if the retained subset contains substantially more benchmark-like text than the baselines' subsets do, the headline margin is contamination rather than evidence of better curation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If joint optimization is correct, web-scale corpora can be made substantially richer without extra compute by recovering long-tail STEM and professional content that uniform filters reject.
  • The 1.8- to 2.0-point average advantage over the strongest baseline persists when training scales from 200B to 1T tokens, suggesting the diversity gains do not saturate early.
  • Soft semantic deduplication cuts false-positive removal in the highest similarity interval from roughly 37.5% to 28%, preserving millions of technical documents that hard thresholds would delete.
  • Domain-aware rule filtering keeps 87.3% of documents versus 32.6% under uniform thresholds while maintaining high recall of quality documents, so the same token budget contains far more usable data.
  • The final corpus at equal volume shows the highest unique n-gram count among compared corpora and domain entropy on par with the broadest baselines, indicating diversity is maintained without sacrificing quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported gains come from a single training run per corpus with no error bars; re-running with multiple seeds and a randomly drawn 200B subset would reveal whether the 1.82-point margin is stable or a selection artifact.
  • Because the quality annotations are distilled from a proprietary teacher model, the 'objective' value scores inherit that teacher's stylistic preferences; the pipeline may over-weight textbook-like content and under-weight genuinely novel or unconventional writing.
  • The power sampler explicitly upweights education, health, and programming content; this could make the corpus stronger on academic benchmarks while reducing performance on open-ended or creative tasks that require different text distributions.
  • A natural test of the joint-optimization thesis is to hold the sampler fixed and vary only the diversity weight α; if benchmark scores are flat across α, the diversity term is not doing the work claimed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes CuraWeb, a web-scale pretraining corpus built from Common Crawl with a pipeline that jointly optimizes quality, redundancy, and diversity. The pipeline comprises dual-track cleaning (domain-adapted rule filters plus a 300M multi-task model distilled from GPT-4o), hybrid deduplication (MinHash-LSH plus a soft semantic deduplication with weighted voting), and a power-sampling stage that balances content value and diversity. The authors claim a 2T-token corpus and report that a 3B model trained on a 200B-token subset achieves 48.07% average on 10 benchmarks, outperforming DCLM by 1.82% absolute and maintaining a similar margin when scaled to 1T tokens. Extensive analyses of domain coverage, entropy, lexical diversity, and quality scores are provided, along with ablations of filtering, deduplication, and sampling components.

Significance. If the central claim holds, CuraWeb would be a meaningful contribution: it addresses a recognized quality–diversity tradeoff in web-scale curation, introduces a soft semantic deduplication mechanism that is more principled than fixed hard thresholds, and provides a large-scale corpus with a comparatively balanced domain distribution. The paper also ships a detailed pipeline description, component ablations, and extensive diversity/quality analyses. However, the headline result rests on a single unseeded training run and an underspecified subset selection protocol; several internal contradictions in the analysis sections undermine the supporting evidence. The underlying ideas and the scale of the resource are potentially valuable, but the empirical support in the current version is incomplete.

major comments (6)
  1. [§3.6, Eq. (3); §4.1] The protocol for drawing the 200B-token training subset from the 2T CuraWeb corpus is not specified. Eq. (3) defines power-sampling weights with hyperparameters n, k, a, and α, but only α=0.6 is reported; the paper does not state whether the 200B set is a random sample from that distribution, a top-scoring slice, or an independent re-run of the sampler. Baseline subsets (DCLM, FineWeb-Edu, etc.) are likewise unspecified. Because the headline 1.82% advantage is measured on these subsets, the corpus-level claim is untestable without this information. Please specify the exact selection procedure, release subsetting code, and evaluate sensitivity to the choice.
  2. [§4.2, Table 3] The principal result is based on a single unseeded training run per corpus, with no error bars, multiple seeds, or significance tests. Given the small average margin (1.82%) and sub-1% differences on several benchmarks (e.g., HellaSwag, PIQA, SCIQ), the state-of-the-art claim may not be robust to training stochasticity. At minimum, report three seeds with mean and standard deviation, or confidence intervals; if that is computationally infeasible, state the limitation explicitly and temper the claim.
  3. [§5.2, Table 5] The text states that CuraWeb's unique n-gram count (5.13M) is 'the highest among all compared baselines,' but Table 5 reports Dolma3 with 5,665,350 unique n-grams, which is larger. This internal contradiction invalidates the stated conclusion and the surrounding paragraph's argument about lexical coverage. Please correct either the text or the table, and re-run the analysis if necessary.
  4. [§5.3, Table 6] The text claims CuraWeb 'outperforms all external datasets in Knowledge (1.164) and PracticalHelpfulness (0.751),' but Table 6 lists FineWeb-Edu with Knowledge 1.387, which exceeds CuraWeb's 1.164. The sentence is factually incorrect as written. Additionally, the composite score for FineWeb-Edu (0.610) is higher than CuraWeb's (0.521); while the text acknowledges this, the prior claim needs correction and the discussion should be revised accordingly.
  5. [§3.3, §3.5; Table 2] The quality annotation model is a 300M student distilled from GPT-4o, and the deduplication false-positive evaluation in Table 2 also uses GPT-4o as ground truth. This means the quality and redundancy signals are aligned with GPT-4o's judgments, which is a potential circularity when claiming corpus superiority. This is not automatically fatal, but it should be acknowledged as a limitation and ideally supplemented with human evaluation on a random sample. In addition, the sampling method is explicitly based on the unpublished SampleMix reference [1], which shares overlapping authors; to make the pipeline reproducible, please provide a self-contained description of the sampling algorithm and its hyperparameters.
  6. [§4, §4.1] No contamination analysis is reported for the 10 evaluation benchmarks. Because the corpus is derived from Common Crawl and several thresholds may have been co-developed with benchmark evaluation, the observed improvements may partly reflect benchmark overlap or unintentional tuning. Please report n-gram overlap between the training subset and the benchmark prompts/answers, and clarify whether any filtering or sampling parameters were chosen based on the evaluation benchmarks.
minor comments (5)
  1. [Table 1 vs. Appendix D] Table 1 lists 'word_count≤50' as the original rule, but Appendix D describes Gopher's rule as a minimum word count of ≥50. The direction of relaxation (≥50 to ≥25) should be stated consistently in both places to avoid confusion.
  2. [§4.2, scaling paragraph] The text contains a placeholder citation: 'Prior studies [?] have demonstrated...' This reference needs to be filled in.
  3. [§3.5, Appendix E] The cross-reference 'details see in Appendix 6' is incorrect; the soft deduplication algorithm is in Appendix E. Please update the reference.
  4. [General] There are several typos: 'CURAWEByields' missing space, 'Doamin' in Table 4 header, 'strategey' in reference [1], and inconsistent capitalization of 'CuraWeb' vs. 'CURAWEB' throughout. A final proofreading pass is needed.
  5. [Figure 2] The caption notes that safety_score is excluded from the heatmap due to scale variance, but safety_score is used in the model filter (§3.4.2). Please clarify how the binary scale is handled in the filtering thresholds, since this affects the interpretation of Figure 2.

Circularity Check

2 steps flagged

External benchmarks keep the central claim independent; internal quality and dedup validations are self-referential.

specific steps
  1. self definitional [Section 3.6 (Data Sampling, Eq. 2) and Section 5.3 (Corpus Quality Assessment)]
    "we define the content value qi by aggregating value-oriented dimensions—including knowledge_score, education_score, helpfulness_score, and reasoning_score via linear summation... Within our pipeline, the overall quality score increases monotonically across stages ... finally reaching 0.521 after importance sampling—the highest among all internal pipeline variants, indicating that each stage contributes positively to data quality."

    Equation (2) sets the sampling weight s_i = α·q_i + (1−α)·d_i with the same q_i, and Eq. (3) over-samples documents with high s_i. Section 5.3 then measures 'quality' with the same annotation model/value scores and reports that they rise after sampling. The reported improvement is the optimization objective itself; it is a selection artifact, not an independent validation. The external benchmark results are independent, so this is partial, not central, circularity.

  2. fitted input called prediction [Section 3.5 (Soft Deduplication Algorithm and Table 2)]
    "Each similarity edge is assigned a weighted penalty according to its similarity interval—empirically determined based on observed false positive rates—e.g., 7 points for [0.98,0.99) and 8 points for [0.99,1.00] . ... Table 2 reports the complete per-interval false positive rates before and after applying soft deduplication, along with the empirically determined interval weights used in the scoring configuration."

    The interval weights are tuned to the observed false-positive rates of the same GPT-4o-annotated similarity benchmark, and Table 2 then reports FPR reductions on that same benchmark. The claimed reduction (e.g., 37.5% to 28.03%) is an in-sample evaluation of a parameter fit to the metric, so it cannot stand as an independent measure of deduplication precision. It is secondary to the main downstream result.

full rationale

The paper's headline result—48.07% average on 10 external benchmarks at 200B tokens and the 1.82% margin over DCLM—is supported by external evaluation, so the central claim is not circular. The main results use fixed public benchmarks and a controlled 3B training setup. However, two internal validation claims are self-referential: (1) the quality analysis in §5.3 reports increases in the same value score the sampler is explicitly maximizing (Eqs. 2–3), so that particular 'quality improvement' is guaranteed by selection; (2) the soft-deduplication FPR table reports improvements on the same GPT-4o-annotated benchmark used to set the interval weights. The sampling strategy is also attributed to 'SampleMix framework [1]', an unpublished reference with overlapping authors; this is a self-citation concern, but because the central result is externally benchmarked it is not load-bearing circularity. Overall, the central derivation is independent; the circularity is confined to internal, non-central evaluations.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The central claim rests on several calibrated thresholds and unreported hyperparameters (rule thresholds, model-filter thresholds, soft-dedup weights, power-sampling n/k/a). The pipeline also assumes GPT-4o annotation quality and a particular diversity proxy without independent validation. No new physical or conceptual entities are introduced.

free parameters (6)
  • rule filter thresholds = alphabetic_char_ratio <0.26; symbol_to_word_ratio >0.38; word_count min 25
    Recalibrated in §3.4.1 Table 1 to preserve STEM content; chosen by hand based on measured over-filtering risk.
  • model filter per-dimension thresholds = not specified
    Empirically determined thresholds applied to writing/coherence/completeness/safety scores in §3.4.2; values are not given.
  • soft dedup interval weights = w=2..8 per similarity interval, target=10
    Weights in Table 2 are explicitly 'empirically determined based on observed false positive rates' using a GPT-4o-annotated benchmark.
  • power sampling hyperparameters = n, k, a unreported
    Eq (3) defines p_i with exponent n, sigmoid steepness k, and gate center a; no values are given in the paper.
  • sampling mix alpha = 0.6
    Eq (2) sets quality-diversity balance parameter α=0.6 with no sensitivity analysis.
  • K-means cluster count = ~49,000
    Diversity scoring in §3.6 relies on K≈49,000 clusters; no justification or sensitivity analysis.
axioms (4)
  • domain assumption GPT-4o annotations are reliable ground truth for quality and domain labels.
    Used in §3.3 to train the student model and in §5.3 to evaluate filtering precision; if GPT-4o is biased, all quality signals are biased.
  • domain assumption The K-means diversity score (product of intra-cluster compactness and inter-cluster separation) is a valid proxy for distributional diversity.
    Eq (1) is asserted without derivation or validation; d_intra,j and d_inter,j are not even defined.
  • domain assumption FastText language classifier with confidence >0.5 retains English documents effectively.
    Used in §3.2 for language filtering; standard practice but not re-validated.
  • domain assumption The 10 benchmarks (MMLU, GSM8K, etc.) are unbiased measures of corpus quality at 3B scale.
    Standard evaluation practice; however the paper does not check whether its curation pipeline or GPT-4o teacher is contaminated with benchmark-like content.

pith-pipeline@v1.3.0-alltime-deepseek · 25714 in / 14205 out tokens · 141091 ms · 2026-08-02T09:35:12.016380+00:00 · methodology

0 comments
read the original abstract

Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance. However, these pipelines typically rely on singular optimization objectives, which inevitably narrows distributional diversity and marginalizes long-tail knowledge, thereby restricting data coverage and underutilizing the vast potential of the open web. To address this limitation, we propose a novel curation paradigm that shifts from linear pruning to the joint optimization of quality, redundancy, and diversity. This framework synergizes dual-track cleaning (rule-based and model-driven) with hybrid deduplication (n-gram and semantic), while employing a multi-objective sampler to balance informational quality with distributional breadth. Applying this framework to Common Crawl, we construct CuraWeb, a 2T-token English corpus. Unlike existing resources, CuraWeb establishes an industrial-grade standard for data curation by recovering a more holistic data distribution with enhanced diversity and minimal redundancy, achieving broader coverage of long-tail knowledge across diverse domains. Experimental evaluations at the 3B scale demonstrate that CuraWeb significantly outperforms state-of-the-art baselines, yielding an average performance gain of 1.8\% across a wide range of benchmarks, particularly in knowledge-intensive and reasoning tasks.

Figures

Figures reproduced from arXiv: 2607.22662 by Gan Dong, Jianxiao Yang, Jian Yang, Jingang Wang, Juncheng Diao, Peiguang Li, Rongxiang Weng, Shuguang Jiao, Xiao Wei, Xunliang Cai, Yongwei Zhou, Yuchun Fan, Zhiye Zou, Zhizhao Zeng, Zhongda Su.

Figure 1
Figure 1. Figure 1: Construction pipeline of CURAWEB. The funnel shows how raw common crawls are progressively refined through exact-deduplication, filtering, deduplication, and data selection. Per￾centages denote the relative document flow or removal ratio at each stage. Fine-Grained Content Optimization Web corpora that have undergone coarse-grained filtering still suffer from excessive low-value content and severe domain d… view at source ↗
Figure 2
Figure 2. Figure 2: Pearson Correlation heatmap display￾ing across the derived quality metrics. Note that the safety_score metric (binary) is ex￾cluded from this heatmap due to its scale vari￾ance from the other 4-point scale metrics. Quality Metrics: To capture multi-faceted text characteristics beyond a single monolithic qual￾ity score—which frequently masks localized doc￾ument defects—we define nine fine-grained evalua￾tio… view at source ↗
Figure 3
Figure 3. Figure 3: Evaluation results and scaling behaviors. (a) Comparison between [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Component-wise ablation studies showing downstream performance trends. The perfor [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Domain L1 distribution before and after rule-based filtering: (a) before filtering, (b) after [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Domain-level effects of model-based filtering: (a) L1-domain distribution across pipeline [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Top-20 domain categories by upsampling ratio in the [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Top-27 L1-domain distributions across five datasets under equal-volume (50M documents) [PITH_FULL_IMAGE:figures/full_fig_p026_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Evolution of first-level domain distribution across the pipeline. [PITH_FULL_IMAGE:figures/full_fig_p027_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 18 linked inside Pith

  1. [1]

    Samplemix: A sample-wise pre-training data mixing strategey by coordinating data quality and diversity

    Xiangyu Xi, Deyang Kong, Jian Yang, Jiawei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, and Wei Ye. Samplemix: A sample-wise pre-training data mixing strategey by coordinating data quality and diversity

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Longcat-flash technical report.arXiv preprint arXiv:2509.01322, 2025

    Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report.arXiv preprint arXiv:2509.01322, 2025

    arXiv 2025

  6. [6]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Nemotron-4 340b technical report.arXiv preprint arXiv:2406.11704, 2024

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report.arXiv preprint arXiv:2406.11704, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    The fineweb datasets: Decanting the web for the finest text data at scale.Advances in Neural Information Processing Systems, 37:30811–30849, 2024

    Guilherme Penedo, Hynek Kydlíˇcek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale.Advances in Neural Information Processing Systems, 37:30811–30849, 2024

    2024

  9. [9]

    Datacomp-lm: In search of the next generation 15 of training sets for language models.Advances in Neural Information Processing Systems, 37: 14200–14282, 2024

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation 15 of training sets for language models.Advances in Neural Information Processing Systems, 37: 14200–14282, 2024

    2024

  10. [10]

    Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2112.11446, 2021

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2112.11446, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only.arXiv preprint arXiv:2306.01116, 2023

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only.arXiv preprint arXiv:2306.01116, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Ccnet: Extracting high quality monolingual datasets from web crawl data

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. InProceedings of the twelfth language resources and evaluation conference, pages 4003–4012, 2020

    2020

  13. [13]

    Bi Huo, Bin Tu, Cheng Qin, Da Zheng, Debing Zhang, Dongjie Zhang, En Li, Fu Guo, Jian Yao, Jie Lou, et al. dots. llm1 technical report.arXiv preprint arXiv:2506.05767, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  15. [15]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and ...

    2019

  16. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021. URL https://openreview.net/forum? id=d7KBjmI3GmQ

    2021

  17. [17]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    2024

  18. [18]

    Race: Large-scale reading comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. InProceedings of the 2017 conference on empirical methods in natural language processing, pages 785–794, 2017

    2017

  19. [19]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019

    2019

  20. [20]

    Piqa: Reasoning about phys- ical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  21. [21]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. InProceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, 2017

    2017

  22. [22]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  23. [23]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391, 2018. 16

    2018

  24. [24]

    Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shan...

    Pith/arXiv arXiv 2025

  25. [25]

    Documenting large webtext corpora: A case study on the colossal clean crawled corpus

    Jesse Dodge, Maarten Sap, Ana Marasovi´c, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. InProceedings of the 2021 conference on empirical methods in natural language processing, pages 1286–1305, 2021

    2021

  26. [26]

    Fineweb2: One pipeline to scale them all — adapting pre-training data processing to every language

    Guilherme Penedo, Hynek Kydlí ˇcek, Vinko Sabol ˇcec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro V on Werra, and Thomas Wolf. Fineweb2: One pipeline to scale them all — adapting pre-training data processing to every language. InSecond Conference on Language Modeling, 2025. URL https://openreview. net/forum?...

    2025

  27. [27]

    On the resemblance and containment of documents

    Andrei Z Broder. On the resemblance and containment of documents. InProceedings. Com- pression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE, 1997

    1997

  28. [28]

    Approximate nearest neighbors: towards removing the curse of dimensionality

    Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. InProceedings of the thirtieth annual ACM symposium on Theory of computing, pages 604–613, 1998

    1998

  29. [29]

    Semd- edup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023

    Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semd- edup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Deduplicating training data makes language mod- els better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language mod- els better. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445, 2022

    2022

  31. [31]

    Slimpajama-dc: Under- standing data combinations for llm training.arXiv preprint arXiv:2309.10818, 2023

    Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, et al. Slimpajama-dc: Under- standing data combinations for llm training.arXiv preprint arXiv:2309.10818, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  33. [33]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Doremi: Optimizing data mixtures speeds up language model pretraining.Advances in Neural Information Processing Systems, 36: 69798–69818, 2023

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining.Advances in Neural Information Processing Systems, 36: 69798–69818, 2023

    2023

  35. [35]

    Organize the web: Constructing domains enhances pre-training data curation.arXiv preprint arXiv:2502.10341, 2025

    Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation.arXiv preprint arXiv:2502.10341, 2025. 17

    Pith/arXiv arXiv 2025

  36. [36]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, D...

    2024

  37. [37]

    Elastic chatnoir: Search engine for the clueweb and the common crawl

    Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. Elastic chatnoir: Search engine for the clueweb and the common crawl. InEuropean conference on information retrieval, pages 820–824. Springer, 2018

    2018

  38. [38]

    Bag of tricks for efficient text classification

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomáš Mikolov. Bag of tricks for efficient text classification. InProceedings of the 15th conference of the European chapter of the association for computational linguistics: volume 2, short papers, pages 427–431, 2017

    2017

  39. [39]

    A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity

    Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computatio...

    2024

  40. [40]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

    Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7482–7491. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/ CVPR.2018.007...

    arXiv 2018

  42. [42]

    How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings

    Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 55–65, 2019

    2019

  43. [43]

    jina- embeddings-v3: Multilingual embeddings with task lora.arXiv preprint arXiv:2409.10173, 2024

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. jina- embeddings-v3: Multilingual embeddings with task lora.arXiv preprint arXiv:2409.10173, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual 18 Meeting of th...

    2025

  46. [46]

    Dolma: An open corpus of three trillion tokens for language model pretraining research

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2024

  47. [47]

    A framework for few-shot language model evaluation, 12 2023

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework...

    2023

  48. [48]

    Evaluation Reasoning

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014. 19 A Training Configuration Table 9 lists the architecture and optimization hyperparameters used for training. Table 9: Training hyperparameters. Hyperparameter Value Parameters 3B Architecture LLaMA (RMSNorm, SwiGLU, RoPE, GQA) Layers 32 Hidde...

    Pith/arXiv arXiv 2014

  49. [49]

    Arts_and_Entertainment

  50. [50]

    Books_and_Literature

  51. [51]

    Business_and_Industrial

  52. [52]

    Computers_and_Electronics

  53. [53]

    Internet_and_Telecom

  54. [54]

    Removed” were eliminated due to an excessive risk of over-filtering high-quality content, while those labeled “Retained

    Travel_and_Transportation The second-level domains are: ... Each domain is structured as: /PARENT_DOMAIN/SUB_DOMAIN D Comprehensive Comparison of Gopher and CuraWeb Heuristic Rules Table 12 presents the complete set of heuristic filtering rules, comparing the original Gopher [10] thresholds with our recalibrated thresholds. Rules designated as “Removed” w...