Pith. sign in

REVIEW 4 major objections 5 minor 9 cited by

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Organizing web pretraining data into topic and format domains and rebalancing their mixture lifts average OLMES accuracy from 51.6% to 54.6% for a 1B-parameter model, and adds about two points on top of quality-filtered corpora.

desk verdict A solid, transparent empirical study of domain-based pretraining data curation; the core claim survives scrutiny, but the scale-transfer assumption and missing error bars keep it from being fully settled. read the letter →

arxiv 2502.10341 v3 pith:HJ3FTMLX submitted 2025-02-14 cs.CL

Alexander Wettig , Kyle Lo , Sewon Min , Hannaneh Hajishirzi , Danqi Chen , Luca Soldaini This is my paper · ORCID
classification cs.CL
keywords pre-trainingdatacurationdomainmixtureoptimizationtopictaxonomyformatknowledgedistillationqualityfilteringRegMixwebcorpusanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the unstructured web corpora used to pretrain language models become much more manageable when they are organized into coarse-grained domains, what a page is about (topic) and how it is written (format), and when those domains are rebalanced toward the tasks the model should perform. It introduces WebOrganizer, which uses two 24-category taxonomies, annotates a 200-billion-token corpus with small classifiers distilled from a large teacher model, and adapts the RegMix mixture-optimization procedure to predict which domains to amplify. The headline result is that simply reweighting domains lifts average OLMES accuracy over nine tasks from 51.6% to 54.6%, 0.4 points above the FineWeb-Edu quality-filter baseline, and that the same domain mixing added on top of FineWeb-Edu raises it from 54.2% to 56.2%. The paper also shows that quality filters are themselves implicit domain mixers: for FineWeb-Edu, reproducing its implicit topic-by-format mixture recovers 84% of its average accuracy gain, while DCLM-fasttext depends more on within-domain document selection. If correct, this establishes domain construction and rebalancing as a distinct, complementary axis of pretraining data curation alongside document-level quality scoring.

What carries the argument

The load-bearing objects are the two human-designed taxonomies of 24 topics (e.g., Science & Technology, Home & Hobbies) and 24 formats (e.g., Academic Writing, Tutorial, Q&A Forum), the distilled classifiers that label the 200B-token corpus, and the RegMix pipeline that turns labels into mixtures. The classifiers are 140M-parameter models fine-tuned from a text-embedding model in two stages, first on 1M annotations from a cheaper teacher, then on 80K annotations from a 405B-parameter teacher, using a soft knowledge-distillation loss and yielding 93.5% average accuracy on confident topic annotations and 91.8% on formats. RegMix trains 512 small models per domain definition and fits gradient-boosted-tree regressors that predict downstream bits-per-byte loss from mixture weights, with the best mixture found by an adaptive search under a KL penalty toward the corpus prior. Topic and format mixtures are combined by assuming the two axes are conditionally independent, $\tilde{P}_{T\times F}(t,f)=\tilde{P}_T(t)\tilde{P}_F(f)$, which the near-zero normalized mutual information between topics and formats ($\mathrm{NMI}\approx 0.10$) supports. This machinery converts an unstructured corpus into a tunable set of levers, and the levers are what the experiments pull.

What would settle it

Train an intermediate-scale model (for example, roughly 400M parameters on about 10B tokens) on the RegMix-predicted Topic × Format mixture and on the baseline corpus. If the predicted mixture no longer beats the baseline at that scale, or a mixture optimized at that scale outperforms it, the paper's claim that small-model mixture preferences transfer to larger training runs would be contradicted.

Watch

Extended reading notes

Core claim

WebOrganizer's central discovery is that constructing meaningful domains and optimizing their mixture toward downstream tasks improves pretraining, and that the topic and format axes are nearly independent and combine multiplicatively. The RegMix procedure, trained on 512 runs of 50M-parameter models and applied to 1B-parameter models, predicts that MMLU benefits from heavy upsampling of Science & Technology and Academic Writing, while HellaSwag benefits from Tutorials and Home & Hobbies; mixing both preferences via the product $\tilde{P}_{T\times F}(t,f)=\tilde{P}_T(t)\tilde{P}_F(f)$ produces gains on 8 of 9 OLMES tasks. Domain mixing raises the baseline corpus from 51.6% to 54.6% average accuracy, slightly exceeds the FineWeb-Edu quality filter, and adds 2.0 points to FineWeb-Edu and 1.0 point to DCLM-fasttext when used as a per-domain pre-sampler. The paper further shows that quality-based selection implicitly rewires the domain distribution: FineWeb-Edu's implicit Topic × Format mixture retains 84% of its average gain, whereas DCLM-fasttext's implicit mixture recovers only 35%, indicating that within-domain document quality is essential there.

Load-bearing premise

The paper's load-bearing premise is that the best domain mixture learned from 50M-parameter models trained on 1B tokens remains the best mixture when training 1B-parameter models on 29B tokens, a transfer that is not validated at an intermediate scale.

Editorial extensions

If this is right

  • Topic × Format domain mixing alone lifts the average OLMES accuracy of a 1B-parameter model from 51.6% to 54.6%, just above the FineWeb-Edu quality-filter baseline of 54.2%.
  • Applying the same domain mixture inside each quality-filtered subset raises FineWeb-Edu from 54.2% to 56.2% and DCLM-fasttext from 55.1% to 56.1%, with FineWeb-Edu's HellaSwag deficit converted from -1.5 points into a +5.0-point gain.
  • FineWeb-Edu's performance is largely a domain effect: its implicit Topic × Format mixture recovers 84% of its average gain and 73% of its MMLU gain without any document-level selection.
  • K-means clusters of document embeddings align mainly with topics (NMI $\approx$ 0.46) rather than formats (NMI $\approx$ 0.13), so the human-designed format taxonomy supplies information that pure clustering misses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mixture predictions transfer to larger models, domain rebalancing would be a near-free addition to trillion-token pretraining pipelines, since the classifiers are cheap to run and the mixture search uses only small models; the scale-transfer question is the main open risk.
  • The same taxonomy could be used as an audit instrument: because quality filters implicitly alter domain proportions, publishing the implicit mixture of any proposed curation method would make its content bias visible before training.
  • Single-label domain assignments miss documents that blend topics or formats; moving to hierarchical or multi-label taxonomies, which the paper names as future work, could sharpen mixture optimization and is a natural next test.
  • Because MMLU and HellaSwag prefer almost opposite mixtures, 'data quality' is task-relative, so a single quality score implicitly encodes a particular mix of downstream preferences, which is worth stating when comparing curation methods.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces WebOrganizer, a framework for organizing web-scale pre-training corpora into 24 topic domains and 24 format domains, annotated by classifiers distilled from Llama-3.1-405B-Instruct. The authors adapt the RegMix mixture-optimization framework to predict domain proportions that minimize bits-per-byte on MMLU and HellaSwag validation sets, using 512 runs of 50M-parameter models trained on 1B tokens each. They then train 1B-parameter models on 29B tokens sampled according to the predicted topic, format, and joint topic-by-format mixtures, and report that the joint mixture raises average OLMES accuracy from 51.6% on the baseline corpus to 54.6%, and that applying the same mixture on top of FineWeb-Edu and DCLM-fasttext raises their averages from 54.2% to 56.2% and from 55.1% to 56.1%, respectively. The paper also analyzes quality filters as implicit domain mixers, showing that the implicit topic-by-format mixture of FineWeb-Edu recovers 84% of its average accuracy gain. The framework, classifiers, annotations, and code are released.

Significance. If the empirical claims hold, the paper makes a useful contribution to pre-training data curation by showing that coarse-grained domain rebalancing, built on two complementary taxonomies, is a viable complement to document-level quality filtering. The two-dimensional topic/format structure is a practical and interpretable organizing principle for CommonCrawl-based corpora, and the observation that topic and format are nearly independent (NMI≈0.10) supports the simple multiplicative combination. The authors deserve credit for releasing the classifiers, the annotated 200B-token corpus, and the training code, and for reporting held-out Spearman correlations (0.86–0.94) between RegMix predictions and small-model evaluations. The analysis of quality filters as implicit domain mixers is insightful and goes beyond a simple accuracy comparison. However, the central attribution of the 1B-scale gains to the RegMix-optimized mixtures currently rests on an unvalidated scale-transfer assumption and on single-run evaluations, so the strength of the contribution is not yet fully established.

major comments (4)
  1. [Section 3 and Section 4.1] The load-bearing claim that the RegMix-selected mixture improves 1B-scale training is not validated across the 20–30x scale gap between the mixture-optimization stage (50M parameters, 1B tokens) and the deployment stage (1B parameters, 29B tokens). The Impact Statement concedes that "it is uncertain how well they transfer across model scales," and the only deployment-scale evidence in Table 10 is that MMLU-targeted and HellaSwag-targeted mixtures move the corresponding scores in the expected directions. That shows the objective is readable, but it does not show that the predicted "Both" mixture is optimal or even near-optimal at the deployment scale. The paper should provide an intermediate-scale validation (e.g., a 300–700M model on 5–10B tokens) or compare the predicted mixture against several random or ablated mixtures at the deployment scale with multiple seeds, so that the headline gains in Table 1 can be attributed to the optimized proportions rather than to generic reweighting.
  2. [Tables 1, 2, and 10] All 1B-parameter models are trained once per configuration, with no error bars or seed variance reported. The headline average gain of 3.0 points combines heterogeneous per-task changes (e.g., SIQA drops by 0.6 points) and some comparisons are close (e.g., Topic 53.7 vs. Format 53.4; DCLM-fasttext 55.1 vs. +Topic×Format 56.1). Without at least three seeds for the baseline corpus, the Topic×Format condition, and the quality-filter conditions, or a reported variance estimate, the magnitude of the claimed improvements is not rigorously established. This is a major issue because the central conclusions are quantitative.
  3. [Appendix E (Evaluation setting)] The OLMES suite is reduced from 10 to 9 tasks by dropping BoolQ, with the explanation that BoolQ produced "unreliable results" (random baseline 63.8% vs. DCLM-fasttext 54.4%). Because excluding this task changes the reported average in a direction that affects the paper's comparisons, the exclusion needs stronger justification. The authors should report BoolQ under the same protocol in an appendix row, or demonstrate that the inconsistency is a known artifact of the task formulation and not a post-hoc selection that favors the proposed method. As written, the "9-task average" is a subset chosen after seeing the results.
  4. [Section 4.2 (Topic × Format selection)] The independence assumption P(topic,format)=P(topic)P(format) is supported only by aggregate NMI≈0.10, and the practical construction then handles pairs whose product exceeds available data by "tak[ing] all available documents and up-sampl[ing] everything else." This correction changes the delivered mixture relative to the target, yet the paper does not report how frequently it triggers or what the actual delivered topic-format distribution is. Since the Topic×Format row is the main result, the paper should quantify the deviation between the intended and delivered mixtures.
minor comments (5)
  1. [Appendix E] There is a typo: "eliminiate" should be "eliminate."
  2. [Appendix A (Full prompt)] The sentence "The prompt template is shown in Table 5 and contains instructions, the text contents and URL of the web page, and the the list of domain descriptions" contains a duplicated "the."
  3. [Section 2.2] The phrase "We use the topic and a format classifiers to annotate" is ungrammatical; it should be "the topic and format classifiers."
  4. [Section 6 (Related Work)] The phrase "two-dimensional structuree" contains a typo; it should be "structure."
  5. [Section 5 (perplexity paragraph)] The sentence "we observe that the values for domain mixing and substantially lower than using the quality filtering" is missing a verb; it should read "are substantially lower than those for quality filtering" or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RegMix mixtures are an empirical extrapolation from 50M/1B-token runs to 1B/29B-token models, with independent support from non-target tasks and external quality filters.

full rationale

The paper's derivation chain is empirical rather than definitional. WebOrganizer taxonomies are human-designed and distilled into classifiers that annotate a 200B-token corpus; this annotation step does not presuppose any downstream gain. The RegMix step fits a gradient-boosted tree to 512 logged 50M-parameter runs with mixture weights as inputs and MMLU/HellaSwag bits-per-byte as targets, then inverts the fitted surface to propose a mixture (Section 3, Appendix C). The headline result is obtained by training 1B-parameter models on 29B tokens selected with that mixture and measuring OLMES accuracy on nine tasks (Section 4). Nothing in this chain makes the final accuracy equal to the fitted loss by construction: the model scale, token budget, and evaluation metric all change, and the paper explicitly concedes that transfer across model scales is uncertain (Impact Statement). The gains on non-target tasks (PIQA, ARC, OBQA, CSQA) and the improvement of external quality filters (FineWeb-Edu, DCLM-fasttext) under domain mixing provide independent evidence that is not an artifact of the RegMix objective. The only in-scope concern is that HellaSwag's validation split is used both in the RegMix objective and in the OLMES evaluation, so the HellaSwag column is a selected optimum rather than a fresh prediction; but the paper's central claim does not rest on HellaSwag alone, and this is an evaluation-overlap risk, not a definitional circularity. Self-citations (e.g., QuRating, Dolma) are contextual and not load-bearing; the load-bearing comparisons are to external, reproducible resources (DCLM, OLMES, FineWeb-Edu, DCLM-fasttext).

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claims rest on the released artifacts and on a modest set of hand-chosen RegMix hyperparameters. The least supported inputs are the scale-transfer and topic-format independence assumptions, both acknowledged or tested by the authors, and the teacher-annotation assumption that underlies all labels.

free parameters (5)
  • RegMix KL coefficient gamma = 0.002
    Added to the search objective to keep predicted mixtures close to the corpus prior; chosen by hand in Appendix C.
  • Dirichlet temperature tau = 2
    Softens the corpus prior when sampling the 512 mixtures for small-model training; chosen by hand following RegMix.
  • Maximum upsampling cap = 6.5x
    Limits the mixture search to proportions achievable when selecting 30B tokens from the 200B corpus; directly shapes every predicted mixture.
  • Search steps T = 15
    Number of adaptive search iterations in the RegMix simulation; chosen by hand in Appendix C.
  • Smoothing eta = 0.2
    Controls how much the search prior moves toward the best current mixture each iteration; chosen by hand in Appendix C.
assumptions (5)
  • domain assumption Bits-per-byte loss on the correct MMLU/HellaSwag responses is a valid proxy for downstream task performance when searching mixtures.
    Used as the RegMix objective in Section 3. The paper reports held-out Spearman correlations of about 0.86-0.94, but the proxy itself is adopted from prior work.
  • ad hoc to paper Topic and format selections are independent, so the combined mixture is P(topic) times P(format).
    Section 4.2 constructs the 576-pair mixture this way. NMI of about 0.10 supports near-independence, but residual dependencies are ignored.
  • ad hoc to paper The best domain mixture is stable when scaling from 50M-parameter models on 1B tokens to 1B-parameter models on 29B tokens.
    The whole RegMix pipeline is trained at the small scale and evaluated at the larger scale. The Impact Statement says scale transfer is uncertain.
  • domain assumption Llama-3.1-405B-Instruct annotations are an adequate ground truth, and the distilled classifiers are accurate enough for corpus-scale annotation.
    Appendix B reports 93.5% topic and 91.8% format average accuracy on confident labels, with about 5% added error relative to the teacher; the teacher itself has order sensitivity.
  • domain assumption The 200B-token base corpus from DCLM's pool with RefinedWeb heuristics and Bloom-filter deduplication is a valid universe for comparing curation methods.
    Appendix E describes preprocessing; the authors could not reproduce the exact DCLM baseline, so the base corpus differs from the official DCLM pool in small but acknowledged ways.
invented entities (1)
  • WebOrganizer topic and format taxonomies (24 categories each) independent evidence
    purpose: Provide human-interpretable, scalable labels of web pages by subject and style to enable domain-aware data curation and corpus documentation.
    The taxonomies ship as trained classifiers and as a 200B-token annotated corpus, and they are tested against k-means clusters and downstream benchmarks, giving independent handles beyond the paper itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Organize the Web: Constructing Domains Enhances Pre-Training Data Curation." pith.science (2026). https://pith.science/paper/HJ3FTMLX

@misc{pith2026250210341,
  author       = {Pith},
  title        = {Pith review of: Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJ3FTMLX}},
  note         = {Machine review of arXiv:2502.10341}
}
read the original abstract

Modern language models are trained on large, unstructured datasets consisting of trillions of tokens and obtained by crawling the web. The unstructured nature makes it difficult to reason about their contents and develop systematic approaches to data curation. In this paper, we unpack monolithic web corpora by developing taxonomies of their contents and organizing them into domains. We introduce WebOrganizer, a framework for organizing web pages in terms of both their topic and format. Using these two complementary notions of domains, we automatically annotate pre-training data by distilling annotations from a large language model into efficient classifiers. This allows us to study how data from different domains should be mixed to improve models on downstream tasks, and we show that we can combine insights about effective topics and formats to further boost performance. We demonstrate that our domain mixing also improves existing methods that select data based on quality. Furthermore, we study and compare how quality-based methods will implicitly change the domain mixture. Overall, our work demonstrates that constructing and mixing domains provides a valuable complement to quality-based data curation methods, opening new avenues for effective and insightful pre-training data curation.

Figures

Figures reproduced from arXiv: 2502.10341 by the authors.

Figure 1
Figure 1. We construct topic domains (left) and format domains (right) to organize pre-training corpora. The areas visualize the number of tokens per domain in a cleaned pre-training corpus based on CommonCrawl. See Appendix A for detailed definitions of the categories. We provide an interactive explorer of the domains at weborganizer.allen.ai. mostly align with topics and do not reveal different formats. How effective are th… view at source ↗
Figure 2
Figure 2. We visualize the 15 highest co-occurrences in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The corpus proportions of our topic domains (left) and formats (right), and the training mixtures predicted by [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The implicit domain compositions from quality filtering compared to the corpus distribution for topic domains [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Frequency statistics of URL domain names in our 200B CommonCrawl corpus. Left: Plotting log document [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: The normalized pointwise-mutual information matrix between all topics (y-axis) and formats (x-axis). A score [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: The normalized pointwise-mutual information (NPMI) matrices between [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: The predicted upsampling by RegMix of our topic domains (top) and formats (bottom), as a proportion of the [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OLMoASR: Open Models and Data for Training Robust Speech Recognition Models

    cs.SD 2025-08 conditional novelty 7.0 of 10

    An open 1M-hour English speech dataset plus Whisper-architecture models trained on it match Whisper's word error rates on short and long-form benchmarks.

  2. CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

    cs.AI 2026-06 conditional novelty 6.0 of 10

    CuraWeb jointly optimizes cleaning, deduplication, and sampling to build a 2T-token pretraining corpus that outperforms FineWeb-Edu, DCLM, Nemotron-CC, and Dolma3 by 1.82% average at 200B tokens.

  3. GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning

    cs.LG 2026-02 reject novelty 6.0 of 10

    GradAlign selects RL training problems whose policy gradients align with validation-set gradients, claiming more stable and better-performing LLM post-training.

  4. Language Models Improve When Pretraining Data Matches Target Tasks

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.

  5. Essential-Web v1.0: 24T tokens of organized web data

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.

  6. The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new 8TB openly-licensed text corpus trains 7B LLMs that are competitive with Llama 1/2, showing that performant models need not depend on unlicensed web data.

  7. Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A paragraph-level LLM-annotated biomedical corpus from PMC-OA, distilled into a classifier, enables targeted clinical and educational filtering that improves medical pretraining efficiency.

  8. DoTA-RAG: Dynamic of Thought Aggregation RAG

    cs.CL 2025-06 conditional novelty 4.0 of 10

    DoTA-RAG combines query rewriting, namespace routing, dense retrieval, BM25 pruning, and reranking to answer questions over a 15M-document corpus, with reported correctness gains but fragile faithfulness under output caps.

  9. Collective Bargaining in the Information Economy Can Address AI-Driven Power Concentration

    cs.CY 2025-06 conditional novelty 4.0 of 10

    A policy agenda proposes collective bargaining by information producers as the principal fix for AI-driven market concentration and collapse of the information commons.

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages · cited by 9 Pith papers

  1. [5]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova

    URL https://openreview.net/forum? id=BDisxnHzRL. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associa- tion for Computational Linguistics: Human Lan...

    arXiv 2019

  2. [13]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groen- eveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al

    URL https://openreview.net/forum? id=j5BuTrEj35. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groen- eveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 OLMo 2 Furi- ous. arXiv preprint arXiv:2501.00656, 2024. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cap- pell...

    arXiv 2024

  3. [18]

    About Page

    URL https://openreview.net/forum? id=6gzPSMUAz2. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 4791–4800, 2019. Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang, Xi...

    arXiv 2023

  4. [164]

    naacl-long.164/

    URL https://aclanthology.org/2024. naacl-long.164/. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Mil- lican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. Colin Raffel, Noam Shazeer, A...

    arXiv 2024

  5. [179]

    naacl-long.179/

    URL https://aclanthology.org/2024. naacl-long.179/. Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren Klein, and Jesse Dodge. AboutMe: Using self-descriptions in webpages to document the effects of english pretraining data filters. arXiv preprint arXiv:2401.06408, 2024. Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, C...

    arXiv 2024

  6. [183]

    emnlp-main.183/

    URL https://aclanthology.org/2023. emnlp-main.183/. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Sc...

    arXiv 2023

  7. [373]

    naacl-main.373/

    URL https://aclanthology.org/2021. naacl-main.373/. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021b. 10 Organize the Web: Constructing Domains Enhances ...

    arXiv 2021

  8. [577]

    acl-long.577

    URL https://aclanthology.org/2022. acl-long.577. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amr...

    arXiv 2022

Show all 18 references
  1. [657]

    emnlp-main.657/

    URL https://aclanthology.org/2023. emnlp-main.657/. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzm ´an, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the T...

    2023 arXiv

  2. [757]

    acl-long.757/

    URL https://aclanthology.org/2024. acl-long.757/. Max Marion, Ahmet ¨Ust¨un, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining LLMs at scale, 2023. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabh...

    2024

  3. [840]

    acl-long.840

    URL https://aclanthology.org/2024. acl-long.840. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neu- rocomputing, 568:127063, 2024. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom....

    2024

  4. [2017]

    cc/paper_files/paper/2017/file/ 6449f44a102fde848669bdd9eb6b76fa-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ 6449f44a102fde848669bdd9eb6b76fa-Paper. pdf. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kris...

    2017

  5. [2019]

    URL https:// aclanthology.org/Q19-1026/

    doi: 10.1162/tacl a 00276. URL https:// aclanthology.org/Q19-1026/. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annu...

    doi

  6. [2020]

    cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper. pdf. Catherine Chen, Kevin Lin, and Dan Klein. Construct- ing taxonomies from pretrained language models. In Kristina Toutanova, Anna Rumshisky, Luke Zettle- moyer, Dilek H...

    2020

  7. [2023]

    Mayee F Chen, Michael Y Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher R ´e

    URL https://openreview.net/forum? id=IoizwO1NLf. Mayee F Chen, Michael Y Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher R ´e. Aioli: A unified optimization framework for language model data mix- ing. arXiv preprint arXiv:2411.05735, 2024. Yangyi Chen, Binxuan Huang, Yifan...

    2024 arXiv

  8. [2024]

    Logan Engstrom, Axel Feldmann, and Aleksander Madry

    URL https://openreview.net/forum? id=RvfPnOkPV4. Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamod- els. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview. net/forum?id=GC8HkKeH8s. Simi...

    2024 arXiv

  9. [2025]

    Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet ¨Ust¨un, and Sara Hooker

    URL https://openreview.net/forum? id=1GTARJhxtq. Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet ¨Ust¨un, and Sara Hooker. To code or not to code? exploring impact of code in pre-training. In The Thir- teenth Internatio...

    2025

  10. [8856]

    Jiachen T

    URL https://openreview.net/forum? id=G7p8djzWOl. Jiachen T. Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. GREATS: Online selection of high- quality data for LLM training in every iteration. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Sy...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.