Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

The paper argues that Classifier-based Quality Filtering (CQF) improves pretraining not by making data resemble a high-quality reference set, but by implicitly filtering that reference set itself, removing web-like noise.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 13:00 UTC pith:JCLUTLQW

arxiv 2510.00866 v4 pith:JCLUTLQW submitted 2025-10-01 cs.LG cs.CL

Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining

Thiziri Nait Saada , Louis Bethune , Michal Klein , David Grangier , Marco Cuturi , Pierre Ablin This is my paper
classification cs.LG cs.CL
keywords data filteringpretrainingclassifier-based quality filteringlikelihood ratiodata conditioningimportance samplingLLM pretrainingdata quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper analyzes Classifier-based Quality Filtering, the standard method for pretraining data selection. It tries to establish that CQF works by removing low-quality web-like documents rather than by drawing the selected data closer to the small high-quality reference set. The evidence is a paradox: training on CQF-filtered data can outperform training directly on the reference set itself, and CQF does not improve language modeling on that reference set. The paper explains this through an implicit filtering of the reference set, where CQF upweights reference documents that lie far from the low-quality bulk, and shows that this alignment with downstream tasks, not a universal notion of quality, drives the gains. It also introduces 'data conditioning' as a formal, optimization-based definition of quality and shows CQF does not satisfy it.

Core claim

The paper's central claim is that CQF's downstream benefits come from an implicit filtering of the high-quality set: under a Bayes-optimal classifier, the CQF score is a monotone function of the density ratio p_HQ/p_LQ, and the filtered dataset's density equals w(x) p_HQ(x) with w(x) ∝ 1_{r(x)≥τ}/r(x). Thus CQF overrepresents HQ documents that are far from the LQ bulk, effectively selecting a task-aligned subset of the HQ set rather than mimicking the whole HQ distribution. This explains why CQF-selected data can beat training on the full HQ set, why HQ loss is U-shaped in the selection fraction, and why CQF differs from importance sampling. The paper further formalizes a notion of quality a

What carries the argument

The likelihood-ratio reweighting identity: with r(x) = p_HQ(x)/p_LQ(x), CQF selects LQ documents with r(x) ≥ τ, so the filtered density can be written p_CQF(x) = (1/Z) 1_{r(x)≥τ} p_LQ(x) = w(x) p_HQ(x), where w(x) ∝ 1_{r(x)≥τ}/r(x). This identity shows CQF reweights the HQ distribution, upweighting HQ points that are unlikely under the LQ distribution. The paper also defines 'data conditioning': D_clean ≻ D_dirty iff L(θ_Dclean, D_dirty) ≤ L(θ_Ddirty, D_dirty), a formalization of quality as easier optimization.

Load-bearing premise

The entire analysis assumes the logistic-regression classifier used by CQF is Bayes-optimal, so its score is a monotone function of the true density ratio p_HQ/p_LQ; if finite data, regularization, or information loss in fixed embeddings break that link, the implicit-filtering interpretation does not follow.

What would settle it

Train a density-ratio estimator that is as close to Bayes-optimal as possible (large capacity, raw text features, calibrated) and check two predictions: (1) the loss on the HQ set should be U-shaped in the selection fraction, with bottom-decile HQ documents losing ground as k shrinks; (2) CQF-selected data should still beat training on the full HQ set. If either fails — especially if the HQ loss becomes monotone or the beat-HQ effect vanishes — the implicit-filtering explanation is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • CQF's downstream gains are tied to the alignment between the chosen high-quality set and the evaluation benchmark, so choosing an HQ set is effectively a benchmark-selection step.
  • Training on a carefully chosen subset of the HQ set (its top decile) can match or beat training on the full HQ set, suggesting HQ set construction matters more than its size.
  • Importance sampling and CQF optimize different objectives: importance sampling better reproduces the HQ distribution, while CQF better aligns with downstream tasks when the HQ set is task-relevant.
  • CQF classifiers can latch onto spurious features such as sequence length, so debiasing the classifier may be necessary to avoid drifting from the intended target distribution.
  • The data-conditioning framework provides a formal, optimization-based definition of quality; synthetic axes like token permutation satisfy it while CQF buckets do not, indicating CQF's quality notion is closer to style or domain similarity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If CQF really is about removing noise rather than imitating a reference set, data-selection scaling laws should be formulated in terms of contamination rate instead of distance to a reference distribution.
  • The implicit-filtering view suggests a cheaper alternative to running a full classifier over the entire low-quality set: directly sample the task-aligned top decile of the HQ set.
  • The data-conditioning comparison could be extended to multi-task or continued-pretraining settings, where the relation between clean and dirty datasets may shift with training horizon and model capacity.
  • The Bayes-optimality assumption is strong; testing with stronger classifiers or calibrated density-ratio estimators would reveal whether the implicit-filtering geometry persists under imperfect scores.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper analyzes Classifier-based Quality Filtering (CQF), a widely used method that trains a binary classifier on low-quality pretraining data versus a small high-quality set and keeps the top-scoring documents. The authors report that CQF improves downstream accuracy but does not consistently reduce language-modeling loss on the HQ set, and they argue that CQF implicitly reweights and filters the HQ set, upweighting HQ documents far from the LQ distribution. They compare CQF with importance sampling (CRISP), finding that CQF better aligns with downstream tasks when the HQ set is a curated corpus. They also propose a new optimization-based notion of data quality—'data conditioning'—and show that CQF-selected subsets do not satisfy the associated ordering, in contrast to synthetic quality axes based on random token permutation. The central claim is that training on CQF-filtered data can beat training directly on the full HQ set.

Significance. The paper is significant if its central claim holds: it would overturn the standard interpretation of CQF as mimicking the HQ distribution and explain why CQF helps on downstream tasks despite not improving HQ perplexity. The implicit-filtering mechanism (Eq. 4.1 and Figure 5) is a novel and potentially useful insight, and the comparison to importance sampling is informative. The data-conditioning framework, while unconventional, provides an optimization-grounded definition of quality that could be useful for designing data-selection objectives. The empirical study covers multiple HQ sets, several model sizes up to 1.3B, and includes careful ablations (e.g., the debiasing analysis in Appendix C). However, as detailed below, the headline comparison—CQF-filtered data versus training on the full HQ set—is never run, and the main theoretical interpretation depends on a Bayes-optimal classifier assumption that is not validated.

major comments (3)
  1. [Abstract; §1; §3; Fig. 6] The abstract and introduction state that 'training on CQF-selected data can outperform training directly on the high-quality set, even when the latter is sufficiently large,' and the conclusion describes this as the paradox that motivates the implicit-filtering explanation. However, no experiment in the paper trains a model on the full HQ set as a pretraining corpus. Section 3 compares CQF fractions against unfiltered RedPajama (k=100%); Section 5 compares CQF to CRISP; Figure 6 is a fine-tuning probe on the top decile versus the whole HQ set, not a pretraining comparison, and the top decile is generated by the same CQF score. Without this direct control, the striking claim is not evidenced. Please add a pretraining experiment that trains on each full HQ set under the same token/compute budget and compares it against CQF-selected subsets.
  2. [§4, Eq. (4.1)] The reweighting p_CQF(x)=w(x)p_HQ(x) and the entire 'implicit filtering' derivation assume the logistic-regression classifier is Bayes-optimal, so the score is a monotone function of the true density ratio p_HQ(x)/p_LQ(x). The actual classifier is an L2-regularized logistic regression on fixed 384-d sBert embeddings, trained on finite samples and tuned for classification accuracy; there is no guarantee that the score approximates the likelihood ratio in document space. If the score is not the true likelihood ratio, Eq. (4.1) does not follow. The empirical decile analysis in Figure 5 may still support the qualitative conclusion, but the strong theoretical interpretation should either be validated (e.g., on synthetic data with a known density ratio) or presented as a heuristic approximation.
  3. [§5, Fig. 7] The comparison with importance sampling uses CRISP, a method introduced by the authors, and the performance gap may depend on the number of clusters C, which is selected manually (C=4096 for OpenOrca, C=260k for ARC-Easy). The choice of C is not justified, and no sensitivity analysis is provided. Moreover, CRISP is only one possible importance-sampling implementation; a direct likelihood-ratio approach (e.g., Xie et al., 2023) may behave differently. This weakens the claim that 'CQF leads to better downstream performance than importance sampling.' Please report the sensitivity of the comparison to C and, if possible, include a second importance-sampling baseline.
minor comments (5)
  1. [Appendix C] Typo: 'This method also shades lights on the difficulty' should be 'sheds light.'
  2. [Related Work] Typo: 'the Dolma dataset iteself' should be 'itself.'
  3. [Table 2] Typo: 'Context lenght' should be 'Context length.'
  4. [§6.1] The relation D_clean ≻ D_dirty is introduced as 'better data-conditioned than,' but the direction is not immediately intuitive; consider restating the definition in words after Eq. (6.2).
  5. [Figure 3 caption] The caption states 'Quality buckets across classifiers (CQF) used in the literature exhibit alignment towards benchmark datasets,' which is confusing; clarify that the plot shows embeddings of selected subsets, not 'quality buckets.'

Circularity Check

0 steps flagged

No significant circularity: Eq 4.1 is a transparent algebraic identity under an explicit Bayes-optimality assumption, and the self-cited CRISP baseline is not load-bearing.

full rationale

The derivation most likely to look circular—the claim that CQF is a reweighting of the HQ distribution—is exactly that: an algebraic restatement of the CQF selection rule. The paper states the assumption explicitly ('Assuming that the binary classifier trained in CQF is Bayes-optimal...') and then writes pCQF(x) = 1/Z 1_{r(x)≥τ} pLQ(x) = w(x)pHQ(x). This follows from the definitions of CQF and r = pHQ/pLQ; it is not a fitted quantity renamed as a prediction. The empirical content lies in the separate measurements of losses on HQ deciles (Figure 5) and the finetuning alignment probe (Figure 6), which could in principle have contradicted the interpretation. The only self-citation used as an experimental tool is CRISP (Grangier et al., 2024, with two current co-authors); it serves as a comparison baseline, and the Section 5 conclusion that CQF differs from importance sampling is already established by Eq 4.1 and does not rest on CRISP's authority. The Chinchilla fits in Figure 9 are extrapolations of measured curves, not a 'prediction' of the very data to which the parameters were fit. The paper's explicit Bayes-optimality assumption is a robustness/correctness caveat, not circularity: if the classifier is not Bayes-optimal, Eq 4.1 fails, but that is an unverified premise rather than a loop back to the conclusion. The skeptic's strongest point—that the abstract's comparison against training on the full HQ set is not directly run—is a genuine missing-support flag for the headline claim, but absence of a control is not a circular derivation; it lowers the evidential weight of that claim, not the circularity score. Overall, the analysis is self-contained and externally testable, so no circular step meeting the quote-and-reduce bar is present.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 1 invented entities

The paper's mechanism rests on a Bayes-optimal classifier and an embedding that preserves document distributions; the data-conditioning conclusion rests on the permutation-quality assumption and the fitted scaling laws. These are stated, but not independently verified, so the ledger is moderate.

free parameters (2)
  • Chinchilla scaling-law coefficients E,A,B,α,β per (k,k') pair = not reported numerically
    Used in Figure 9 to transform the empirical data-conditioning matrix into a 'dynamic' view at N=1B,D=20N/1000N and N=D=∞; these coefficients are fit to the paper's finite pretraining runs and the infinite-compute limit uses the fitted irreducible error E.
  • Exclusive-CQF quality bucket boundaries = 1%,2%,5%,10%,25%,50%,100% score thresholds
    The data-conditioning graphs for CQF depend on how the score axis is partitioned into 'quality' buckets; these boundaries are chosen by hand and are not derived from the downstream tasks.
axioms (5)
  • domain assumption The trained CQF classifier is Bayes-optimal, so its score equals phi(p_HQ/p_LQ)
    Invoked in Section 4, Eq 4.1; if logistic regression on finite sBert embeddings does not realize the true likelihood ratio, the reweighting and implicit-filtering conclusions do not follow.
  • domain assumption sBert embeddings preserve the document-level distributions relevant to quality
    The entire CQF score operates in a 384-dim embedding space (Section 2); if embedding loses or distorts quality-relevant information, the analysis describes embedding-space quality, not document quality.
  • domain assumption Trained 350M/1.3B models approximate infinite-capacity, converged predictors for the KL interpretation
    Section 4.1 uses H(D_HQ)+KL(D_HQ||D_CQF) to interpret HQ loss; the paper itself notes this requires infinite capacity, while its models are finite and trained for 8B-120B tokens.
  • ad hoc to paper Random token permutation with probability k yields a monotone ground-truth quality axis
    The semi-synthetic Perm(k) family (Section 6.2) is the paper's calibration for a 'true' quality ordering; the data-conditioning test's validity depends on this assumption.
  • domain assumption Imperfect optimization is the mechanism that permits data conditioning; fixed learning rule and horizon
    Section 6.1 defines quality relative to algorithm A and horizon n; the empirical matrices are snapshots for one training budget and may not reflect other scales.
invented entities (1)
  • Data-conditioning relation ≻ (D_clean ≻ D_dirty iff L(θ_Dclean, D_dirty) ≤ L(θ_Ddirty, D_dirty)) independent evidence
    purpose: A formal, testable definition of data quality as an optimization catalyst, used to probe whether CQF's quality axis is meaningful.
    The relation yields falsifiable ordering predictions (Eq 6.2) measured in Figures 8-9, so it is not a purely post hoc construct.

pith-pipeline@v1.3.0-alltime-deepseek · 16342 in / 15612 out tokens · 140638 ms · 2026-08-04T13:00:03.456719+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining." pith.science (2026). https://pith.science/paper/JCLUTLQW

@misc{pith2026251000866,
  author       = {Pith},
  title        = {Pith review of: Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JCLUTLQW}},
  note         = {Machine review of arXiv:2510.00866}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish between pretraining data and a small, high-quality set. It assigns each pretraining document a quality score defined as the classifier's score and retains only the top-scoring ones. We provide an in-depth analysis of CQF. We show that while CQF improves downstream task performance, it does not necessarily enhance language modeling on the high-quality set. Importantly, we find that training on CQF-selected data can outperform training directly on the high-quality set, even when the latter is sufficiently large. This finding alone is particularly striking, given the substantial effort and cost recently devoted to augmenting high-quality data. We explain this paradox by the fact that CQF implicitly filters the high-quality dataset as well as the low-quality one. Finally, we introduce an optimization-driven notion of data quality and demonstrate that it can be reliably estimated using small-scale proxy experiments. Altogether, our results both elucidate the mechanisms behind CQF and deepen our understanding of data selection methods widely used in practice.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Is a Document Educational or Just Wikipedia-Style? -- Pitfalls of Classifier-Based Quality Filtering

    cs.CL 2026-05 unverdicted novelty 6.0

    Wikipedia-style reformatting reverses FineWeb-Edu CQF filtering decisions for ~7% of documents, admitting otherwise excluded low-quality content.

Reference graph

Works this paper leans on

24 extracted references · 13 linked inside Pith · cited by 1 Pith paper

  1. [1]

    A survey on data selection for language models.arXiv preprint arXiv:2402.16827,

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models.arXiv preprint arXiv:2402.16827,

    Pith/arXiv arXiv

  2. [4]

    This method also shades lights on the difficulty of properly projecting and representing in 2D a 384-dim geometry. D No HQ set is superior to all others across all tasks While various HQ sets are used in the literature for CQF, no single HQ consistently outperforms others across all downstream tasks. Figure 16 shows that varying HQ sets yield various perf...

    2024

  3. [8]

    Fasttext

    Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext. zip: Compressing text classification models.arXiv preprint arXiv:1612.03651,

    Pith/arXiv arXiv

  4. [10]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling. InFindings of the Association for Computational Linguistics: NAACL 2025, pp. 1755–1797. Association for Computa...

    2025

  5. [11]

    We show in Figure 12 the result of such experiments, averaged across 3 runs

    We train 350M models on the resulting partitions of RedPajama and evaluate them on ARC (Clark et al., 2018), MMLU (Hendrycks et al., 2021), and Reward Bench (Lambert et al., 2025). We show in Figure 12 the result of such experiments, averaged across 3 runs. Beyond this specific case of sequence length bias, we investigate whether CQF classifiers exhibit s...

    2018

  6. [12]

    A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity

    11 Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computa...

    2024

  7. [13]

    Arctic-embed: Scalable, efficient, and accurate text embedding models.arXiv preprint arXiv:2405.05374,

    Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos. Arctic-embed: Scalable, efficient, and accurate text embedding models.arXiv preprint arXiv:2405.05374,

    Pith/arXiv arXiv

  8. [14]

    Language models improve when pretraining data matches target tasks.arXiv preprint arXiv:2507.12466,

    David Mizrahi, Anders Boesen Lindbo Larsen, Jesse Allardice, Suzie Petryk, Yuri Gorokhov, Jeffrey Li, Alex Fang, Josh Gardner, Tom Gunter, and Afshin Dehghan. Language models improve when pretraining data matches target tasks.arXiv preprint arXiv:2507.12466,

    Pith/arXiv arXiv

  9. [15]

    Scaling laws for optimal data mixtures.arXiv preprint arXiv:2507.09404,

    Mustafa Shukor, Louis Bethune, Dan Busbridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby, and Pierre Ablin. Scaling laws for optimal data mixtures.arXiv preprint arXiv:2507.09404,

    arXiv

  10. [16]

    co/datasets/teknium/OpenHermes-2.5

    URLhttps://huggingface. co/datasets/teknium/OpenHermes-2.5. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    Pith/arXiv arXiv

  11. [17]

    Dataset distillation.arXiv preprint arXiv:1811.10959,

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation.arXiv preprint arXiv:1811.10959,

    Pith/arXiv arXiv

  12. [18]

    This aligns with recent concurrent work from Mizrahi et al

    We observe that quality buckets across classifiers tend to align with specific benchmark datasets, indicating that classifiers—implicitly or explicitly—favor data that resembles their respective supervision targets. This aligns with recent concurrent work from Mizrahi et al. (2025), who show that direct supervision using explicitly target benchmark data c...

    2025

  13. [19]

    Challenges in detoxifying language models

    Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2447–2469,

    2021

  14. [20]

    12 Xinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Jiantao Qiu, Chi Zhang, Ying Qian, and Conghui He

    URLhttps: //openreview.net/forum?id=kJ5i29FejW. 12 Xinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Jiantao Qiu, Chi Zhang, Ying Qian, and Conghui He. Meta-rater: A multi-dimensional data selection method for pre-training language models.arXiv preprint arXiv:2504.14194,

    Pith/arXiv arXiv

  15. [21]

    13 Contents 1 Introduction 1 1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 Classifier-Based Quality-filtering 3 3 CQF improves model evaluations 4 4 CQF does not select data that resemble the high-quality set 5 4.1 Kullback-Leibler divergence between datasets . . . . . . . . . . . . . . . . ...

    2025

  16. [2009]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  17. [2016]

    Mission: Impossible language models.arXiv preprint arXiv:2401.06416,

    Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. Mission: Impossible language models.arXiv preprint arXiv:2401.06416,

    Pith/arXiv arXiv

  18. [2018]

    Ultra-fineweb: Efficient data filtering and verification for high-quality llm training data.arXiv preprint arXiv:2505.05427,

    Yudong Wang, Zixuan Fu, Jie Cai, Peijun Tang, Hongya Lyu, Yewei Fang, Zhi Zheng, Jie Zhou, Guoyang Zeng, Chaojun Xiao, et al. Ultra-fineweb: Efficient data filtering and verification for high-quality llm training data.arXiv preprint arXiv:2505.05427,

    Pith/arXiv arXiv

  19. [2019]

    URL https://doi.org/10.18653/v1/p19-1346

    doi: 10.18653/v1/p19-1346. URL https://doi.org/10.18653/v1/p19-1346. Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Hang Yan, Xipeng Qiu, and Dahua Lin. Query of cc: Unearthing large scale domain-specific knowledge from public corpora.arXiv preprint arXiv:2401.14624,

    Pith/arXiv arXiv

  20. [2021]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Pith/arXiv arXiv

  21. [2022]

    ELI5: long form question answering

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: long form question answering. In Anna Korhonen, David R. Traum, and Lluís Màrquez (eds.),Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 3558–3567...

    2019

  22. [2023]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1,

    Pith/arXiv arXiv

  23. [2024]

    Task-adaptive pretrained language models via clustered- importance sampling.arXiv preprint arXiv:2410.03735,

    David Grangier, Simin Fan, Skyler Seto, and Pierre Ablin. Task-adaptive pretrained language models via clustered- importance sampling.arXiv preprint arXiv:2410.03735,

    Pith/arXiv arXiv

  24. [2025]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al

    doi: 10.18653/v1/2025.findings-naacl.96. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models.Advances in Neural Information Processing Systems, 37:14200–14282,

    doi:10.18653/v1/2025.findings-naacl.96 2025