Pith. sign in

REVIEW 3 major objections 4 minor 6 cited by

Language Models Improve When Pretraining Data Matches Target Tasks

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Ranking pretraining documents by embedding similarity to benchmark training examples—instead of by generic quality scores—consistently improves benchmark performance at every scale tested, worth a 2.1x compute multiplier over a strong…

desk verdict A serious, large-scale empirical study with a clean method and an honest limitations section; the headline compute multiplier partly rests on an undecontaminated pool, so read the Nemotron numbers with care. read the letter →

arxiv 2507.12466 v1 pith:NQ6X7PIE submitted 2025-07-16 cs.CL cs.LG

David Mizrahi , Anders Boesen Lindbo Larsen , Jesse Allardice , Suzie Petryk , Yuri Gorokhov , Jeffrey Li , Alex Fang , Josh Gardner
show 2 more authors
Tom Gunter Afshin Dehghan
This is my paper
classification cs.CLcs.LG MSC 68T5068T07
keywords benchmark-targetedrankingBETRdataselectionfilteringscalinglawscomputemultiplierlanguagemodelpretrainingGoodhart'slaw
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the implicit loop researchers already run—pick data, train, measure benchmarks, refine—works better when made explicit. The authors propose BETR, which ranks pretraining documents by their embedding similarity to benchmark training examples, trains a lightweight classifier to apply that ranking to billions of documents, and keeps only the top-ranked fraction for training. Training over 500 models from $10^{19}$ to $10^{22}$ FLOPs, they find BETR beats strong existing filters by a 1.8–2.8x compute multiplier (2.1x on average over DCLM-Baseline, 4.7x over unfiltered data) and improves 9 of 10 evaluated tasks. They also find the best filtering rate grows with scale, roughly as $F_{\mathrm{opt}}(C) = 4\times 10^{-5} C^{0.25}$, so small models want the top 3% of documents while larger models want the top 30%. If right, this reframes data curation as an explicit optimization problem with a predictable, scale-dependent answer rather than a fixed notion of quality.

What carries the argument

The load-bearing object is the score: for each sampled document, BETR takes its best rank across all benchmark-example embeddings (max aggregation with value function $v(r) = 1/r$), which rewards a document for being extremely close to at least one target example rather than moderately close to many. A FastText classifier trained on those scores then labels the full corpus, making web-scale filtering cheap despite the fact that direct similarity search over billions of documents is intractable. The evaluation machinery is a two-stage scaling-law pipeline—a Chinchilla-style loss law $L(N, D)$ per benchmark, then a sigmoid mapping from bits-per-byte to accuracy—that converts 53 trained models per dataset into predicted accuracy at any compute budget, from which compute multipliers and the $F_{\mathrm{opt}}(C) = 4\times 10^{-5} C^{0.25}$ power law are derived.

What would settle it

One decisive check: at a compute budget where the predicted optimal filter retains fewer tokens than the training run consumes, forcing repeated epochs, compare BETR filtering against the unfiltered baseline. If the compute multiplier shrinks or reverses once data repetition is required, or if the best filtering rate stops following $F_{\mathrm{opt}}(C) = 4\times 10^{-5} C^{0.25}$, the unlimited-data premise on which the scaling claims rest is falsified.

Watch

Extended reading notes

Core claim

BETR (benchmark-targeted ranking) is the central proposal: take the training examples of a benchmark set, embed them together with roughly 0.1% of a web-scale corpus in a shared space, score each sampled document by its best similarity rank against any benchmark example, and train a FastText classifier to predict those scores so the full corpus can be filtered. The paper's claim is that this direct matching—requiring no hand-chosen notion of quality—consistently outperforms both unfiltered data and established filters such as DCLM-Baseline and Nemotron-CC HQ, by a 2.1x compute multiplier on average, improving 9 of 10 Core tasks at every scale from $10^{19}$ to $10^{22}$ FLOPs and holding up at the 7B-10x overtrained scale. Targeting a disjoint set of 39 diverse benchmarks still matches or beats baselines on the held-out Core suite, showing the method generalizes rather than merely memorizing target tasks. The scaling analysis adds a second claim: the optimal filtering rate is not a constant but grows with compute, fitted as $F_{\mathrm{opt}}(C) = 4\times10^{-5} C^{0.25}$ on Nemotron-CC—from top 3% at $10^{20}$ FLOPs to top 30% at $10^{23}$ FLOPs—with a weaker trend $F_{\mathrm{opt}} \propto C^{0.14}$ on DCLM-RefinedWeb.

Load-bearing premise

The scaling analysis assumes an effectively unlimited pretraining pool, so its recommended filtering rates are extrapolated from a range where filtering never forces data repetition; at trillion-token budgets the top-3% filter would exhaust a pool like Nemotron-CC and require multiple epochs, a regime the $F_{\mathrm{opt}}(C)$ curve does not model.

Editorial extensions

If this is right

  • Data selection can be made an explicit optimization: choosing a benchmark set determines the capability profile of the resulting model, and jointly targeting diverse benchmarks produces generalists rather than narrow specialists.
  • Fixed filtering rates are suboptimal: because the best rate grows with compute, a single percentage chosen at one scale will be wrong at others, and compute multipliers translate directly into training-cost savings of 35–55% of compute for the same accuracy.
  • Benchmarks do not merely measure progress; they steer it: optimizing for Core benchmarks raised Core scores but dropped held-out Noncore performance below two baselines, a concrete illustration of Goodhart's law in pretraining.
  • Individual task targeting yields specialists: each of the 10 Core benchmarks, targeted alone, produced the best score on that benchmark, with predictable spillover to related tasks and predictable losses elsewhere.
  • The gains persist in the overtrained regime: at 7B parameters trained on 1.4T tokens, BETR Target-Core still leads Core averages by +1.6 to +1.8 points over baselines on both data pools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: if the $F_{\mathrm{opt}}(C)$ trend holds beyond the fitted range, then 'quality' is not a property of documents alone but of the (model scale, target) pair, and curation pipelines should expose the filtering rate as a hyperparameter that tracks roughly $C^{0.25}$ rather than fixing it once.
  • My inference: because the paper measures only before post-training and only on English text, the strongest untested extension is whether BETR's gains survive instruction tuning and transfer to multilingual or code pretraining; the FastText whitespace-tokenizer limitation suggests a BPE-based scorer may be needed for that transfer.
  • My inference: the Goodhart asymmetry documented here implies a collective-action problem: if many labs train on benchmark-aligned data, saturation of common benchmarks will outpace genuine capability growth, and only continually refreshed or held-out benchmark sets can detect the divergence.
  • My inference: the unexplained success of the lower-accuracy FastText scorer over LM scorers suggests the ranking signal is coarse and distributed; a testable extension would be to use BETR scores as soft weights or as a curriculum ordering rather than hard cuts, which the paper lists as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes benchmark-targeted ranking (BETR), a document-level pretraining data selection method that scores documents by embedding-space similarity to benchmark training examples, trains a FastText classifier to predict those scores over a full corpus, and filters by a token threshold. The authors train over 500 models spanning roughly 1e19 to 1e22 FLOPs, fit two-stage per-benchmark scaling laws (loss in model size and tokens, then accuracy in loss), and compare BETR against DCLM-Baseline and Nemotron-CC HQ on two pools, DCLM-RefinedWeb and Nemotron-CC. They report a 1.8x–2.8x compute multiplier over DCLM-Baseline (abstract: 2.1x average), improvement on 9 of 10 Core tasks, generalization to held-out tasks when targeting a disjoint Noncore benchmark set, and a power-law relationship Fopt(C) = 4e-5 C^0.25 between the optimal filtering rate and training FLOPs.

Significance. The central claim—that explicitly aligning pretraining data with target benchmarks yields consistent, scale-dependent compute savings—is important and, if correct, has practical implications for data curation. The paper's empirical scale is a clear strength: over 500 models across two data pools, direct 7B-1x and 7B-10x experiments, bootstrap-uncertainty propagation through the scaling fits, and explicit integrity checks (train vs test targeting, decontamination of DCLM-RefinedWeb, evaluation-blind Noncore targeting). The DCLM-RefinedWeb results are internally consistent and support the direction of the claim. However, the headline numbers rely on undecontaminated Nemotron-CC results and on extrapolated scaling fits beyond the observed range, so the significance is currently conditional rather than fully established.

major comments (3)
  1. [Section 5.1, Table 1, Table 7] The claim that BETR Target-Core outperforms all baselines on 9 out of 10 Core tasks on both data pools is contradicted by the fixed-scale tables on Nemotron-CC. In Table 1 (7B-10x), BETR Target-Core scores 96.4 on SciQ versus 97.0 for DCLM-Baseline, 25.5 on WebQuestions versus 27.6 for No Filter and 27.5 for DCLM-Baseline, and 74.0 on WinoGrande versus 74.7 for DCLM-Baseline. At 7B-1x (Table 7), SciQ and WinoGrande are also below DCLM-Baseline. Thus 'all except Winogrande on Nemotron-CC' is inaccurate, and the abstract's '9 out of 10 tasks across all scales' is not supported by the fixed-scale results. Please correct the claim and report per-task win/loss counts with uncertainty taken into account.
  2. [Section 4.2, Tables 1 and 5, Figure 1] The Nemotron-CC results are exposed to a contamination route that the paper explicitly acknowledges but does not bound. Section 4.2 states that Nemotron-CC is not decontaminated because n-gram matching cannot detect rephrased text, and that the pool includes 1.9T synthetic tokens created by model-based rephrasing. BETR Target-Core selects documents by similarity to Core training examples; if Nemotron-CC contains rephrased Core test or validation items, this selection would preferentially retrieve them. The 2.5x Nemotron-CC multiplier is the larger component of the 2.1x average headline, and the Table 4 decontamination ablation is run only on DCLM-RefinedWeb, so it does not address this route. Please decontaminate Nemotron-CC with a rephrase-resistant method (e.g., embedding-based near-duplicate search against Core test/validation sets) and re-report the affected results, or restrict the headline compute-multiplier and 9-of-10 claims to the decontaminated DCLM-RefinedWeb pool.
  3. [Section 6.2, Figure 10, Limitations] The power law Fopt(C) = 4e-5 C^0.25 is fitted to four discrete filtering rates (3%, 10%, 30%, 100%) over the observed range of roughly 1e19–1e22 FLOPs and then extrapolated to 1e23–1e24 FLOPs, where the 'unlimited data' assumption breaks. Top 3% of Nemotron-CC is about 190B tokens and top 10% about 630B tokens, which is below the multi-trillion-token training budgets typical at those compute levels. The Limitations paragraph acknowledges this, but the main-text takeaway and the Introduction state the extrapolated trend without a data-constraint caveat. Please either explicitly restrict the Fopt claim to the observed range or incorporate a data-constrained scaling model (e.g., with repetition) before presenting the power law as a general result.
minor comments (4)
  1. [Table 2] Several entries in Table 2 appear corrupted or misformatted (e.g., '80.3 3', '54.18', '15.611 7.810'); the table needs to be regenerated with clean numeric formatting.
  2. [Section 4.1 (Lambada OpenAI)] The paper reserves one-third of the Lambada OpenAI test set for targeting and evaluates on the remaining two-thirds; please clarify whether the targeting subset is disjoint from the evaluation subset and whether any benchmark examples overlap across the two splits.
  3. [Section C.4 (accuracy-to-loss fits)] The sigmoid accuracy mapping includes an appended point at L=0, Acc=1.0 for every fit; please report sensitivity to this anchor, since it may systematically affect the steepness and location of the fitted curves, especially for tasks with narrow observed loss ranges.
  4. [Figure 17 and Section 6.2] The bootstrapped confidence bands for the accuracy-to-loss fits of different filtering rates overlap substantially (e.g., TriviaQA); the claim that the filtering-rate transitions are 'clear' would be strengthened by reporting the bootstrap distribution of the Fopt exponent itself and the probability that Fopt increases with C.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: BETR is evaluated on held-out test sets and the scaling-law findings are empirical fits; the main caveat is a data-integrity risk on Nemotron-CC, not a derivation loop.

full rationale

The paper's central BETR claims are not equivalent to their inputs by construction. BETR targets benchmark training examples (Section 3.4) and evaluates on separate Core test sets; the authors directly ablate train-vs-test targeting (Table 4: 62.3 vs 62.4) and decontamination on DCLM-RefinedWeb (-0.2), so the evaluation-aware result does not reduce to the selection criterion. The evaluation-blind result targets 39 Noncore benchmarks disjoint from the Core suite (Sections 3.6 and 4.1), so the held-out gain is not self-defined. The compute multipliers and Fopt(C) trend come from explicit two-stage scaling-law fits (Equations 2 and 4; Section 6.2), and the paper openly states 'We fit a power law for its simplicity' rather than presenting the fit as an independent prediction. Citations to the authors' prior work (Gunter et al. for Core evaluation settings; Li et al. for DCLM baselines) supply benchmarks and baselines, not a uniqueness theorem or a forced choice, so no self-citation chain is load-bearing. Two caveats are worth stating but are not circularity. First, Section 4.2 says 'we do not decontaminate Nemotron-CC' and that the pool contains 1.9T synthetic rephrased tokens; if rephrased Core test content is present, BETR's similarity-to-train-example selection could retrieve it and inflate the larger Nemotron gains. That is a contamination risk requiring external verification, not a by-construction equivalence, because the target is train examples and the paper shows on the decontaminated pool that train and test targeting behave identically. Second, the unlimited-data assumption behind Fopt(C) is explicitly acknowledged in the limitations ('Our scaling analysis assumes unlimited data'); it bounds the extrapolation but does not make the fit circular. Overall the derivation chain is self-contained against standard held-out benchmarks, so the correct circularity finding is low.

Assumptions & free parameters 10 free parameters · 10 assumptions · 0 invented entities

The method contributes a ranking procedure but relies on a series of fitted thresholds, scaling-law coefficients, and extrapolations. The compute multipliers and the filtering-rate law are empirical fits rather than first-principles derivations, and the data-constrained regime is explicitly not modeled.

free parameters (10)
  • Filtering rate threshold = top 3%, 10%, 30%, 100%; default 10%
    Main results use top 10% of tokens; Section 6.2 fits Fopt to choose rates per compute level, making the threshold a fitted quantity.
  • Rank score function = max(1/rank)
    Selected after ablation (Figure 8c); mean-based alternatives score lower.
  • Target granularity = per-example embeddings
    Per-example outperforms centroids and k-means in Figure 8a.
  • Embedding model = Arctic-Embed L v2
    Best among tested embedders (Figure 8b); differences are small.
  • FastText hyperparameters = lr=0.03, dim=128, window=10, min count=5, epochs=5
    Tuned with 1000 Optuna trials on a representative configuration.
  • Classifier label threshold = top 10% vs bottom 90% of rank scores
    Chosen as the default filtering rate; classifier is trained to predict this binary label.
  • Document sample size = 10M documents (~0.1% of pool)
    Chosen to balance compute and coverage; representativeness is assumed.
  • Loss scaling law parameters per benchmark = A, B, E, alpha, beta (Appendix Table 12)
    Chinchilla-form coefficients are fit per dataset and per benchmark; they drive compute multiplier calculations.
  • Accuracy-to-loss sigmoid parameters per benchmark = c1, c2, k, L0 (Appendix Table 13)
    Fitted per benchmark and per dataset to map bits-per-byte to accuracy.
  • Filtering-rate power law exponent = 0.25 on Nemotron-CC, 0.14 on DCLM-RefinedWeb (Fopt = k * C^exponent)
    Empirical fit over four filtering rates; presented as a scaling law but is a fit.
assumptions (10)
  • domain assumption Chinchilla scaling law form L(N,D)=E+A/N^alpha+B/D^beta holds for each benchmark BPB and validation loss.
    Used as the loss model in Appendix C.3; extrapolation beyond the trained grid depends on this form.
  • domain assumption Sigmoid mapping from bits-per-byte to accuracy is stable per benchmark and per dataset.
    Appendix C.4; fits independently per configuration and is applied to compute-optimal loss values.
  • domain assumption Embedding similarity between web documents and benchmark examples is a valid proxy for training usefulness.
    Section 3.4; the whole method rests on this premise, though the embedding-model ablation shows moderate robustness.
  • domain assumption The 10M document sample is representative of the full pool.
    Section 3.3; scores computed on the sample are used to train the full-corpus scorer.
  • domain assumption FastText classifier predictions transfer from the scored sample to the entire corpus.
    Section 3.5; classifier validation accuracy is 75.4%, and the paper argues exact accuracy is not the key metric.
  • domain assumption Compute-optimal model size is identified from validation loss rather than task accuracy.
    Appendix C.3; used to place models on the compute-optimal curve before mapping to benchmark accuracy.
  • domain assumption Critical batch size law from Zhang et al. controls training efficiency for all runs.
    Appendix C.2; batch sizes are set 20% below B* = d * D^(-gamma).
  • domain assumption The pretraining pool is effectively unlimited when deriving optimal filtering rates.
    Section 6.2 and limitations; aggressive filters yield 190B-630B token budgets, so data constraints eventually bind.
  • domain assumption Nemotron-CC has no meaningful benchmark contamination despite no decontamination.
    Section 4.2; authors state n-gram matching cannot detect rephrased text, so they skip decontamination.
  • standard math FLOPs are approximated as 6ND.
    Standard approximation used in Appendix C.3 for compute-optimality calculations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language Models Improve When Pretraining Data Matches Target Tasks." pith.science (2026). https://pith.science/paper/NQ6X7PIE

@misc{pith2026250712466,
  author       = {Pith},
  title        = {Pith review of: Language Models Improve When Pretraining Data Matches Target Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQ6X7PIE}},
  note         = {Machine review of arXiv:2507.12466}
}
abstract

Every data selection method inherently has a target. In practice, these targets often emerge implicitly through benchmark-driven iteration: researchers develop selection strategies, train models, measure benchmark performance, then refine accordingly. This raises a natural question: what happens when we make this optimization explicit? To explore this, we propose benchmark-targeted ranking (BETR), a simple method that selects pretraining documents based on similarity to benchmark training examples. BETR embeds benchmark examples and a sample of pretraining documents in a shared space, scores this sample by similarity to benchmarks, then trains a lightweight classifier to predict these scores for the full corpus. We compare data selection methods by training over 500 models spanning $10^{19}$ to $10^{22}$ FLOPs and fitting scaling laws to them. From this, we find that simply aligning pretraining data to evaluation benchmarks using BETR achieves a 2.1x compute multiplier over DCLM-Baseline (4.7x over unfiltered data) and improves performance on 9 out of 10 tasks across all scales. BETR also generalizes well: when targeting a diverse set of benchmarks disjoint from our evaluation suite, it still matches or outperforms baselines. Our scaling analysis further reveals a clear trend: larger models require less aggressive filtering. Overall, our findings show that directly matching pretraining data to target tasks precisely shapes model capabilities and highlight that optimal selection strategies must adapt to model scale.

Figures

Figures reproduced from arXiv: 2507.12466 by the authors.

Figure 1
Figure 1. Benchmark-targeted ranking (BETR) achieves a 1.8x–2.8x compute multiplier over strong baselines. Scaling curves show accuracy on Core (10 standard benchmarks) at compute￾optimality from 1019 to 1022 FLOPs. Target-Core directly optimizes for evaluated benchmarks, while Target-Noncore targets distinct benchmarks. Both outperform DCLM-Baseline at all scales. †Work done during an internship at Apple. ‡Work done while at… view at source ↗
Figure 2
Figure 2. BETR method overview. We embed benchmark examples and a small sample of pretraining documents (∼0.1% of pool) in a shared space, score the sampled documents by their similarity to benchmarks, then train a classifier on these scores to efficiently rank and filter the entire document pool. 2.2 Data scaling laws Scaling laws enable prediction of model behavior across compute scales, initially focused on how loss decrea… view at source ↗
Figure 3
Figure 3. BETR scoring distributions in practice. Left: Best rank assigned by any benchmark example (BE). With >14,000 benchmark examples competing to score documents, only those ranked in the top 0.002% by some benchmark example reach the top 10% of BETR scores. Right: Cosine similarity to the benchmark example that assigned the best rank. Even top-ranked documents show only moderate similarities (∼0.5) to benchmark examples… view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Comparing datasets using scaling laws. For each dataset, we use a two-stage scaling law approach to predict per-benchmark accuracy [Gadre et al., 2024; Meta AI, 2024; Bhagia et al., 2024]. 1) Fit a loss scaling law as a function of model size and training tokens (color…
Figure 5
Figure 5. Figure 5: Performance scales with benchmark di￾versity. After an initial plateau, targeting additional diverse benchmarks (Noncore) consistently improves performance on held-out tasks (Core). 5.3 Does targeting diverse benchmarks generalize to held-out tasks? While individual be…
Figure 6
Figure 6. Figure 6: Core vs Noncore performance. At 7B￾10x scale, BETR Target-Core achieves the highest Core performance but falls to third place on held￾out Noncore tasks, showing the trade-off of targeted optimization. BETR Target-Core BETR Target-Noncore DCLM-Baseline 0.0% 2.0% 4.0% 6.…
Figure 8
Figure 8. Figure 8: BETR ablation studies showing the impact of (a) target granularity, (b) embedding model choice, (c) scoring function, and (d) elementwise scorer on model performance. All experiments use DCLM-RefinedWeb at 7B-1x scale. Highlighted cells indicate selected settings for o…
Figure 9
Figure 9. Figure 9: Per-benchmark scaling laws. We compare BETR Target-Core against no filtering across all Core benchmarks on Nemotron-CC, showing compute-optimal performance from 1019 to 1022 FLOPs. The scaling laws use two-stage fitting: first from model size and training tokens to per…
Figure 10
Figure 10. Figure 10: Optimal data filtering rate scaling. Left: Compute-optimal Core accuracy for varying BETR Target-Core filtering rates on Nemotron-CC, along with scaling law fits. Center: Probability that each filtering rate is optimal at a given compute scale, estimated from bootstra…
Figure 11
Figure 11. Figure 11: MMLU task accuracy vs. bits-per-byte scaling. Comparison between the original MMLU evaluation format (i.e. by reference)and the reformulated MMLU-Direct variant. Left: Original MMLU accuracy as a function of bits-per-byte shows irregular patterns. Middle: MMLU￾Direct …
Figure 12
Figure 12. Figure 12: Batch size selection for scaling experiments. The dashed gray line shows the critical batch size B∗ as a function of training tokens following Zhang et al. [2024a]. The light gray region indicates our 20% safety buffer below B∗ . Blue circles mark our selected batch s…
Figure 13
Figure 13. Figure 13: Validation loss scaling with model size and training tokens. Left: Validation loss surface as a function of model size and training FLOPs for models trained on BETR Target-Core (Nemotron-CC, top 10% filtering). Contours show fitted scaling law values; the blue curve s…
Figure 14
Figure 14. Figure 14: Task loss scaling with model size and training tokens. Bits-per-byte loss surfaces for individual benchmarks and pretraining validation loss (bottom right) for BETR Target-Core (Nemotron-CC, top 10% filtering). Each panel shows how we fit scaling laws to every benchma…
Figure 15
Figure 15. Figure 15: Task loss scaling comparison across data selection methods. Compute-optimal performance on each benchmark for models trained on Nemotron-CC with different data se￾lection methods. Lines show fitted scaling laws with 95% confidence intervals (shaded). These per-benchma…
Figure 16
Figure 16. Figure 16: Task accuracy as a function of bits-per-byte. Scaling law fits from the second stage of our two-step prediction pipeline, relating benchmark accuracy to task loss (in bits-per-byte) for models trained on the Nemotron-CC BETR Target-Core dataset. Each subplot shows dat…
Figure 17
Figure 17. Figure 17: Impact of data selection methods and filtering intensity on accuracy-to-loss scaling. Each plot shows the fitted relationship between TriviaQA bits-per-byte (BPB) and TriviaQA accuracy for models trained on Nemotron-CC with varying selection approaches. Left: Comparis…
Figure 18
Figure 18. Figure 18: shows the optimal training tokens to model size ratio D/Nopt as a function of compute across different data pools and selection methods. While Hoffmann et al. [2022] identified an optimal ratio of ∼20 tokens per parameter, our models consistently exhibit lower values,…
Figure 19
Figure 19. Figure 19: Per-benchmark accuracy scaling on DCLM-RefinedWeb. Compute-optimal task accuracy for models trained on DCLM-RefinedWeb with different data selection methods. 1020 1022 60 80 Acc ARC-Easy 1020 1022 20 40 60 ARC-Challenge 1020 1022 40 60 HellaSwag 1020 10 22 25 50 75 La…
Figure 20
Figure 20. Figure 20: Per-benchmark accuracy scaling on Nemotron-CC. Compute-optimal task accuracy for models trained on Nemotron-CC with different data selection methods. D.3 Data pool comparison [PITH_FULL_IMAGE:figures/full_fig_p034_20.png]
Figure 21
Figure 21. Figure 21: Data pool comparison with and without BETR filtering. Mean benchmark accuracy (Core average) as a function of compute for models trained on DCLM-RefinedWeb (orange) and Nemotron-CC (blue), with (solid) and without (dashed) BETR Target-Core filtering. Lines show scalin…
Figure 22
Figure 22. Figure 22: extends our filtering rate analysis to the DCLM-RefinedWeb data pool. As with Nemotron￾CC ( [PITH_FULL_IMAGE:figures/full_fig_p035_22.png]
Figure 23
Figure 23. Figure 23: compares mean benchmark accuracy across data selection methods for a fixed model size of 6.6B parameters as a function of training tokens. This analysis extends to the overtraining regime commonly used in production LLMs. The results show that the performance ranking …
Figure 24
Figure 24. Figure 24: Optimal data filtering rate scaling at fixed model size. Left: Mean task accuracy (Core average) for 6.6B models trained with different BETR Target-Core filtering rates on Nemotron￾CC, along with scaling law fits. Center: Probability that each filtering rate is optima…
Figure 25
Figure 25. Figure 25: Topic and format distributions on DCLM-RefinedWeb. 1 We show the WebOr￾ganizer [Wettig et al., 2025] topic and format distributions for four datasets: unfiltered DCLM￾RefinedWeb, DCLM-Baseline, BETR Target-Noncore, and BETR Target-Core (the latter three filtered to to…

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 conditional novelty 8.0 of 10

    DataComp-VLM benchmark shows instruction-heavy data mixing outperforms filtering for VLM training, with DCVLM-Baseline achieving 63.6% on 33 tasks for 8B models (+5.4pp over FineVision).

  2. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    DataComp-VLM benchmark shows instruction-heavy data mixtures outperform caption-heavy ones for VLM training, with DCVLM-Baseline reaching 63.6% on 33 tasks using 200B tokens, +5.4pp over FineVision.

  3. Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality

    cs.CL 2026-06 conditional novelty 6.0 of 10

    Web graph centrality from Common Crawl supplies an orthogonal signal for pretraining data selection that improves language model performance when central and peripheral hosts are balanced.

  4. Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization

    cs.LG 2026-03 unverdicted novelty 6.0 of 10

    CAMEL is a scaling law capturing nonlinear model-size and mixture interactions to extrapolate optimal data mixtures for large LLMs from small-model experiments, reducing optimization cost by 50% and improving benchmar...

  5. Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

    cs.LG 2026-02 conditional novelty 6.0 of 10

    For most benchmarks, the best achievable post-training accuracy follows a stable sigmoid curve in pre-training compute; math reasoning is the exception, with a boundary that keeps rising over time.

  6. Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Classifier-based quality filtering for LLM pretraining improves downstream tasks by implicitly filtering the reference high-quality set rather than by mimicking it, and its quality scores fail a data-conditioning test.

Reference graph

Works this paper leans on

122 extracted references · 1 canonical work pages · cited by 5 Pith papers

  1. [1]

    Semdedup: Data-efficient learning at web-scale through semantic deduplication

    Amro Abbas, Kushal Tirumala, D \'a niel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023

    arXiv 2023

  2. [2]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr'on, and Sumit K. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  3. [3]

    Optuna: A next-generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623--2631, 2019

    2019

  4. [4]

    Efficient online data mixing for language model pre-training

    Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. arXiv preprint arXiv:2312.02406, 2023

    arXiv 2023

  5. [5]

    A survey on data selection for language models

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint arXiv:2402.16827, 2024

    arXiv 2024

  6. [6]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019

    arXiv 1905

  7. [7]

    On deepseek and export controls, January 2025

    Dario Amodei. On deepseek and export controls, January 2025. URL https://www.darioamodei.com/post/on-deepseek-and-export-controls

    2025

  8. [8]

    Perplexed by perplexity: Perplexity-based data pruning with small reference models

    Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541, 2024

    arXiv 2024

Show all 122 references
  1. [9]

    System card: Claude opus 4 & claude sonnet 4

    Anthropic . System card: Claude opus 4 & claude sonnet 4. https://www.anthropic.com/claude-4-system-card, May 2025. Claude 4 family system card

    2025

  2. [10]

    Semantic parsing on F reebase from question-answer pairs

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on F reebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533--1544, Seattle, Washington, USA, October 2013. Associatio...

    2013

  3. [11]

    Chinchilla scaling: A replication attempt

    Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102, 2024

    2024 arXiv

  4. [12]

    Compute multipliers, 2023

    James Betker. Compute multipliers, 2023. URL https://nonint.com/2023/11/05/compute-multipliers/. Blog post

    2023

  5. [13]

    When you think it through, there are only 2 fundamental approaches to data selection

    Lucas Beyer. When you think it through, there are only 2 fundamental approaches to data selection. Tweet, March 2025. URL https://x.com/giffmana/status/1898664177452953701. Twitter/X

    2025

  6. [14]

    Establishing task scaling laws via compute-efficient model ladders

    Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A Smith, Dirk Groeneveld, Pang Wei Koh, et al. Establishing task scaling laws via compute-efficient model ladders. arXiv preprint arXiv:2412.04403, 2024

    2024 arXiv

  7. [15]

    Deepseek llm: Scaling open-source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024

    2024 arXiv

  8. [16]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432--7439, 2020

    2020

  9. [17]

    Does your data spark joy? performance gains from domain upsampling at the end of training

    Cody Blakeney, Mansheej Paul, Brett W Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training. arXiv preprint arXiv:2406.03476, 2024

    2024 arXiv

  10. [18]

    Loss-to-loss prediction: Scaling laws for all datasets

    David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach, and Sham Kakade. Loss-to-loss prediction: Scaling laws for all datasets. arXiv preprint arXiv:2411.12925, 2024 a

    2024 arXiv

  11. [19]

    Color-filter: Conditional loss reduction filtering for targeted language model pre-training

    David Brandfonbrener, Hanlin Zhang, Andreas Kirsch, Jonathan Richard Schwarz, and Sham Kakade. Color-filter: Conditional loss reduction filtering for targeted language model pre-training. Advances in Neural Information Processing Systems, 37: 0 97618--97649, 2024 b

    2024

  12. [20]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  13. [21]

    Distillation scaling laws

    Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws. arXiv preprint arXiv:2502.08606, 2025

    2025 arXiv

  14. [22]

    Datarater: Meta-learned dataset curation

    Dan A Calian, Gregory Farquhar, Iurii Kemaev, Luisa M Zintgraf, Matteo Hessel, Jeremy Shar, Junhyuk Oh, Andr \'a s Gy \"o rgy, Tom Schaul, Jeffrey Dean, et al. Datarater: Meta-learned dataset curation. arXiv preprint arXiv:2505.17895, 2025

    2025

  15. [23]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024 a

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024 a

    2024

  16. [24]

    Skill-it! a data-driven skills framework for understanding and training language models

    Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher R \'e . Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36: 0 36000--36040, 2023

    2023

  17. [25]

    Aioli: A unified optimization framework for language model data mixing

    Mayee F Chen, Michael Y Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher R \'e . Aioli: A unified optimization framework for language model data mixing. arXiv preprint arXiv:2411.05735, 2024 b

    2024 arXiv

  18. [26]

    Take the bull by the horns: Hard sample-reweighted continual training improves llm generalization

    Xuxi Chen, Zhendong Wang, Daouda Sow, Junjie Yang, Tianlong Chen, Yingbin Liang, Mingyuan Zhou, and Zhangyang Wang. Take the bull by the horns: Hard sample-reweighted continual training improves llm generalization. arXiv preprint arXiv:2402.14270, 2024 c

    2024 arXiv

  19. [27]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  20. [28]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  21. [29]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  22. [30]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  23. [31]

    Deepseek-v3 technical report

    DeepSeek-AI . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  24. [32]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

    2019

  25. [33]

    Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training

    Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. arXiv preprint arXiv:2504.13161, 2025

    2025 arXiv

  26. [34]

    Eli5: Long form question answering

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190, 2019

    1907 arXiv

  27. [35]

    Doge: Domain reweighting with generalization estimation

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393, 2023

    2023 arXiv

  28. [36]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Representations, 2024

    2024

  29. [37]

    Language models scale reliably with over-training and on downstream tasks

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540, 2024

    2024 arXiv

  30. [38]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  31. [39]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gemini Team . Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025

    2025 arXiv

  32. [40]

    Scaling laws for data filtering--data curation cannot be compute agnostic

    Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling laws for data filtering--data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22702--22711, 2024

    2024

  33. [41]

    Task-adaptive pretrained language models via clustered-importance sampling

    David Grangier, Simin Fan, Skyler Seto, and Pierre Ablin. Task-adaptive pretrained language models via clustered-importance sampling. arXiv preprint arXiv:2410.03735, 2024

    2024 arXiv

  34. [42]

    Data selection via optimal control for language models

    Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=dhAL5fy8wS

    2025

  35. [43]

    Apple intelligence foundation language models

    Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, et al. Apple intelligence foundation language models. arXiv preprint arXiv:2407.21075, 2024

    2024

  36. [44]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  37. [45]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  38. [46]

    Essential-web v1

    Andrew Hojel, Michael Pust, Tim Romanski, Yash Vanjani, Ritvik Kapila, Mohit Parmar, Adarsh Chaluvaraju, Alok Tripathy, Anil Thomas, Ashish Tanwer, et al. Essential-web v1. 0: 24t tokens of organized web data. arXiv preprint arXiv:2506.14111, 2025

    2025 arXiv

  39. [47]

    NeMo-Curator : a toolkit for data curation

    Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Shrimai Prabhumoye, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ryan Wolf, Sarah Yurick, and Varun Singh. NeMo-Curator : a toolkit for data curation. https://github.com/NVIDIA/NeMo-Curator, 2025

    2025

  40. [48]

    Adaptive data optimization: Dynamic sample selection with scaling laws

    Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and J Zico Kolter. Adaptive data optimization: Dynamic sample selection with scaling laws. arXiv preprint arXiv:2410.11820, 2024

    2024 arXiv

  41. [49]

    Pubmedqa: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natur...

    2019

  42. [50]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017

    2017 arXiv

  43. [51]

    Bag of tricks for efficient text classification

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016

    2016 arXiv

  44. [52]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  45. [53]

    Large language model-guided document selection

    Xiang Kong, Tom Gunter, and Ruoming Pang. Large language model-guided document selection. arXiv preprint arXiv:2406.04638, 2024

    2024 arXiv

  46. [54]

    Axlearn: Modular large model training on heterogeneous infrastructure

    Mark Lee, Tom Gunter, Chang Lan, John Peebles, Hanzhi Zhou, Kelvin Zou, Sneha Bangalore, Chung-Cheng Chiu, Nan Du, Xianzhi Du, et al. Axlearn: Modular large model training on heterogeneous infrastructure. arXiv preprint arXiv:2507.05411, 2025

    2025

  47. [55]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processin...

    2024

  48. [56]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023

    2023 arXiv

  49. [57]

    Rho-1: Not all tokens are what you need

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965, 2024

    2024 arXiv

  50. [58]

    Program induction by rationale generation: Learning to solve and explain algebraic word problems

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146, 2017

    2017 arXiv

  51. [59]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020

    2007 arXiv

  52. [60]

    Regmix: Data mixture as regression for language model pre-training

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024

    2024 arXiv

  53. [61]

    Datadecide: How to predict best pretraining data with small experiments

    Ian Magnusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, et al. Datadecide: How to predict best pretraining data with small experiments. arXiv preprint arXiv:2504.11393, 2025

    2025 arXiv

  54. [62]

    Rephrasing the web: A recipe for compute and data-efficient language modeling

    Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380, 2024

    2024 arXiv

  55. [63]

    When less is more: Investigating data pruning for pretraining llms at scale

    Max Marion, Ahmet \"U st \"u n, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564, 2023

    2023 arXiv

  56. [64]

    Llms on the line: Data determines loss-to-loss scaling laws

    Prasanna Mayilvahanan, Thadd \"a us Wiedemer, Sayak Mallick, Matthias Bethge, and Wieland Brendel. Llms on the line: Data determines loss-to-loss scaling laws. arXiv preprint arXiv:2502.12120, 2025

    2025 arXiv

  57. [65]

    An empirical model of large-batch training

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018

    2018 arXiv

  58. [66]

    The llama 3 herd of models

    Meta AI . The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  59. [67]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  60. [68]

    llm-foundry: Eval\_gauntlet.md

    MosaicML . llm-foundry: Eval\_gauntlet.md. https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md, 2023

    2023

  61. [69]

    Scaling data-constrained language models

    Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36: 0 50358--50376, 2023

    2023

  62. [70]

    Gpt-4o system card

    OpenAI . Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  63. [71]

    gzip predicts data-dependent scaling laws

    Rohan Pandey. gzip predicts data-dependent scaling laws. arXiv preprint arXiv:2405.16684, 2024

    2024 arXiv

  64. [72]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germ \'a n Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \'a ndez. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016

    2016 arXiv

  65. [73]

    Bbq: A hand-built bias benchmark for question answering

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021

    2021 arXiv

  66. [74]

    Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021

    2021 arXiv

  67. [75]

    Patronus ai launches enterprisepii, the industry's first llm dataset for detecting business-sensitive information

    PatronusAI . Patronus ai launches enterprisepii, the industry's first llm dataset for detecting business-sensitive information. https://www.patronus.ai/announcements/patronus-ai-launches-enterprisepii-the-industrys-first-llm-dataset-for-detecting-business-sensitive-information, 2023

    2023

  68. [76]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv pr...

    2023 arXiv

  69. [77]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37: 0 30811--30849, 2024

    2024

  70. [78]

    Computational optimal transport: With applications to data science

    Gabriel Peyr \'e , Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning , 11 0 (5-6): 0 355--607, 2019

    2019

  71. [79]

    Qwen3 technical report

    Qwen Team . Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  72. [80]

    Scaling language models: Methods, analysis & insights from training gopher

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  73. [81]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  74. [82]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  75. [83]

    Coqa: A conversational question answering challenge

    Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7: 0 249--266, 2019

    2019

  76. [84]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  77. [85]

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Association for the Advancement of Artificial Intelligence (AAAI) Spring Symposium, 2011. URL https://people.ict.usc.edu/ gordon/copa.html

    2011

  78. [86]

    How to train data-efficient llms

    Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient llms. arXiv preprint arXiv:2402.09668, 2024

    2024 arXiv

  79. [87]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  80. [88]

    Socialiqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  81. [89]

    Analysing mathematical reasoning abilities of neural models

    David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. arXiv preprint arXiv:1904.01557, 2019

    1904 arXiv

  82. [90]

    Pretraining on the test set is all you need

    Rylan Schaeffer. Pretraining on the test set is all you need. arXiv preprint arXiv:2309.08632, 2023

    2023 arXiv

  83. [91]

    Why has predicting downstream capabilities of frontier ai models with scale remained elusive? arXiv preprint arXiv:2406.04391, 2024

    Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier ai models with scale remained elusive? arXiv preprint arXiv:2406.04391, 2024

    2024 arXiv

  84. [92]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  85. [93]

    Noam M. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  86. [94]

    Scaling laws for optimal data mixtures

    Mustafa Shukor, Louis Bethune, Dan Busbridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby, and Pierre Ablin. Scaling laws for optimal data mixtures. arXiv preprint arXiv:2507.09404, 2025

    2025

  87. [95]

    Predictive data selection: The data that predicts is the data that teaches

    Kashun Shum, Yuzhen Huang, Hongjian Zou, Qi Ding, Yixuan Liao, Xiaoxin Chen, Qian Liu, and Junxian He. Predictive data selection: The data that predicts is the data that teaches. arXiv preprint arXiv:2503.00808, 2025

    2025 arXiv

  88. [96]

    Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model

    Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. ...

    2022 arXiv

  89. [97]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on mach...

    2023

  90. [98]

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. arXiv preprint arXiv:2412.02595, 2024 a

    2024 arXiv

  91. [99]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024 b

    2024

  92. [100]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  93. [101]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL https://huggingface.co/datasets/teknium/OpenHermes-2.5

    2023

  94. [102]

    Improving pretraining data using perplexity correlations

    Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. Improving pretraining data using perplexity correlations. arXiv preprint arXiv:2409.05816, 2024

    2024 arXiv

  95. [103]

    Zyda-2: a 5 trillion token high-quality dataset

    Yury Tokpanov, Paolo Glorioso, Quentin Anthony, and Beren Millidge. Zyda-2: a 5 trillion token high-quality dataset. arXiv preprint arXiv:2411.06068, 2024

    2024 arXiv

  96. [104]

    200,000+ jeopardy! questions

    Bojan Tunguz. 200,000+ jeopardy! questions. Kaggle, 2019. URL https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions

    2019

  97. [105]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017

    2017

  98. [106]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017

    2017 arXiv

  99. [107]

    Ccnet: Extracting high quality monolingual datasets from web crawl data

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm \'a n, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019

    1911 arXiv

  100. [108]

    QuRating : Selecting high-quality data for training language models

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating : Selecting high-quality data for training language models. In International Conference on Machine Learning (ICML), 2024

    2024

  101. [109]

    Organize the web: Constructing domains enhances pre-training data curation

    Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation. arXiv preprint arXiv:2502.10341, 2025

    2025 arXiv

  102. [110]

    Small-scale proxies for large-scale transformer training instabilities

    Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023

    2023 arXiv

  103. [111]

    C-pack: Packed resources for general chinese embeddings

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641--649, 2024

    2024

  104. [112]

    Doremi: Optimizing data mixtures speeds up language model pretraining

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36: 0 69798--69818, 2023 a

    2023

  105. [113]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023 b

    2023

  106. [114]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022

    2022 arXiv

  107. [115]

    Arctic-embed 2.0: Multilingual retrieval without compromise

    Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. Arctic-embed 2.0: Multilingual retrieval without compromise. arXiv preprint arXiv:2412.04506, 2024 a

    2024 arXiv

  108. [116]

    Mates: Model-aware data selection for efficient pretraining with data influence models

    Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. Advances in Neural Information Processing Systems, 37: 0 108735--108759, 2024 b

    2024

  109. [117]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  110. [118]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  111. [119]

    How does critical batch size scale in pre-training? arXiv preprint arXiv:2410.21676, 2024 a

    Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham Kakade. How does critical batch size scale in pre-training? arXiv preprint arXiv:2410.21676, 2024 a

    2024 arXiv

  112. [120]

    mgte: Generalized long-context text representation and reranking models for multilingual text retrieval

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. arXiv preprint arXiv:2407.19669, 2024 b

    2024 arXiv

  113. [121]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023

    2023 arXiv

  114. [122]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.