Pith. sign in

REVIEW 4 major objections 5 minor 63 references

Data Efficacy for Language Model Training

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reordering a fixed training corpus by gradient-derived scores improves language model accuracy with no added data or parameters.

desk verdict Folding ordering is a solid, non-circular contribution, but the paper's headline gains are undercut by single-run DELT numbers against a seed-averaged baseline. read the letter →

arxiv 2506.21545 v1 pith:NRLL5TVL submitted 2025-06-26 cs.CL cs.AIcs.LGcs.PF

Yalun Dai , Yangyu Huang , Xin Zhang , Wenshan Wu , Chong Li , Wenhui Lu , Shijie Cao , Li Dong
show 1 more author
Scarlett Li
This is my paper
classification cs.CLcs.AIcs.LGcs.PF
keywords dataefficacyorderingscoringselectionlearnability-qualityfoldingcurriculumlearninglanguagemodelpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper defines 'data efficacy' as improving language models by changing how fixed training data is organized, rather than by choosing which data to keep or adding more data. It proposes a three-component pipeline it calls DELT — data scoring, optional data selection, and data ordering — and designs two new pieces: Learnability-Quality Scoring (LQS), which rates each sample by the way its gradient changes across training steps, and Folding Ordering (FO), which interleaves a score-sorted list in several passes. On the OLMo benchmark suite, the LQS+FO combination beats random shuffling, plain curriculum sorting, and the KenLM and PDS baselines across models from 160M to 1B parameters and training sets from 1B to 50B tokens, and it also improves math and code post-training. The same pipeline applied on top of data selection improves both accuracy and data efficiency, showing that ordering and selection are compatible rather than competing.

What carries the argument

The machinery that carries the argument is the LQS score of Equation 9: $$\gamma_n = \sum_{t=1}^{T-1} \frac{\lambda_{t+1}^\top \nabla \ell(x_n, \theta_t)}{\|\nabla \ell(x_n, \theta_{t+1})\|},$$ where $\lambda_t$ is the target gradient vector of Equation 8, obtained by propagating the downstream loss $J(\theta)$ backward through the training trajectory. The numerator charges a sample for aligning with the direction that reduces the downstream loss; the denominator charges it for remaining hard, so high-scoring samples are both high-quality and highly learnable. Because computing this on the full corpus is expensive, the authors annotate a small proxy subset with these scores, train a small scorer to predict them, and then score the whole corpus. Folding Ordering then converts the score vector into a training order: sort ascending and take every $L$-th element in $L$ consecutive passes, with $L=3$ chosen as the default, so the model gets a repeated easy-to-hard progression without duplicating data.

What would settle it

Train one fixed corpus and model with LQS+Folding and with random shuffling, repeated over at least three seeds; the claim predicts a clear average gain on the OLMo benchmarks. The claim is falsified if the gain disappears within seed noise, or if the score ranking computed by the 125M proxy model fails to transfer when the same ordering is applied to a 1B target model in a controlled head-to-head.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that the presentation order of training data is a first-class source of language model performance. Reordering the same corpus — no samples added, no parameters changed — yields measurable gains on downstream reasoning benchmarks, with the largest gains coming from combining a gradient-consistency scorer with a multi-pass interleaved ordering. The authors argue that pure score-sorted curriculum learning suffers from forgetting and distribution bias because the model sees all easy examples first and all hard examples last, and their folding schedule answers that by cycling through the sorted list, so every part of the score range appears in each pass. They also claim this ordering benefit survives when data selection is applied, meaning a user can simultaneously train on fewer tokens and get better results.

Load-bearing premise

The load-bearing premise is that the gradient geometry of a small 125M-parameter proxy model, measured against a small curated reference set, predicts which ordering helps much larger models on unrelated benchmarks; if that transfer fails, the scores carry no ordering information.

Editorial extensions

If this is right

  • Any existing pretraining run can be improved by reordering its own data; the only extra cost is a one-time scoring pass with a small model.
  • The ordering gain does not wash out as the model or corpus grows: the paper reports the effect at 160M, 470M, and 1B parameters and at 1B, 10B, and 50B tokens.
  • Ordering and selection are compatible: putting DELT on top of KenLM, PDS, or the proposed scorer improves both accuracy and the fraction of the corpus that must be kept.
  • The method transfers to domain-specific continued training, with gains reported on math and code benchmarks when ordering is applied after pretraining.
  • The default of $L=3$ folding layers outperforms $L=1$ (pure curriculum) on the reported benchmarks, so even the ordering component alone is an improvement over sorting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the score is computed from gradients of a small proxy model, the transfer claim could be tested directly by re-running the ordering with the target model's own gradients; if the gap widens, the scoring step is even more useful than the paper shows.
  • Inference: the reference-set choice (LIMA for general data) is likely a sensitive knob; swapping the 1,030-example reference set for another small set of equal quality would show whether ordering gains are robust or reference-specific.
  • Inference: folding layer count $L$ is empirically peaked at 3 for these sizes, but the mechanism suggests $L$ should roughly track the number of distinct difficulty levels in the corpus; an adaptive $L$ could be the natural scaling extension to terabyte-scale pretraining.
  • Inference: the same learnability signal that orders data could be reused mid-run to identify samples that have already been mastered, connecting data efficacy to dynamic truncation or on-the-fly data selection during training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper defines "data efficacy" as improving language model performance by reorganizing training data without changing dataset content or model architecture, and proposes a DELT paradigm with three components: data scoring, optional data selection, and data ordering. It introduces Learnability-Quality Scoring (LQS), which combines per-sample gradient-magnitude ratios and gradient alignment with a target vector, and Folding Ordering (FO), which interleaves curriculum-sorted data over L folds. The authors report pre-training experiments on Mistral-style models from 160M to 1B parameters on RedPajama subsets, post-training experiments on OpenWebMath and The-Stack-v2, and evaluations on OLMo, code, and math benchmarks. They claim DELT consistently outperforms baselines, and that data efficacy and data efficiency are compatible.

Significance. If the reported effect is real, the claim that reordering existing training data alone improves downstream LM performance across model sizes and data scales is significant and practically valuable, since it is almost cost-free relative to data collection or architecture changes. The paper also proposes a general paradigm that could unify curriculum learning, annealing, and data selection. Strengths include public code, a broad evaluation suite, multiple model sizes and data scales, and the use of held-out evaluation benchmarks distinct from the reference set used for scoring. The significance is conditional, however, on resolving the variance and hyperparameter-selection issues detailed below; the current evidence does not yet support the central "consistently outperforms" claim.

major comments (4)
  1. [§5.2, Table 1 caption and Table 9] The proposed DELT rows are reported without any variance estimate, while only the conventional baseline is averaged over three random seeds. For example, at 160M/1B tokens the average gain is about 1.7 points (36.37 to 38.08), and individual benchmarks move in opposite directions across settings (e.g., Wino improves from 50.72 to 53.04 at 160M but declines from 52.07 to 51.28 at 1B; ARC-e declines from 41.88 to 41.07 at 50B in Table 9b). Run-to-run variation at this scale on small multiple-choice benchmarks can easily be of this magnitude. Because the baseline is stabilized by averaging but the proposed method is not, the comparison is asymmetric and the "consistently outperforms" statement is not distinguishable from seed noise. The authors should report means and standard deviations over at least three seeds for the proposed pipeline, or provide confidence intervals and paired tests.
  2. [§5.3, Table 4 and Figure 7] The ordering direction (ascending vs. descending) and the folding layer L are selected using the same OLMo evaluation benchmarks on which final results are reported. Table 4 shows that ascending sorting helps while descending sorting hurts, and Figure 7/Table 12 show that L=3 is chosen because it peaks on those benchmarks. With gains of roughly 1-2 average points, selecting these free hyperparameters on the test benchmarks can inflate the apparent effect. The paper should either prespecify these choices, select them on a held-out validation split, or demonstrate that the results are stable across a range of L and both directions.
  3. [Appendix D, Eqs. (11)-(14)] The reliability score R(θ_{t+1}) = ||λ_{t+1}|| is introduced as a weighting that reflects model capability, but it cancels exactly in the derivation: Eq. (12) simplifies to Eq. (14), which is the same as Eq. (9) in the main text. Consequently, the stated motivation that "a stronger model will assign more weight to the scores" is not implemented by the final scoring function. This is not a numerical error, but it is a mismatch between the narrative and the method. Either R should be retained as an explicit weighting factor, or the claim that reliability is part of LQS should be removed.
  4. [§5.1 and Appendix E] The entire scoring pipeline rests on a 125M proxy model and a tiny reference set (1,030 LIMA examples for general data), and the resulting scores are used to order training for models up to 1B parameters. The paper does not provide any sensitivity analysis with respect to proxy model size, reference set size, or training trajectory length T. Since the central claim is that gradient-geometry scores transfer across model scales and downstream tasks, the authors should at least report an ablation varying proxy capacity or reference data to show that the ordering benefit is not an artifact of a particular proxy choice.
minor comments (5)
  1. [§1, Introduction] The sentence "This area complements data efficacy" appears to be a typo; it should read "data efficiency," since the surrounding text contrasts with data efficiency.
  2. [Table 12 caption] The phrase "which is three times the random average results" is unclear; it likely means the conventional baseline is averaged over three random seeds. Please rephrase.
  3. [Table 2] The rows for "LQS - Folding" and "LQS ✓ Folding" are numerically identical across all eight benchmarks. If the selection row reports the best result over selection ratios and that best occurs at r=1.0 (i.e., no actual selection), this should be stated explicitly; otherwise the duplicate rows look like an error.
  4. [Figure 5] Figure 5 is never referenced in the main text and no numeric values or selection ratios are given in the caption. Please add a pointer in the text and summarize the shown ratios.
  5. [Eq. (8)] The target-vector recursion involves a Hessian-vector product ∇²L(θ_t, γ)λ_{t+1}; the paper does not describe how these products are computed or approximated in practice, nor how the computational cost compares with PDS [10].

Circularity Check

2 steps flagged · score 4.0 of 10

LQS algebraically collapses to a reweighted PDS score imported from self-cited work, and the folding layer L and ordering direction are selected on the same OLMo benchmarks later used to report gains; the core ordering comparison retains independent empirical content.

  1. renaming known result [Section 4.1.2, Eqs. (8)-(9); Appendix D, Eqs. (12)-(14)]
    "As defined in [10], the target vector λ_t is: λ_t = λ_{t+1} + ∇J(θ_t) − η · ∇^2L(θ_t, γ) · λ_{t+1}, if t < T ... γ_n = ∑_{t=1}^{T−1} λ_{t+1}^⊤∇ℓ(x_n, θ_t)/∥∇ℓ(x_n, θ_{t+1})∥"

    LQS is introduced as a two-factor score combining learnability L(x_n) and quality Q(x_n), but the product R·Q·L in Eq. (12) cancels to Eq. (14): γ_n = ∑ λ_{t+1}^⊤∇ℓ(x_n,θ_t)/∥∇ℓ(x_n,θ_{t+1})∥. This is the PDS cosine-alignment score (Eq. 7, with denominator ∥∇ℓ(x_n,θ_t)∥) multiplied by the learnability ratio ∥∇ℓ(x_n,θ_t)∥/∥∇ℓ(x_n,θ_{t+1})∥ and by ∥λ_{t+1}∥. The target vector λ is taken verbatim from PDS [10], a prior paper with overlapping authorship (Li Dong). Thus the claimed new scoring mechanism is a reweighting/reparameterization of a self-cited prior score rather than an independent construction; comparisons of LQS against PDS measure this rescaling.

  2. fitted input called prediction [Section 5.3, Figure 7 caption; Tables 1 and 9]
    "The average performance initially increases and then gradually declines, peaking at L = 3. In the experiments conducted in this paper, L is set to a default value of 3. ... Benchmarks from OLMo [46] are applied."

    The headline 'Ours' rows in Tables 1 and 9 use L=3, and L=3 was selected by inspecting average performance on the OLMo benchmark suite (Figure 7 uses those same OLMo benchmarks). The ascending versus descending ordering direction was likewise chosen from Table 4 on the same benchmarks. The reported DELT gains are therefore partly the result of selecting hyperparameters on the evaluation set that is later used to claim consistent improvement; the 'prediction' is not fully out-of-sample for these configuration choices.

full rationale

The central empirical claim—that reorganizing data with the DELT pipeline improves LM performance—is not by itself circular: the final models are trained and evaluated on OLMo benchmarks that are distinct from the downstream loss J(θ) used in scoring, and the Folding ordering rule is a concrete permutation operation. However, the paper's new scoring method has a partially circular status: its 'learnability-quality' product cancels algebraically, leaving a reweighted PDS score, and the target-vector recursion is imported from PDS [10], a self-cited prior work by overlapping authors. In addition, the folding layer L (and the ordering direction) are tuned on the same OLMo benchmarks used to report the main gains, so part of the reported advantage reflects selection on the evaluation set rather than an independent prediction. These issues do not collapse the whole derivation—the comparison of ordering methods and the multi-epoch/domain results have independent content—but they warrant a moderate circularity score of 4.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central dependency is the transfer of proxy-model gradient scores to target models and benchmarks. The only numeric parameter actually fitted to the evaluation benchmarks is the folding layer L, chosen as 3, plus the ascending ordering direction and the selection ratio when reported 'across all selection ratios.' The reliability score R is defined and then cancels in the final equation, so it is not an independent component.

free parameters (3)
  • Folding layers L = 3
    Set to 3 after observing the peak in average accuracy on the OLMo benchmarks in Figure 7 and Table 12; this is a hyperparameter selected on the evaluation set, not a derived constant.
  • Ordering direction = ascending
    Ablation in Table 4 shows descending order reduces accuracy; ascending is chosen based on test-set performance, making the direction a test-fitted choice.
  • Selection ratio r (when data selection is enabled) = swept
    Results in Tables 2 and 10 report 'the highest scores across all selection ratios,' so the reported efficiency gain is the best over the swept ratio, a form of test-set selection.
assumptions (4)
  • domain assumption Gradient-based scores computed with a 125M proxy model transfer to target models of up to 1B parameters.
    Scores are computed on Fairseq-Dense-125M (Appendix E) and then used to order data for 160M, 470M, and 1B target models (Section 5.1).
  • domain assumption The downstream loss J(theta) on a small high-quality dataset (LIMA, MiniF2F, Epicoder) is a valid proxy for general model quality on the evaluation benchmarks.
    Section 5.1 defines J(theta) on these small sets; the evaluation uses OLMo, HumanEval, MBPP, MathQA, GPQA, which are different distributions.
  • domain assumption The optimal-control target vector recursion in Equation 8, taken from PDS, provides a meaningful reference direction for sample quality.
    Section 4.1.2 adopts this recursion without independently validating it; it is the backbone of the quality score Q(x_n).
  • domain assumption Training for one epoch is the regime where data ordering matters most; multi-epoch behavior is extrapolated from small-scale experiments.
    The introduction motivates the work from one-epoch training; Figure 6 shows multi-epoch results only up to 5 epochs at 160M scale.
invented entities (1)
  • Reliability score R(theta)
    purpose: Measure model capability to weight LQS scores
    Defined in Eq. 11 as ||lambda_{t+1}||, claimed to let stronger models assign more weight, but it cancels algebraically in Eq. 14 and is absent from the final gamma in Eq. 9. No independent evidence supports its role.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data Efficacy for Language Model Training." pith.science (2026). https://pith.science/paper/NRLL5TVL

@misc{pith2026250621545,
  author       = {Pith},
  title        = {Pith review of: Data Efficacy for Language Model Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NRLL5TVL}},
  note         = {Machine review of arXiv:2506.21545}
}
read the original abstract

Data is fundamental to the training of language models (LM). Recent research has been dedicated to data efficiency, which aims to maximize performance by selecting a minimal or optimal subset of training data. Techniques such as data filtering, sampling, and selection play a crucial role in this area. To complement it, we define Data Efficacy, which focuses on maximizing performance by optimizing the organization of training data and remains relatively underexplored. This work introduces a general paradigm, DELT, for considering data efficacy in LM training, which highlights the significance of training data organization. DELT comprises three components: Data Scoring, Data Selection, and Data Ordering. Among these components, we design Learnability-Quality Scoring (LQS), as a new instance of Data Scoring, which considers both the learnability and quality of each data sample from the gradient consistency perspective. We also devise Folding Ordering (FO), as a novel instance of Data Ordering, which addresses issues such as model forgetting and data distribution bias. Comprehensive experiments validate the data efficacy in LM training, which demonstrates the following: Firstly, various instances of the proposed DELT enhance LM performance to varying degrees without increasing the data scale and model size. Secondly, among these instances, the combination of our proposed LQS for data scoring and Folding for data ordering achieves the most significant improvement. Lastly, data efficacy can be achieved together with data efficiency by applying data selection. Therefore, we believe that data efficacy is a promising foundational area in LM training.

Figures

Figures reproduced from arXiv: 2506.21545 by the authors.

Figure 1
Figure 1. Average result across 8 benchmarks for different methods. High performance at the same [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Paradigm of Data Efficacy for LM training. The blue box represents the paradigm DELT. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of LQS scoring method. The left part demonstrates the calculation of the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Illustration of ordering methods. The right one is Folding method, an advanced multi-fold [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The performance of KenLM [19] and PDS [10] under different data selection ratios, both with and without the DELT paradigm. Data efficiency is enhanced when integrated into DELT. Domain Robustness in Post-training ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Influence of the folding layers L. Benchmarks from OLMo [46] are applied. 6 Conclusion Regarding the underexplored research on data efficacy, we propose a general paradigm, DELT, for enhancing data efficacy in language model training. Meanwhile, we explore the relation…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 35 canonical work pages

  1. [1]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730–27744, 2022

    2022

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  4. [4]

    Advances in natural language processing

    Julia Hirschberg and Christopher D Manning. Advances in natural language processing. Science, 349(6245):261–266, 2015

    work page 2015

  5. [5]

    Exploring Sentiment Analysis Techniques in Natural Language Processing: A Comprehensive Review

    Karthick Prasad Gunasekaran. Exploring sentiment analysis techniques in natural language processing: A comprehensive review. arXiv preprint arXiv:2305.14842, 2023

    work page Pith review arXiv 2023

  6. [6]

    Natural language reasoning, a survey.ACM Computing Surveys, 56(12):1–39, 2024

    Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, a survey.ACM Computing Surveys, 56(12):1–39, 2024

    2024

  7. [7]

    Ai- based conversational agents: a scoping review from technologies to future directions

    Sheetal Kusal, Shruti Patil, Jyoti Choudrie, Ketan Kotecha, Sashikala Mishra, and Ajith Abraham. Ai- based conversational agents: a scoping review from technologies to future directions. IEEE Access , 10:92337–92356, 2022

    work page 2022

  8. [8]

    A survey on data selection for language models

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint arXiv:2402.16827, 2024

    arXiv 2024

Show all 63 references
  1. [9]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems , 36:34201–34227, 2023

    2023

  2. [10]

    Data selection via optimal control for language models

    Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models. arXiv preprint arXiv:2410.07064, 2024

    2024 arXiv

  3. [11]

    Curriculum learning for language modeling

    Daniel Campos. Curriculum learning for language modeling. arXiv preprint arXiv:2108.02170, 2021

    2021 arXiv

  4. [12]

    A survey on curriculum learning.IEEE transactions on pattern analysis and machine intelligence, 44(9):4555–4576, 2021

    Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning.IEEE transactions on pattern analysis and machine intelligence, 44(9):4555–4576, 2021

    2021

  5. [13]

    hello-gpt-4o

    OpenAI. hello-gpt-4o. (2024)

    2024

  6. [14]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  7. [15]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014

    2014 arXiv

  8. [16]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

    1997

  9. [17]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  10. [18]

    Scaling laws for data filtering–data curation cannot be compute agnostic

    Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling laws for data filtering–data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22702–22711, 2024

    2024

  11. [19]

    KenLM: Faster and smaller language model queries

    Kenneth Heafield. KenLM: Faster and smaller language model queries. In Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan, editors, Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland, July 2011. Associat...

    2011

  12. [20]

    Claude 3 haiku: our fastest model yet

    Anthropic. Claude 3 haiku: our fastest model yet. (2024). 10

    2024

  13. [21]

    Paml 4: phylogenetic analysis by maximum likelihood

    Ziheng Yang. Paml 4: phylogenetic analysis by maximum likelihood. Molecular biology and evolution, 24(8):1586–1591, 2007

    2007

  14. [22]

    Common crawl – building an open web-scale crawl using hadoop, 2010

    Ahad Rana. Common crawl – building an open web-scale crawl using hadoop, 2010

    2010

  15. [23]

    Project gutenberg, 2004

    Michael Hart. Project gutenberg, 2004

    2004

  16. [24]

    Synthetic data for deep learning , volume 174

    Sergey I Nikolenko et al. Synthetic data for deep learning , volume 174. Springer, 2021

    2021

  17. [25]

    Virtual sensors: Abstracting data from physical sensors

    Sanem Kabadayi, Adam Pridgen, and Christine Julien. Virtual sensors: Abstracting data from physical sensors. In 2006 International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM’06), pages 6–pp. IEEE, 2006

    2006

  18. [26]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  19. [27]

    The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only. Advances in Neural Inf...

    2023

  20. [28]

    Redpajama: an open dataset for training large language models

    Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. Advances in neural information processing systems , 37:116462–116492, 2024

    2024

  21. [29]

    Redstone: Curating general, code, math, and qa data for large language models

    Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, et al. Redstone: Curating general, code, math, and qa data for large language models. arXiv preprint arXiv:2412.03398, 2024

    2024 arXiv

  22. [30]

    Mates: Model-aware data selection for efficient pretraining with data influence models

    Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. Advances in Neural Information Processing Systems , 2024

    2024

  23. [31]

    Training-free dataset pruning for instance segmentation

    Yalun Dai, Lingao Xiao, Ivor Tsang, and Yang He. Training-free dataset pruning for instance segmentation. In The Thirteenth International Conference on Learning Representations

  24. [32]

    P-diff+: Improving learning classifier with noisy labels by noisy negative learning loss

    QiHao Zhao, Wei Hu, Yangyu Huang, and Fan Zhang. P-diff+: Improving learning classifier with noisy labels by noisy negative learning loss. Neural Networks, 144:1–10, 2021

    2021

  25. [33]

    P-diff: Learning classifier with noisy labels based on probability difference distributions

    Wei Hu, QiHao Zhao, Yangyu Huang, and Fan Zhang. P-diff: Learning classifier with noisy labels based on probability difference distributions. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 1882–1889. IEEE, 2021

    2020

  26. [34]

    Semdedup: Data-efficient learning at web-scale through semantic deduplication

    Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023

    2023 arXiv

  27. [35]

    D4: Improving llm pretraining via document de-duplication and diversification

    Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving llm pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems , 36:53983–53995, 2023

    2023

  28. [36]

    Strategic data ordering: Enhancing large language model performance through curriculum learning

    Jisu Kim and Juhwan Lee. Strategic data ordering: Enhancing large language model performance through curriculum learning. arXiv preprint arXiv:2405.07490, 2024

    2024 arXiv

  29. [37]

    Does the order of training samples matter? improving neural data-to-text generation with curriculum learning

    Ernie Chang, Hui-Syuan Yeh, and Vera Demberg. Does the order of training samples matter? improving neural data-to-text generation with curriculum learning. arXiv preprint arXiv:2102.03554, 2021

    2021 arXiv

  30. [38]

    DoReMi: Optimizing data mixtures speeds up language model pretraining

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. DoReMi: Optimizing data mixtures speeds up language model pretraining. In Proceedings of NeurIPS, 2024

    2024

  31. [39]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. In Proceedings of NeurIPS, 2024

    2024

  32. [40]

    Openwebmath: An open dataset of high-quality mathematical web text, 2023

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023

    2023

  33. [41]

    Minif2f: a cross-system benchmark for formal olympiad-level mathematics

    Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: a cross-system benchmark for formal olympiad-level mathematics. arXiv preprint arXiv:2109.00110, 2021. 11

    2021 arXiv

  34. [42]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  35. [43]

    Epicoder: Encompassing diversity and complexity in code generation

    Yaoxiang Wang, Haoling Li, Xin Zhang, Jie Wu, Xiao Liu, Wenxiang Hu, Zhongxin Guo, Yangyu Huang, Ying Xin, Yujiu Yang, et al. Epicoder: Encompassing diversity and complexity in code generation. arXiv preprint arXiv:2501.04694, 2025

    2025

  36. [44]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  37. [45]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  38. [46]

    OLMo: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. OLMo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024

    2024 arXiv

  39. [47]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of ACL, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of ACL, 2019

    2019

  40. [48]

    The winograd schema challenge

    Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Proceedings of KR, 2012

    2012

  41. [49]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of ACL, 2016

    2016

  42. [50]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of EMNLP, 2018

    2018

  43. [51]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    2018 arXiv

  44. [52]

    Piqa: Reasoning about physical common- sense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical common- sense in natural language. In Proceedings of AAAI, 2020

    2020

  45. [53]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text (ACL 2017) , 2017

    2017

  46. [54]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT, 2019

    2019

  47. [55]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  48. [56]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms, 2019

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms, 2019

    2019

  49. [57]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  50. [58]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. 12

    2021 arXiv

  51. [59]

    Efficient large scale language modeling with mixtures of experts

    Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. In Proceedings EMNLP, 2022

    2022

  52. [60]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of ICLR, 2019

    2019

  53. [61]

    Comparing the pearson and spearman correla- tion coefficients across distributions and sample sizes: A tutorial using simulations and empirical data

    Joost CF De Winter, Samuel D Gosling, and Jeff Potter. Comparing the pearson and spearman correla- tion coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychological methods, 21(3):273, 2016. 13 Appendix: Data Efficacy for L...

    2016

  54. [62]

    As described in algorithm 1, we apply a linear transformation to the mean-pooled representations of instances along the sequence length

    on the solved data weights γ. As described in algorithm 1, we apply a linear transformation to the mean-pooled representations of instances along the sequence length. The hidden state size is set to 768. The optimization of algorithm 1 is performed using the AdamW optimizer [6...

  55. [63]

    I am overpowered by the discovery of my own genius for management

    Data scorer training. The data scorer, typically a small LM, is fine-tuned on the automatically annotated data samples in Dprx to predict data scores effectively. (see Section E.2) 4) Full data scoring. The trained data scorer is then applied to infer scores for the entire pre...

    1968

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.