Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Down-weighting low-loss samples during LLM pretraining is claimed to add about a point of average accuracy at 1.4B and 7B scale, backed by new convergence bounds and 124M-to-7B experiments.

desk verdict A useful empirical paper on online loss-based reweighting for LLM pretraining, but the headline theory does not yet attach to the implemented algorithm. read the letter →

arxiv 2502.06733 v1 pith:APR7QEVT submitted 2025-02-10 cs.LG cs.AI

Daouda Sow , Herbert Woisetschläger , Saikiran Bulusu , Shiqiang Wang , Hans-Arno Jacobsen , Yingbin Liang This is my paper · ORCID
classification cs.LGcs.AI
keywords largelanguagemodelpretrainingsamplereweightingloss-basedinstance-leveldataimportanceconvergenceboundsLinUpperstrategydown-weightinglow-losssamples
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that an LLM pretraining pipeline can be improved by a rule that runs per batch for free: reweight each sample by its current loss, down-weighting the samples the model already fits well. Its central empirical claim is that the LinUpper strategy — a capped linear up-weighting of higher-loss samples — beats uniform sampling by about one percentage point on average across 19 reasoning and commonsense tasks at both 1.4B and 7B scale, while adding negligible compute. Its theoretical claim is the first explicit characterization of how loss-based reweighting enters the convergence bound of gradient methods: down-weighting samples with small loss gaps makes the extra term negative and tightens the bound. If true, the method offers an online, curation-free lever on pretraining efficiency that also stacks on top of domain-level reweighting methods.

What carries the argument

The key object is the per-step reweighted gradient update, where each sample's weight $w(x_i; \theta_t)$ is derived purely from that sample's current loss. The machinery has three parts: normalization of raw losses to $[-\alpha, \alpha]$; a strategy function $s_i$, of which LinUpper's $s_i = \min\{h_i + \alpha, \alpha\}$ is the empirically winning choice; and a temperature-controlled softmax $w_i = e^{s_i/r}/\sum_j e^{s_j/r}$ subject to a cap $w_i \leq 2/M$ that the theory requires. The analytic engine is the decomposition $\delta_t = \sum_{i \in B} (1/b - w_{i,t})(f_i(\theta_t) - f_i(\theta^*))$ inside the convergence bound: uniform weighting gives $\delta_t = 0$, and any scheme that consistently down-weights small loss gaps gives $\delta_t \leq 0$, improving the $O(1/\sqrt{T})$ bound. Proposition 1 connects the theory to the algorithm by showing that the KL-regularized minimizer of $\delta_t$ is a capped exponential of the normalized loss, which is exactly the shape LinUpper applies.

What would settle it

On a finite-sum problem where the per-sample optima $f_i(\theta^*)$ are known and differ across samples (for example, linear regression with per-sample irreducible noise), compute the weights Algorithm 1 assigns from normalized raw losses and compare them with the claimed optimum $w_i = C\min\{\exp(\Delta_i/r), 2/M\}$ computed from the true loss gaps; if the two weight vectors are not ordered alike, the theoretical explanation of LinUpper fails in that setting. A second check: train the same model on a heavily deduplicated corpus and on the raw corpus; if LinUpper's gain over uniform sampling does not shrink on the deduplicated corpus, the redundancy-suppression reading of the mechanism is wrong.

Watch

Extended reading notes

Core claim

The core claim is that sample importance during pretraining can be read off the per-sample losses already computed in the forward pass, and that the productive direction is to down-weight low-loss samples. Formally, the paper analyzes the reweighted update $\theta_{t+1} = \theta_t - \eta \sum_{i \in B} w(x_i; \theta_t)\nabla f(x_i; \theta_t)$ and shows that its convergence bound carries an extra term $\delta_t = \sum_{i \in B} (1/b - w_{i,t})(f_i(\theta_t) - f_i(\theta^*))$; assigning above-uniform weight to samples whose loss gap $f_i(\theta_t) - f_i(\theta^*)$ is small makes $\delta_t \geq 0$ and loosens the bound, while down-weighting those samples makes $\delta_t \leq 0$ and tightens it. This justifies LinUpper, which normalizes losses to $[-\alpha, \alpha]$, applies $s_i = \min\{h_i + \alpha, \alpha\}$, and feeds $s_i$ through a temperature-controlled softmax with a cap $w_i \leq 2/M$. The paper further proves (Proposition 1) that the KL-regularized optimizer of $\delta_t$ has the form $w_i = C\min\{\exp(h_i/r), 2/M\}$, matching LinUpper's shape. Empirically, LinUpper improves per-domain perplexity on 5–6 of 7 SlimPajama domains for 124M–300M GPT2 models, improves mean accuracy by 0.99 points (1.4B) and 1.35 points (7B) over 19 FineWeb-trained benchmarks, and improves DoGE and DoReMi domain reweighting when layered on top.

Load-bearing premise

The load-bearing premise is that the raw per-sample loss used by the algorithm orders samples the same way as the true loss gap $f_i(\theta_t) - f_i(\theta^*)$; the paper does not prove that, and if samples differ in their irreducible loss, the optimality proof does not cover the implemented LinUpper.

Editorial extensions

If this is right

  • Down-weighting low-loss samples is claimed to help across the whole tested scale spectrum, from a synthetic linear-regression problem with outliers up to 7B-parameter LLM pretraining.
  • The instance-level gains are claimed to be additive with domain-level reweighting: LinUpper on top of DoGE or DoReMi improves mean 5-shot accuracy in every reported comparison.
  • Because the weights reuse forward-pass losses, the method inserts into an existing pretraining loop with near-zero extra cost and no per-sample statistics stored across epochs.
  • The weight cap $w_i \leq 2/M$ is claimed to be what keeps the method stable where distributionally robust reweighting fails: the capped scheme does not collapse onto the hardest samples and avoids divergence on outlier-heavy data.
  • Benchmark gains are larger for the 1.4B and 7B models than for the 124M–300M models, which the paper attributes to smaller models having limited capacity to exploit reweighting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference beyond the paper: Proposition 1 is proven for loss gaps $\Delta_i = f_i(\theta_t) - f_i(\theta^*)$, while Algorithm 1 computes weights from normalized raw losses $h_i$; if $h_i$ is not proportional to, or at least ordered like, $\Delta_i$ across samples, the optimality proof does not strictly cover the implemented algorithm, so the empirical gains are better read as evidence for the down
  • Inference beyond the paper: if the mechanism works by suppressing redundant or repetitive content, LinUpper's gain over uniform sampling should shrink on aggressively deduplicated corpora and grow on raw web dumps — a testable prediction the paper did not run.
  • Inference beyond the paper: the temperature schedule (near-uniform early in training, sharp later) is itself a curriculum; an adaptive schedule tied to batch loss statistics might capture similar gains without the tuned hyperparameter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes fully online, instance-level, loss-based sample reweighting strategies for LLM pretraining. The main method, LinUpper, down-weights low-loss samples by applying a capped linear transform to normalized per-sample losses and then a temperature-annealed softmax; two alternatives (Quadratic and Extremes) are also studied. The authors derive convergence bounds for reweighted gradient descent (Theorems 1 and 2) and claim that Proposition 1 characterizes LinUpper as the KL-regularized minimizer of the bound's reweighting term. Empirically, they compare against uniform sampling for GPT2-size models on SlimPajama and for 1.4B and 7B Llama models on FineWeb, reporting average improvements across 19 downstream tasks, plus a synthetic linear-regression study and comparisons with DRO baselines.

Significance. If the central claims were fully supported, the paper would make a useful practical contribution: a zero-overhead, fully online reweighting rule that improves LLM pretraining and a convergence-bound analysis of loss-based reweighting. The paper ships a public codebase and detailed hyperparameters, and the breadth of experiments from 124M to 7B parameters is a strength. However, the current version has a load-bearing gap between the theory and the implemented algorithm, and the headline empirical gains are fragile because they rely on single runs and on averages that are dominated by one outlier task. With those issues repaired, the work could be a solid contribution; as written, the claims are stronger than the evidence.

major comments (4)
  1. [4.2, Eq. (5), Algorithm 1, Appendix C.2] Proposition 1 is not established for the algorithm actually implemented. The proof in Appendix C.2 minimizes the KL-regularized δ_t in terms of loss gaps Δ_i = f_i(θ_t) − f_i(θ*), yielding weights proportional to min{exp(Δ_i/r), 2/M}; Eq. (5) instead replaces Δ_i by the normalized raw loss h_i from Algorithm 1, and no argument shows that h_i and Δ_i are monotonically related under Assumption 3. A simple counterexample is M=2 with f_1(θ)=θ² and f_2(θ)=θ²+100 at θ_t=1, where Δ_1=Δ_2=1 but the raw losses 1 and 101 produce different LinUpper weights. Moreover, even if h_i=Δ_i, the softmax over the capped LinUpper scores in Algorithm 1 line 7 and Eq. (4) gives weights proportional to min{exp(h_i/r), 1}, not C min{exp(h_i/r), 2/M}. Thus the optimality result does not explain the implemented LinUpper rule as written; the theory is currently detached from the algorithm.
  2. [6.4, Tables 10 and 11] The reported 7B LUR average gain of 1.44 points is driven almost entirely by one task: SST-2 improves by +19.50 points. Excluding SST-2, the mean difference over the remaining 12 LUR tasks is approximately −0.06. The same task regresses by −7.45 points for the 1.4B model in Table 8. Since all reported numbers appear to come from single runs with no seeds or error bars, the abstract and conclusion statements that LinUpper 'consistently outperforms' uniform sampling are not supported. The authors should report multiple seeds or otherwise quantify run-to-run variance, and should present aggregate gains with and without the SST-2 outlier.
  3. [4.2, Theorem 1, Appendix C.1] Theorem 1 as stated assumes only convexity and the interpolation condition, but its proof invokes Lemma 1, which requires L-smoothness (Assumption 2). The O(1/T) convergence rate is not available for general convex, non-smooth losses, so the theorem statement is missing a necessary smoothness assumption. This is a one-line fix, but it is needed for the stated result to be correct.
  4. [Appendix C.3, Eqs. (35)-(38)] The minibatch SGD proof does not make the conditioning structure explicit: E[||θ_{t+1}−θ*||²] is written as an unconditional expectation on the left while the right-hand side still contains the random quantities f(θ_t) and δ_t without conditioning on θ_t. The argument can be repaired by taking conditional expectations given θ_t and then applying the tower property and Jensen's inequality, but as written the telescoping step is not rigorous. Since Theorem 2 is a central theoretical result, this gap should be fixed in the revision.
minor comments (5)
  1. [A.2] The appendix states that all models are trained with masked language modeling, but the models are autoregressive decoder-only transformers; this should read 'causal language modeling'.
  2. [A.2, Table 6] Table 6 lists the embedding dimension as 786 for both GPT2-small and GPT2-medium; standard GPT2-small uses 768 and GPT2-medium uses 1024, so these entries appear to be typos.
  3. [Algorithm 1] Algorithm 1 uses fmin and fmax as inputs without defining them in the main text. The reference implementation in Listing 2 computes them from the gathered batch, and this should be stated explicitly in the main text.
  4. [7 and abstract] The phrase 'first explicit characterization' / 'first formal characterization' is stronger than what the paper establishes once the Proposition 1 mismatch is accounted for; please temper these claims or justify them after repairing the theory.
  5. [A.4, Figure 5] The caption of Figure 5 says r=1 is a large value, while Section 6.1 says the temperature is annealed from 100 to 1; the scale of r used in the GPT2 experiments should be clarified.

Circularity Check

1 steps flagged · score 6.0 of 10

Proposition 1 derives softmax-capped weights from loss gaps Δ_i, then restates them with normalized raw losses h_i; without an identity h_i=Δ_i the optimality claim is LinUpper's own formula, not an independent prediction.

  1. self definitional [Section 4.2 (Proposition 1) and Appendix C.2 (proof), with Algorithm 1 step 5]
    "Proposition 1. The optimal strategy that minimizes the KL-divergence regularized δt, i.e., δt + r PM i=1 wi log(M wi), is given by wi = C min{exp(hi/r), 2/M}. ... Proof. ... Let ∆i = f (xi; θt) − f (xi; θ∗) be the loss gap. ... we obtain wi = min{C exp(∆i/r), 2/M}. ... Algorithm 1: Normalize losses into interval [−α, α] using linear transform: hi,t = 2(fi,t−fmin)/(fmax−fmin) − 1."

    The proof's Lagrangian solution is over loss gaps Δ_i and delivers weights proportional to exp(Δ_i/r), capped at 2/M. The proposition statement replaces Δ_i with h_i, the batch-min/max normalized raw loss from Algorithm 1, without any lemma identifying h_i with Δ_i or even preserving its order. Under Assumption 3, a common θ* minimizes every f_i, but f_i(θ*) need not be zero or equal across samples, so h_i is not an affine or monotone function of Δ_i. Thus Eq. (5) is not the derived minimizer for the implemented LinUpper weights; the claimed 'optimal strategy' is essentially the algorithm's own softmax-capped formula (Eq. 4 applied to s_i = min{h_i + α, α}) rewritten as a solution.

full rationale

The empirical benchmark comparisons are self-contained: Tables 4 and 5 compare LinUpper directly against uniform sampling on held-out tasks, and these measurements do not depend on the theory. The convergence theorems are generic bias-decomposition results and are not themselves circular. However, the paper's central theoretical bridge, Proposition 1, is not an independent derivation of LinUpper's optimality. The proof solves for weights over loss gaps Δ_i, while the stated result and Algorithm 1 use normalized raw losses h_i; the paper provides no proof that these quantities coincide or are even order-equivalent under its own Assumption 3. Consequently the optimality claim reduces to substituting the algorithm's own softmax-capped formula into a theorem proven for a different input, making the theoretical justification partially fitted to the method rather than derived for it. Because the empirical claims stand independently, the circularity is partial, not total.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The theory depends on the interpolation, convexity, and smoothness assumptions from Section 5, and on an unstated equivalence between raw loss and loss gap. The method's main free parameters are the temperature r and the normalization bound alpha, both chosen by hand.

free parameters (2)
  • temperature r = 100 during warmup, then 1 (Table 7)
    Controls softmax sharpness of the reweighting; sensitivity shown in Fig. 5 where r=1 is essentially uniform and r=0.2 degrades, so the operating regime is chosen by hand.
  • normalization bound alpha (delta) = 1.0 (code default)
    Sets the scaling of normalized losses in the strategies; not tuned but is a design choice that affects the shape of all reweighting functions.
assumptions (3)
  • domain assumption Assumption 3 (interpolation): there exists theta* in argmin f_i for all i
    Used throughout the proofs via Lemma 1 and to set sigma* = 0; not exactly satisfied in LLM pretraining, which is non-convex and only approximately overparameterized.
  • domain assumption Assumptions 1-2: each f_i is convex and L-smooth
    Foundation of the convergence analysis in Theorems 1, 2, and 3; LLM loss surfaces are non-convex.
  • ad hoc to paper Raw per-sample loss h_i is a suitable proxy for the loss gap f_i(theta_t) - f_i(theta*)
    Algorithm 1 reweights using normalized raw losses, while Proposition 1's derivation and the convergence bounds are stated for loss gaps. The equivalence is asserted without proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining." pith.science (2026). https://pith.science/paper/APR7QEVT

@misc{pith2026250206733,
  author       = {Pith},
  title        = {Pith review of: Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/APR7QEVT}},
  note         = {Machine review of arXiv:2502.06733}
}
read the original abstract

Pretraining large language models (LLMs) on vast and heterogeneous datasets is crucial for achieving state-of-the-art performance across diverse downstream tasks. However, current training paradigms treat all samples equally, overlooking the importance or relevance of individual samples throughout the training process. Existing reweighting strategies, which primarily focus on group-level data importance, fail to leverage fine-grained instance-level information and do not adapt dynamically to individual sample importance as training progresses. In this paper, we introduce novel algorithms for dynamic, instance-level data reweighting aimed at improving both the efficiency and effectiveness of LLM pretraining. Our methods adjust the weight of each training sample based on its loss value in an online fashion, allowing the model to dynamically focus on more informative or important samples at the current training stage. In particular, our framework allows us to systematically devise reweighting strategies deprioritizing redundant or uninformative data, which we find tend to work best. Furthermore, we develop a new theoretical framework for analyzing the impact of loss-based reweighting on the convergence of gradient-based optimization, providing the first formal characterization of how these strategies affect convergence bounds. We empirically validate our approach across a spectrum of tasks, from pretraining 7B and 1.4B parameter LLMs to smaller-scale language models and linear regression problems, demonstrating that our loss-based reweighting approach can lead to faster convergence and significantly improved performance.

Figures

Figures reproduced from arXiv: 2502.06733 by the authors.

Figure 1
Figure 1. Left: Geometric curves of the different reweighting functions. Right: Shape of the LinUpper strategy after applying Eq. (4) on top of it for different values of r. These plots are obtained for a batch of 128 uniformly drawn losses. As r increases, LinUpper converges to the uniform averaging method. of loss-based reweighting on the convergence bound of reweighted full gradient descent when the sample losses f(xi ; θ … view at source ↗
Figure 2
Figure 2. Per-domain perplexities on hold-out validation sets under the uniform domain sampling [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Per-domain perplexities on hold-out validation sets under the uniform domain sampling [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Per-domain perplexities on hold-out validation sets under the uniform domain sampling [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Sensitivity of our method LinUpper to the value of r. When the r is large (e.g. r = 1) the performance of our method becomes closer to that of the uniform baseline. Decreasing the value of r leads to diminished effect of low-loss samples, but can also have a negative e…
Figure 6
Figure 6. Figure 6: Distribution of maximum weight values when training our 7B parameter model with Llama architecture. The maximum weight values always stay well below the theoretical upper bound defined in Theorem 2. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Left: Test loss vs training steps for the toy regression problem. Our LinUpper strategy achieves faster convergence compared to all the other methods. Notably, the Quadratic strategy also outperforms the uniform baseline, further highlighting the benefit of selective r…
Figure 8
Figure 8. Figure 8: Regression problem with DRO-KL. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining

    cs.LG 2025-05 conditional novelty 5.0 of 10

    ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.

Reference graph

Works this paper leans on

48 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [5]

    Skill-it! a data-driven skills framework for understanding and training language models

    Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christopher Re. Skill-it! a data-driven skills framework for understanding and training language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=IoizwO1NLf

    work page 2023

  6. [6]

    Take the bull by the horns: Hard sample-reweighted continual training improves llm generalization

    Xuxi Chen, Zhendong Wang, Daouda Sow, Junjie Yang, Tianlong Chen, Yingbin Liang, Mingyuan Zhou, and Zhangyang Wang. Take the bull by the horns: Hard sample-reweighted continual training improves llm generalization. arXiv preprint arXiv:2402.14270, 2024

    arXiv 2024

  7. [7]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  8. [8]

    A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning

    Yehuda Dar, Vidya Muthukumar, and Richard G Baraniuk. A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning. arXiv preprint arXiv:2109.02355, 2021

    arXiv 2021

Show all 48 references
  1. [9]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    Irreducible curriculum for language model pretraining

    Simin Fan and Martin Jaggi. Irreducible curriculum for language model pretraining. arXiv preprint arXiv:2310.15389, 2023

    2023 arXiv

  3. [11]

    DOGE : Domain reweighting with generalization estimation

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. DOGE : Domain reweighting with generalization estimation. In Second Agent Learning in Open-Endedness Workshop, 2023. URL https://openreview.net/forum?id=qiKqsqwYXm

    2023

  4. [12]

    Rethinking importance weighting for deep learning under distribution shift

    Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. Advances in neural information processing systems, 33: 0 11996--12007, 2020

    2020

  5. [13]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  6. [14]

    Adaptive training distributions with scalable online bilevel optimization

    David Grangier, Pierre Ablin, and Awni Hannun. Adaptive training distributions with scalable online bilevel optimization. arXiv preprint arXiv:2311.11973, 2023

    2023 arXiv

  7. [15]

    Fedexp: Speeding up federated averaging via extrapolation

    Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. In The Eleventh International Conference on Learning Representations, 2023

    2023

  8. [16]

    Accelerating deep learning by focusing on the biggest losers

    Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762, 2019

    1910 arXiv

  9. [17]

    Importance weighting can help large language models self-improve

    Chunyang Jiang, Chi-min Chan, Wei Xue, Qifeng Liu, and Yike Guo. Importance weighting can help large language models self-improve. arXiv preprint arXiv:2408.09849, 2024

    2024 arXiv

  10. [18]

    Instance weighting for domain adaptation in NLP

    Jing Jiang and ChengXiang Zhai. Instance weighting for domain adaptation in NLP . In Annie Zaenen and Antal van den Bosch (eds.), Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp.\ 264--271, June 2007

    2007

  11. [19]

    Not all samples are created equal: Deep learning with importance sampling

    Angelos Katharopoulos and Fran c ois Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp.\ 2525--2534. PMLR, 2018

    2018

  12. [20]

    Stochastic re-weighted gradient descent via distributionally robust optimization

    Ramnath Kumar, Kushal Majmundar, Dheeraj Nagaraj, and Arun Sai Suggala. Stochastic re-weighted gradient descent via distributionally robust optimization. arXiv preprint arXiv:2306.09222, 2023

    2023 arXiv

  13. [21]

    Probabilistic margins for instance reweighting in adversarial training

    Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, Masashi Sugiyama, et al. Probabilistic margins for instance reweighting in adversarial training. Advances in Neural Information Processing Systems, 34: 0 23258--23269, 2021

    2021

  14. [22]

    Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding

    Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023

    2023

  15. [23]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020

    2007 arXiv

  16. [24]

    A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity

    Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv...

    2023 arXiv

  17. [25]

    Online batch selection for faster training of neural networks

    Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343, 2015

    2015 arXiv

  18. [26]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv pr...

    2023 arXiv

  19. [27]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  20. [28]

    An online method for a class of distributionally robust optimization with non-convex objectives

    Qi Qi, Zhishuai Guo, Yi Xu, Rong Jin, and Tianbao Yang. An online method for a class of distributionally robust optimization with non-convex objectives. Advances in Neural Information Processing Systems, 34: 0 10067--10080, 2021

    2021

  21. [29]

    Robust optimization over multiple domains

    Qi Qian, Shenghuo Zhu, Jiasheng Tang, Rong Jin, Baigui Sun, and Hao Li. Robust optimization over multiple domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.\ 4739--4746, 2019

    2019

  22. [30]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  23. [31]

    Overparameterized neural networks implement associative memory

    Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Overparameterized neural networks implement associative memory. Proceedings of the National Academy of Sciences, 117 0 (44): 0 27162--27170, 2020

    2020

  24. [32]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 0 (1): 0 5485--5551, 2020

    2020

  25. [33]

    Learning to reweight examples for robust deep learning

    Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pp.\ 4334--4343. PMLR, 2018

    2018

  26. [34]

    Almost sure convergence rates for stochastic gradient descent and stochastic heavy ball

    Othmane Sebbouh, Robert M Gower, and Aaron Defazio. Almost sure convergence rates for stochastic gradient descent and stochastic heavy ball. In Conference on Learning Theory, pp.\ 3935--3971. PMLR, 2021

    2021

  27. [35]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama . https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, June 2023. U...

    2023

  28. [36]

    Doubly robust instance-reweighted adversarial training

    Daouda Sow, Sen Lin, Zhangyang Wang, and Yingbin Liang. Doubly robust instance-reweighted adversarial training. arXiv preprint arXiv:2308.00311, 2023

    2023 arXiv

  29. [37]

    Self-influence guided data reweighting for language model pre-training

    Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pre-training. arXiv preprint arXiv:2311.00913, 2023

    2023 arXiv

  30. [38]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  31. [39]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  32. [40]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pp.\ 94--106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics

    2017

  33. [41]

    Q u R ating: Selecting high-quality data for training language models

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Q u R ating: Selecting high-quality data for training language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings...

    2024

  34. [42]

    Doremi: Optimizing data mixtures speeds up language model pretraining

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305.10429, 2023

    2023 arXiv

  35. [43]

    Reweighting augmented samples by minimizing the maximal expected loss

    Mingyang Yi, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Zhi-Ming Ma. Reweighting augmented samples by minimizing the maximal expected loss. arXiv preprint arXiv:2103.08933, 2021

    2021 arXiv

  36. [44]

    Are adversarial examples created equal? a learnable weighted minimax risk for robustness under non-uniform attacks

    Huimin Zeng, Chen Zhu, Tom Goldstein, and Furong Huang. Are adversarial examples created equal? a learnable weighted minimax risk for robustness under non-uniform attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 10815--10823, 2021

    2021

  37. [45]

    Geometry-aware instance-reweighted adversarial training

    Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. arXiv preprint arXiv:2010.01736, 2020

    2010 arXiv

  38. [46]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  39. [47]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  40. [48]

    誁ڞYBq0 w @ \˂bF 3`^) f ` WޏTb]tQ Z Ы;]2Zj8ݐlW c kX֏'y S! QfĊ GYs) W 4 P0,Or (W ; )C NƢ). ; W `m GN 徐ݏիhF ސWW xu3ur]!54#VO=? ±* ^p rx ]K f hFIf QY b g+ <mcOt3Gܘ tX

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2000

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.