Pith. sign in

REVIEW 3 major objections 5 minor 188 references

Pretraining loss for fresh, repeated, or paraphrased tokens obeys a single compute-data scaling law, with token value diminishing as models and data budgets grow.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:56 UTC pith:5FSPIXSV

load-bearing objection A real unification of compute- and data-optimal scaling with careful in-range fits; the 3B–7B recommendations are extrapolation, not measurement. the 3 major comments →

arxiv 2607.25271 v1 pith:5FSPIXSV submitted 2026-07-28 cs.LG cs.AIcs.PF

Bridging Compute- and Data-Optimal Pretraining

Tian Qin , Kimia Hamidieh , David Alvarez-Melis This is my paper
classification cs.LG cs.AIcs.PF
keywords scaling lawspretrainingtoken effectivenessdata-optimal scalingcompute-optimal scalingmulti-epoch trainingparaphrasinglanguage models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that pretraining loss for any data-expansion strategy—reusing the same corpus for extra epochs or paraphrasing it—obeys a single scaling law that sits between two known limits: compute-optimal scaling (fresh data freely available) and data-optimal scaling (fixed corpus, unlimited extra compute). The unifying device is a token-effectiveness function η that converts each derived token into a fraction of a fresh token; fitted across 14M–600M parameter models, η shrinks with model size and data abundance and saturates as derived data grows. With that one function, the framework recovers the classical compute-optimal law, the repeated-data saturation law, and the infinite-compute data-optimal limit as special cases, and predicts held-out losses across model sizes at log-RMSE 0.079. If correct, it gives practitioners a joint compute–data Pareto frontier, a way to tell which resource (compute, data, or model size) is binding, and concrete guidance on when repetition beats paraphrasing and when extra epochs stop paying.

Core claim

The paper's central discovery is that a single parametric family describes how much repeated or paraphrased tokens are worth relative to fresh tokens. The loss takes L = E + A/N^α + B/(D + ηD')^β, with η = (R*/r)(1 − e^(−r/R*)) and R* = K(D/N)^ρ N^σ. Here r = D'/D and R* is the saturation ceiling: the maximum number of fresh-equivalent tokens a unit of corpus can yield no matter how much derived data is produced. Fitting this form to 356 runs shows that R* falls as both model size and tokens-per-parameter grow, so the value of substituting compute for data diminishes with scale and data abundance; the law collapses multi-epoch and paraphrase runs onto the same 1-epoch scaling curve when plot

What carries the argument

The load-bearing object is the token-effectiveness function η, defined as the fraction of a fresh token's value carried by a derived token, together with its saturation ceiling R* = K(D/N)^ρ N^σ. The exponential form η = (R*/r)(1 − e^(−r/R*)) is chosen so that η → 1 for the first derived tokens and η·r → R* for infinitely many derived tokens, meaning effective data saturates at D(1 + R*). R* is the mechanism that carries the argument: it makes token value depend jointly on model size N and tokens-per-parameter D/N, and the power-law exponents determine the crossover between repetition and paraphrasing and the recommended epoch count.

Load-bearing premise

The load-bearing premise is that the power-law saturation ceiling R* = K(D/N)^ρ N^σ, fitted on models up to 600M parameters on one corpus, continues to hold at 3B–7B scale where the epoch and strategy recommendations are made; a second premise is that the fitted η measures the strategy's true token value rather than absorbing artifacts of the 1-epoch baseline and fixed hyperparameters.

What would settle it

Train a 3B model on a fixed corpus at roughly the Chinchilla tokens-per-parameter ratio, repeat the data for 2, 4, 8, 16, and 32 epochs, measure the empirical η from observed loss using the paper's 1-epoch constants, and compare the saturation ceiling against the extrapolated power law; a departure beyond the bootstrapped 95% confidence interval would falsify the cross-scale extrapolation. A second check: rerun the 14M–600M sweep on a different corpus and see whether ρ, σ, and the repetition/paraphrase crossover shift materially.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the CD scaling law holds, single-epoch compute-optimal pretraining leaves loss on the table whenever fresh data is not freely available; modest extra compute on derived tokens lowers loss until the saturation ceiling is approached.
  • The three-regime decomposition lets a practitioner identify the binding constraint: compute-bound (more derived tokens help), data-bound (derived tokens have saturated and only fresh data helps), and model-bound (neither helps; increase model size).
  • Strategy choice follows the fitted R* crossover: paraphrasing wins for small models and sub-Chinchilla data budgets, repetition wins for larger models and data-rich budgets, and the 4-epoch rule holds only near 3B parameters at roughly 1× Chinchilla.
  • A fit on 14M and 30M models alone predicts held-out 60M–600M losses at log-RMSE 0.079, so small, cheap sweeps can be used to set production-scale data-expansion budgets.
  • Validation loss is sufficient for downstream capability across strategies: multi-epoch and paraphrase runs fall on a single loss-to-benchmark curve.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editor inference: if R* keeps its power-law form above 600M, the steep negative exponents imply paraphrasing becomes nearly worthless at 7B and beyond; propagating fit uncertainty into that extrapolation would show a wide band, so the practical threshold should be treated as a range, not a point.
  • Editor inference: the same η form may describe other derived-token strategies (self-distillation, structured synthetic data, or mixtures of strategies), and a mixture could push effective tokens above any single strategy's ceiling R*D; that is testable with the existing framework.
  • Editor inference: because η is fitted from loss after a 1-epoch baseline, it absorbs any systematic mismatch in the baseline itself; comparing fitted η across corpora or training schedules would separate strategy-intrinsic value from setup-specific artifacts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a Compute-Data (CD) scaling law that extends the classical Chinchilla scaling law to settings where a fixed pretraining corpus is supplemented by derived tokens D′ (multi-epoch repetition or paraphrasing). The law is L(N,D,D′) = E + A/N^α + B/(D + η·D′)^β, where η is a token-effectiveness function with the exponential-saturation form η = (R*/r)(1 − e^{−r/R*}) and R*(D,N) = K·(D/N)^ρ·N^σ. The authors fit the joint model to 356 runs on Dolma-3 across 14M–600M parameters, report bootstrap CIs, ablations over functional forms, and a cross-scale holdout (fit at 14M/30M predicting 60M–600M at log-RMSE 0.079). They then use the fitted law to identify three training regimes (compute-, data-, and model-bound), construct a compute–data Pareto frontier, and make practical recommendations, including that the 4-epoch rule holds only near 3B parameters and that paraphrasing is ineffective for N≥7B or budgets ≥4× Chinchilla.

Significance. If the CD scaling law holds beyond the fitted range, the paper would make a valuable contribution: it unifies the compute-optimal (Chinchilla), data-constrained (Muennighoff), and data-optimal (Kim) regimes within a single parametric family, and it provides practitioners with concrete guidance on when to repeat, paraphrase, or add fresh data. The strengths are substantial: the fitting procedure is careful (joint Huber regression on log-loss, bootstrap CIs, iterative residual trimming), the functional-form ablation is informative, and the small-to-medium cross-scale holdout is a genuine out-of-distribution test. The paper also makes falsifiable predictions about crossover boundaries, which is commendable. However, the headline practical claims rely on extrapolating an empirically fitted power law from 600M to 3B–7B, and the paper does not propagate the substantial uncertainty of that extrapolation.

major comments (3)
  1. [§5.3, Eq. (5)] The practical claims — '4-epoch rule holds only near 3B', 'paraphrasing ineffective for N≥7B', and 'large budgets ≥4× Chinchilla' — are obtained by evaluating R*(D,N) = K·(D/N)^ρ·N^σ at N=3e9–7e9, which is 5–12× beyond the largest fitted model size (600M). This extrapolation is extremely sensitive to the fitted exponents: for paraphrase σ=−1.30 with 95% CI [−1.35,−1.26], so the CI width alone changes R* at 7B by a factor of ~7.7, and a 0.1 change in σ changes R* by ~9.7. Since R* enters η and hence the entire CD law, the crossover boundaries and epoch recommendations in §5.3 are not supported by the fitted uncertainty. The cross-scale validation in Table 5/6 validates only up to 600M; it provides no evidence for the 600M→7B step. Please either restrict the practical recommendations to the fitted range, propagate parameter uncertainty into the thresholds, or provide validation at or near
  2. [§4.1, Eq. (2)] η_emp is defined as the unique value that, substituted into Eq. (2) with the fitted 1-epoch constants, reproduces the observed loss. Consequently, the collapse of repetition and paraphrase runs onto the classical scaling line in Figure 3 (right) is partly a definitional identity, not an independent validation. The paper's own §6 concedes that η absorbs any residual misspecification of the 1-epoch fit. To make the collapse informative, the authors should present a strictly predictive check — e.g., fix the 1-epoch constants and the parametric η form from a training subset, then predict held-out runs without per-run η_emp — or explicitly re-label Figure 3 (right) as a consistency check. This is load-bearing because the paper uses the collapse to argue that CD scaling 'prescribes' behavior across strategies.
  3. [§3.2, §6] The fitted law is based on a single corpus (Dolma-3), a single architecture (OLMo3), and fixed hyperparameters (batch size 512, sequence length 4096, cosine schedule with 1B warmup tokens); only learning rate and weight decay were swept. §6 acknowledges that other hyperparameters 'could in principle interact with η'. As a result, the abstract's claim that CD scaling is a 'unified framework' bridging compute- and data-optimal pretraining is stronger than the evidence: the framework is demonstrated for two expansion strategies on one corpus in a narrow hyperparameter envelope. The paper should either qualify the scope of the unification or add a sensitivity analysis showing η is stable across batch size, sequence length, and schedule.
minor comments (5)
  1. [Abstract] Typo: 'effectivness' should be 'effectiveness'.
  2. [Appendix A.1] The 'standard Hampel outlier rule' citation is missing — it appears as '(?)'. Please add a proper reference.
  3. [Appendix A.5 / B.1] Tables 5 and 6 are identical (same content, same caption). Consolidate to a single table to avoid redundancy.
  4. [§4.4 / Table 3] The LOO RMSE values are reported for a frozen backbone; the text should clarify that these are conditional on the Stage-1 anchors and not fully joint fits, to avoid overstating the ablation.
  5. [Fig. 4 caption] The caption says 'grey region marks where the two CIs overlap' but in panel (b) the grey region appears at small TPP; consider annotating the panels directly to make the crossover region unambiguous.

Circularity Check

0 steps flagged

No significant circularity: the CD law is fit and held-out validated; the η_emp definition is a residual reparameterization, not the load-bearing prediction.

full rationale

The paper's load-bearing claim is that Eqs. (3)+(5) define a parametric token-effectiveness function that, inserted into Eq. (2), fits all 356 runs and predicts held-out losses across model sizes. This is genuinely empirical: the cross-scale validation in §4.4/Table 5 refits on N≤30M and predicts 60M–600M runs with held-out RMSE 0.079 on logL, and the functional form is selected by leave-one-out comparison against eight alternatives (Appendix A.3), not by construction. The one definitional caveat is §4.1: η_emp is defined as the unique value that reproduces the observed loss in Eq. (2), so the empirical characterization of η is a reparameterization of the loss residuals relative to the 1-epoch Chinchilla baseline. The paper explicitly concedes this in §6 ('η absorbs any residual misspecification of the 1-epoch fit'). However, this caveat does not make the central derivation circular, because the fitted η_strat is not obtained by inverting Eq. (2) per run, and the predictive validation is held-out rather than in-sample. The §5.3 recommendations at 3B–7B are an extrapolation beyond the fitted 14M–600M range with steep sensitivity to σ; that is a correctness/uncertainty risk, not a circularity, since the extrapolated values are not statistically forced by the fit within range. The only self-citation, Hamidieh et al. (2026), appears as a related-work pointer and is not load-bearing; no uniqueness theorem or ansatz is imported from the authors' own prior work. Overall, the paper is self-contained against its held-out data, so I find no significant circularity.

Axiom & Free-Parameter Ledger

12 free parameters · 6 axioms · 3 invented entities

The central law (Eq. 2) is a four-term additive fit whose three data/model terms come from Chinchilla (re-fit, not derived); the effectiveness function η is defined to absorb residuals; the saturation ceiling R* is a fitted power law whose exponents (ρ, σ negative) do the work of all headline conclusions; and the abstract-level recommendations require extrapolating those exponents 5-12x beyond the fit range. The paper contributes 8 fitted constants plus a hand-chosen trimming knob, and two constructed quantities (η, R*), against which it offers one genuine out-of-sample check (cross-scale holdout) and ablations against a constant-R* baseline.

free parameters (12)
  • E (irreducible loss) = 1.35 [1.0, 1.6]
    Constant term in Eq. (2), jointly fitted with all other parameters on 1-epoch + derived-token runs.
  • A (model-size coefficient) = 205 [90, 676]
    Coefficient of N^(-α) in Eq. (2).
  • B (data coefficient) = 16,597 [7,677, 34,475]
    Coefficient of the effective-token term in Eq. (2).
  • alpha (model-size exponent) = 0.283 [0.22, 0.37]
    Exponent of N in Eq. (2).
  • beta (data exponent) = 0.435 [0.40, 0.47]
    Exponent of effective tokens in Eq. (2); shifts strongly with trimming k (0.298 at k=0, 0.482 at k=30, Table 2).
  • logK (repetition) = 10.93 [9.7, 13.5]
    Saturation scale K in Eq. (5) for multi-epoch repetition; moves from 15.92 to 10.64 across the trimming sweep, so headline R* statements depend on the trimming choice.
  • rho (repetition) = -0.42 [-0.71, -0.15]
    Exponent of TPP=D/N in Eq. (5) for repetition; negative sign drives the 'effectiveness shrinks with data availability' claim.
  • sigma (repetition) = -0.41 [-0.55, -0.35]
    Exponent of N in Eq. (5) for repetition; negative sign drives the 'effectiveness shrinks with model size' claim and sits in tension with Yan et al. (2025).
  • logK (paraphrase) = 30.50 [30.4, 30.8]
    Saturation scale K in Eq. (5) for paraphrasing; combined with the steep negative exponents it yields the extrapolated 'paraphrase dies at 7B' claim.
  • rho (paraphrase) = -1.52 [-1.71, -1.22]
    Exponent of TPP for paraphrasing in Eq. (5).
  • sigma (paraphrase) = -1.30 [-1.35, -1.26]
    Exponent of N for paraphrasing in Eq. (5); at N=7e9, each 0.1 change in σ moves R*_para by a factor ~9.7, so the 7B threshold is extremely sensitive to this fitted value.
  • k (residual-trim count) = 15
    Number of iterative residual-trimming iterations; removes ~4% of runs (all small-scale 14M/30M), materially changing fitted parameters; justified via a Hampel rule cited with an unfinished '(?)' placeholder (Appendix A.1).
axioms (6)
  • domain assumption The Chinchilla parametric form L = E + A/N^α + B/D^β holds on 1-epoch runs.
    Borrowed from Hoffmann et al. (2022) and re-fit here; every η_emp value is defined relative to this baseline (Eq. 1, §3.1), so misspecification of this form is absorbed into η rather than detected.
  • ad hoc to paper Derived tokens enter the loss only through the additive effective-token count D + η·D'.
    Eq. (2): η multiplies only the data term; the paper does not test couplings of η with the irreducible loss E or the model term A/N^α.
  • ad hoc to paper η·r saturates to a finite ceiling as r→∞ (requirement ii, §4.2).
    The functional family is restricted to saturating forms, so 'diminishing returns' is partly a consequence of the chosen ansatz; the largest observed r is 63, so the infinite-r limit is an extrapolation.
  • domain assumption Paraphrased tokens generated by SmolLM2-1.7B with the 4-style mixture are representative of 'paraphrasing' as a general strategy.
    One generator, one style recipe (§3.2); the fitted K_para, ρ_para, σ_para are specific to this pipeline and may not transfer to other generators or style mixes.
  • ad hoc to paper The exponents ρ, σ fitted at N ≤ 600M on Dolma-3 remain valid at 3B-7B and on other corpora.
    All abstract-level strategy guidance (§5.3, Fig. 8) depends on this extrapolation; 7B is ~12x beyond the largest fitted model, and corpus composition is held fixed (Dolma-3), so corpus generality is untested.
  • domain assumption Fresh-data scarcity is modeled by taking D as a subset of the 150B-token Dolma-3 corpus.
    The 'fixed corpus' regime is simulated by restriction to nested subsets (§3.2); results may depend on subset composition and corpus choice.
invented entities (3)
  • η — token-effectiveness function no independent evidence
    purpose: Converts each derived token into a fresh-token equivalent in the loss law (Eq. 2).
    Defined in §4.1 as the unique value making Eq. (2) reproduce the observed loss, so it is a residual-absorber by construction; the only out-of-sample check is of the parametric form (cross-scale holdout), not of the construct itself.
  • R* — saturation ceiling no independent evidence
    purpose: Maximum fresh-equivalent value a corpus can yield via an expansion strategy, as a multiple of D; determines the data-optimal limit L_D (Eq. 4).
    Fitted power law (Eq. 5); its behavior at 3B-7B — which drives the headline claims — is an extrapolation with no external benchmark.
  • Three operational regimes (compute-bound, data-bound, model-bound) no independent evidence
    purpose: Diagnostic partition of training states identifying the binding resource.
    Regime boundaries are level sets of the fitted law (L_Chin, L_D, L_N); they restate the chosen saturating η form rather than being independently measured.

pith-pipeline@v1.3.0-alltime-deepseek · 35190 in / 29433 out tokens · 274706 ms · 2026-08-01T02:56:45.517083+00:00 · methodology

0 comments
read the original abstract

Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound. CD scaling extends classical scaling laws by introducing a token-effectiveness function, $\eta$, which quantifies the value of a derived token-produced, for example, through multi-epoch repetition or paraphrasing-relative to a fresh token, ranging from a perfect substitute to having no value. We fit $\eta$ for two data-expansion strategies, multi-epoch repetition and paraphrasing, across model sizes from 14M to 600M parameters using the Dolma-3 corpus. We find that token effectiveness is far from constant: it depends jointly on model size, the tokens-per-parameter ratio, and the amount of derived data, and it saturates as the corpus is expanded. The functional form of $\eta$ implies diminishing returns when substituting compute for data as either model size or data availability increases. It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound---and shows that classical compute-optimal allocation is suboptimal across most practically relevant settings.

Figures

Figures reproduced from arXiv: 2607.25271 by David Alvarez-Melis, Kimia Hamidieh, Tian Qin.

Figure 1
Figure 1. Figure 1: CD-scaling law overview. (a) For a model of size N, classical scaling laws assume 1-epoch training on fresh data, which coincides with the compute-optimal Pareto frontier. Spending additional compute on derived tokens D′ produced from D, via data expansion strategies such as multi-epoch repetition, paraphrasing, or distillation, departs from this frontier and reaches lower loss without curating more fresh … view at source ↗
Figure 2
Figure 2. Figure 2: Empirical characterization of η (y-axis is ηemp throughout). (a) η vs. TPP at fixed D′/D: η decreases monotonically with TPP, so the portion of fresh data in the pretraining set matters. (b) η vs. D′/D at fixed (N, D): monotone decay; together with (c), this is decay toward a finite saturation ceiling, not decay to zero. (c) η · D′/D vs. r at fixed (N, D): extra effective tokens plateau as D′ grows. The pl… view at source ↗
Figure 3
Figure 3. Figure 3: Fitted CD-scaling law across model sizes and strategies. Data-reducible loss L − (E + A/Nα) across model sizes and training settings (1-epoch, repetition, and paraphrase). The solid line is the classic scaling-law fit on 1-epoch runs. Left: Data-reducible loss against fresh tokens D. Repetition and paraphrase runs train on extra D′ tokens, pushing their loss below what fresh D alone achieves, so these runs… view at source ↗
Figure 4
Figure 4. Figure 4: Saturation ceiling R∗ across strategies. Lines show fitted R∗ , shaded bands show 95% CIs, and the grey region marks where the two CIs overlap. (a) R∗ vs. N at TPP = 20. Paraphrasing significantly wins for small models (N ≲ 50M) and repetition for large ones (N ≳ 190M). (b) R∗ vs. TPP for N = 600M. The CIs overlap at small TPP, so paraphrasing does not meaningfully beat repetition. Beyond TPP ≈ 8, repetiti… view at source ↗
Figure 5
Figure 5. Figure 5: Downstream performance as a function of validation BPB. Left: average BPB on benchmarks recast as language-modeling tasks (GSM8K, TriviaQA, HumanEval, etc.). Right: mean accuracy on LAMBADA, HellaSwag, OpenBookQA, RACE, and SQuAD. In both panels, points from multi-epoch and paraphrase runs collapse onto a single curve, indicating that validation loss is a sufficient statistic for downstream capability rega… view at source ↗
Figure 6
Figure 6. Figure 6: Optimal compute-data allocation on (C, D) loss surfaces. Left: 1-epoch compute￾optimal training scales compute C proportionally with fresh data D, reducing the (C, D) loss surface into a single ray (black). Loss contour lines around it indicates that small increase in C would significantly lower the loss. Tracing the sharpest direction of descent, the empirical optimal scaling path (teal) prefers scaling c… view at source ↗
Figure 7
Figure 7. Figure 7: The CD-scaling law captures three training regimes. Left: N = 30M empirical losses overlaid with CD-scaling law predictions. By sweeping D′ from 0 (L Chin, compute-optimal limit) to ∞ (L D, data-optimal limit), CD-scaling law traces out all achievable losses from L Chin to L D. Right: The same decomposition across model sizes N . As N grows, the compute-bound region bounded by L Chin and L D tightens, indi… view at source ↗
Figure 8
Figure 8. Figure 8: When to repeat, when to paraphrase, and how much. Top: Saturation ceiling R∗ across N and TPP for repetition (left) and paraphrase (right). The boundary marks where the winning strategy switches. Paraphrase wins in the lower-left (small N, low data budget), repetition wins in the upper-right. Bottom: Training compute to reach 90% of R∗ . We provide recommended epochs (left) for repetition and passes K (rig… view at source ↗
Figure 9
Figure 9. Figure 9: Per-task breakdown of accuracy-based benchmarks versus validation BPB. Each subplot shows one task against validation BPB, decomposing the accuracy side of [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Per-task breakdown of BPB-based benchmarks versus validation BPB. Each subplot shows one task (recast as language modeling and scored in BPB) against validation BPB, decomposing the BPB side of [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Optimal compute-data allocation on (C, D) loss surfaces for N = 370M. Replicating results for [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Hyperparameter grid example for 30M and 370M model. More data repetition requires much higher weight decay especially for small models [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

188 extracted references · 10 canonical work pages · 5 internal anchors

  1. [1]

    doi:10.5281/zenodo.12608602 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang...

    doi:10.5281/zenodo.12608602

  2. [2]

    2019 , eprint=

    Decoupled Weight Decay Regularization , author=. 2019 , eprint=

    2019

  3. [3]

    Position: Will we run out of data?

    Villalobos, Pablo and Ho, Anson and Sevilla, Jaime and Besiroglu, Tamay and Heim, Lennart and Hobbhahn, Marius , booktitle =. Position: Will we run out of data?. 2024 , editor =

    2024

  4. [4]

    arXiv preprint arXiv:2202.05924 , year=

    Compute Trends Across Three Eras of Machine Learning , author=. arXiv preprint arXiv:2202.05924 , year=

    Pith/arXiv arXiv

  5. [5]

    2024 , eprint =

    Language Models Scale Reliably with Over-Training and on Downstream Tasks , author =. 2024 , eprint =

    2024

  6. [6]

    and Le, Quoc V

    Xie, Sang Michael and Pham, Hieu and Dong, Xinyun and Du, Nan and Liu, Hanxiao and Lu, Yifeng and Liang, Percy S. and Le, Quoc V. and Ma, Tengyu and Yu, Adams Wei , booktitle =. 2023 , url =

    2023

  7. [7]

    2025 , url =

    Liu, Qian and Zheng, Xiaosen and Muennighoff, Niklas and Zeng, Guangtao and Dou, Longxu and Pang, Tianyu and Jiang, Jing and Lin, Min , booktitle =. 2025 , url =

    2025

  8. [8]

    2024 , eprint =

    Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance , author =. 2024 , eprint =

    2024

  9. [9]

    arXiv preprint arXiv:2607.11052 , year=

    Domain-aware scaling laws uncover data synergy , author=. arXiv preprint arXiv:2607.11052 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2404.10102 , year=

    Chinchilla scaling: A replication attempt , author=. arXiv preprint arXiv:2404.10102 , year=

    Pith/arXiv arXiv

  11. [11]

    2025 , eprint=

    SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model , author=. 2025 , eprint=

    2025

  12. [12]

    The Synthetic Data Playbook: Generating Trillions of the Finest Tokens , author=

  13. [13]

    2026 , eprint=

    Training Language Models via Neural Cellular Automata , author=. 2026 , eprint=

    2026

  14. [14]

    2026 , eprint=

    Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression , author=. 2026 , eprint=

    2026

  15. [15]

    arXiv preprint arXiv:2409.02257 , year=

    Mmlu-pro+: Evaluating higher-order reasoning and shortcut learning in llms , author=. arXiv preprint arXiv:2409.02257 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:1606.06031 , year=

    The LAMBADA dataset: Word prediction requiring a broad discourse context , author=. arXiv preprint arXiv:1606.06031 , year=

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:1905.07830 , year=

    Hellaswag: Can a machine really finish your sentence? , author=. arXiv preprint arXiv:1905.07830 , year=

    Pith/arXiv arXiv 1905

  18. [18]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:1803.05457 , year=

    Think you have solved question answering? try arc, the ai2 reasoning challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    Pith/arXiv arXiv

  21. [21]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  23. [23]

    Proceedings of the 2017 conference on empirical methods in natural language processing , pages=

    Race: Large-scale reading comprehension dataset from examinations , author=. Proceedings of the 2017 conference on empirical methods in natural language processing , pages=

    2017

  24. [24]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension , author=. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  25. [25]

    Know What You Don ' t Know: Unanswerable Questions for SQ u AD

    Rajpurkar, Pranav and Jia, Robin and Liang, Percy. Know What You Don ' t Know: Unanswerable Questions for SQ u AD. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2018. doi:10.18653/v1/P18-2124. 1806.03822 , archivePrefix=

    Pith/arXiv arXiv 2018

  26. [26]

    EMNLP , year=

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. EMNLP , year=

  27. [27]

    2026 , eprint=

    Prescriptive Scaling Laws for Data Constrained Training , author=. 2026 , eprint=

    2026

  28. [28]

    Bailey, Luke and Wen, Kaiyue and Dong, Kefan and Hashimoto, Tatsunori and Ma, Tengyu , month = apr, year =. Scaling. doi:10.48550/arXiv.2604.20209 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2604.20209

  29. [29]

    Nature , author =

    Loss of plasticity in deep continual learning , volume =. Nature , author =. 2024 , pages =. doi:10.1038/s41586-024-07711-7 , abstract =

    doi:10.1038/s41586-024-07711-7 2024

  30. [30]

    Yan, Tingkai and Wen, Haodong and Li, Binghui and Luo, Kairong and Chen, Wenguang and Lyu, Kaifeng , month = nov, year =. Larger. doi:10.48550/arXiv.2511.13421 , abstract =

    doi:10.48550/arxiv.2511.13421

  31. [31]

    Niklaus, Joel and Penedo, Guilherme and Kydlicek, Hynek and Bakouch, Elie and Tunstall, Lewis and Beeching, Ed and Frere, Thibaud and Raffel, Colin and Werra, Leandro von and Wolf, Thomas , month = mar, year =. The

  32. [32]

    Bose, Avinandan and Li, Shuyue Stella and Brahman, Faeze and Koh, Pang Wei and Du, Simon Shaolei and Tsvetkov, Yulia and Fazel, Maryam and Xiao, Lin and Celikyilmaz, Asli , month = feb, year =. Cold-. doi:10.48550/arXiv.2602.15012 , abstract =

    work page doi:10.48550/arxiv.2602.15012

  33. [33]

    Consistently

    Abdulhai, Marwa and Cheng, Ryan and Clay, Donovan and Althoff, Tim and Levine, Sergey and Jaques, Natasha , month = oct, year =. Consistently. doi:10.48550/arXiv.2511.00222 , abstract =

    doi:10.48550/arxiv.2511.00222

  34. [34]

    Post-training

    Chen, Yilei and Chakraborty, Souradip and Wolf, Lorenz and Paschalidis, Yannis and Pacchiano, Aldo , month = mar, year =. Post-training. doi:10.48550/arXiv.2509.04784 , abstract =

    doi:10.48550/arxiv.2509.04784

  35. [35]

    Where does output diversity collapse in post-training? , url =

    Karouzos, Constantinos and Tan, Xingwei and Aletras, Nikolaos , month = apr, year =. Where does output diversity collapse in post-training? , url =. doi:10.48550/arXiv.2604.16027 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2604.16027

  36. [36]

    Learning

    Berant, Jonathan and Chen, Maximillian and Fisch, Adam and Aghajani, Reza and Huot, Fantine and Lapata, Mirella and Eisenstein, Jacob , month = jan, year =. Learning. doi:10.48550/arXiv.2512.04068 , abstract =

    doi:10.48550/arxiv.2512.04068

  37. [37]

    Andrew and Singh, Aarti and Zanette, Andrea , month = feb, year =

    Song, Yuda and Chen, Lili and Tajwar, Fahim and Munos, Remi and Pathak, Deepak and Bagnell, J. Andrew and Singh, Aarti and Zanette, Andrea , month = feb, year =. Expanding the. doi:10.48550/arXiv.2602.02482 , abstract =

    doi:10.48550/arxiv.2602.02482

  38. [38]

    Straight to

    Bergsma, Shane and Dey, Nolan and Gosal, Gurpreet and Gray, Gavia and Soboleva, Daria and Hestness, Joel , month = nov, year =. Straight to. doi:10.48550/arXiv.2502.15938 , abstract =

    doi:10.48550/arxiv.2502.15938

  39. [39]

    doi:10.48550/arXiv.2411.07501 , abstract =

    Menghani, Gaurav and Kumar, Ravi and Kumar, Sanjiv , month = jun, year =. doi:10.48550/arXiv.2411.07501 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.07501

  40. [40]

    Provably Efficient Maximum Entropy Exploration

    Hazan, Elad and Kakade, Sham M. and Singh, Karan and Soest, Abby Van , month = jan, year =. Provably. doi:10.48550/arXiv.1812.02690 , abstract =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1812.02690

  41. [41]

    Learning to

    Chen, Maximillian and Sun, Ruoxi and Pfister, Tomas and Arık, Sercan Ö , month = jul, year =. Learning to. doi:10.48550/arXiv.2406.00222 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.00222

  42. [42]

    Flipping the

    Naous, Tarek and Laban, Philippe and Xu, Wei and Neville, Jennifer , month = mar, year =. Flipping the. doi:10.48550/arXiv.2510.06552 , abstract =

    doi:10.48550/arxiv.2510.06552

  43. [43]

    Meterez, Alexandru and Morwani, Depen and Wu, Jingfeng and Oncescu, Costin-Andrei and Pehlevan, Cengiz and Kakade, Sham , month = oct, year =. Seesaw:. doi:10.48550/arXiv.2510.14717 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2510.14717

  44. [44]

    Meterez, Alexandru and Nair, Pranav Ajit and Morwani, Depen and Pehlevan, Cengiz and Kakade, Sham , month = feb, year =. Anytime. doi:10.48550/arXiv.2602.03702 , abstract =

    doi:10.48550/arxiv.2602.03702

  45. [45]

    Zhang, Hanlin and Morwani, Depen and Vyas, Nikhil and Wu, Jingfeng and Zou, Difan and Ghai, Udaya and Foster, Dean and Kakade, Sham , month = apr, year =. How. doi:10.48550/arXiv.2410.21676 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.21676

  46. [46]

    and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , month = mar, year =

    Yang, Greg and Hu, Edward J. and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , month = mar, year =. Tensor. doi:10.48550/arXiv.2203.03466 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2203.03466

  47. [47]

    and Meng, Fanqing and Hong, Chao and Xie, Xiaotong and Liu, Shaowei and Lu, Enzhe and Tai, Yunpeng and Chen, Yanru and Men, Xin and Guo, Haiqing and Charles, Y

    Team, Kimi and Chen, Guangyu and Zhang, Yu and Su, Jianlin and Xu, Weixin and Pan, Siyuan and Wang, Yaoyu and Wang, Yucheng and Chen, Guanduo and Yin, Bohong and Chen, Yutian and Yan, Junjie and Wei, Ming and Zhang, Y. and Meng, Fanqing and Hong, Chao and Xie, Xiaotong and Liu, Shaowei and Lu, Enzhe and Tai, Yunpeng and Chen, Yanru and Men, Xin and Guo, H...

    Pith/arXiv arXiv doi:10.48550/arxiv.2603.15031

  48. [48]

    Distillation

    Busbridge, Dan and Shidani, Amitis and Weers, Floris and Ramapuram, Jason and Littwin, Etai and Webb, Russ , month = jul, year =. Distillation. doi:10.48550/arXiv.2502.08606 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.08606

  49. [49]

    and Hinton, Geoffrey E

    Anil, Rohan and Pereyra, Gabriel and Passos, Alexandre and Ormandi, Robert and Dahl, George E. and Hinton, Geoffrey E. , month = aug, year =. Large scale distributed neural network training through online distillation , url =. doi:10.48550/arXiv.1804.03235 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.1804.03235

  50. [50]

    Distilling the

    Hinton, Geoffrey and Vinyals, Oriol and Dean, Jeff , month = mar, year =. Distilling the. doi:10.48550/arXiv.1503.02531 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.1503.02531

  51. [51]

    and Chen, Weizhu and Cheng, Minhao and Wei, Furu , month = oct, year =

    Qin, Zeyu and Dong, Qingxiu and Zhang, Xingxing and Dong, Li and Huang, Xiaolong and Yang, Ziyi and Khademi, Mahmoud and Zhang, Dongdong and Awadalla, Hany Hassan and Fung, Yi R. and Chen, Weizhu and Cheng, Minhao and Wei, Furu , month = oct, year =. Scaling. doi:10.48550/arXiv.2503.19551 , abstract =

    doi:10.48550/arxiv.2503.19551

  52. [52]

    Training

    Lee, Dan and Han, Seungwook and Kumar, Akarsh and Agrawal, Pulkit , month = mar, year =. Training. doi:10.48550/arXiv.2603.10055 , abstract =

    doi:10.48550/arxiv.2603.10055

  53. [53]

    Synthetic bootstrapped pretraining , url =

    Yang, Zitong and Zhang, Aonan and Liu, Hong and Hashimoto, Tatsunori and Candès, Emmanuel and Wang, Chong and Pang, Ruoming , month = dec, year =. Synthetic bootstrapped pretraining , url =. doi:10.48550/arXiv.2509.15248 , abstract =

    work page doi:10.48550/arxiv.2509.15248

  54. [54]

    Improving the

    Askari-Hemmat, Reyhane and Pezeshki, Mohammad and Dohmatob, Elvis and Bordes, Florian and Astolfi, Pietro and Hall, Melissa and Verbeek, Jakob and Drozdzal, Michal and Romero-Soriano, Adriana , month = feb, year =. Improving the. doi:10.48550/arXiv.2502.15588 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.15588

  55. [55]

    Rephrasing the

    Maini, Pratyush and Seto, Skyler and Bai, He and Grangier, David and Zhang, Yizhe and Jaitly, Navdeep , month = jan, year =. Rephrasing the. doi:10.48550/arXiv.2401.16380 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.16380

  56. [56]

    and Dao, Tri , month = sep, year =

    Wang, Junxiong and Li, Wen-Ding and Paliotta, Daniele and Ritter, Daniel and Rush, Alexander M. and Dao, Tri , month = sep, year =. M1:. doi:10.48550/arXiv.2504.10449 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.10449

  57. [57]

    Zhu, Rui-Jie and Wang, Zixuan and Hua, Kai and Zhang, Tianyu and Li, Ziniu and Que, Haoran and Wei, Boyi and Wen, Zixin and Yin, Fan and Xing, He and Li, Lu and Shi, Jiajun and Ma, Kaijing and Li, Shanda and Kergan, Taylor and Smith, Andrew and Qu, Xingwei and Hui, Mude and Wu, Bohong and Min, Qiyang and Huang, Hongzhi and Zhou, Xun and Ye, Wei and Liu, J...

    Pith/arXiv arXiv doi:10.48550/arxiv.2510.25741

  58. [58]

    Koishekenov, Yeskendir and Lipani, Aldo and Cancedda, Nicola , month = oct, year =. Encode,. doi:10.48550/arXiv.2510.07358 , abstract =

    doi:10.48550/arxiv.2510.07358

  59. [59]

    Zhu, Rui-Jie and Peng, Tianhao and Cheng, Tianhao and Qu, Xingwei and Huang, Jinfa and Zhu, Dawei and Wang, Hao and Xue, Kaiwen and Zhang, Xuanliang and Shan, Yong and Cai, Tianle and Kergan, Taylor and Kembay, Assel and Smith, Andrew and Lin, Chenghua and Nguyen, Binh and Pan, Yuqi and Chou, Yuhong and Cai, Zefan and Wu, Zhenhe and Zhao, Yongchi and Liu,...

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.06203

  60. [60]

    Demystifing

    Wang, Ruisi and Cai, Zhongang and Pu, Fanyi and Xu, Junxiang and Yin, Wanqi and Wang, Maijunxian and Ji, Ran and Gu, Chenyang and Li, Bo and Huang, Ziqi and Deng, Hokin and Lin, Dahua and Liu, Ziwei and Yang, Lei , month = mar, year =. Demystifing. doi:10.48550/arXiv.2603.16870 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2603.16870

  61. [61]

    Decoupled

    Loshchilov, Ilya and Hutter, Frank , month = jan, year =. Decoupled. doi:10.48550/arXiv.1711.05101 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.1711.05101

  62. [62]

    Teaching

    Sundaram, Shobhita and Quan, John and Kwiatkowski, Ariel and Ahuja, Kartik and Ollivier, Yann and Kempe, Julia , year =. Teaching. doi:10.48550/ARXIV.2601.18778 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2601.18778

  63. [63]

    Dong, Xin and Fu, Yonggan and Diao, Shizhe and Byeon, Wonmin and Chen, Zijia and Mahabaleshwarkar, Ameya Sunil and Liu, Shih-Yang and Keirsbilck, Matthijs Van and Chen, Min-Hung and Suhara, Yoshi and Lin, Yingyan and Kautz, Jan and Molchanov, Pavlo , month = nov, year =. Hymba:. doi:10.48550/arXiv.2411.13676 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.13676

  64. [64]

    Gu, Albert and Dao, Tri , month = may, year =. Mamba:. doi:10.48550/arXiv.2312.00752 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.00752

  65. [65]

    Pre-training under infinite compute , url =

    Kim, Konwoo and Kotha, Suhas and Liang, Percy and Hashimoto, Tatsunori , month = sep, year =. Pre-training under infinite compute , url =. doi:10.48550/arXiv.2509.14786 , abstract =

    doi:10.48550/arxiv.2509.14786

  66. [66]

    Replaying pre-training data improves fine-tuning , url =

    Kotha, Suhas and Liang, Percy , month = mar, year =. Replaying pre-training data improves fine-tuning , url =. doi:10.48550/arXiv.2603.04964 , abstract =

    doi:10.48550/arxiv.2603.04964

  67. [67]

    doi:10.48550/arXiv.2410.01560 , abstract =

    Toshniwal, Shubham and Du, Wei and Moshkov, Ivan and Kisacanin, Branislav and Ayrapetyan, Alexan and Gitman, Igor , month = oct, year =. doi:10.48550/arXiv.2410.01560 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.01560

  68. [68]

    doi:10.48550/arXiv.2502.02737 , abstract =

    Allal, Loubna Ben and Lozhkov, Anton and Bakouch, Elie and Blázquez, Gabriel Martín and Penedo, Guilherme and Tunstall, Lewis and Marafioti, Andrés and Kydlíček, Hynek and Lajarín, Agustín Piqueres and Srivastav, Vaibhav and Lochner, Joshua and Fahlgren, Caleb and Nguyen, Xuan-Son and Fourrier, Clémentine and Burtenshaw, Ben and Larcher, Hugo and Zhao, Ha...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.02737

  69. [69]

    Nemotron-

    Mahabadi, Rabeeh Karimi and Satheesh, Sanjeev and Prabhumoye, Shrimai and Patwary, Mostofa and Shoeybi, Mohammad and Catanzaro, Bryan , month = aug, year =. Nemotron-. doi:10.48550/arXiv.2508.15096 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.15096

  70. [70]

    Lin, Zhenghao and Gou, Zhibin and Gong, Yeyun and Liu, Xiao and Shen, Yelong and Xu, Ruochen and Lin, Chen and Yang, Yujiu and Jiao, Jian and Duan, Nan and Chen, Weizhu , month = jan, year =. Rho-1:. doi:10.48550/arXiv.2404.07965 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.07965

  71. [71]

    Deriving

    Cagnetta, Francesco and Raventós, Allan and Ganguli, Surya and Wyart, Matthieu , month = feb, year =. Deriving. doi:10.48550/arXiv.2602.07488 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2602.07488

  72. [72]

    How do language models learn facts?

    Zucchet, Nicolas and Bornschein, Jörg and Chan, Stephanie and Lampinen, Andrew and Pascanu, Razvan and De, Soham , month = jul, year =. How do language models learn facts?. doi:10.48550/arXiv.2503.21676 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.21676

  73. [73]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and Yang, Amy and Fan, Angela and Goyal, Anirudh and Hartshorn, Anthony and Yang, Aobo and Mitra, Archi and Sravankumar, Archie and Korenev, Artem and Hinsvark, A...

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783

  74. [74]

    and Soldaini, Luca and Bhagia, Akshita and Liu, Jiacheng and Groeneveld, Dirk and Tafjord, Oyvind and Smith, Noah A

    Magnusson, Ian and Tai, Nguyen and Bogin, Ben and Heineman, David and Hwang, Jena D. and Soldaini, Luca and Bhagia, Akshita and Liu, Jiacheng and Groeneveld, Dirk and Tafjord, Oyvind and Smith, Noah A. and Koh, Pang Wei and Dodge, Jesse , month = jul, year =. doi:10.48550/arXiv.2504.11393 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.11393

  75. [75]

    and Murray, Tyler and Heineman, David and Jordan, Matt and Hajishirzi, Hannaneh and Ré, Christopher and Soldaini, Luca and Lo, Kyle , month = feb, year =

    Chen, Mayee F. and Murray, Tyler and Heineman, David and Jordan, Matt and Hajishirzi, Hannaneh and Ré, Christopher and Soldaini, Luca and Lo, Kyle , month = feb, year =. Olmix:. doi:10.48550/arXiv.2602.12237 , abstract =

    doi:10.48550/arxiv.2602.12237

  76. [76]

    Lu, Wenquan and Huang, Hai and Balestriero, Randall , month = feb, year =. Prompt. doi:10.48550/arXiv.2602.03190 , abstract =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2602.03190

  77. [77]

    and Wiegreffe, Sarah and Elazar, Yanai , month = apr, year =

    Merullo, Jack and Smith, Noah A. and Wiegreffe, Sarah and Elazar, Yanai , month = apr, year =. On. doi:10.48550/arXiv.2504.12459 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.12459

  78. [78]

    What is the objective of reasoning with reinforcement learning? , url =

    Davis, Damek and Recht, Benjamin , month = oct, year =. What is the objective of reasoning with reinforcement learning? , url =. doi:10.48550/arXiv.2510.13651 , abstract =

    doi:10.48550/arxiv.2510.13651

  79. [79]

    Tajwar, Fahim and Zeng, Guanning and Zhou, Yueer and Song, Yuda and Arora, Daman and Jiang, Yiding and Schneider, Jeff and Salakhutdinov, Ruslan and Feng, Haiwen and Zanette, Andrea , month = feb, year =. Maximum. doi:10.48550/arXiv.2602.02710 , abstract =

    doi:10.48550/arxiv.2602.02710

  80. [80]

    Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning

    Mircea, Andrei and Chakraborty, Supriyo and Chitsazan, Nima and Naphade, Milind and Sahu, Sambit and Rish, Irina and Lobacheva, Ekaterina , month = jul, year =. Training. doi:10.48550/arXiv.2506.05447 , abstract =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.05447

Showing first 80 references.