Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Larger pretraining weight decay improves a language model's downstream adaptability, even when pretraining loss gets slightly worse.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pretrained models trained with larger weight decay fine-tune better on downstream tasks, so the best pretraining checkpoint by loss is not always the best starting point for later training.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A plausible but under-powered empirical claim that pretraining weight decay above the default improves downstream fine-tuning; worth refereeing, but I wouldn't take Finding 1 at face value without repeated seeds and a varied fine-tuning recipe. the 3 major comments →

arxiv 2602.11137 v2 pith:TQP7M23Z submitted 2026-02-11 cs.LG cs.AIcs.CL

Weight Decay Improves Language Model Plasticity

Tessa Han , Sebastian Bordt , Hanlin Zhang , Sham Kakade This is my paper
classification cs.LG cs.AIcs.CL
keywords weight decaymodel plasticitylanguage model pretrainingfine-tuningvalidation losshyperparameter optimizationchain-of-thought tasksAdamW
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that weight decay during pretraining is a key determinant of language-model plasticity, the base model's ability to improve through later fine-tuning, and that the weight decay value which maximizes downstream performance is larger than the standard 0.1 default. The evidence comes from systematic sweeps over weight decay across two model families, sizes up to 4 billion parameters, compute-optimal (20 tokens-per-parameter) and overtrained (140 TPP) regimes, and six chain-of-thought tasks. A direct consequence is that base-model validation loss is not a reliable proxy for downstream quality: in some regimes, models with higher pretraining loss fine-tune into better downstream models. The paper also identifies three mechanisms through which weight decay may preserve plasticity: more linearly separable representations, lower-rank attention matrices, and reduced overfitting on the pretraining data. If the claim holds, pretraining hyperparameter selection and scaling-law analyses should measure downstream adaptability, not only cross-entropy loss.

Core claim

On the paper's terms: pretraining weight decay controls a property the field has not optimized for—base-model plasticity. In their setup, larger weight decay (about 0.3–1.0, above the 0.1 default) yields larger fine-tuning gains on six chain-of-thought tasks across Llama-2 and OLMo-2 models up to 4B. The best downstream value was 1.0 at 20 tokens-per-parameter and 0.3 at 140 TPP. This produces a trade-off: a base model with worse validation loss can become the better fine-tuned model. The paper further reports that the loss-minimizing and downstream-maximizing weight decays differ, and offers three mechanisms: more linearly separable representations, lower-rank attention, and less overfittin

What carries the argument

The core object is the AdamW weight-decay hyperparameter λ, the multiplicative decay applied to weights at each optimizer step after the gradient update. The experimental machinery is an end-to-end comparison: pretrain variants with different λ, fine-tune all of them with one fixed recipe, and rank them by downstream accuracy. The explanatory mechanism is λ's effect on the learned model: higher λ yields last-token embeddings that linear probes classify more accurately (linearly separated representations), lowers the pseudo-rank of the QK and value-projection matrices in attention (the fraction of singular values needed to capture 95 percent of the total), and shrinks the train-minus-validati

Load-bearing premise

The load-bearing premise is that a single fine-tuning recipe—three epochs, batch size 64, learning rate 1e-5 for 1B-and-under models and 7.5e-6 for 4B—measures plasticity fairly across all pretrained variants, so the observed ordering would not reverse if each weight-decay value received its own optimal fine-tuning settings.

What would settle it

Use the OLMo-2-1B-20x checkpoints pretrained with weight decay 0.1 and 1.0. Fine-tune both on one task, such as MetaMathQA, over a grid spanning learning rates 1e-5, 3e-5, and 1e-4 and epoch counts 1, 3, and 6, with batch size 64 held fixed. The paper's claim predicts the weight-decay-1.0 checkpoint wins under all those settings; if any reasonable setting erases or reverses the ordering, the reported plasticity advantage is an artifact of the single fixed recipe.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Pretrained checkpoints should be selected with downstream fine-tuning performance in mind, because the weight decay that minimizes pretraining loss differs from the weight decay that maximizes downstream accuracy.
  • The common 0.1 default weight decay is likely leaving downstream performance on the table for models trained near the compute-optimal regime; values of 0.3–1.0 did better in the tested setups.
  • Scaling-law and hyperparameter-sweep studies that report only validation loss can systematically misorder base models from the perspective of the final, fine-tuned model.
  • A model with higher pretraining validation loss can fine-tune into a better downstream model, so validation loss alone is not a sufficient selection criterion.
  • The optimal pretraining weight decay for downstream performance depends on training duration: it fell from 1.0 at 20 TPP to 0.3 at 140 TPP in these experiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a practical protocol suggested by this result is to run a small multi-task fine-tuning probe on candidate checkpoints before committing large compute to post-training, since validation loss alone can pick the wrong base model.
  • Editorial inference: weight decay may also buffer plasticity across multiple rounds of post-training, such as successive fine-tuning or alignment stages, because lower-rank, less-overfit representations are what continual-learning accounts identify as protecting future learning—but the paper only tests a single fine-tuning round.
  • Editorial inference: the drop in optimal weight decay from 20 to 140 TPP hints at a two-dimensional scaling law in training tokens and weight decay, with the downstream-optimal λ decreasing as training grows; the paper's three-point sweep at 140 TPP is too sparse to confirm this, but it is directly testable.
  • Editorial inference: since the ordering rests on one fixed fine-tuning recipe, the cleanest robustness test is a learning-rate and epoch-count grid; if any plausible grid setting reverses the ordering, plasticity needs a definition over a set of recipes rather than one recipe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies how the weight decay hyperparameter used during LLM pretraining affects model plasticity, measured by downstream performance after supervised fine-tuning (SFT) on six chain-of-thought tasks. Across five model setups (Llama-2 0.5B/1B/4B and OLMo-2 1B at 20 or 140 tokens-per-parameter), the authors sweep pretraining weight decay and report that larger-than-default values (e.g., 1.0 at 20 TPP, 0.3 at 140 TPP) improve fine-tuned performance, even when such values give worse pretraining validation loss. They also report mechanistic observations: higher weight decay improves linear-probe accuracy of internal representations, reduces attention-matrix pseudo-rank, and reduces the train-validation gap. The paper concludes that pretraining hyperparameter selection based solely on validation loss may not select the best downstream model.

Significance. If the main finding holds, it is practically important: it challenges the common assumption that lower pretraining validation loss should guide hyperparameter selection for models that will later be fine-tuned. The paper's empirical scope is substantial — multiple model families, sizes, training regimes, six tasks, and six evaluation metrics — and the authors are appropriately cautious about the instability of the pretraining-loss/downstream correlation (Section 4.3, Appendix Figure 9). The paper also ships a large amount of experimental detail and explicitly identifies limitations, including the use of prior checkpoints and a fixed fine-tuning recipe. The central claim is, however, currently contingent on a single fine-tuning configuration and single-run measurements, so the quantitative strength of the claim exceeds what the evidence supports.

major comments (3)
  1. [Section 4.2 and Appendix B.2] Finding 1 — that pretraining weight decay >0.1 improves plasticity — is measured entirely under one fixed SFT recipe: LR 1e-5 (7.5e-6 for the 4B model), 3 epochs, batch size 64, cosine schedule, warmup 0.1. No fine-tuning hyperparameter is varied, and Appendix B.2 states the recipe is inherited from Qi et al. (2025), not tuned for these pretrained checkpoints. If pretraining weight decay changes the scale of the weights or the effective step size during fine-tuning, the optimal SFT learning rate or epoch count may differ across weight-decay values. Then the reported ranking (e.g., WD=1.0 beating WD=0.1) could be an artifact of this one recipe rather than evidence about plasticity. Because the headline claim is a general statement about plasticity, this interaction must be ruled out or acknowledged as a boundary condition. A targeted fine-tuning hyperparameter sweep (at least LR and epoch
  2. [Section 4.2, Figures 2 and 7] All reported downstream results come from single fine-tuning runs without repeated seeds or error bars. Many comparisons between adjacent weight-decay values are small in magnitude (e.g., differences around 0.01–0.02 on average accuracy), while seed noise in SFT of models of this scale is typically at least that large. Additionally, two of the five setups have only two (Llama-2-4B-20x) or three (OLMo-2-1B-140x) weight-decay values, so the claim that 'the optimal pretraining weight decay value for plasticity is larger than 0.1' is not equally supported across setups. The authors' own Section 4.3 shows that correlation coefficients are unstable under leave-one-out re-computation, which reinforces the need for uncertainty quantification before drawing quantitative conclusions about optimal values.
  3. [Section 5, especially Figure 15] The mechanistic analyses in Section 5 are presented as 'effects' and 'mechanisms,' but the evidence is correlational and based on the same checkpoints that were used to establish the main result. For example, the high correlation between probing accuracy and downstream performance (Appendix Figure 15) may reflect a common cause — e.g., both are driven by the same weight-norm/regularization differences — rather than a mechanism through which linearly separable representations causally improve plasticity. The paper appropriately hedges in places ('potential explanation,' 'we conjecture'), but the section title and Finding 3 assert stronger causal language than the experiments support. The central claim of the paper does not depend on these mechanisms, so this is a framing issue, but it should be tightened.
minor comments (6)
  1. [Abstract and Section 6] Typo: 'the multifaceted role of that a single optimization hyperparameter plays' should read 'the multifaceted role that a single optimization hyperparameter plays.' The same phrase appears twice.
  2. [Appendix A.2, Table 3] The weight-decay grids are highly unbalanced across model setups: Llama-2-0.5B and 1B cover nine values, while Llama-2-4B has only {0.1, 1.0} and OLMo-2-1B-140x has only {0.1, 0.3, 1.0}. This should be stated clearly in the main text, since it affects how much weight the reader can give to the 'consistent across setups' claim.
  3. [Appendix B.2, Table 7] The fine-tuning batch size differs from Qi et al. (2025) (64 vs. their value) and is chosen 'due to computational constraints.' This is a reasonable practical choice, but it should be discussed as a possible source of discrepancy if results are compared directly to that prior work.
  4. [Figure 1] Figure 1b and 1c appear to have very compressed y-axis ranges for OLMo-2-1B-20x; consider adding a zoomed inset or annotating the loss differences, which are on the order of 0.01–0.02 and may be within optimization noise.
  5. [Appendix D.2.2, Figure 17] The figure caption and surrounding text discuss the 140 TPP OLMo models but the axis label says 'OLMo-2-1B-7x'; please correct the label or the description to avoid confusion.
  6. [General] The paper does not state whether code, configs, and model checkpoints will be released. Given the heavy reliance on pretraining runs and custom pipelines, a reproducibility statement would strengthen the manuscript.

Circularity Check

0 steps flagged

No significant circularity: the central finding is a direct empirical sweep, and self-citations are only pipeline/checkpoint reuse.

full rationale

The paper's central claim is an empirical comparison: pretrained models varying only in weight decay are fine-tuned under a fixed recipe and evaluated. The 'optimal' weight decay is read off measured accuracy curves; no parameter is fitted to the downstream result and then re-predicted, and no equation in the paper reproduces the target finding. The mechanistic sections (linear probing, attention pseudo-rank, train-val gap) are direct measurements or import external theory (Kobayashi et al. 2024); they do not define the main claim. Self-citations appear as infrastructure: the fine-tuning hyperparameters and some checkpoints come from Qi et al. (2025) and Bordt & Pawelczyk (2025), which overlap with the author list, but these are experimental inputs rather than load-bearing justifications for the plasticity result. The fixed fine-tuning recipe (Appendix B.2) is a potential confound about whether the ranking generalizes to other SFT configurations, but that is a correctness/external-validity risk, not circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper contributes an empirical sweep; no quantities are fitted to reproduce the target result, so the ledger is light. The central load is carried by domain assumptions about how plasticity is measured and whether a single fine-tuning recipe is valid across base models.

axioms (4)
  • domain assumption Fine-tuning performance on the chosen six CoT tasks measures model plasticity.
    Section 3 defines plasticity operationally as fine-tuning performance; the validity of this proxy is assumed, not derived.
  • domain assumption A single fixed fine-tuning recipe is appropriate for all pretrained weight-decay variants.
    Appendix B.2 inherits all fine-tuning hyperparameters from Qi et al. (2025); no per-model tuning or sensitivity analysis is reported.
  • domain assumption The AdamW decoupled weight-decay implementation in the pretraining repositories follows the stated equations.
    Section 3, Eqs. 1-2, with implementation details in Appendix A.2; correctness of the PyTorch/other implementations is assumed.
  • domain assumption Pseudo-rank with a 95% energy threshold is a meaningful summary of attention-matrix rank.
    Appendix D.2.1, Eq. 4, threshold borrowed from Kobayashi et al. (2024); chosen rather than independently justified for this setting.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Weight Decay Improves Language Model Plasticity." pith.science (2026). https://pith.science/paper/TQP7M23Z

@misc{pith2026260211137,
  author       = {Pith},
  title        = {Pith review of: Weight Decay Improves Language Model Plasticity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TQP7M23Z}},
  note         = {Machine review of arXiv:2602.11137}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models are typically trained in two broad phases: pretraining to produce a base model, followed by further training to improve downstream performance. However, hyperparameter optimization and scaling laws are studied primarily from the perspective of the base model's validation loss, overlooking a crucial model property: downstream adaptability. In this work, we study pretraining from the perspective of model plasticity, that is, the ability of the base model to successfully adapt to downstream tasks upon additional training. We focus on the role of weight decay, a key regularization parameter during pretraining, and show through systematic experiments that larger weight decay increases the plasticity of the pretrained model, resulting in greater performance gains downstream after fine-tuning. This effect can lead to counterintuitive trade-offs where base models that perform worse after pretraining can perform better after further training. Further investigation of weight decay's mechanistic effects on model behavior reveals that it encourages linearly separable representations, regularizes attention matrices, and reduces overfitting on the training data. Together, these findings highlight the importance of pretrained model plasticity, the limits of using cross-entropy loss as the sole metric for hyperparameter optimization, and the multifaceted role that a single optimization hyperparameter plays in shaping model behavior.

Figures

Figures reproduced from arXiv: 2602.11137 by Hanlin Zhang, Sebastian Bordt, Sham Kakade, Tessa Han.

Figure 1
Figure 1. Figure 1: Pretraining validation cross-entropy loss of models pretrained with varying weight decay. The weight decay value that minimizes pretraining loss may be equal to or larger than the standard default value of 0.1 depending on the training regime. 10 −4 10 −3 10 −2 10 −1 10 0 3 10 1 Weight Decay 0.150 0.175 0.200 0.225 0.250 0.275 0.300 Downstream Accuracy Greedy 10 −4 10 −3 10 −2 10 −1 10 0 3 10 1 Weight Deca… view at source ↗
Figure 2
Figure 2. Figure 2: Weight decay during pretraining improves language model plasticity and downstream performance. This figure plots the average accuracy after fine-tuning for models pretrained with varying weight decay. The results indicate that weight decay leads to better downstream model per￾formance, suggesting it enables the pretrained model to learn better during fine-tuning and improves model plasticity. In these expe… view at source ↗
Figure 3
Figure 3. Figure 3: A model’s performance after pretraining is not perfectly predictive of its performance downstream. Models with similar pretraining losses can perform differently downstream, and models with lower validation cross-entropy loss after pretraining can perform better or worse downstream (i.e., after fine-tuning) than models with higher pretraining losses. possible that as models are trained for even longer (i.e… view at source ↗
Figure 4
Figure 4. Figure 4: Weight decay encourages linearly separated representations. This figure depicts the accuracy of linear probes for sentiment and topic for models pretrained with different weight decay values. We observe that linear probing achieves better accuracy when models are pretrained with a weight decay greater than the default 0.1. 2500 5000 7500 1000012500 Training Step 0.0 0.2 0.4 0.6 0.8 1.0 Pseudo-Rank Layer 5 … view at source ↗
Figure 5
Figure 5. Figure 5: Weight decay reduces the rank of attention matrices. This figure depicts the average pseudo-rank (Appendix D.2.1) of the query-key (WQK) and value projection (WV P ) matrices in layers 5 and 15 during the training of OLMo-2-1B models at 20 TPP. probing accuracies, indicating that these representations are more linearly separated and suggesting that models pretrained with higher weight decay form more struc… view at source ↗
Figure 6
Figure 6. Figure 6: Weight decay reduces overfitting on training data. The figure depicts the train-val gap (Equation 3) for OLMo-2-1B models trained at 20 TPP. Ibrahim et al., 2024; Kirkpatrick et al., 2017; Riemer et al., 2018). Building on these insights, we investigate how weight decay influences overfitting, which is closely related to the forgetting of training data, in pretrained models. To measure the degree to which … view at source ↗
Figure 7
Figure 7. Figure 7: Model performance on each fine-tuning task evaluated using six metrics. Weight decay during pretraining improves model plasticity. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Pretraining validation cross-entropy loss vs. fine-tuning accuracy for individual tasks. Pretraining validation cross-entropy loss (pretraining performance) is not fully predictive of model performance after fine-tuning (downstream performance). 23 [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Stability analysis for Pearson correlation coefficient. Pearson correlation is computed for each leave-one-out (LOO) subset in Figure 8g. The LOO correlation can change noticeably in magnitude and sign, suggesting that the correlation for the full set of data points in Figure 8g is rather unstable, which further supports the finding in that pretraining validation cross-entropy loss (pretraining performance… view at source ↗
Figure 10
Figure 10. Figure 10: Linear probing experiments for Llama-2-0.5B-20x. The train and test accuracies of the linear probes for the SST and AG News datasets and the average train and test accuracy over the two datasets. 4 8 12 16 20 Model Layer 0.80 0.82 0.84 0.86 0.88 0.90 0.92 Probing Accuracy Train Accuracy = 0.0001 = 0.001 = 0.01 = 0.1 = 0.5 = 1.0 = 1.5 = 3.0 = 10.0 4 8 12 16 20 Model Layer 0.80 0.82 0.84 0.86 0.88 0.90 0.92… view at source ↗
Figure 11
Figure 11. Figure 11: Linear probing experiments for Llama-2-1B-20x. The train and test accuracies of the linear probes for the SST and AG News datasets and the average train and test accuracy over the two datasets. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Linear probing experiments for Llama-2-4B-20x. The train and test accuracies of the linear probes for the SST and AG News datasets and the average train and test accuracy over the two datasets. 4 8 12 16 Model Layer 0.87 0.88 0.89 0.90 0.91 0.92 0.93 0.94 Probing Accuracy Train Accuracy = 0.1 = 0.3 = 1.0 4 8 12 16 Model Layer 0.87 0.88 0.89 0.90 0.91 0.92 0.93 0.94 Probing Accuracy Test Accuracy = 0.1 = 0… view at source ↗
Figure 13
Figure 13. Figure 13: Linear probing experiments for OLMo-2-1B-20x. The train and test accuracies of the linear probes for the SST and AG News datasets and the average train and test accuracy over the two datasets. 4 8 12 16 Model Layer 0.86 0.88 0.90 0.92 0.94 Probing Accuracy Train Accuracy = 0.1 = 0.3 = 1.0 4 8 12 16 Model Layer 0.86 0.88 0.90 0.92 0.94 Probing Accuracy Test Accuracy = 0.1 = 0.3 = 1.0 olmo-1B-210BT, ag_news… view at source ↗
Figure 14
Figure 14. Figure 14: Linear probing experiments for OLMo-2-1B-140x. The train and test accuracies of the linear probes for the SST and AG News datasets and the average train and test accuracy over the two datasets. 0.86 0.88 0.90 0.92 Probing Accuracy 0.18 0.20 0.22 0.24 0.26 0.28 0.30 Performance on FT Task r (overall) = 0.93 Greedy r = -0.02, 0.39, 1.00, 0.77, 0.96 0.86 0.88 0.90 0.92 Probing Accuracy 0.20 0.22 0.24 0.26 0.… view at source ↗
Figure 15
Figure 15. Figure 15: Probing accuracy is highly predictive of downstream model performance. The x-axis is the best average probing accuracy of the model (highest probing accuracy out of all model layers). The y-axis the average accuracy of the model over all tasks after fine-tuning. Pretrained models with higher probing accuracies from the linear probing experiments tend to perform better downstream after fine-tuning. 25 [PI… view at source ↗
Figure 16
Figure 16. Figure 16: Weight decay reduces the rank of attention matrices. The figure depicts the average pseudo-rank (Appendix D.2.1) of the query-key (WQK) and value projection (WV P ) matrices in layers 5 and 15 of the fully-trained Llama-2 models at 20 TPP. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Weight decay reduces the rank of attention matrices. This is for the OLMo models trained at 140 TPP. We observe that the rank of attention for weight decay 0.1 is generally smaller than that for both the 20 TPP and the fully trained OLMo-2-1B-0425 model. Hence, we conjecture that this is because the 140 TPP models were trained with a warmup-stable-decay learning rate schedule, whereas the 1x and 144x mode… view at source ↗
Figure 18
Figure 18. Figure 18: Training time does not reduce the rank of attention matrices. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Weight decay reduces the norm of the weights of the model. The effect does not occur for the input layer, where the weights are not being decayed. This is for OLMo-2-1B models trained at 20 TPP. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff

    cs.LG 2026-06 unverdicted novelty 5.0

    Excessive SFT reduces LLM plasticity for RL; Rejuvenation restores it via base-anchored fusion and targeted neuron resets, yielding better RL performance and OOD generalization.

Reference graph

Works this paper leans on

32 extracted references · 23 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker

    Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. One tokenizer to rule them all: Emergent language plasticity via multilingual tokenizers.arXiv preprint arXiv:2506.10766,

    Pith/arXiv arXiv

  2. [5]

    (2025) and pretrain the weight decay 1.0 model

    Model Weight Decay Llama-2-0.5B-20x 9 values: {0.0001, 0.001, 0.01, 0.1, 0.5, 1.0, 1.5, 3.0, 10.0} Llama-2-1B-20x 9 values: {0.0001, 0.001, 0.01, 0.1, 0.5, 1.0, 1.5, 3.0, 10.0} Llama-2-4B-20x 2 values: {0.1, 1.0} OLMo-2-1B-20x 10 values: {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0} OLMo-2-1B-140x 3 values: {0.1, 0.3, 1.0} Table 3:Weight decay values...

    2025

  3. [8]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  4. [9]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  5. [11]

    Wide neural networks trained with weight decay provably exhibit neural collapse.arXiv preprint arXiv:2410.04887,

    Arthur Jacot, Peter Súkeník, Zihan Wang, and Marco Mondelli. Wide neural networks trained with weight decay provably exhibit neural collapse.arXiv preprint arXiv:2410.04887,

    Pith/arXiv arXiv

  6. [13]

    Pre-training under infinite compute.arXiv preprint arXiv:2509.14786,

    Konwoo Kim, Suhas Kotha, Percy Liang, and Tatsunori Hashimoto. Pre-training under infinite compute.arXiv preprint arXiv:2509.14786,

    arXiv

  7. [15]

    Weight decay may matter more than muP for learning rate transfer in practice.arXiv preprint arXiv:2510.19093,

    Atli Kosson, Jeremy Welborn, Yang Liu, Martin Jaggi, and Xi Chen. Weight decay may matter more than muP for learning rate transfer in practice.arXiv preprint arXiv:2510.19093,

    arXiv

  8. [17]

    Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

    Pith/arXiv arXiv

  9. [18]

    Rethinking evaluation protocols of visual representations learned via self-supervised learning.arXiv preprint arXiv:2304.03456,

    Jae-Hun Lee, Doyoung Yoon, ByeongMoon Ji, Kyungyul Kim, and Sangheum Hwang. Rethinking evaluation protocols of visual representations learned via self-supervised learning.arXiv preprint arXiv:2304.03456,

    Pith/arXiv arXiv

  10. [19]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    14 Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  11. [20]

    2 OLMo 2 Furious.arXiv preprint arXiv:2501.00656,

    Team OLMo Team, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 OLMo 2 Furious.arXiv preprint arXiv:2501.00656,

    Pith/arXiv arXiv

  12. [21]

    Generalization guaran- tees for neural networks via harnessing the low-rank structure of the Jacobian.arXiv preprint arXiv:1906.05392,

    Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via harnessing the low-rank structure of the Jacobian.arXiv preprint arXiv:1906.05392,

    Pith/arXiv arXiv 1906

  13. [22]

    Learning to learn without forgetting by maximizing transfer and minimizing interference.arXiv preprint arXiv:1810.11910,

    Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference.arXiv preprint arXiv:1810.11910,

    Pith/arXiv arXiv

  14. [24]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Pith/arXiv arXiv

  15. [25]

    Do large language model benchmarks test reliability?arXiv preprint arXiv:2502.03461,

    Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability?arXiv preprint arXiv:2502.03461,

    Pith/arXiv arXiv

  16. [26]

    How to set AdamW’s weight decay as you scale model and dataset size.arXiv preprint arXiv:2405.13698,

    Xi Wang and Laurence Aitchison. How to set AdamW’s weight decay as you scale model and dataset size.arXiv preprint arXiv:2405.13698,

    Pith/arXiv arXiv

  17. [27]

    MetaMath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284,

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284,

    Pith/arXiv arXiv

  18. [28]

    Three mechanisms of weight decay regularization.arXiv preprint arXiv:1810.12281,

    Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization.arXiv preprint arXiv:1810.12281,

    Pith/arXiv arXiv

  19. [29]

    For each model setup, we pretrain variants with varying weight decay values. Llama-2-0.5B Llama-2-1B Llama-2-4B OLMo-2-1B Model size 0.5B 1B 4B 1.5B Hidden size 1536 2048 4096 2048 Intermediate size 3216 4896 7792 16384 Vocab size 32000 32000 32000 100278 Context length 2048 2048 2048 4096 # Heads 32 32 32 16 # Layers 20 22 28 16 # Query groups 4 4 4 16 T...

    2048

  20. [64]

    (2025) due to computational constraints

    than Qi et al. (2025) due to computational constraints. B.3 Template We use the following template for supervised fine-tuning. Human: {question} Assistant: {response} C Evaluation 21 10 4 10 3 10 2 10 1 100 3 101 Weight Decay of PT Model 0.025 0.050 0.075 0.100 0.125 0.150 Performance on FT T ask Greedy 10 4 10 3 10 2 10 1 100 3 101 Weight Decay of PT Mod...

    arXiv 2025

  21. [1991]

    Maintaining plasticity in continual learning via regenerative regularization.arXiv preprint arXiv:2308.11958,

    Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization.arXiv preprint arXiv:2308.11958,

    Pith/arXiv arXiv

  22. [2010]

    Training dynamics impact post-training quantization robustness.arXiv preprint arXiv:2510.06213,

    Albert Catalan-Tatjer, Niccolò Ajroldi, and Jonas Geiping. Training dynamics impact post-training quantization robustness.arXiv preprint arXiv:2510.06213,

    arXiv

  23. [2017]

    Plasticity loss in deep reinforcement learning: A survey.arXiv preprint arXiv:2411.04832,

    Timo Klein, Lukas Miklautz, Kevin Sidak, Claudia Plant, and Sebastian Tschiatschek. Plasticity loss in deep reinforcement learning: A survey.arXiv preprint arXiv:2411.04832,

    Pith/arXiv arXiv

  24. [2018]

    OpenAI GPT-5 system card.arXiv preprint arXiv:2601.03267,

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. OpenAI GPT-5 system card.arXiv preprint arXiv:2601.03267,

    Pith/arXiv arXiv

  25. [2019]

    Brown, et al

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  26. [2020]

    A study on the plasticity of neural networks.arXiv preprint arXiv:2106.00042,

    Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks.arXiv preprint arXiv:2106.00042,

    Pith/arXiv arXiv

  27. [2021]

    Power lines: Scaling laws for weight decay and batch size in LLM pre-training.arXiv preprint arXiv:2505.13738,

    Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Power lines: Scaling laws for weight decay and batch size in LLM pre-training.arXiv preprint arXiv:2505.13738,

    arXiv

  28. [2022]

    Simple and scalable strategies to continually pre-train large language models.arXiv preprint arXiv:2403.08763,

    Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models.arXiv preprint arXiv:2403.08763,

    Pith/arXiv arXiv

  29. [2023]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  30. [2024]

    Train once, answer all: Many pretraining experiments for the cost of one.arXiv preprint arXiv:2509.23383,

    Sebastian Bordt and Martin Pawelczyk. Train once, answer all: Many pretraining experiments for the cost of one.arXiv preprint arXiv:2509.23383,

    arXiv

  31. [2025]

    Deepseek LLM: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954,

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek LLM: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954,

    Pith/arXiv arXiv

  32. [4934]

    (2021); Vendrow et al

    SimpleScalingn= 54,484GSM8KPlatinum (n= 1,209) + MATH (n= 5,000) Table 6:Fine-tuning and evaluation datasets.MetaMathQA and SimpleScaling are evaluated on test sets of the GSM8KPlatinum Cobbe et al. (2021); Vendrow et al. (2025) and MATH Hendrycks et al. (2021) datasets because MetaMathQA and SimpleScaling contain questions that are augmented from the tra...

    2021

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.