{"id":"4ecbc28a-8736-4ad9-812e-1ba70a852aa8","arxiv_id":"2602.11137","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pretrained models trained with larger weight decay fine-tune better on downstream tasks, so the best pretraining checkpoint by loss is not always the best starting point for later training.","lead":"This paper tests whether increasing weight decay during language-model pretraining makes the resulting model easier to fine-tune on downstream tasks, and reports that it does—even when the higher-decay base model has slightly worse pretraining loss. The finding suggests that picking pretraining hyperparameters by validation loss alone can miss models that adapt better later.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finding 1 rests on a single fine-tuning recipe; if pretraining weight decay shifts the optimal fine-tuning LR or epoch count, the downstream ranking is an artifact of that recipe.","rationale":"The reader identified this as the weakest assumption, and I agree. The paper's internal evidence is otherwise reasonably consistent: the effect appears across two model families, several sizes, and both TPP regimes, and the mechanistic analyses (linear separability, attention rank, train-val gap) are independent of the fine-tuning recipe. That breadth is real credit. However, none of that breadth addresses the fine-tuning interaction, because all downstream numbers come from the same SFT configuration. The claim 'optimal pretraining WD >0.1' is a claim about a point estimate; without error bars and with coarse grids (e.g., 2-3 points for the 4B and 140x models), a few lucky or unlucky seeds could shift the apparent optimum. The requested check is feasible and the effect is plausible, so the appropriate verdict remains CONDITIONAL rather than REJECT. If the check fails, the paper should narrow the claim to the fixed SFT recipe; if it passes, the central claim is substantially strengthened.","tokens_in":27850,"tokens_out":5571,"duration_ms":53681,"concrete_test":"Take Llama-2-1B-20x (WD in {0.1, 0.5, 1.0}) and OLMo-2-1B-20x (WD in {0.1, 0.3, 1.0}); fine-tune on two tasks (e.g., MetaMathQA and MedMCQA) with a grid of learning rates {1e-5, 3e-5, 1e-4} and epochs {1, 3, 6}, batch 64, cosine, all else as in Appendix B.2. For each pretrained WD, take the best downstream accuracy over the grid; check whether the argmax WD remains >0.1 and whether pairwise ordering is stable across grid points. If best-over-grid argmax WD drops to 0.1 or flips by task/model, Finding 1 needs to be restricted to the fixed recipe; if it stays at >=0.5, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Finding 1 (Section 4.2) claims that pretraining weight decay >0.1 improves plasticity based on average downstream accuracy after SFT. The SFT recipe is fixed: LR 1e-5 (7.5e-6 for the 4B model), 3 epochs, batch 64, cosine schedule (Appendix B.2). No fine-tuning hyperparameter is varied, and no repeated seeds are reported. The load-bearing step is the implicit identification of 'plasticity' with performance under this one recipe. If higher pretraining WD changes the scale of the weights or the effective step size during fine-tuning, the optimal SFT LR/epoch count can shift with pretraining WD. Then the WD=1.0 model could win under the fixed recipe even if WD=0.1 models are equally or more plastic under their own optimal recipe. Because the paper's headline is a general claim about plasticity, not about one fine-tuning configuration, this interaction is a confound that must be ruled out. Single-seed runs make this worse: the differences between adjacent WD values are small relative to typical SFT seed noise, so the 'optimal >0.1' may not be stable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how the weight decay hyperparameter used during LLM pretraining affects model plasticity, measured by downstream performance after supervised fine-tuning (SFT) on six chain-of-thought tasks. Across five model setups (Llama-2 0.5B/1B/4B and OLMo-2 1B at 20 or 140 tokens-per-parameter), the authors sweep pretraining weight decay and report that larger-than-default values (e.g., 1.0 at 20 TPP, 0.3 at 140 TPP) improve fine-tuned performance, even when such values give worse pretraining validation loss. They also report mechanistic observations: higher weight decay improves linear-probe accuracy of internal representations, reduces attention-matrix pseudo-rank, and reduces the train-validation gap. The paper concludes that pretraining hyperparameter selection based solely on validation loss may not select the best downstream model.","tokens_in":28122,"tokens_out":2358,"duration_ms":26786,"significance":"If the main finding holds, it is practically important: it challenges the common assumption that lower pretraining validation loss should guide hyperparameter selection for models that will later be fine-tuned. The paper's empirical scope is substantial — multiple model families, sizes, training regimes, six tasks, and six evaluation metrics — and the authors are appropriately cautious about the instability of the pretraining-loss/downstream correlation (Section 4.3, Appendix Figure 9). The paper also ships a large amount of experimental detail and explicitly identifies limitations, including the use of prior checkpoints and a fixed fine-tuning recipe. The central claim is, however, currently contingent on a single fine-tuning configuration and single-run measurements, so the quantitative strength of the claim exceeds what the evidence supports.","major_comments":[{"comment":"Finding 1 — that pretraining weight decay >0.1 improves plasticity — is measured entirely under one fixed SFT recipe: LR 1e-5 (7.5e-6 for the 4B model), 3 epochs, batch size 64, cosine schedule, warmup 0.1. No fine-tuning hyperparameter is varied, and Appendix B.2 states the recipe is inherited from Qi et al. (2025), not tuned for these pretrained checkpoints. If pretraining weight decay changes the scale of the weights or the effective step size during fine-tuning, the optimal SFT learning rate or epoch count may differ across weight-decay values. Then the reported ranking (e.g., WD=1.0 beating WD=0.1) could be an artifact of this one recipe rather than evidence about plasticity. Because the headline claim is a general statement about plasticity, this interaction must be ruled out or acknowledged as a boundary condition. A targeted fine-tuning hyperparameter sweep (at least LR and epoch","section":"Section 4.2 and Appendix B.2"},{"comment":"All reported downstream results come from single fine-tuning runs without repeated seeds or error bars. Many comparisons between adjacent weight-decay values are small in magnitude (e.g., differences around 0.01–0.02 on average accuracy), while seed noise in SFT of models of this scale is typically at least that large. Additionally, two of the five setups have only two (Llama-2-4B-20x) or three (OLMo-2-1B-140x) weight-decay values, so the claim that 'the optimal pretraining weight decay value for plasticity is larger than 0.1' is not equally supported across setups. The authors' own Section 4.3 shows that correlation coefficients are unstable under leave-one-out re-computation, which reinforces the need for uncertainty quantification before drawing quantitative conclusions about optimal values.","section":"Section 4.2, Figures 2 and 7"},{"comment":"The mechanistic analyses in Section 5 are presented as 'effects' and 'mechanisms,' but the evidence is correlational and based on the same checkpoints that were used to establish the main result. For example, the high correlation between probing accuracy and downstream performance (Appendix Figure 15) may reflect a common cause — e.g., both are driven by the same weight-norm/regularization differences — rather than a mechanism through which linearly separable representations causally improve plasticity. The paper appropriately hedges in places ('potential explanation,' 'we conjecture'), but the section title and Finding 3 assert stronger causal language than the experiments support. The central claim of the paper does not depend on these mechanisms, so this is a framing issue, but it should be tightened.","section":"Section 5, especially Figure 15"}],"minor_comments":[{"comment":"Typo: 'the multifaceted role of that a single optimization hyperparameter plays' should read 'the multifaceted role that a single optimization hyperparameter plays.' The same phrase appears twice.","section":"Abstract and Section 6"},{"comment":"The weight-decay grids are highly unbalanced across model setups: Llama-2-0.5B and 1B cover nine values, while Llama-2-4B has only {0.1, 1.0} and OLMo-2-1B-140x has only {0.1, 0.3, 1.0}. This should be stated clearly in the main text, since it affects how much weight the reader can give to the 'consistent across setups' claim.","section":"Appendix A.2, Table 3"},{"comment":"The fine-tuning batch size differs from Qi et al. (2025) (64 vs. their value) and is chosen 'due to computational constraints.' This is a reasonable practical choice, but it should be discussed as a possible source of discrepancy if results are compared directly to that prior work.","section":"Appendix B.2, Table 7"},{"comment":"Figure 1b and 1c appear to have very compressed y-axis ranges for OLMo-2-1B-20x; consider adding a zoomed inset or annotating the loss differences, which are on the order of 0.01–0.02 and may be within optimization noise.","section":"Figure 1"},{"comment":"The figure caption and surrounding text discuss the 140 TPP OLMo models but the axis label says 'OLMo-2-1B-7x'; please correct the label or the description to avoid confusion.","section":"Appendix D.2.2, Figure 17"},{"comment":"The paper does not state whether code, configs, and model checkpoints will be released. Given the heavy reliance on pretraining runs and custom pipelines, a reproducibility statement would strengthen the manuscript.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the journal's audience. The main uncertainty is the fixed fine-tuning recipe: the authors reuse a recipe from their own prior work (Qi et al. 2025) and do not demonstrate that the downstream ranking is stable under reasonable variations of SFT hyperparameters. This is a fixable weakness — a focused LR/epoch sweep on one or two model setups would substantially de-risk the central claim. The absence of repeated seeds is a second concern that should be addressed at least by reporting seed sensitivity for a subset of configurations. I would not recommend rejection, as the qualitative pattern is consistent across several setups and the paper is transparent about the instability of correlations. However, the current wording of Finding 1 overstates the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper claims that pretraining weight decay larger than the default 0.1 improves the plasticity of LLM base models, meaning they fine-tune better downstream even when their pretraining validation loss is worse. That claim is plausible, internally consistent, and backed by sweeps across five model setups, six tasks, and six metrics. But the central result rests on single runs and a single fixed fine-tuning recipe, so I'm not ready to treat the effect as established. It's a serious empirical paper, not a dud.\n\nWhat's new: prior work tied weight decay to pretraining loss or studied plasticity in RL/continual learning. This is the first systematic look at how pretraining weight decay shapes downstream fine-tuning in modern LLMs, with the end-to-end framing that hyperparameter selection should care about downstream performance. That's a good question, and the paper is honest when the answer is messy — the correlation between pretraining loss and downstream accuracy is sign-unstable, and they say so explicitly rather than sweeping it under the rug. The mechanistic section (linear separability, attention rank, train-val gap) is secondary evidence and mostly confirms existing theory, but it's a reasonable supporting cast.\n\nThe soft spots are real and load-bearing, though none are fatal by themselves. First, every point in the downstream accuracy curves is a single run. Adjacent weight decay values differ by small amounts; typical SFT seed noise could flip the 'optimal WD > 0.1' conclusion. The LOO stability analysis in the appendix addresses the correlation plot, not the main WD sweep. Second, the fine-tuning recipe is fixed: LR 1e-5/7.5e-6, 3 epochs, batch 64, inherited from prior work. If pretraining weight decay changes the effective step size or moves the optimal SFT LR/epoch count, the downstream ranking in Finding 1 is partly an artifact of that recipe. The stress-test concern on this is on point; the paper doesn't rule it out. Third, coverage is thin where it matters: Llama-2-4B has only two WD values and the 140 TPP model has three, so the 'optimal' claim is a coarse grid. Fourth, no code or data is released in the arXiv send, so reproducibility is uncheckable from the text.\n\nWeighting all that: the evidence is suggestive, not conclusive. I'd send this to peer review — it's a timely, useful empirical claim with honest reporting — but I'd ask for repeated seeds, at least one variation of the fine-tuning hyperparameters, and release of code/data before accepting the headline claim. For a reading group, it's worth an hour discussing how much we trust single-recipe, single-seed sweeps when they contradict validation-loss-based selection.","headline":"A plausible but under-powered empirical claim that pretraining weight decay above the default improves downstream fine-tuning; worth refereeing, but I wouldn't take Finding 1 at face value without repeated seeds and a varied fine-tuning recipe.","tokens_in":28593,"tokens_out":2536,"would_cite":true,"duration_ms":25321,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Larger pretraining weight decay improves a language model's downstream adaptability, even when pretraining loss gets slightly worse.","keywords":["weight decay","model plasticity","language model pretraining","fine-tuning","validation loss","hyperparameter optimization","chain-of-thought tasks","AdamW"],"falsifier":"Use the OLMo-2-1B-20x checkpoints pretrained with weight decay 0.1 and 1.0. Fine-tune both on one task, such as MetaMathQA, over a grid spanning learning rates 1e-5, 3e-5, and 1e-4 and epoch counts 1, 3, and 6, with batch size 64 held fixed. The paper's claim predicts the weight-decay-1.0 checkpoint wins under all those settings; if any reasonable setting erases or reverses the ordering, the reported plasticity advantage is an artifact of the single fixed recipe.","tokens_in":1494,"feed_emoji":"⚖️","tokens_out":1806,"duration_ms":83817,"temperature":0.7,"pith_summary":"The paper sets out to establish that weight decay during pretraining is a key determinant of language-model plasticity, the base model's ability to improve through later fine-tuning, and that the weight decay value which maximizes downstream performance is larger than the standard 0.1 default. The evidence comes from systematic sweeps over weight decay across two model families, sizes up to 4 billion parameters, compute-optimal (20 tokens-per-parameter) and overtrained (140 TPP) regimes, and six chain-of-thought tasks. A direct consequence is that base-model validation loss is not a reliable proxy for downstream quality: in some regimes, models with higher pretraining loss fine-tune into better downstream models. The paper also identifies three mechanisms through which weight decay may preserve plasticity: more linearly separable representations, lower-rank attention matrices, and reduced overfitting on the pretraining data. If the claim holds, pretraining hyperparameter selection and scaling-law analyses should measure downstream adaptability, not only cross-entropy loss.","feed_headline":"Higher pretraining weight decay yields better fine-tuned models","feed_subtitle":"Base models with worse pretraining loss can fine-tune into better downstream models—validation loss isn't the selection metric.","key_machinery":"The core object is the AdamW weight-decay hyperparameter λ, the multiplicative decay applied to weights at each optimizer step after the gradient update. The experimental machinery is an end-to-end comparison: pretrain variants with different λ, fine-tune all of them with one fixed recipe, and rank them by downstream accuracy. The explanatory mechanism is λ's effect on the learned model: higher λ yields last-token embeddings that linear probes classify more accurately (linearly separated representations), lowers the pseudo-rank of the QK and value-projection matrices in attention (the fraction of singular values needed to capture 95 percent of the total), and shrinks the train-minus-validati","core_discovery":"On the paper's terms: pretraining weight decay controls a property the field has not optimized for—base-model plasticity. In their setup, larger weight decay (about 0.3–1.0, above the 0.1 default) yields larger fine-tuning gains on six chain-of-thought tasks across Llama-2 and OLMo-2 models up to 4B. The best downstream value was 1.0 at 20 tokens-per-parameter and 0.3 at 140 TPP. This produces a trade-off: a base model with worse validation loss can become the better fine-tuned model. The paper further reports that the loss-minimizing and downstream-maximizing weight decays differ, and offers three mechanisms: more linearly separable representations, lower-rank attention, and less overfittin","pith_inferences":["Editorial inference: a practical protocol suggested by this result is to run a small multi-task fine-tuning probe on candidate checkpoints before committing large compute to post-training, since validation loss alone can pick the wrong base model.","Editorial inference: weight decay may also buffer plasticity across multiple rounds of post-training, such as successive fine-tuning or alignment stages, because lower-rank, less-overfit representations are what continual-learning accounts identify as protecting future learning—but the paper only tests a single fine-tuning round.","Editorial inference: the drop in optimal weight decay from 20 to 140 TPP hints at a two-dimensional scaling law in training tokens and weight decay, with the downstream-optimal λ decreasing as training grows; the paper's three-point sweep at 140 TPP is too sparse to confirm this, but it is directly testable.","Editorial inference: since the ordering rests on one fixed fine-tuning recipe, the cleanest robustness test is a learning-rate and epoch-count grid; if any plausible grid setting reverses the ordering, plasticity needs a definition over a set of recipes rather than one recipe."],"forward_implications":["Pretrained checkpoints should be selected with downstream fine-tuning performance in mind, because the weight decay that minimizes pretraining loss differs from the weight decay that maximizes downstream accuracy.","The common 0.1 default weight decay is likely leaving downstream performance on the table for models trained near the compute-optimal regime; values of 0.3–1.0 did better in the tested setups.","Scaling-law and hyperparameter-sweep studies that report only validation loss can systematically misorder base models from the perspective of the final, fine-tuned model.","A model with higher pretraining validation loss can fine-tune into a better downstream model, so validation loss alone is not a sufficient selection criterion.","The optimal pretraining weight decay for downstream performance depends on training duration: it fell from 1.0 at 20 TPP to 0.3 at 140 TPP in these experiments."],"fun_headline_variants":["Weight decay boosts plasticity, yielding better fine-tuned models","Pretraining weight decay: the overlooked lever for model adaptability","More weight decay, better fine-tuning even with worse base loss","Weight decay enhances model plasticity for fine-tuning gains","Higher weight decay during pretraining improves downstream adaptation"],"cache_read_input_tokens":29952,"weakest_assumption_plain":"The load-bearing premise is that a single fine-tuning recipe—three epochs, batch size 64, learning rate 1e-5 for 1B-and-under models and 7.5e-6 for 4B—measures plasticity fairly across all pretrained variants, so the observed ordering would not reverse if each weight-decay value received its own optimal fine-tuning settings.","fun_headline_variants_meta":{"raw":{"variants":["Weight decay boosts plasticity, yielding better fine-tuned models","Pretraining weight decay: the overlooked lever for model adaptability","More weight decay, better fine-tuning even with worse base loss","Weight decay enhances model plasticity for fine-tuning gains","Higher weight decay during pretraining improves downstream adaptation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000924,"raw_usage":{"total_tokens":3809,"prompt_tokens":765,"completion_tokens":3044,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":2966}},"tokens_in":509,"tokens_out":3044,"duration_ms":23010,"temperature":1.0,"reasoning_tokens":2966,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:11:00.801877+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the OLMo-2-1B-20x checkpoints pretrained with weight decay 0.1 and 1.0. Fine-tune both on one task, such as MetaMathQA, over a grid spanning learning rates 1e-5, 3e-5, and 1e-4 and epoch counts 1, 3, and 6, with batch size 64 held fixed. The paper's claim predicts the weight-decay-1.0 checkpoint wins under all those settings; if any reasonable setting erases or reverses the ordering, the reported plasticity advantage is an artifact of the single fixed recipe.","supporting_citations":[],"review_version":1}