{"id":"7d29f56a-7ffe-44ee-ab3f-be53d80bcd2b","arxiv_id":"2502.06042","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"Finetuning forgetting follows a multiplicative scaling law in model size, finetuning tokens, and injected pretraining fraction, with 1% injection nearly eliminating forgetting.","lead":"This paper measures how much a language model forgets its pretraining data when finetuned on a smaller domain, and fits scaling laws that predict that forgetting from model size, finetuning data size, and the fraction of pretraining data mixed in. It finds that injecting around 1% pretraining data into the finetuning mixture largely prevents the pretraining loss from rising, which gives practitioners a cheap regularizer against forgetting.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (7) omits the size of the pretraining reservoir; the paper's own §4.4 shows 1% injection fails to prevent forgetting when unique pretraining tokens are scarce, so the headline 1% rule is conditional and the law is misspecified in a realistic regime.","rationale":"The reader's verdict is CONDITIONAL, and my read does not overturn it; the scaling-law fits, bootstrap errors, and extrapolation table are genuine supporting evidence. I looked for the least secure condition for the central claim and found a different soft spot than the reader's isocurve concern. The D=100N isocurve is acknowledged in §4.5 and partially mitigated by the D=10N check in Appendix A, so it is a known extrapolation limit. The stronger issue is that Eq. (7) has no variable for the number of unique pretraining tokens available for injection, and the paper's own §4.4/Figure 6 shows that this quantity changes the outcome at fixed p. The headline '1% prevents forgetting' is therefore not supported in the finite-reservoir regime, which is a realistic deployment condition. This is an internal, concrete limitation rather than a disagreement with consensus. I give credit for the held-out extrapolation in Table 4 and the downstream ARC-easy check, which independently support the mitigation effect, but neither tests the reservoir-size dependence that Figure 6 exposes. A single targeted experiment with 32KT vs 3,200KT reservoirs and a comparison to Eq. (7) would settle whether the law needs a reservoir term. If the 32KT point deviates beyond the reported MRE, the abstract and Eq. (7) should be revised to state the reservoir requirement explicitly.","tokens_in":23357,"tokens_out":11696,"duration_ms":114017,"concrete_test":"Re-run the §4.4 experiment (Arxiv, Tiny, p=1%, Dft=9,000K) with reservoir sizes 32KT and 3,200KT, and compare the measured pretraining loss to the prediction of Eq. (7) using the Table 2 Arxiv coefficients. If the 32KT point deviates by more than the reported 0.40% MRE, or if adding an explicit reservoir-size factor (e.g., multiplying the forgetting term by R^γ) significantly reduces the MRE on a leave-one-reservoir-out evaluation, then Eq. (7) is misspecified for finite reservoirs and the abstract's 1% claim must be qualified to large/diverse pretraining pools.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central forgetting law, Eq. (7), predicts Lpt from model size N, finetuning tokens Dft, injection fraction p, and a measured L0pt. It contains no term for the number of unique pretraining tokens available for injection. Section 4.4 explicitly states that the main experiments stream pretraining data 'without repetition,' i.e., from an effectively infinite reservoir. However, Figure 6 varies the reservoir from 32KT to 3,200KT unique pretraining tokens at fixed p=1% and shows that the post-finetuning pretraining loss depends strongly on reservoir size: with a small reservoir, repeated injected tokens overfit the pretraining set and forgetting is not prevented. Thus Eq. (7) is fitted and validated only in the infinite-reservoir regime, while the abstract states unconditionally that 'injecting as little as 1% of pretraining data prevents the model from forgetting the pretraining set.' In the common practical case where only a finite cache of pretraining data is available, the 1% rule can fail, and Eq. (7) cannot predict or even express that failure. This is not an external extrapolation worry: the paper's own experiment demonstrates the missing variable, and the absence of a reservoir term makes the law incomplete as a general predictor of forgetting.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies catastrophic forgetting during full-parameter finetuning of GPT-style language models and proposes two per-domain scaling laws: one for the finetuning validation loss (Eq. 6, following Zhang et al.) and one for the pretraining loss after finetuning as a measure of forgetting (Eq. 7). The forgetting law predicts L_pt from the pretrained baseline L0_pt, model size N, finetuning tokens D_ft, and injection fraction p, with a multiplicative factor (1+Bp)N. The authors fit these laws on 125 configurations per domain (5 model sizes × 5 finetuning dataset sizes × 5 injection fractions) across 12 domains of The Pile, report mean relative errors of 0.89% (finetuning) and 0.40% (forgetting), and validate extrapolation to larger models and larger finetuning datasets. The headline practical claim is that injecting p=1% pretraining data into the finetuning mixture mitigates forgetting with negligible cost to finetuning loss.","tokens_in":23743,"tokens_out":9088,"duration_ms":74551,"significance":"The work is a careful, large-scale empirical study in a practically important setting. Its strengths include the out-of-sample extrapolation checks in Table 4, the use of multiple domains, the comparison to alternative law forms, and the explicit measurement of the rewarming offset L0_pt. If the forgetting law survives the caveats below, it would give practitioners a cheap way to predict forgetting from small-scale runs. However, the law's scope is narrower than the abstract suggests: it is fitted only on the D=100N pretraining isocurve and only for streaming (infinite-reservoir) pretraining data injection.","major_comments":[{"comment":"The forgetting law in Eq. (7) does not include the number of unique pretraining tokens available for injection. The authors' own experiment in §4.4 (Figure 6) shows that at fixed p=1%, the post-finetuning pretraining loss depends strongly on the reservoir size: with only 32KT-320KT unique pretraining tokens, repeated injection of the same tokens overfits the pretraining set and forgetting is not prevented. Since Eq. (7) contains no reservoir term, it is only valid in the infinite-reservoir streaming regime, while the abstract states that 'injecting as little as 1% of pretraining data... prevents the model from forgetting the pretraining set' without this caveat. This is a load-bearing limitation of the central claim; the law cannot predict the finite-reservoir failure mode shown in the paper itself. The authors should either extend the law (e.g., with a repetition or reservoir-size factor) or explicitly limit the scope of the claim and the law to streaming injection, and revise the abstract accordingly.","section":"Section 4.4, Eq. (7)"},{"comment":"All pretrained checkpoints lie on the isocurve D=100N (Section 3.1, Table 1), so the fitted exponent α in Eq. (7) conflates model size N with the number of pretraining tokens D_pretrain. The statement in the Analysis that 'forgetting is primarily attributed to network capacity' is therefore not identified: larger models also received more pretraining data, and the D=10N check in Figure 11 does not separate the two covariates because it only changes the isocurve ratio, not N and D_pretrain independently. The authors should rephrase the capacity interpretation, or add experiments that vary N and D_pretrain separately, or at least prominently flag the confound as a limitation.","section":"Section 4.5, Eq. (7) and Table 2"}],"minor_comments":[{"comment":"The injection fractions are listed as {0%, 0.1%, 0.5%, 1%, 5%}, but Figure 15 shows 0.2% in the legend; please reconcile this discrepancy.","section":"Section 3.1 / Figure 15"},{"comment":"The text states pretraining uses 100 tokens per parameter, but the D/N ratios in Table 1 range from about 79 to 124; please clarify whether the isocurve is approximate.","section":"Section 3.1 / Table 1"},{"comment":"The sentence 'they lose up to 95% (!) of the pretraining progress' appears to conflict with Figure 14's caption ('up to 80%'); please clarify which number is correct or specify the domain and condition.","section":"Section 4.5 / Figure 14"},{"comment":"The sentence 'Remarkably, only 0.3 unique pretraining tokens per unique finetuning token are sufficient to avoid forgetting' is difficult to reconcile with the main text's finding that limited unique tokens increase pretraining loss; please rephrase to state the threshold condition and its dependence on D_ft.","section":"Figure 6 caption"},{"comment":"The phrase 'prevents the model from forgetting' is too absolute given the finite-reservoir results in §4.4; I suggest using 'mitigates' or adding a qualified statement.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical study, but the abstract's unconditional 1% claim and the missing reservoir term in the law are significant issues. I believe a major revision is appropriate, ideally asking the authors to add a reservoir-aware variant of the law or to reframe the contribution within the streaming-injection scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a solid empirical scaling-law study with a genuinely new forgetting law and tight fits, but the headline '1% injection prevents forgetting' is conditional on an effectively infinite pretraining reservoir. The paper's own Section 4.4 shows that when the unique pretraining token pool is small, repeated injected tokens overfit and forgetting comes back; Equation (7) has no reservoir term, so the law can't express or predict that failure.\n\nWhat is new: the forgetting law Lpt = L0pt + A Dft^beta / ((1+Bp)N)^alpha, including the injection fraction p and the rewarming offset from L0pt, extends Zhang et al.'s multiplicative finetuning law and goes beyond Kalajdzievski, who didn't study injection or full-parameter finetuning. The fits are tight—MRE 0.89% for finetuning, 0.40% for forgetting—and the extrapolation test (fit on models up to Medium and 3M finetune tokens, predict Large/XL at 9M/30M) is a real out-of-sample check. The D=10N retraining in the appendix is a nice robustness point, and the citation pattern is fair.\n\nSoft spots, in proportion. The missing reservoir term is the main one; the abstract overclaims relative to what the law covers. The D=100N isocurve means alpha conflates model size with pretraining token count; the paper acknowledges it, and the D=10N check mitigates it, but the exponents shouldn't be read as cleanly separable. The p grid is coarse (five values) and single-seed, so the precise 1% is approximate. No code or artifacts are released, which makes the fits harder to reproduce.\n\nWho this is for: practitioners finetuning small-to-mid-size LMs on a target domain who want a cheap rule for mixing in pretraining data. The law is descriptive, fitted on GPT-2-scale models up to 1.3B, so extrapolation to frontier models is a guess. It deserves a serious referee; a good revision would add a reservoir term to Eq. (7) or at least state the regime, and soften the abstract's unconditional claim.","headline":"Solid scaling-law study with a genuinely new forgetting law and a real extrapolation check, but the '1% injection prevents forgetting' headline only holds with an effectively infinite pretraining reservoir, which the paper's own Section 4.4 demonstrates.","tokens_in":24276,"tokens_out":3064,"would_cite":true,"duration_ms":26776,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A scaling law predicts how much finetuning makes language models forget, and 1% pretraining data in the mix prevents most of the damage.","keywords":["scaling laws","catastrophic forgetting","finetuning","pretraining data injection","language models","next-token prediction","overfitting","extrapolation"],"falsifier":"Pretrain a 334M-parameter model on both 3.3B and 33B tokens, finetune both on the same target domain with $p=1\\%$, and check whether the same $\\alpha$ in Equation 7 predicts both pretraining losses; a systematic mismatch would falsify the claim that $\\alpha$ captures scale alone.","tokens_in":23137,"feed_emoji":"🛡️","tokens_out":7303,"duration_ms":62628,"temperature":0.7,"pith_summary":"This paper aims to turn two failure modes of finetuning — overfitting to a small target dataset and forgetting the pretraining distribution — into quantities that can be predicted before a run. The authors propose a scaling law for the pretraining loss after finetuning, measured at the best finetuning-validation checkpoint, as a function of model size, number of finetuning tokens, and the fraction of pretraining data mixed into finetuning. Across twelve target domains the law predicts the actual pretraining loss with a mean relative error of 0.40%, and it extrapolates from small models and small datasets to larger ones within about 1–2% error. A second, simpler law predicts the finetuning validation loss, which is essentially independent of the injection fraction. The headline practical result is that injecting just 1% pretraining data suppresses most forgetting with no measurable cost to finetuning loss.","feed_headline":"One formula predicts finetuning forgetfulness to 0.4 percent error","feed_subtitle":"Injecting just 1% pretraining data shields language models from forgetting at no measurable cost to finetuning loss.","key_machinery":"The carrying object is the modified multiplicative scaling law for the pretraining loss (Equation 7 of the paper), $L_{\\mathrm{pt}} = L^0_{\\mathrm{pt}} + A\\, D_{\\mathrm{ft}}^{\\beta} / ((1+Bp)N)^{\\alpha}$, with the term $(1+Bp)N$ as the mechanism that prices in pretraining data injection: a fraction $p$ of parameters is \"owned\" by the pretraining task and is $B$ times more efficient at preserving pretraining loss than parameters devoted to finetuning. The law is evaluated at the checkpoint that minimizes finetuning validation loss, i.e. the bottom of the overfitting U-curve, and $L^0_{\\mathrm{pt}}$ is measured on the rewarmed model — the pretrained model at the constant learning rate used for finetuning — because the rewarming itself increases the pretraining loss. Coefficients are fitted per domain by Huber loss in log space, following the standard scaling-law fitting procedure.","core_discovery":"The central discovery is that forgetting during finetuning is not erratic but obeys the multiplicative law $L_{\\mathrm{pt}} = L^0_{\\mathrm{pt}} + A\\, D_{\\mathrm{ft}}^{\\beta} / ((1+Bp)N)^{\\alpha}$, where $L^0_{\\mathrm{pt}}$ is the pretraining loss of the pretrained model before finetuning, $D_{\\mathrm{ft}}$ is the number of finetuning tokens, $N$ is model size, $p$ is the fraction of pretraining data injected, and $A$, $B$, $\\alpha$, $\\beta$ are domain-dependent positive constants. The factor $(1+Bp)N$ encodes the idea that a fraction $p$ of the model's parameters is allocated to the pretraining task, and these parameters are $B$ times more efficient at retaining pretraining performance than those allocated to finetuning; $B\\gg 1$ because pretrained features are already aligned with the pretraining loss. The paper reports a bootstrapped mean relative error of 0.40% across twelve domains of The Pile, and shows the law still holds when pretraining checkpoints are taken on the $D=10N$ isocurve and when instruction finetuning is used. For the finetuning loss itself, the paper confirms the multiplicative law $L_{\\mathrm{ft}} = A/(N^{\\alpha} D_{\\mathrm{ft}}^{\\beta}) + E$ with 0.89% error and essentially no dependence on $p$, and shows that injecting 1% pretraining data does not hurt finetuning performance while strongly reducing forgetting.","pith_inferences":["I infer that the fitted coefficient $B$ can serve as a cheap, quantitative measure of a target domain's distance from the pretraining distribution: domains far from the pretraining mix (Dm mathematics, $B \\approx 10^4$) need injection most, while similar domains (Wikipedia) need it least — a relationship the paper observes but does not formalize.","Because the pretraining-token budget is tied to model size ($D=100N$), the law's $\\alpha$ conflates scale with pretraining data volume; I expect the exponents would need re-fitting for models trained at compute-optimal or other token budgets.","The paper measures forgetting through pretraining loss; a natural testable extension is to check whether the same law predicts accuracy drops on held-out benchmarks, grounding the loss-based proxy in task-level forgetting.","The observed rule that roughly 0.3 unique pretraining tokens per finetuning token suffices to avoid forgetting suggests a diversity threshold, which could be probed directly by ablating the size of the injected pretraining pool."],"forward_implications":["Forgetting before a finetuning run can be predicted from three inputs — model size, target token count, and injection fraction — with no need to train the model first, once the domain coefficients are fitted on small runs.","The injection fraction $p$ can be chosen from the law: $p=1\\%$ already shields pretraining loss, and larger $p$ improves finetuning validation loss only mildly before a U-curve makes it worse.","Small-scale experiments (models up to 334M parameters, datasets up to 3,000K tokens) predict the behavior of models up to 1.3B parameters and 30,000K tokens within about 2% on finetuning loss and 1% on forgetting.","Forgetting is primarily a capacity phenomenon: small models lose up to 95% of pretraining progress while large models lose about 20%, so bigger models are more robust to sequential finetuning at higher compute cost.","The finetuning loss obeys a multiplicative law with no measurable dependence on $p$, meaning pretraining data injection is a nearly free regularizer for the target domain."],"supporting_citations":[{"why":"Supplies the multiplicative finetuning scaling law (Equation 6) that the paper extends to forgetting, and the finding that full finetuning outperforms parameter-efficient methods.","marker":"(Zhang et al., 2024)"},{"why":"Provides the pretraining scaling-law context, the isocurve $D=100N$ convention, and the cost analysis used to quantify forgetting's GPU-hour price.","marker":"(Hoffmann et al., 2022a)"},{"why":"The closest prior characterization of forgetting during finetuning, which this paper contrasts with its multi-scale, injection-aware law.","marker":"(Kalajdzievski, 2024)"},{"why":"Establishes that a small percentage of pretraining data mitigates forgetting in continual pretraining, motivating the injection study here.","marker":"(Ibrahim et al., 2024)"},{"why":"Introduces scaling laws for transfer with repeated data, the foundation for treating finetuning's data-repetition regime.","marker":"(Hernandez et al., 2021)"},{"why":"Supplies the data-constrained scaling-law fitting procedure (Huber loss, log-space optimization) used to estimate coefficients.","marker":"(Muennighoff et al., 2023)"},{"why":"Provides the twelve The Pile domains used as finetuning target datasets.","marker":"(Gao et al., 2020)"},{"why":"Provides the RedPajamaV2 pretraining corpus on which all models are pretrained and forgetting is measured.","marker":"(Weber et al., 2024)"}],"fun_headline_variants":["1% pretraining data stops finetuning forgetfulness","Formula predicts finetuning forgetting to 0.4% error","Inject 1% pretraining data to keep model knowledge","Scaling law reveals finetuning forgetting and its fix","Tiny pretraining injection prevents catastrophic forgetting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model-size exponent in the forgetting law is estimated from models that were always pretrained on 100 tokens per parameter, so the law does not separate model scale from pretraining data volume.","fun_headline_variants_meta":{"raw":{"variants":["1% pretraining data stops finetuning forgetfulness","Formula predicts finetuning forgetting to 0.4% error","Inject 1% pretraining data to keep model knowledge","Scaling law reveals finetuning forgetting and its fix","Tiny pretraining injection prevents catastrophic forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000548,"raw_usage":{"total_tokens":2668,"prompt_tokens":1044,"completion_tokens":1624,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":1544}},"tokens_in":660,"tokens_out":1624,"duration_ms":11188,"temperature":1.0,"reasoning_tokens":1544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:56:05.967804+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pretrain a 334M-parameter model on both 3.3B and 33B tokens, finetune both on the same target domain with $p=1\\%$, and check whether the same $\\alpha$ in Equation 7 predicts both pretraining losses; a systematic mismatch would falsify the claim that $\\alpha$ captures scale alone.","supporting_citations":[{"cited_title":"When scaling meets LLM finetuning: The effect of data, model and finetuning method","cited_arxiv_id":null,"evidence_quote":"Supplies the multiplicative finetuning scaling law (Equation 6) that the paper extends to forgetting, and the finding that full finetuning outperforms parameter-efficient methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RedPajamaV2 pretraining corpus on which all models are pretrained and forgetting is measured."}],"review_version":1}