Pith. sign in

REVIEW 4 major objections 6 minor 10 references

Stacked pretraining curbs forgetting during continual fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Growth-based pretraining (StackLLM) forgets less in continual fine-tuning, especially on reading comprehension, but reduces social bias less than standard training.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Honest but statistically weak empirical note on whether stack-pretrained LLMs forget less; the headline RACE gap is within sampling noise, so treat it as suggestive. the 4 major comments →

arxiv 2509.01213 v1 pith:F5KUIBGF submitted 2025-09-01 cs.CL

Mitigating Catastrophic Forgetting in Continual Learning through Model Growth

Ege S\"ualp , Mina Rezaei This is my paper
classification cs.CL
keywords catastrophic forgettingcontinual learningmodel growthlayer stackinglarge language modelsfine-tuningreading comprehensionsocial bias
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether model growth—pretraining a large language model by stacking layers of a smaller trained model—helps resist catastrophic forgetting when the model is later fine-tuned on a sequence of new tasks. Comparing a 7-billion-parameter stack-grown model with a from-scratch twin trained on the same token budget, the authors run both through three continual fine-tuning tasks and evaluate domain knowledge, reasoning, reading comprehension, and social bias after each step. Both models gain domain knowledge and lose ground on reasoning and reading comprehension, but the stack-grown model degrades less—most clearly on reading comprehension, where it scores a forgetting value of 6.8 versus 11.9 for the from-scratch model. The from-scratch model becomes more neutral in bias evaluations over time, while the stack-grown model stays near 60–61%. The paper's claim is that growth-based pretraining offers modest retention benefits, with trade-offs in bias handling.

Core claim

The central claim is that a decoder-only 7B language model pretrained by layer-stacking retains previously learned capabilities better than an architecturally identical from-scratch model when both are continually fine-tuned on the same three tasks. The paper reports forgetting on reading comprehension of 6.8 for the stacked model versus 11.9 for the baseline, and on reasoning of 7.3 versus 8.2, while domain-knowledge improves for both (negative forgetting values). The stacked model also shows lower training and evaluation loss during the fine-tuning tasks, and a 10.1% BERTScore gain on the third task compared with the baseline's 5.7%. The interpretation is that growth-based pretraining may

What carries the argument

The mechanism is model growth by depth-wise layer stacking: a small model is trained on a small fraction of the token budget, its layers are stacked to the target depth, and the enlarged model continues training on the full budget. The paper compares the stack-grown 7B model with its from-scratch counterpart trained on the same 300-billion-token budget, and applies a continual fine-tuning protocol with three sequential tasks (text simplification, empathetic dialogue, and inquisitive question generation). The retention comparison is carried by the forgetting metric FG, which averages the relative drop in each evaluation category across the fine-tuning steps.

Load-bearing premise

The load-bearing premise is that the two 7-billion-parameter checkpoints are a matched pair—same architecture and similar 300-billion-token pretraining budget, differing only in the layer-stacking growth method—so any retention gap can be attributed to growth; the authors' own Limitations section notes that fine-tuning used different GPU counts, batch sizes, gradient accumulation steps, and ZeRO configurations, which can confound the comparison.

What would settle it

Replicate the three-task continual fine-tuning for both models with identical GPU count, batch size, gradient-accumulation steps, and ZeRO configuration, computing FG on RACE after each task; if the 6.8-versus-11.9 gap between the stacked and from-scratch models narrows to near zero, the claimed retention benefit of layer-stacking is an artifact of unequal training conditions rather than a property of model growth.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, stack-grown models are a cost-effective choice for continual fine-tuning pipelines: they reach comparable performance at lower pretraining cost and retain more of their starting capability across successive tasks.
  • Applications that depend on reading comprehension, such as long-document question answering, would see less capability erosion when models are repeatedly adapted to new instructions.
  • The bias asymmetry implies that continual fine-tuning of a stack-grown model cannot be assumed to reduce stereotypical outputs; bias mitigation may need to be an explicit objective for such models.
  • Reporting forgetting per category matters more than aggregate scores: both models improve on domain knowledge while losing on reasoning and comprehension, so an average retention score would hide the losses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper does not run: vary the growth factor (e.g., stacking a model trained on a quarter vs. a half of the target depth) to see whether retention advantages scale with how much structure the small model provides.
  • A controlled replication that fixes GPU count, batch size, gradient-accumulation steps, and ZeRO configuration for both models is the direct way to pin down the size of the retention advantage, since the authors varied these settings across tasks.
  • If layer stacking regularizes representations by reusing layers, it may also slow the unlearning of biased associations; testing this by measuring bias-relevant internal activations before and after each fine-tuning step could connect the two findings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether model-growth pretraining, specifically layer-stacking, reduces catastrophic forgetting during sequential fine-tuning of large language models. Using two 7B models from Du et al. (2024) — StackLLM (stacking-based pretraining) and LLM (standard pretraining) — the authors fine-tune each model on three tasks (text simplification, empathetic dialogue generation, inquisitive question generation) and measure forgetting on evaluation categories: domain knowledge (MMLU), reasoning (BoolQ, PIQA, Winogrande, HellaSwag, MathQA, Mutual), reading comprehension (RACE-high), and bias (CrowS-Pairs). They report that both models improve in domain knowledge, but both show degradation in reasoning and reading comprehension; StackLLM shows less degradation, especially on RACE (FG 6.8 vs 11.9), which they interpret as enhanced retention. They also report that the baseline LLM becomes more neutral on bias measures over continual fine-tuning, while StackLLM stays around 60–61% biased, a trade-off the paper discusses.

Significance. If the reported effects are real, the paper provides a useful empirical datapoint: layer-stacking, which is already known to accelerate pretraining, may also confer modest retention benefits during subsequent continual fine-tuning. The study reuses publicly available checkpoints, follows an established continual-learning protocol, and uses a transparent, parameter-free forgetting metric (Eq. 1), which is a strength. The paper is also honest about several threats to validity in its Limitations section. However, the central retention claim rests on single-run comparisons without error bars, selected checkpoints, and a small evaluation set for the headline reading-comprehension result. As such, the study is more suggestive than conclusive; with additional uncertainty quantification or multi-seed runs it could become a solid empirical contribution.

major comments (4)
  1. [§IV-C, Table IV, Eq. (1)] The headline claim of enhanced retention in reading comprehension rests on a single pair of FG values (6.8 vs 11.9) computed from RACE-high accuracies at R0–R3. With a test set of approximately 1,049 examples, the standard error of each accuracy near p≈0.34 is roughly 1.5 percentage points; the final accuracy difference (33.7% vs 31.4%) is about 2.3 points, or ≈1.5 SE. The 5.1-point FG gap is therefore within the range of sampling noise. No confidence intervals, bootstrap estimates, or repeated-seed runs are provided. The phrase 'consistently shows less degradation' is not supported by a single run. Please provide uncertainty estimates or multiple runs, or explicitly soften the conclusion to a preliminary observation.
  2. [§IV-A and §VI] The evaluation checkpoints were selected after observing the loss curves. The paper notes a clear overfitting pattern (validation loss increases after epoch 1) yet uses the end-of-epoch-3 model 'to ensure comparability with prior work by Luo et al.' This post-hoc checkpoint choice can differentially affect the FG values of the two models, and the overfitting acknowledged in Section IV-A undermines the reliability of the reported retention advantage. Please report results for all epoch checkpoints or justify a pre-specified selection criterion, and show that the main conclusion is robust to the choice.
  3. [§VI (Limitations)] The Limitations state that 'our MMLU evaluation results and observed patterns differ significantly from those reported in Luo et al.' Since the paper's continual-learning protocol and evaluation categories are explicitly adopted from that study, a significant discrepancy suggests differences in evaluation setup or implementation that could affect the forgetting comparison. The paper should analyze and explain this discrepancy, and indicate whether the StackLLM-vs-LLM differences persist under the original evaluation configuration or with an alternative MMLU harness.
  4. [§VI, Table V] Fine-tuning configurations are not constant across the experiment. Section VI mentions different GPU counts, batch sizes, gradient accumulation steps, and ZeRO optimizer configurations across tasks. Although Table V shows matching settings for the two models within each task, it does not list ZeRO configuration details, and the text suggests further unlisted variation. Because the FG metric compares performance across tasks, variable compute settings could confound the forgetting trajectories. Please clarify exactly which settings differ across models versus across tasks, and ideally provide at least a sensitivity check showing the main results are unchanged under fixed settings.
minor comments (6)
  1. [§II-a] Typo: 'depthhwise' should be 'depthwise'.
  2. [§IV-A] The text refers to 'Empd' but Table I and the method section use 'Emdg' (empathetic dialogue generation). Please make the abbreviation consistent.
  3. [§III-B] Equation (1) is not typeset clearly in the preprint; the fraction inside the sum can be misread. Please format it with an explicit numerator/denominator and define the index of summation over fine-tuning steps.
  4. [§IV-B] 'Bert Score' should be 'BERTScore' for consistency with the references and the table.
  5. [§IV-A] Minor phrasing: 'the results are promising, indicating...' is a new sentence and should begin with a capital letter. Also consider adding a comma after 'promising'.
  6. [§III-A] The phrase 'categorized under 3:' is awkward; use 'categorized into three groups:'.

Circularity Check

0 steps flagged

No significant circularity; the comparison is computed directly from externally supplied checkpoints and measured accuracies.

full rationale

The paper's derivation chain is self-contained with respect to the claim being made. The two models (StackLLM and LLM) are externally pretrained checkpoints from Du et al. [2024]; the continual-learning tasks and formatted datasets come from Scialom et al. [2022]; the forgetting metric FG is Eq. (1), taken from Luo et al. [2025], and is computed directly from measured evaluation accuracies at checkpoints R0-R3. There are no fitted parameters, no constants tuned to the data, and no prediction that is defined in terms of the outcome it is supposed to explain. The central comparison—StackLLM's FG of 6.8 vs. LLM's 11.9 for reading comprehension—is a direct arithmetic consequence of the reported RACE accuracies, not of any assumption that already contains the conclusion. The cited prior works are external to the authors and are used for methodology and models, not as a self-referential justification of the result. The limitations acknowledged in Section VI (different GPU counts, batch sizes, gradient accumulation, ZeRO configurations, and the choice of epoch-3 checkpoints despite observed overfitting) undermine the statistical strength of the claim, but they do not make the derivation circular. Concerns about sampling error or confounding are validity issues, not circularity issues. Thus the paper does not exhibit any of the identified circularity patterns.

Axiom & Free-Parameter Ledger

3 free parameters · 2 axioms · 0 invented entities

The central comparison rests on the matched-pair assumption and the reproduced evaluation protocol; these are domain assumptions, not standard math. The forgetting metric has no fitted parameters, but procedural choices, including checkpoints, regularization, and compute configuration, vary across runs and can confound the conclusion.

free parameters (3)
  • Evaluation epoch checkpoints = M1: E3; M3: E2/E3
    Chosen after observing overfitting to match prior work; changes the reported forgetting scores.
  • M3 regularization hyperparameters = cosine schedule, warmup 100 steps, weight decay 0.01, LR 2e-5
    Introduced ad hoc for the third task only, creating a confound with M1 and M2.
  • Per-task compute configuration = 6-8 GPUs, batch size 8-16, grad accum 1-2, ZeRO-3
    Adjusted per task and model to avoid OOM; not held constant across compared runs.
axioms (2)
  • domain assumption The two 7B checkpoints from Du et al. are a matched pair (same architecture, data, and token budget), so retention differences are attributable to layer stacking.
    Section III-C states that architecture and dataset details are the same and both models used 300B tokens; this is the basis for a causal reading of the comparison.
  • domain assumption The evaluation protocol of Luo et al. is reproduced faithfully, including task order, metric definitions, and benchmark versions.
    The paper reuses Luo et al.'s categories and forgetting metric (Eq. 1), but acknowledges in Limitations that MMLU results differ significantly, so protocol equivalence is not established.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Catastrophic Forgetting in Continual Learning through Model Growth." pith.science (2026). https://pith.science/paper/F5KUIBGF

@misc{pith2026250901213,
  author       = {Pith},
  title        = {Pith review of: Mitigating Catastrophic Forgetting in Continual Learning through Model Growth},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5KUIBGF}},
  note         = {Machine review of arXiv:2509.01213}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Catastrophic forgetting is a significant challenge in continual learning, in which a model loses prior knowledge when it is fine-tuned on new tasks. This problem is particularly critical for large language models (LLMs) undergoing continual learning, as retaining performance across diverse domains is important for their general utility. In this paper, we explore model growth, a promising strategy that leverages smaller models to expedite and structure the training of larger ones for mitigating the catastrophic forgetting problem. Although growth-based pretraining, particularly via transformer stacking, has shown promise in accelerating convergence, its impact on forgetting remains under-explored. Therefore, we evaluate whether growth-based models can retain previously learned capabilities more effectively across a sequence of fine-tuning tasks involving domain knowledge, reasoning, reading comprehension, and bias. Our findings show that both models -- one trained with growth (Stack LLM) and one without (LLM) -- exhibit improvements in domain knowledge. However, reasoning and reading comprehension degrade over time, indicating signs of catastrophic forgetting. Stack LLM consistently shows less degradation, especially in reading comprehension, suggesting enhanced retention capabilities. Interestingly, in bias evaluation, the baseline LLM becomes progressively more neutral with continued fine-tuning, while Stack LLM maintains a steady bias ratio around 60--61\%. These results indicate that growth-based pretraining may deliver modest improvements in resisting catastrophic forgetting, though trade-offs remain in handling social biases.

Figures

Figures reproduced from arXiv: 2509.01213 by Ege S\"ualp, Mina Rezaei.

Figure 1
Figure 1. Figure 1: Training and validation loss during fine-tuning for [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance in Evaluation Tasks during Continual Learning. Domain Knowledge, Reasoning, Reading Comprehension in respective order. Metric [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Bias Percentage: the percentage of examples for which a model assigns a higher (pseudo-)likelihood to the stereotyping sentence [Nangia et al., 2020]. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Forgetting (FG) as the average decrease in performance across tasks, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

10 extracted references · 1 canonical work pages

  1. [5]

    Chin-Yew Lin

    URL https://arxiv.org/abs/2001.08361. Chin-Yew Lin. ROUGE: A package for automatic eval- uation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop , pages 74–81, Barcelona, Spain, July

    Pith/arXiv arXiv 2001

  2. [7]

    doi: 10.18653/v1/2020.emnlp-main.154

    Association for Computa- tional Linguistics. doi: 10.18653/v1/2020.emnlp-main.154. URL https://aclanthology.org/2020.emnlp-main.154/. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , pages 311...

    doi:10.18653/v1/2020.emnlp-main.154 2020

  3. [8]

    doi: 10.18653/v1/P19-1534

    Association for Computational Linguistics. doi: 10.18653/v1/P19-1534. URL https://aclanthology.org/ P19-1534/. Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners,

    doi:10.18653/v1/p19-1534

  4. [9]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey

    URL https://arxiv.org/abs/2205.12393. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://cerebras.ai/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, June

    Pith/arXiv arXiv

  5. [11]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q

    URL https://arxiv.org/abs/ 2401.02385. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Wein- berger, and Yoav Artzi. BERTScore: Evaluating text gener- ation with BERT. In International Conference on Learning Representations,

    Pith/arXiv arXiv

  6. [12]

    URL https://openreview.net/forum? id=SkeHuCVFDr. VII. *A PPENDIX A. Implementation Details For the training tasks, we used 6 to 8 NVIDIA A100 40GB GPUs (see Table V), whereas for the evaluation tasks, we used only 2 GPUs. All fine-tuning tasks were parallelized using the Deepspeed ZeRO-3 optimizer. B. Model Diagnosis Details for the StackLLM model are lis...

    work page 2022

  7. [2019]

    doi: 10.18653/v1/P19-1346

    Association for Computational Linguistics. doi: 10.18653/v1/P19-1346. URL https://aclanthology.org/P19-1346/. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Gold- ing, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle Mc- Donell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Sc...

    doi:10.18653/v1/p19-1346

  8. [2022]

    Wenyu Du, Tongxu Luo, Zihan Qiu, Zeyu Huang, Yikang Shen, Reynold Cheng, Yike Guo, and Jie Fu

    URL https://arxiv.org/abs/2210.11416. Wenyu Du, Tongxu Luo, Zihan Qiu, Zeyu Huang, Yikang Shen, Reynold Cheng, Yike Guo, and Jie Fu. Stacking your transformers: A closer look at model growth for efficient llm pre-training,

    Pith/arXiv arXiv

  9. [2024]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli

    URL https://arxiv.org/abs/2405.15319. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form ques- tion answering. In Anna Korhonen, David Traum, and Llu´ıs M `arquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3558–3567, Florence, Italy, July

    Pith/arXiv arXiv

  10. [2025]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R

    URL https://arxiv.org/abs/2308.08747. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pag...

    Pith/arXiv arXiv 2020

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.