REVIEW 4 major objections 7 minor 2 cited by
How to Alleviate Catastrophic Forgetting in LLMs Finetuning? Hierarchical Layer-Wise and Element-Wise Regularization
T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A cheap importance-recording pass can keep a large language model's general knowledge intact while it is fine-tuned to a specialized domain.
desk verdict A plausible SI-for-LoRA adaptation whose main empirical claim rests on a circular Pile-PPL metric; needs held-out evaluation and code before it can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the per-parameter importance score $\Omega_i^{\nu} = \frac{\omega_i^{\nu}}{(\Delta_i^{\nu})^2 + \xi}$, where $\omega_i^{\nu} = -\int_{t_0}^{t_1} \tilde{g}_i(t) \tilde{\theta}'_i(t)\,dt$ is a path integral of the loss gradient over the LoRA update trajectory, and $\Delta_i^{\nu}$ is the total change of that parameter during the general-knowledge recording phase. The argument carries through a derived 'hypothetical gradient' $\tilde{g}(t) = g_B(t)A(t) + B(t)g_A(t) - \eta g_B(t)g_A(t)$ for the LoRA update rule, which lets the path-integral idea from full fine-tuning apply to low-rank adapters. This importance score then defines a per-layer regularizer $L_{\text{reg},l} = \sum_i \Omega_i^{\nu} (\theta_i^t - \theta_i^{\nu})^2$, and the layer-wise coefficients $\text{softmax}(\|\Omega_l^{\nu}\|_2)$ decide how strongly each layer is held near its recorded value. The whole mechanism is a quadratic safeguard around the $\nu$-task solution, weighted by how much each parameter contributed to learning that task.
What would settle it
Run the same two-phase procedure but evaluate the fine-tuned model on a broad held-out suite of general-knowledge benchmarks that do not overlap with Pile; if accuracy there drops by the same amount as unregularized LoRA while Pile perplexity stays low, the claim that importance-based regularization protects general knowledge is falsified.
Extended reading notes
Core claim
The central claim is that catastrophic forgetting during LoRA fine-tuning can be alleviated without a Fisher matrix, by computing a Synaptic-Intelligence-style importance score for each LoRA parameter during a general-language recording phase and then regularizing the fine-tuning objective with a weighted quadratic penalty on those parameters. The paper further claims that treating each layer's regularization strength separately, via a softmax over the L2 norms of per-layer importance, yields better retention than a uniform penalty. Empirically, the method—called HLoRA—achieves the lowest perplexity on the Pile corpus among all compared methods on three domain QA datasets across two backbone models, while improving or matching domain accuracy; the importance-recording phase is reported to take about 1.2 hours versus roughly 26 hours for EWCLoRA and to store 1.3–3.5 GB rather than 12–23 GB.
Load-bearing premise
The empirical claim depends on Pile perplexity being a faithful measure of general ability, and it is the same corpus used to record importance, so the results do not by themselves show that other general capabilities survive fine-tuning.
Editorial extensions
If this is right
- A single short recording pass over general text yields importance scores that can then regularize fine-tuning on any domain task.
- Removing the Fisher matrix cuts the storage footprint of importance-based regularization from 12–23 GB to 1.3–3.5 GB and the recording time from roughly 26 hours to about 1.2 hours.
- Layer-wise softmax weighting means different layers receive different regularization strengths, with value-projection layers in early and final blocks flagged as most important for general knowledge.
- The method retains or improves domain accuracy while lowering perplexity, so the protection of general knowledge does not come at the cost of domain adaptation.
- Because the importance-recording phase is a standard LoRA training run rather than a Fisher computation, it requires no additional backpropagation pass and no samples from the model distribution.
Reading between the lines
- A natural next test is whether importance scores recorded on the Pile corpus transfer to protecting general knowledge measured on entirely different benchmarks; the paper does not establish this transfer, since Pile is used both for recording and for evaluating perplexity.
- The reported importance heatmaps suggest a cheaper variant: freeze or down-weight the value-projection LoRA parameters in the first four and last layers during domain fine-tuning, instead of computing full importance matrices.
- The quadratic regularizer assumes the ν-task loss landscape is locally quadratic; one could test whether higher-order terms matter by checking whether the optimal regularization coefficient changes systematically as the distance between the recorded parameters and the domain-tuned parameters grows.
- The layer-wise coefficients are computed from the same importance matrices, so the method could be extended to per-module or per-head regularization, e.g., protecting only attention blocks or only MLP blocks, which the paper does not explore.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HLoRA, a regularization method for LoRA-based fine-tuning of LLMs. It first records element-wise parameter importance during a preliminary fine-tuning stage on a general corpus (Pile), using a path-integral/Synaptic-Intelligence-style formula adapted to the LoRA parameterization. During subsequent domain fine-tuning, it adds a quadratic regularization term that penalizes deviations from the post-ν checkpoint, weighted element-wise by the recorded importance and layer-wise by an L2-norm softmax coefficient. Experiments on GPT-J-6B and LLaMA-3-3B over SciQ, PiQA, and MedMCQA compare the method with LoRA, rsLoRA, EWCLoRA, and sequential fine-tuning baselines, and report lower Pile perplexity, competitive or better domain accuracy, and large speed and storage savings for the importance computation relative to EWCLoRA.
Significance. If the empirical results hold, the method offers a cheap alternative to Fisher-based regularization for LoRA fine-tuning: parameter importance is obtained as a byproduct of the ν-stage training rather than through an additional backward pass, and the storage requirement is per-LoRA-element rather than a full Fisher matrix. The derivation in Eqs. (6)-(13) is a reasonable adaptation of Synaptic Intelligence to the LoRA parameterization, and the layer-wise reweighting in Eq. (15) is a simple, interpretable addition. The main weakness is that the current experimental design does not isolate retention of pretrained general knowledge from continued training on the importance-recording distribution, so the central claim of mitigating catastrophic forgetting is not yet fully established.
major comments (4)
- [Section 5.2, Table 1] The general-ability metric is PPL on Pile, which is exactly the corpus used in the ν importance-recording stage, and Eq. (14) regularizes toward θ_ν, the checkpoint obtained after Pile fine-tuning. Thus the reported improvement for Ours (LLaMA-3 PPL 4.64 vs. Base 4.94) may simply reflect additional training on Pile during the ν stage rather than preservation of pretrained general knowledge. This is especially problematic because Base is never trained on Pile, so the comparison is not apples-to-apples. The paper should report at least one held-out general benchmark that is not used in the ν stage (e.g., MMLU, HellaSwag, ARC, or a disjoint language-modeling corpus); without such evidence, the central claim of mitigating catastrophic forgetting is not established.
- [Section 6.2, Figure 5] The claimed 20x speedup compares the EWCLoRA Fisher computation on 20,000 Pile samples with the authors' 5-epoch ν-stage LoRA training, but the paper does not state how many Pile tokens or examples were used in the 5-epoch ν stage for HLoRA. If the two computations use very different amounts of data, the ratio is not a fair comparison. Please report the data volume, batch size, total optimizer steps, and hardware for both methods, and specify the numerical precision used for storing the importance matrix Ω, since the storage comparison (10%-15%) depends on this.
- [Eq. (14), Section 4.2] The notation Σ_{ν<t<µ} is not defined and appears to mix task indices with time indices; if the sum is intended to accumulate per-step contributions during µ training, it should be written as a sum over optimization steps t. Additionally, in LoRA the full weight θ_t differs from θ_ν only through the product B_tA_t, so the regularization should explicitly state whether the sum runs over entries of the LoRA matrices or over entries of the full weight matrix, since this affects how Ω_i is indexed, stored, and compared with methods such as EWCLoRA.
- [Section 5.2, Table 1, Figure 3] No seed information or error bars are reported for Table 1, so it is unclear whether the point estimates are means over multiple runs. Figure 3 reports independent-samples t-tests, but the text does not state what the samples in the violin plots are (e.g., multiple seeds, multiple test batches, or bootstrap resamples); if they are test-batch perplexities from a single run, a t-test does not support a claim about method-level differences. Please report the number of seeds, standard deviations, and confidence intervals for the main results.
minor comments (7)
- [Eq. (2)] The notation y_k should be y_{i,k} (or similar) to match the example index i in the same equation; as written, the index k is unclear.
- [Eq. (13)] The summation Σ_ν is confusing because ν denotes the single general task; clarify that the sum runs over the optimization steps or epochs of the ν-stage training.
- [Section 5.2] The phrase 'we take Pile as the evaluation datasets for LLM general ability' should be justified, since Pile is a training corpus; using the training distribution as an evaluation set inflates apparent retention and should at least be acknowledged as a limitation.
- [Figure 3] The text states that p-values are below 0.05 for several comparisons but does not report the actual p-values; please include them in the caption or text.
- [Section 6.2] The heading 'Time Complexity' measures wall-clock time, not asymptotic complexity; rename it to something like 'Computation Time' to avoid confusion.
- [Table 2] The ablation row '- layer, element' is equivalent to the LoRA(ν+µ) baseline in Table 1; state this explicitly so the ablation table is self-contained.
- [Section 5.1, Abstract] The model is described as 'LLaMA-3' in the abstract but as 'LLaMA 3.2-3B' in Section 5.2; use one consistent name and specification.
Circularity Check
General-ability result is measured on the same Pile corpus used to fit the importance scores, making the forgetting claim partly by construction.
-
fitted input called prediction
[Section 5.2 (nu Task) and Table 1; Eq. (14)]
"nu Task (General Ability): The nu task focuses on learning which parameters are important for general tasks. Following previous work (Xiang et al., 2024), we take Pile (Gao et al., 2020) as the evaluation datasets for LLM general ability. LoRA is applied to fine-tune the model on the nu task, and parameter importance for Synaptic Intelligence (SI) is recorded during this stage."
The element-wise importance Omega_i^nu in Eq. (13) is accumulated from gradients of the nu-task loss (Eqs. 6 and 12), and the nu task is LoRA fine-tuning on Pile. The reported 'general ability' is PPL on Pile (Table 1 caption), which is exactly the language-modeling loss minimized in the nu stage. Eq. (14) then penalizes deviations from the Pile-fine-tuned checkpoint theta_nu, weighted by Omega_i^nu. Thus a low PPL on Pile after mu-tuning is not an independent test of whether pretrained general knowledge was retained; it is the direct objective of the regularization. The comparison against Base is also confounded because Ours has seen Pile in the nu stage while Base has not, and no held-out benchmark such as MMLU or HellaSwag is reported.
full rationale
The derivation of the importance estimator itself is not circular: Eqs. (4)-(6) reproduce Synaptic Intelligence, and the LoRA-specific gradient algebra in Eqs. (8)-(12) is a reasonable adaptation. The circularity lies in the evaluation of the central claim. 'General ability' is operationalized solely as perplexity on Pile (Table 1), while the nu stage that produces the importance scores Omega is LoRA fine-tuning on Pile. Eq. (14) then regularizes mu-tuning toward theta_nu with weights derived from Pile-loss gradients. Therefore Ours's low PPL on Pile is substantially by construction: it is the same loss that the method is explicitly designed to keep low. The comparison against Base is not controlled for extra Pile exposure; LoRA(nu+mu) is the correct control and shows smaller gaps, but even that control measures only preservation of the Pile-optimized checkpoint, not of pretrained general knowledge. No external benchmarks (MMLU, HellaSwag, etc.) are reported. Domain-task accuracy and the 20x speed / 10-15% storage efficiency comparisons are independent and are not affected by this circularity, which is why the score is 6 (partial) rather than higher. There is no load-bearing self-citation chain.
Assumptions & free parameters
free parameters (7)
- Regularization coefficient φ =
1e-3
- LoRA rank r =
8
- LoRA alpha =
32
- Learning rate =
8e-4
- Training epochs for ν and µ tasks =
5
- Batch size =
20
- Stability constant ξ =
unspecified
assumptions (6)
- domain assumption SI path integral w_i = -∫ g_i θ'_i dt on the ν task measures parameter importance for general knowledge.
- domain assumption The effective LoRA gradient \tilde g(t) = g_B A + B g_A - η g_B g_A fully characterizes parameter updates, so the importance computation in Eq. 12 is valid.
- domain assumption Weighted L2 regularization around θ_ν is sufficient to preserve task ν performance, implying a locally quadratic loss landscape.
- domain assumption Perplexity on Pile is an adequate measure of general ability and catastrophic forgetting.
- ad hoc to paper Layer-wise L2 norm of importance followed by softmax weighting in Eq. 15 correctly balances layers.
- standard math Path integral identity L(t1)-L(t0)=∫ g(θ)θ'(t)dt and its decomposition over parameters is valid.
Cite this review
Pith. "Pith review of How to Alleviate Catastrophic Forgetting in LLMs Finetuning? Hierarchical Layer-Wise and Element-Wise Regularization." pith.science (2026). https://pith.science/paper/AUHZIGKL
@misc{pith2026250113669,
author = {Pith},
title = {Pith review of: How to Alleviate Catastrophic Forgetting in LLMs Finetuning? Hierarchical Layer-Wise and Element-Wise Regularization},
year = {2026},
howpublished = {\url{https://pith.science/paper/AUHZIGKL}},
note = {Machine review of arXiv:2501.13669}
}
read the original abstract
Large Language Models (LLMs) exhibit strong general language capabilities. However, fine-tuning these models on domain-specific tasks often leads to catastrophic forgetting, where the model overwrites or loses essential knowledge acquired during pretraining. This phenomenon significantly limits the broader applicability of LLMs. To address this challenge, we propose a novel approach to compute the element-wise importance of model parameters crucial for preserving general knowledge during fine-tuning. Our method utilizes a dual-objective optimization strategy: (1) regularization loss based on element-wise parameter importance, which constrains the updates to parameters crucial for general knowledge; (2) cross-entropy loss to adapt to domain-specific tasks. Additionally, we introduce layer-wise coefficients to account for the varying contributions of different layers, dynamically balancing the dual-objective optimization. Extensive experiments on scientific, medical, and physical tasks using GPT-J and LLaMA-3 demonstrate that our approach mitigates catastrophic forgetting while enhancing model adaptability. Compared to previous methods, our solution is approximately 20 times faster and requires only 10-15% of the storage, highlighting the practical efficiency. The code will be released.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
Time Series Foundation Models for Multivariate Financial Time Series Forecasting
Pretrained TTM shows large transfer and sample-efficiency gains in three financial forecasting tasks relative to training from scratch, but methodological flaws including possible look-ahead bias weaken the quantitati...
-
Domain-Aware RAG: MoL-Enhanced RL for Efficient Training and Scalable Retrieval
A two-stage RAG training pipeline, MoL continual pre-training plus GRPO reinforcement learning with single-passage training and multi-passage inference, reports state-of-the-art retrieval recall on NFCORPUS and SCIFAC...
Reference graph
Works this paper leans on
-
[4]
J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W
Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,
-
[5]
A rank stabilization scaling factor for fine-tuning with lora
Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732,
-
[8]
Qin, C. and Joty, S. Lfpt5: A unified framework for lifelong few-shot language learning based on prompt tuning of t5. ICLR 2022 - 10th International Conference on Learning Representations,
work page 2022
-
[9]
Analyzing and reducing catastrophic forgetting in parameter efficient tuning
Ren, W., Li, X., Wang, L., Zhao, T., and Qin, W. Analyzing and reducing catastrophic forgetting in parameter efficient tuning. arXiv preprint arXiv:2402.18865,
-
[10]
9 Shao, C. and Feng, Y . Overcoming catastrophic forgetting beyond continual learning: Balanced training for neural machine translation. Proceedings of the Annual Meeting of the Association for Computational Linguistics, 1:2023 – 2036,
work page 2023
-
[11]
Llama: Open and efficient foundation lan- guage models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023a. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, ...
-
[12]
Learning to prompt for continual learning
Wang, Z., Zhang, Z., Lee, C.-Y ., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V ., Dy, J., and Pfister, T. Learning to prompt for continual learning. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022-June:139 – 149,
work page 2022
-
[13]
Lora- pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.18242,
Wang, Z., Liang, J., He, R., Wang, Z., and Tan, T. Lora- pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.18242,
Show all 14 references
-
[14]
F., and Gardner, M
Welbl, J., Liu, N. F., and Gardner, M. Crowdsourc- ing multiple choice science questions. arXiv preprint arXiv:1707.06209,
-
[2020]
Continual learning of neural machine translation within low forgetting risk regions
Gu, S., Hu, B., and Feng, Y . Continual learning of neural machine translation within low forgetting risk regions. Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022, pp. 1707 – 1718,
2022
-
[2021]
Full parameter fine-tuning for large language models with lim- ited resources
Lv, K., Yang, Y ., Liu, T., Gao, Q., Guo, Q., and Qiu, X. Full parameter fine-tuning for large language models with lim- ited resources. arXiv preprint arXiv:2306.09782,
-
[2022]
The llama 3 herd of models
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
- [2023]
-
[2024]
The pile: An 800gb dataset of diverse text for language modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027,
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.