Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Catastrophic forgetting can be reduced to 0.25% by pruning task-vector parameters whose change is large relative to pre-trained weights.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 20:55 UTC pith:YPZ6ZMJH

load-bearing objection Useful empirical trick, but the paper's central mechanistic claim is undone by an algebraic identity the authors missed: Eq. (1) reduces to magnitude pruning reweighted by pretrained magnitudes, so the 'relative change' story is confounded and needs a control experiment. the 4 major comments →

arxiv 2509.08255 v1 pith:YPZ6ZMJH submitted 2025-09-10 cs.LG

Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning

Wei Huang , Anda Cheng , Yinggui Wang This is my paper
classification cs.LG
keywords catastrophic forgettinglarge language modelstask vectorweight pruningfine-tuningrelative change magnitudeforgetting-aware pruning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that catastrophic forgetting during full fine-tuning of large language models is governed by the relative size of the task-vector change compared with the pre-trained weight, not just the absolute change. On that basis it proposes a pruning metric, FAPM, that scores each parameter by its absolute change and penalizes parameters whose change is large relative to their pre-trained value. Pruning the fine-tuned model's task vector with this score, the authors report, reduces forgetting on general benchmarks to 0.25% while keeping 99.67% of downstream accuracy, without retraining or extra data. The significance is a post-hoc, no-training way to restore generality after fine-tuning.

Core claim

The central claim is that the ratio ΔW/W_pre — the relative change magnitude of each parameter from pre-trained to fine-tuned weights — is the quantity that governs catastrophic forgetting. Since the fine-tuned weights factor as W_pre(1 + ΔW/W_pre), and W_pre itself cannot cause forgetting, the paper argues the relative-change term is what disrupts pre-trained knowledge. FAPM uses a hybrid score S = |ΔW| − |W_pre|_avg * |ΔW| / |W_pre| to prune the task vector: it keeps parameters with large absolute change (needed for the downstream task) while removing those whose relative deviation is largest (blamed for forgetting). The paper reports this yields a better balance than magnitude pruning alo

What carries the argument

The central object is the Forgetting-Aware Pruning Metric (FAPM), an unstructured pruning score applied to the task vector ΔW = W_ft − W_pre. It subtracts a scaled relative-change term from the absolute-change term so that a parameter with both a large absolute update and a large update relative to its pre-trained value is penalized and pruned. The scaling by the average pre-trained magnitude |W_pre|_avg brings the two criteria onto a comparable numerical range.

Load-bearing premise

The claim collapses if catastrophic forgetting is not primarily caused by parameters with large relative change |ΔW/W_pre|, so pruning those parameters would not reliably restore generality at a given sparsity.

What would settle it

On a benchmark where the coordinates with largest |ΔW/W_pre| are few, compare FAPM against pruning the same number of random coordinates. The paper's mechanism predicts FAPM's mask specifically recovers general-task accuracy while a random mask does not; if random masks match its general-task recovery at equal downstream accuracy, the relative-change driver is unsupported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • FAPM can be applied after any fine-tuning run as a post-processing step, requiring only the pre-trained and fine-tuned checkpoints and no training data.
  • At 90% sparsity it preserves near-full downstream accuracy while restoring general capability to nearly pre-trained levels, so models can be specialized without sacrificing generality.
  • The same relative-change score extends to LoRA fine-tuning by treating the merged low-rank update as the task vector, reducing forgetting there too.
  • FAPM also mitigates forgetting in sequential fine-tuning, where a model trained on one task is then trained on a second, suggesting it can be applied repeatedly.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • I infer that the relative-change criterion could serve as a cheap diagnostic for which parameters a fine-tuning run has 'hijacked', potentially guiding data selection or hyperparameter choices rather than only pruning.
  • I infer that FAPM's gains should shrink on tasks where large absolute changes and large relative changes occur in the same coordinates, because there is little to prune that separates the two objectives.
  • I infer that the metric could be combined with training-based methods for mitigating catastrophic forgetting, a direction the paper explicitly leaves open, to push forgetting below the already-low reported level.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FAPM, a post-hoc pruning method that sparsifies the task vector ΔW = W_ft − W_pre after full fine-tuning in order to reduce catastrophic forgetting (CF) while preserving downstream task accuracy. The pruning score in Eq. (1) subtracts a pretrained-magnitude-scaled relative-change term from the absolute task-vector magnitude. Experiments on Llama3-8B and Qwen2-7B across eight downstream datasets and four general-capability benchmarks compare FAPM with L1 regularization, WiSE-FT, V-SoftMask, CoFiTune, and LoRA, and report that FAPM achieves a favorable trade-off. The paper also applies an adapted version to LoRA fine-tuning and to sequential fine-tuning. The central empirical claim is that FAPM limits CF to 0.25% while maintaining 99.67% accuracy on downstream tasks, and that the relative-change magnitude |ΔW|/|W_pre| is a key driver of CF.

Significance. If the empirical results are sound, FAPM is a simple, practical, post-hoc method for mitigating CF: it requires no changes to training, no auxiliary data, no architectural modification, and only uses the pre-trained and fine-tuned weights. The paper includes code, experiments on two model families and eight datasets, and comparisons with several baselines. These are genuine strengths. However, the central theoretical claim is currently not supported: Eq. (1) algebraically reduces to magnitude pruning reweighted by a monotone function of |W_pre|, so the proposed 'relative-change' mechanism is confounded with pretrained-magnitude gating. In addition, the headline numbers in the abstract do not match the numbers reported in Section 5.1. The paper may still describe a useful empirical method, but the explanatory contribution needs substantial rework and additional controlled experiments.

major comments (4)
  1. [Abstract and §5.1] The abstract claims 'FAPM limits CF to just 0.25% while maintaining 99.67% accuracy on downstream tasks.' In §5.1, the 0.25% is the decrease in downstream-task performance relative to Full SFT (0.816 → 0.814), while CF—the loss on general tasks—is reported as 0.33% (0.6132 → 0.6112). The figure 99.67% is the retained fraction of pre-trained general-task performance, not downstream accuracy. The headline therefore misstates both the CF rate and the downstream accuracy and must be corrected.
  2. [Eq. (1), §3] Let c_i = |W_i_pre|_avg, which is a per-layer scalar. Then Eq. (1) becomes S_i = |ΔW_i|(1 − c_i / |W_i_pre|). The 'relative change' term is therefore a monotone function of |W_i_pre| alone; it does not create a genuine joint absolute+relative criterion. FAPM is magnitude pruning reweighted by pretrained weight magnitude. Consequently, the asserted mechanism—that large relative change identifies CF-prone parameters—is confounded with the pretrained-magnitude distribution. The paper needs a control experiment, e.g., comparing FAPM with magnitude pruning at matched sparsity and with a criterion |ΔW|·h(|W_pre|) for a different monotone h, or stratifying by |W_pre| to test whether relative change adds predictive power beyond |W_pre|.
  3. [§5.3, Table 3] The experiment comparing retention by |ΔW|/|W_pre| with retention by |ΔW| does not isolate relative change. Parameters with large |ΔW|/|W_pre| are disproportionately those with small |W_pre|; without controlling for |W_pre|, the observed difference from magnitude pruning may be driven by the denominator alone. The statement that the results 'reveal a strong relationship between |ΔW|/|W_pre| and CF' is not supported by this comparison. A proper control would bin parameters by |W_pre| and vary |ΔW| within bins, or equivalently compare against a criterion that uses |W_pre| independently of |ΔW|.
  4. [§2.2] The 'Theoretical analysis' is an algebraic identity plus a conjecture. The factorization W_ft = W_pre(1 + ΔW/W_pre) is trivially true, and the claim that 'CF may be more closely related to the last term ΔW/W_pre' is not derived; it is an assumption. The heading overstates the content. If the authors wish to make a theoretical claim, a formal statement with proof, or at least a clearly labeled hypothesis with dedicated tests, is required. Otherwise the narrative should be reframed as an empirical heuristic.
minor comments (5)
  1. [Abstract / §5.1] The abstract's '99.67% accuracy on downstream tasks' should be reworded; as written it is contradicted by the body's downstream-task numbers. Please report the correct quantity (retained general-task performance) and align the CF percentage with the body.
  2. [Tables 1 and 6] The table headers are hard to parse: 'Avg. Results' appears twice, and the Pre-trained rows appear to contain numbers that are not downstream-task accuracies. Please clarify the column layout and define what the last numeric column in each block represents.
  3. [General] Only point estimates are reported; no standard deviations, confidence intervals, or repeated-seed results are given. Since many differences are on the order of 0.001–0.005, the significance of the comparisons is unclear.
  4. [§3, Eq. (1)] The notation i is used both for a layer and for an element within a layer. Please clarify that |W_i_pre|_avg is a per-layer scalar and that the score is computed elementwise.
  5. [Appendix D] The global 90% sparsity ratio is applied across all datasets and models. The ablation shows trends but does not provide a principled selection rule or sensitivity analysis. Since the headline numbers depend on this choice, report how 90% was selected and whether the conclusions are robust to it.

Circularity Check

0 steps flagged

No significant circularity: FAPM is a hand-designed pruning score evaluated on held-out benchmarks; no fitted parameter or self-citation chain carries the central claim.

full rationale

The derivation chain is self-contained. Equation (1) defines S_i = |ΔW_i| − |W^i_pre|_avg · |ΔW_i|/|W^i_pre| using only the fine-tuned and pre-trained weights. The evaluation targets (MMLU, C-Eval, GSM8K, HumanEval, and downstream accuracy) are external held-out benchmarks; they are not used to construct or fit the pruning score. The only tunable quantity is the global sparsity ratio, which is a standard hyperparameter and is not renamed as a prediction. The algebraic identity S_i = |ΔW_i|(1 − c_i/|W^i_pre|) shows that the relative-change term acts as a per-layer rescaling by pretrained magnitude; this is a potential confound for the paper's mechanistic claim that relative change is the driver of forgetting, but it is a correctness/identification concern, not circularity. The Limitations section (no integration with training-based CF methods) is a scope limitation and does not reveal a circular step. No load-bearing self-citations were found. The claim that FAPM reduces forgetting is an empirical, falsifiable result, not a tautology.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical or architectural entities. It relies on the task-vector definition, a heuristic assumption about relative change magnitude, and the empirically motivated choice to prune ΔW and restore W_pre. The single significant free parameter is the sparsity ratio, chosen by test-set inspection.

free parameters (2)
  • sparsity ratio (full fine-tuning) = 90%
    Set uniformly across models and datasets after inspecting ablation curves in Appendix D; the paper states this 'may represent an optimal balance'.
  • LoRA pruning rate = 3-10% (per dataset)
    Table 2 reports results at several pruning rates (0%, 3%, 10%, 90%); the text notes the rate must be small, implying per-dataset selection.
axioms (4)
  • standard math Task vector ΔW = W_ft - W_pre captures the fine-tuning change (Ilharco et al., 2022).
    Definition borrowed from prior work, used as the object to prune.
  • domain assumption CF is related to the relative change magnitude |ΔW/W_pre| rather than only |ΔW|.
    Section 2.2 proposes this based on the algebraic identity and intuitions; no proof is given.
  • domain assumption Pruning task vector entries and restoring W_pre recovers general knowledge while retaining task accuracy.
    Section 2.1 and Figure 2; assumes redundancy in ΔW and that restored W_pre values are beneficial.
  • domain assumption Magnitude pruning of ΔW preserves downstream task accuracy up to high sparsity.
    Empirically observed in Figure 2 and used as the base of FAPM.

pith-pipeline@v1.3.0-alltime-deepseek · 19183 in / 14247 out tokens · 131423 ms · 2026-08-04T20:55:22.458285+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning." pith.science (2026). https://pith.science/paper/YPZ6ZMJH

@misc{pith2026250908255,
  author       = {Pith},
  title        = {Pith review of: Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YPZ6ZMJH}},
  note         = {Machine review of arXiv:2509.08255}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advancements in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning. In this paper, we propose the Forgetting-Aware Pruning Metric (FAPM), a novel pruning-based approach to balance CF and downstream task performance. Our investigation reveals that the degree to which task vectors (i.e., the subtraction of pre-trained weights from the weights fine-tuned on downstream tasks) overlap with pre-trained model parameters is a critical factor for CF. Based on this finding, FAPM employs the ratio of the task vector to pre-trained model parameters as a metric to quantify CF, integrating this measure into the pruning criteria. Importantly, FAPM does not necessitate modifications to the training process or model architecture, nor does it require any auxiliary data. We conducted extensive experiments across eight datasets, covering natural language inference, General Q&A, Medical Q&A, Math Q&A, reading comprehension, and cloze tests. The results demonstrate that FAPM limits CF to just 0.25\% while maintaining 99.67\% accuracy on downstream tasks. We provide the code to reproduce our results.

Figures

Figures reproduced from arXiv: 2509.08255 by Anda Cheng, Wei Huang, Yinggui Wang.

Figure 1
Figure 1. Figure 1: Illustration of the issues in CF, the desired [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The relationship between the magnitude pruning sparsity ratio, general capability, and downstream task [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of the weight matrices in different layers of Llama3-8B fine-tuned on RTE dataset. From left [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of pruning with FAPM. If |∆Wi | is large (it will be retained by magnitude pruning) and |∆Wi | |Wi pre| is also large, FAPM will penalize and possibly prune this parameter, e.g., the value 1.1 in the middle of ∆W. By doing so, most large-magnitude parameters in ∆W are retained, while only a small subset are replaced by parameters with smaller |∆Wi | |Wi pre| values. approximately 1 × 10−2 , wh… view at source ↗
Figure 5
Figure 5. Figure 5: Performance of FAPM on downstream task accuracy and mitigation of catastrophic forgetting with different sparsity ratios on Llama3-8B. 50 55 60 65 70 75 80 85 90 95 Sparsity Ratio (%) 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 Performance Downstream Task Performance MRPC RTE WikiQA Winogrande QASC SQuAD 50 55 60 65 70 75 80 85 90 95 Sparsity Ratio (%) 0.600 0.625 0.650 0.675 0.700 0.725 0.750 Performance Down… view at source ↗
Figure 6
Figure 6. Figure 6: Performance of FAPM on downstream task accuracy and mitigation of catastrophic forgetting with different sparsity ratios on Qwen2-7B. gradually increases. These observations suggest that a 90% sparsity ratio may represent an opti- [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of the weight matrices in different layers of Qwen2-7B fine-tuned on RTE dataset. From left [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The relationship between the magnitude prun [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model

    cs.LG 2026-06 unverdicted novelty 6.0

    Boundary-aware Curriculum RL raises average pass@256 by 9.8 points over base models and 10.3 points over vanilla RLVR on Qwen, Llama, and DeepSeek families.

  2. Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining

    cs.LG 2026-05 unverdicted novelty 6.0

    DG-Hard uses Donoho-Gavish hard thresholding on the fine-tuning weight delta to separate task-aligned signal from noise-like residual, recovering damaged capabilities while preserving target-task gains.

Reference graph

Works this paper leans on

41 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712

    Pith/arXiv arXiv 2023

  2. [2]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    Pith/arXiv arXiv 2021

  3. [3]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    Pith/arXiv arXiv 2021

  4. [4]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  5. [5]

    Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28

    work page 2015

  6. [6]

    Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608

    Pith/arXiv arXiv 2024

  7. [7]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  8. [8]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    Pith/arXiv arXiv 2021

  9. [9]

    Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. 2024. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXiv:2403.01244

    Pith/arXiv arXiv 2024

  10. [10]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322

    Pith/arXiv arXiv 2023

  11. [11]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations

    work page 2022

  12. [12]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  13. [13]

    Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual pre-training of language models. In The Eleventh International Conference on Learning Representations

    work page 2023

  14. [14]

    Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. arXiv:1910.11473v2

    Pith/arXiv arXiv 2020

  15. [15]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521--3526

    2017

  16. [16]

    Yajing Kong, Liu Liu, Huanhuan Chen, Janusz Kacprzyk, and Dacheng Tao. 2023. Overcoming catastrophic forgetting in continual learning by exploring eigenvalues of hessian matrix. IEEE Transactions on Neural Networks and Learning Systems

    work page 2023

  17. [17]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611--626

    work page 2023

  18. [18]

    Yann LeCun, John Denker, and Sara Solla. 1989. https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf Optimal brain damage . In Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann

    work page 1989

  19. [19]

    Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. 2023. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.06256

    Pith/arXiv arXiv 2023

  20. [20]

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747

    Pith/arXiv arXiv 2023

  21. [21]

    Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. 2024. Lottery ticket adaptation: Mitigating destructive interference in llms. arXiv preprint arXiv:2406.16797

    Pith/arXiv arXiv 2024

  22. [22]

    Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. 2023. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pages 27011--27033. PMLR

    work page 2023

  23. [23]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  24. [24]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--2392, Austin, Texas. Association for Computational Linguistics

    doi:10.18653/v1/d16-1264 2016

  25. [25]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  26. [26]

    Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107--6122

    work page 2022

  27. [27]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695

    Pith/arXiv arXiv 2023

  28. [28]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR

    work page 2019

  29. [29]

    Zhicheng Wang, Yufang Liu, Tao Ji, Xiaoling Wang, Yuanbin Wu, Congcong Jiang, Ye Chao, Zhencong Han, Ling Wang, Xu Shao, et al. 2023. Rehearsal-free continual language learning via efficient parameter isolation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10933--10946

    work page 2023

  30. [30]

    Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. 2021. https://api.semanticscholar.org/CorpusID:237420687 Robust fine-tuning of zero-shot models . 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7949--7961

    work page 2021

  31. [31]

    Mitigating catastrophic forgetting in online continual learning by modeling previous task interrelations via pareto optimization

    Yichen Wu, Hong Wang, Peilin Zhao, Yefeng Zheng, Ying Wei, and Long-Kai Huang. Mitigating catastrophic forgetting in online continual learning by modeling previous task interrelations via pareto optimization. In Forty-first International Conference on Machine Learning

    work page

  32. [32]

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024. Reft: Representation finetuning for language models. arXiv preprint arXiv:2404.03592

    Pith/arXiv arXiv 2024

  33. [33]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36

    work page 2024

  34. [34]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024 a . Qwen2 technical report. arXiv preprint arXiv:2407.10671

    Pith/arXiv arXiv 2024

  35. [35]

    Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. 2024 b . Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19368--19376

    work page 2024

  36. [36]

    Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. https://doi.org/10.18653/v1/D15-1237 W iki QA : A challenge dataset for open-domain question answering . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013--2018, Lisbon, Portugal. Association for Computational Linguistics

    doi:10.18653/v1/d15-1237 2015

  37. [37]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

    Pith/arXiv arXiv 2023

  38. [38]

    Hengyuan Zhang, Yanru Wu, Dawei Li, Zacc Yang, Rui Zhao, Yong Jiang, and Fei Tan. 2024. Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model. arXiv preprint arXiv:2404.10306

    Pith/arXiv arXiv 2024

  39. [39]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372

    Pith/arXiv arXiv 2024

  40. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  41. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...