Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

SelfAug claims that a KL-divergence penalty on input-token logits added to the fine-tuning objective mitigates catastrophic forgetting in RAG settings while preserving downstream task gains.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A simple, plausible regularizer for RAG fine-tuning with genuinely useful ablations, but the central transfer claim—that aligning logits on RAG inputs preserves general abilities—is never directly measured. the 3 major comments →

arxiv 2509.03934 v1 pith:XA3422FO submitted 2025-09-04 cs.CL cs.AI

SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment

Yuqing Huang , Rongyang Zhang , Qimeng Wang , Chengqiang Lu , Yan Gao , Yi Wu , Yao Hu , Xuyang Zhi
show 4 more authors
Guiquan Liu Xin Li Hao Wang Enhong Chen
This is my paper
classification cs.CL cs.AI
keywords catastrophic forgettingretrieval-augmented generationlogits distribution alignmentKL divergenceknowledge distillationfine-tuninginstruction-followingdistribution shift
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that catastrophic forgetting during task-specific fine-tuning can be reduced by aligning the fine-tuned model's probability distribution over input tokens to the original model's distribution, using KL divergence as a penalty term added to the negative log-likelihood loss. The target setting is retrieval-augmented generation, where long retrieved contexts create large distribution shifts and severe forgetting of general abilities, especially instruction following. SelfAug requires no extra instruction data, no response regeneration, and no parameter orthogonality constraints; it uses only the model's own logits on the training inputs. The paper argues this works because final-layer logits carry the model's semantic distribution, and constraining them keeps the adapted model close to its prior while still learning the downstream task. If correct, any fine-tuning pipeline can gain a forgetting-mitigation term at the cost of one extra forward pass.

Core claim

On the paper's own terms, the central claim is that catastrophic forgetting in task-specific fine-tuning is a distribution-shift problem, and that the shift can be measured and controlled through the logits the model produces over input tokens. In RAG fine-tuning, the inputs are prompts plus retrieved documents; the paper shows that this distribution is far from the general instruction distribution the model was aligned on, so the longer the context, the larger the shift, and the worse the forgetting of instruction-following ability. SelfAug adds to the standard negative log-likelihood loss a KL-divergence term L_KL = D_KL(p_ft(x_t) || p_o(x_t)) between the softmax distributions of the fine-

What carries the argument

The load-bearing object is the input-sequence logit distribution: the probability vector the language model assigns to each token of the prompt-plus-retrieved-document sequence before any response is generated. SelfAug uses the KL divergence between these distributions for the fine-tuned model and the original model, D_KL(p_ft(x_t) || p_o(x_t)), as a regularizer weighted by alpha. This term forces the adapting model to keep its per-token predictive distribution close to the original model's on exactly the inputs it is learning from, while NLL on response tokens drives downstream task learning. The paper's stated rationale is an information-bottleneck argument: the final logits retain the ess

Load-bearing premise

The KL penalty is computed only on the fine-tuning batch's input sequences; if those RAG prompts and retrieved documents are not representative of the model's broad input distribution, keeping the model close to itself on those inputs may not preserve general instruction-following ability.

What would settle it

Run SelfAug with a fixed alpha on a RAG dataset such as CRAG under the authors' settings, then measure KL divergence between the original and fine-tuned model on a held-out set of general instruction prompts with no retrieved documents (for example, IFEval prompts). If the held-out KL grows as fast as with vanilla LoRA while in-batch KL stays low, or if IFEval accuracy still collapses, the distribution-alignment assumption is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any RAG fine-tuning run can add SelfAug without extra data or validation; the only overhead is one additional forward pass through the original model.
  • Because the penalty is on input logits only, downstream learning on response tokens is not directly constrained, so task performance can equal or exceed vanilla LoRA while instruction-following ability is retained.
  • The paper's shift-forgetting correlation implies that monitoring KL divergence of logits during training can serve as an early warning for looming catastrophic forgetting.
  • Longer retrieved contexts (2K to 8K tokens) cause larger distribution shifts, and SelfAug consistently recovers instruction-following accuracy at every tested length.
  • The method transfers across base models, model sizes (3B to 72B), LoRA ranks, and low-shift tasks like math and code fine-tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the KL term is only computed on the task's own input distribution, SelfAug's protection of general abilities is only as strong as the overlap between RAG prompts and ordinary instructions; a held-out set of general instructions could directly test whether the constraint transfers.
  • The same input-logit alignment could be applied to other sequence-to-sequence adaptations such as preference tuning or continual learning, acting as a cheap prior that anchors the model to its own pretraining distribution.
  • For inputs beyond 32K tokens, whole-sequence logit alignment becomes expensive; attention-based selection of the most informative tokens is a natural next experiment to keep the method scalable.
  • The method's success suggests output-distribution regularization may be a cheaper substitute for data replay in some settings, though combining it with a small replay buffer could cover cases where task inputs are far from general instructions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SelfAug, a fine-tuning regularizer for RAG adaptation. During supervised fine-tuning, SelfAug adds to the standard NLL loss a KL-divergence term between the fine-tuned model's softmax logits and the original model's softmax logits, computed on the input sequences (prompts plus retrieved documents) of the downstream training data. The total loss is L_total = L_NLL + alpha * L_KL. The authors argue that this 'self-distribution alignment' preserves the original model's semantic distribution, mitigates catastrophic forgetting, and improves downstream performance. Experiments are conducted with Qwen2.5-7B-Instruct and Llama-3-8B-Instruct on CRAG and RAG-Instruct fine-tuning, with evaluation on RAG benchmarks (CRAG, ChatRAGBench, BioASQ, OmniEval) and general capabilities (IFEval, MMLU, ARC-C, HellaSwag, MATH, HumanEval). The paper also presents epoch-wise KL/IFEval/CRAG curves, a hyperparameter sensitivity analysis, context-length and model-scale studies, and an ablation over the location of the distribution constraint.

Significance. If the central claim holds, SelfAug is an attractive method: it requires no extra data, no architecture changes, and only one additional forward pass per training batch, while reducing forgetting of instruction-following abilities. The manuscript has concrete strengths: the method is simple and reproducible, code is released, experiments cover two base models and multiple RAG datasets, and the authors provide an ablation over constraint location and weight. The empirical support, however, is weakened by the evaluation protocol (best-of-five selection and grid search on the evaluation benchmarks), by the absence of error bars or statistical tests, and by the fact that the proposed distribution-preservation mechanism is only measured on the fine-tuning input distribution, not on held-out general instructions. These issues are central to the paper's causal claim that distribution shift drives catastrophic forgetting and that SelfAug alleviates it.

major comments (3)
  1. [§4.1, Table 1] The reported results are based on 'five repeated experiments to obtain the best value' and 'hyperparameters ... determined through a hyperparameter grid search.' Since alpha is selected using the evaluation benchmarks (including IFEval), and only the best run is reported, the large IFEval improvements attributed to SelfAug (e.g., 48.80 to 62.11 in the CRAG block) may reflect selection noise rather than a genuine distribution-preserving effect. No variance or statistical significance is reported. This is load-bearing for the claim that SelfAug 'mitigates catastrophic forgetting.' Please report mean and standard deviation over the five runs, or a fixed validation split, and use a proper protocol for hyperparameter selection.
  2. [§3.3, §4.3, Figure 2] The KL regularizer L_KL = D_KL(p_ft(x_t) || p_o(x_t)) is computed only on the downstream RAG inputs x_t. Figure 2 measures KL divergence on the training set, not on held-out general instruction inputs. Thus the mechanism claim—that aligning logits on RAG inputs preserves the model's behavior on general instructions—is not directly tested. The observed decrease in KL under SelfAug is partly by construction, since L_KL is explicitly minimized. To support the transfer assumption, the authors should measure KL divergence (or a related distributional distance) on a held-out general instruction set (e.g., IFEval prompts or an instruction-following corpus) before and after fine-tuning.
  3. [§4.3.1, Figure 2] The paper claims a 'direct link' or 'strong correlation' between distribution shift and catastrophic forgetting severity, but this is supported only by a visual comparison of three curves over 10 epochs on a single training run. There is no quantitative correlation coefficient, no multiple seeds, and no control for confounds such as training loss or learning rate. Given that the KL term is part of the objective for SelfAug, the comparison between LoRA and SelfAug in Figure 2 is also not apples-to-apples. I recommend a more rigorous analysis, e.g., reporting correlation across checkpoints or runs, and measuring distribution shift on held-out instructions.
minor comments (5)
  1. [Table 1] The table formatting is confusing: many cells contain concatenated values and arrows (e.g., '4.344.42↓', '68.020.80↑') that are not defined in the caption. Please clarify the notation and ensure the baseline and delta conventions are immediately understandable.
  2. [§3.3, Eq. (6)] The derivation of the Bayesian prior p(θ) = exp(−α·Dist(f_θ, f_θ0)) is informal: this expression is not a properly normalized prior over parameters, and the step from the posterior objective to L_NLL + α·Dist is presented as an equality rather than as a regularization heuristic. This is not fatal, but the text should be clearer that this is motivation, not a formal Bayesian derivation.
  3. [§4.1, Implementation Details] The hyperparameter grid range for alpha is not reported; only '0.5' is mentioned as the setting used in the main experiments, and the ablation in Figure 3 shows alpha in [0.3, 0.5] as reasonable. Please specify the full grid and the selection criteria for each benchmark.
  4. [§B, Computational Cost] The cost analysis is qualitative. Since SelfAug requires an additional forward pass through the reference model, it would be useful to report wall-clock time or FLOPs relative to LoRA and the other baselines.
  5. [Table 4] The CRAG base scores for larger models are very low (e.g., -46.82 for 3B, -20.30 for 72B), which the paper attributes to hallucination cases. This is plausible, but the table would benefit from a breakdown of CRAG sub-scores (correct, incorrect, missing) to make the interpretation transparent.

Circularity Check

1 steps flagged

Mostly self-contained; the only tautological element is the claim that SelfAug reduces the KL term it explicitly minimizes, while the central forgetting-mitigation result rests on independent held-out metrics.

specific steps
  1. self definitional [Section 4.3.2 / Figure 2(b); cf. Section 3.3 objective L_total = L_NLL + alpha * L_KL]
    "after applying the SelfAug constraint, the KL divergence of model logits significantly decreases and remains at a stable level."

    The paper's training objective directly includes this KL term: L_total = L_NLL + alpha * L_KL, with L_KL = D_KL(p_ft(x_t) || p_o(x_t)) over the fine-tuning inputs (Section 3.3). Therefore observing that the KL divergence decreases after SelfAug training is, at least in part, a statement about the objective being optimized rather than an independent empirical discovery. However, this by-construction observation is not the paper's main load-bearing claim: catastrophic forgetting is measured on IFEval and other held-out general benchmarks, which are not the KL-optimized quantity, so the forgetting-mitigation result retains independent content.

full rationale

SelfAug's derivations are largely self-contained. The KL term is defined as D_KL(p_ft(x_t) || p_o(x_t)) over the fine-tuning inputs and is added directly to the training loss (Section 3.3), so the observation in Section 4.3.2 that this KL decreases during SelfAug training is partly by construction. This is a mild self-definitional element, but it does not drive the paper's central claim: catastrophic forgetting is evaluated on IFEval (and other held-out general benchmarks), which are outside the optimized KL term, and the reported improvement there (e.g., IFEval 48.80 -> 62.11 in Table 1) is an independent empirical outcome. No parameter is fitted and then renamed a prediction; alpha is a grid-searched hyperparameter, and the method's mechanism does not rely on self-citations or an imported uniqueness theorem. The best-of-five reporting is a statistical-selection caveat, not a circularity. Overall, the core derivation (Bayesian prior plus KL regularization) is a standard modeling choice, and the forgetting-mitigation claim has independent content.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claim rests on one fitted hyperparameter (alpha) and four assumptions: the Bayesian prior framing, the validity of input-token KL as a shift measure, the transferability of the constraint from task-specific to general inputs, and the reliability of the frozen reference model. No new physical or architectural entities are introduced.

free parameters (1)
  • KL loss weight alpha = 0.5 default; grid-searched
    Alpha balances task loss and distribution-preservation loss. The paper sets 0.5 as default after 'hyperparameter grid search' and reports best-of-five results, so the reported performance depends on this fitted weight.
axioms (4)
  • ad hoc to paper p(theta) = exp(-alpha * Dist(f_theta, f_theta0)) is a valid Bayesian prior over parameters
    Section 3.3 introduces this prior to justify the regularized objective. It is asserted, not derived, and the step from this prior to the loss is a framing rather than a mathematical consequence.
  • domain assumption KL divergence between model logit distributions on input tokens is a faithful measure of distribution shift relevant to forgetting
    Section 3.3 and Section 4.3 use D_KL(p_ft(x_t) || p_o(x_t)) as the operational measure of distribution shift. No theoretical or empirical justification is given that this specific distance captures the semantic capability loss.
  • domain assumption Task-specific input sequences are sufficient anchors for preserving general abilities
    The KL term is applied only to the fine-tuning batch inputs. The paper assumes that aligning logits on these task inputs transfers to general instruction inputs, but it never evaluates logit similarity on general data.
  • domain assumption The frozen original model's logits are a reliable reference for the desired distribution
    SelfAug treats the pre-fine-tuning model as the oracle for general capabilities. This is standard in distillation, but it is an assumption about the base model's distribution being the one worth preserving.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment." pith.science (2026). https://pith.science/paper/XA3422FO

@misc{pith2026250903934,
  author       = {Pith},
  title        = {Pith review of: SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XA3422FO}},
  note         = {Machine review of arXiv:2509.03934}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advancements in large language models (LLMs) have revolutionized natural language processing through their remarkable capabilities in understanding and executing diverse tasks. While supervised fine-tuning, particularly in Retrieval-Augmented Generation (RAG) scenarios, effectively enhances task-specific performance, it often leads to catastrophic forgetting, where models lose their previously acquired knowledge and general capabilities. Existing solutions either require access to general instruction data or face limitations in preserving the model's original distribution. To overcome these limitations, we propose SelfAug, a self-distribution alignment method that aligns input sequence logits to preserve the model's semantic distribution, thereby mitigating catastrophic forgetting and improving downstream performance. Extensive experiments demonstrate that SelfAug achieves a superior balance between downstream learning and general capability retention. Our comprehensive empirical analysis reveals a direct correlation between distribution shifts and the severity of catastrophic forgetting in RAG scenarios, highlighting how the absence of RAG capabilities in general instruction tuning leads to significant distribution shifts during fine-tuning. Our findings not only advance the understanding of catastrophic forgetting in RAG contexts but also provide a practical solution applicable across diverse fine-tuning scenarios. Our code is publicly available at https://github.com/USTC-StarTeam/SelfAug.

Figures

Figures reproduced from arXiv: 2509.03934 by Chengqiang Lu, Enhong Chen, Guiquan Liu, Hao Wang, Qimeng Wang, Rongyang Zhang, Xin Li, Xuyang Zhi, Yan Gao, Yao Hu, Yi Wu, Yuqing Huang.

Figure 1
Figure 1. Figure 1: An illustration of full fine-tuning, LoRA, and methods for catastrophic forgetting mitigation. (a) SFT: [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Epoch-wise Performance and Logits Diver [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Model Performance with Respect to Weight [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Model Performance with Respect to LoRA Rank. Increasing trainable parameters through LoRA rank amplifies catastrophic forgetting severity. tional expectations, our experiments demonstrate that the relationship between model size and CRAG performance is not monotonically positive for base models. This counter-intuitive phenomenon can be attributed primarily to the prevalence of hallucina￾tion cases in the C… view at source ↗
Figure 5
Figure 5. Figure 5: Evaluation Results of Math and Code Tasks. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. IE as Cache: Information Extraction Enhanced Agentic Reasoning

    cs.CL 2026-04 unverdicted novelty 7.0

    IE-as-Cache framework repurposes information extraction as a dynamic cognitive cache to improve agentic reasoning accuracy in LLMs on challenging benchmarks.

  2. Rethinking the Necessity of Adaptive Retrieval-Augmented Generation through the Lens of Adaptive Listwise Ranking

    cs.IR 2026-04 unverdicted novelty 5.0

    AdaRankLLM shows adaptive listwise reranking outperforms fixed-depth retrieval for most LLMs by acting as a noise filter for weak models and an efficiency optimizer for strong ones, with lower context use.

Reference graph

Works this paper leans on

105 extracted references · 25 canonical work pages · cited by 2 Pith papers · 9 internal anchors

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  4. [4]

    Divyanshu Aggarwal, Sankarshan Damle, Navin Goyal, Satya Lokam, and Sunayana Sitaram. 2024. Exploring continual fine-tuning for enhancing language ability in large language model. arXiv preprint arXiv:2410.16006

    Pith/arXiv arXiv 2024

  5. [5]

    Anton Alexandrov, Veselin Raychev, Mark Niklas M \"u ller, Ce Zhang, Martin Vechev, and Kristina Toutanova. 2024. Mitigating catastrophic forgetting in language transfer via model merging. arXiv preprint arXiv:2407.08699

    Pith/arXiv arXiv 2024

  6. [6]

    Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2020. Open-domain question answering goes conversational via question rewriting. arXiv preprint arXiv:2010.04898

    Pith/arXiv arXiv 2020

  7. [7]

    Andrew Bai, Chih-Kuan Yeh, Cho-Jui Hsieh, and Ankur Taly. 2024. Which pretrain samples to rehearse when finetuning pretrained models? arXiv preprint arXiv:2402.08096

    Pith/arXiv arXiv 2024

  8. [8]

    Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. 2022. Recent advances in retrieval-augmented text generation. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 3417--3419

    2022

  9. [9]

    Howard Chen, Jiayi Geng, Adithya Bhaskar, Dan Friedman, and Danqi Chen. 2024 a . Continual memorization of factoids in large language models. arXiv preprint arXiv:2411.07175

    Pith/arXiv arXiv 2024

  10. [10]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024 b . Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754--17762

    2024

  11. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    Pith/arXiv arXiv 2021

  12. [12]

    Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac: Question answering in context. arXiv preprint arXiv:1808.07036

    Pith/arXiv arXiv 2018

  13. [13]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  14. [14]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

    2024

  15. [15]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    Pith/arXiv arXiv 2018

  16. [16]

    OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass

    2023

  17. [17]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233

    Pith/arXiv arXiv 2023

  18. [18]

    Franke, Michael Hefenbrock, and Frank Hutter

    J \"o rg K.H. Franke, Michael Hefenbrock, and Frank Hutter. 2024. https://openreview.net/forum?id=XoWtroECJU Preserving principal subspaces to reduce catastrophic forgetting in fine-tuning . In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models

    2024

  19. [19]

    Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128--135

    1999

  20. [20]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    Pith/arXiv arXiv 2023

  21. [21]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, and 1 others. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    Pith/arXiv arXiv 2024

  22. [22]

    Hongchao Gu, Dexun Li, Kuicai Dong, Hao Zhang, Hang Lv, Hao Wang, Defu Lian, Yong Liu, and Enhong Chen. 2025. https://doi.org/10.18653/v1/2025.findings-acl.859 RAPID : Efficient retrieval-augmented long text generation with writing planning and information discovery . In Findings of the Association for Computational Linguistics: ACL 2025, pages 16742--167...

    doi:10.18653/v1/2025.findings-acl.859 2025

  23. [23]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929--3938. PMLR

    2020

  24. [24]

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608

    Pith/arXiv arXiv 2024

  25. [25]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    Pith/arXiv arXiv 2020

  26. [26]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    Pith/arXiv arXiv 2021

  27. [27]

    Geoffrey Hinton. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    Pith/arXiv arXiv 2015

  28. [28]

    Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33:9782--9793

    2020

  29. [29]

    Yen-Chang Hsu, James Smith, Yilin Shen, Zsolt Kira, and Hongxia Jin. 2022. A closer look at knowledge distillation with features, logits, and gradients. arXiv preprint arXiv:2203.10163

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  30. [30]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    Pith/arXiv arXiv 2021

  31. [31]

    Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. 2024 a . Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXiv:2403.01244

    Pith/arXiv arXiv 2024

  32. [32]

    Yuqing Huang, Rongyang Zhang, Xuesong He, Xuyang Zhi, Hao Wang, Xin Li, Feiyang Xu, Deguang Liu, Huadong Liang, Yi Li, and 1 others. 2024 b . Chemeval: a comprehensive multi-level chemical evaluation for large language models. arXiv preprint arXiv:2409.13989

    Pith/arXiv arXiv 2024

  33. [33]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    Pith/arXiv arXiv 2024

  34. [34]

    Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. 2024. Genegpt: Augmenting large language models with domain tools for improved access to biomedical information. Bioinformatics, 40(2):btae075

    2024

  35. [35]

    Demystifying language model forgetting with low-rank example associations

    Xisen Jin and Xiang Ren. Demystifying language model forgetting with low-rank example associations. In NeurIPS 2024 Workshop on Scalable Continual Learning for Lifelong Foundation Models

    2024

  36. [36]

    Xisen Jin and Xiang Ren. 2024 a . https://openreview.net/forum?id=bzNwexOPWm What will my model forget? forecasting forgotten examples in language model refinement . In Forty-first International Conference on Machine Learning

    2024

  37. [37]

    Xisen Jin and Xiang Ren. 2024 b . What will my model forget? forecasting forgotten examples in language model refinement. arXiv preprint arXiv:2402.01865

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  38. [38]

    Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. 2018. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32

    2018

  39. [39]

    Md Kowsher, Nusrat Jahan Prottasha, and Prakash Bhat. 2024. Propulsion: Steering llm with tiny fine-tuning. arXiv preprint arXiv:2409.10927

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [40]

    Minh Le, An Nguyen, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Van Ngo, and Nhat Ho. 2024. Mixture of experts meets prompt-based continual learning. arXiv preprint arXiv:2405.14124

    Pith/arXiv arXiv 2024

  41. [41]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459--9474

    2020

  42. [42]

    Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. 2024 a . Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint arXiv:2404.15159

    Pith/arXiv arXiv 2024

  43. [43]

    Tianhao Li, Shangjie Li, Binbin Xie, Deyi Xiong, and Baosong Yang. 2024 b . Moe-ct: a novel approach for large language models training with resistance to catastrophic forgetting. arXiv preprint arXiv:2407.00875

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  44. [44]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance, pages 374--382

    2023

  45. [45]

    Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pages 20852--20867. PMLR

    2023

  46. [46]

    Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, and 1 others. 2024. Mitigating the alignment tax of rlhf. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 580--606

    2024

  47. [47]

    Chengyuan Liu, Yangyang Kang, Shihang Wang, Lizhi Qing, Fubang Zhao, Changlong Sun, Kun Kuang, and Fei Wu. 2024 a . More than catastrophic forgetting: Integrating general capabilities for domain-specific llms. arXiv preprint arXiv:2405.17830

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [48]

    Wanlong Liu, Junying Chen, Ke Ji, Li Zhou, Wenyu Chen, and Benyou Wang. 2024 b . Rag-instruct: Boosting llms with diverse retrieval-augmented instructions. arXiv preprint arXiv:2501.00353

    Pith/arXiv arXiv 2024

  49. [49]

    Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. 2019. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2604--2613

    2019

  50. [50]

    Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. 2024 c . Chatqa: Surpassing gpt-4 on conversational qa and rag. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  51. [51]

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023 a . Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583

    Pith/arXiv arXiv 2023

  52. [52]

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023 b . An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747

    Pith/arXiv arXiv 2023

  53. [53]

    Hang Lv, Sheng Liang, Hao Wang, Hongchao Gu, Yaxiong Wu, Wei Guo, Defu Lian, Yong Liu, and Enhong Chen. 2025. Costeer: Collaborative decoding-time personalization via local delta steering. arXiv preprint arXiv:2507.04756

    arXiv 2025

  54. [54]

    Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Vardaan Pahuja, Tanya Berger-Wolf, Song Gao, Charles Stewart, Yu Su, and 1 others. 2024. Fine-tuning is fine, if calibrated. arXiv preprint arXiv:2409.16223

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  55. [55]

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and B Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. URL: https://github. com/huggingface/peft

    2022

  56. [56]

    Daniel Marczak, Bart omiej Twardowski, Tomasz Trzci \'n ski, and Sebastian Cygert. 2025. Magmax: Leveraging model merging for seamless continual learning. In European Conference on Computer Vision, pages 379--395. Springer

    2025

  57. [57]

    Anastasios Nentidis, Georgios Katsimpras, Anastasia Krithara, Salvador Lima-L \'o pez, Eul \`a lia Farr \'e -Maduell, Martin Krallinger, Natalia Loukachevitch, Vera Davydova, Elena Tutubalina, and Georgios Paliouras. 2024. Overview of bioasq 2024: the twelfth bioasq challenge on large-scale biomedical semantic indexing and question answering. In Internati...

    2024

  58. [58]

    Yao Ni, Shan Zhang, and Piotr Koniusz. 2024. Pace: marrying generalization in parameter-efficient fine-tuning with consistency regularization. arXiv preprint arXiv:2409.17137

    work page arXiv 2024

  59. [59]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730--27744

    2022

  60. [60]

    Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. 2024. Lottery ticket adaptation: Mitigating destructive interference in llms. arXiv preprint arXiv:2406.16797

    Pith/arXiv arXiv 2024

  61. [61]

    Learn more, but bother less: parameter efficient continual learning

    Fuli Qiao and Mehrdad Mahdavi. Learn more, but bother less: parameter efficient continual learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page

  62. [62]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    Pith/arXiv arXiv 2023

  63. [63]

    Gobinda Saha, Isha Garg, and Kaushik Roy. 2021. Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762

    Pith/arXiv arXiv 2021

  64. [64]

    Tingjia Shen, Hao Wang, Chuan Qin, Ruijun Sun, Yang Song, Defu Lian, Hengshu Zhu, and Enhong Chen. 2025. Genki: Enhancing open-domain question answering with knowledge integration and controllable generation in large language models. arXiv preprint arXiv:2505.19660

    work page arXiv 2025

  65. [65]

    Tingjia Shen, Hao Wang, Chuhan Wu, Jin Yao Chin, Wei Guo, Yong Liu, Huifeng Guo, Defu Lian, Ruiming Tang, and Enhong Chen. 2024 a . Optimizing sequential recommendation models with scaling laws and approximate entropy. arXiv preprint arXiv:2412.00430

    Pith/arXiv arXiv 2024

  66. [66]

    Tingjia Shen, Hao Wang, Jiaqing Zhang, Sirui Zhao, Liangyue Li, Zulong Chen, Defu Lian, and Enhong Chen. 2024 b . Exploring user retrieval integration towards large language models for cross-domain sequential recommendation. arXiv preprint arXiv:2406.03085

    Pith/arXiv arXiv 2024

  67. [67]

    Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. 2024. Continual learning of large language models: A comprehensive survey. arXiv preprint arXiv:2404.16789

    Pith/arXiv arXiv 2024

  68. [68]

    Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. 2021. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5311--5320

    work page 2021

  69. [69]

    Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. 2024. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15731--15740

    work page 2024

  70. [70]

    Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. 2024. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884

    Pith/arXiv arXiv 2024

  71. [71]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  72. [72]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  73. [73]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    Pith/arXiv arXiv 2023

  74. [74]

    Sheng Wang, Liheng Chen, Jiyue Jiang, Boyang Xue, Lingpeng Kong, and Chuan Wu. 2024 a . Lora meets dropout under a unified framework. arXiv preprint arXiv:2403.00812

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  75. [75]

    Shuting Wang, Jiejun Tan, Zhicheng Dou, and Ji-Rong Wen. 2024 b . Omnieval: An omnidirectional and automatic rag evaluation benchmark in financial domain. arXiv preprint arXiv:2412.13018

    Pith/arXiv arXiv 2024

  76. [76]

    Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. 2019. Distilling object detectors with fine-grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4933--4942

    work page 2019

  77. [77]

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2023. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.14152

    Pith/arXiv arXiv 2023

  78. [78]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560

    Pith/arXiv arXiv 2022

  79. [79]

    Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. 2020. Intra-class feature variation distillation for semantic segmentation. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VII 16, pages 346--362. Springer

    work page 2020

  80. [80]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120

    Pith/arXiv arXiv 2023

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.