pith. sign in

arxiv: 2605.16865 · v2 · pith:KWFZ7LKCnew · submitted 2026-05-16 · 💻 cs.CL

MixSD: Mixed Contextual Self-Distillation for Knowledge Injection

Jiarui Liu , Lechen Zhang , Yongjin Yang , Yinghui He , Yingheng Wang , Weihao Xuan , Zhijing Jin , Mona Diab This is my paper

Pith reviewed 2026-05-22 10:07 UTC · model grok-4.3

classification 💻 cs.CL
keywords knowledge injectioncatastrophic forgettingself-distillationsupervised fine-tuninglanguage modelsdistribution alignmentfactual recallknowledge editing
0
0 comments X

The pith

MixSD mixes tokens from a base model's expert and naive conditionals to inject facts while preserving original capabilities.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard supervised fine-tuning injects new knowledge into language models but often erases pretrained reasoning and general-domain skills because its targets diverge from the model's natural generation distribution. MixSD instead builds supervision on the fly by mixing tokens sampled from the same base model under an expert conditional that sees the injected fact and a naive conditional that reflects its original prior. The resulting sequences carry the factual signal yet stay much closer to the model's autoregressive distribution. Across synthetic factual-recall and arithmetic tasks plus open-domain QA and knowledge-editing benchmarks, this yields near-perfect training accuracy together with retention of up to 100 percent of held-out capability, while ordinary fine-tuning can retain as little as 1 percent.

Core claim

MixSD constructs supervision dynamically by mixing tokens from an expert conditional that observes the injected fact in context and a naive conditional that reflects the model's original prior. The resulting supervision sequences preserve the factual learning signal while remaining substantially closer to the base model's autoregressive distribution. This produces lower negative log-likelihood targets under the base model and reduces harmful movement along Fisher-sensitive parameter directions.

What carries the argument

MixSD, a token-mixing procedure that blends outputs from the base model's expert conditional (observing the new fact) and naive conditional (original prior) to form distribution-aligned training targets.

If this is right

  • MixSD achieves a better memorization-retention trade-off than SFT and on-policy self-distillation across multiple model scales.
  • It retains up to 100 percent of the base model's held-out capability while maintaining near-perfect training accuracy on injected facts.
  • Standard SFT retains as little as 1 percent of held-out capability under comparable conditions.
  • MixSD produces substantially lower-NLL supervision targets under the base model.
  • It reduces harmful movement along Fisher-sensitive parameter directions during optimization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Distribution alignment via self-generated mixed targets may serve as a general principle for other fine-tuning regimes that seek to limit capability loss.
  • The same mixing approach could be tested in sequential knowledge-editing settings to check whether cumulative forgetting is reduced without external teachers.
  • Extending the method to non-fact domains such as code or mathematical reasoning might reveal whether the retention benefit is specific to factual injection.
  • Direct comparison of gradient directions under MixSD versus SFT on held-out benchmarks would provide a mechanistic test of the Fisher-sensitive movement claim.

Load-bearing premise

Mixing tokens from the expert conditional that observes the injected fact and the naive conditional produces supervision that preserves the factual learning signal while remaining substantially closer to the base model's autoregressive distribution.

What would settle it

A controlled run on a new model scale or domain in which MixSD mixed sequences fail to show both lower negative log-likelihood under the base model and higher held-out retention than standard SFT targets while still reaching high training accuracy.

Figures

Figures reproduced from arXiv: 2605.16865 by Jiarui Liu, Lechen Zhang, Mona Diab, Weihao Xuan, Yingheng Wang, Yinghui He, Yongjin Yang, Zhijing Jin.

Figure 1
Figure 1. Figure 1: Overview of MIXSD and the two datasets KGFACT and KGFUNC we construct. Given an input prompt and a ground-truth target, MIXSD samples token-level supervision from two base-model conditionals: an expert rollout conditioned on the injected knowledge and a naive rollout conditioned only on the original prompt. At each decoding step, MIXSD selects the expert token with probability 1 − λ and the naive token wit… view at source ↗
Figure 1
Figure 1. Figure 1: Overview of MIXSD and the two datasets KGFACT and KGFUNC we construct. Given an input prompt and a ground-truth target, MIXSD samples token-level supervision from two base-model conditionals: an expert rollout conditioned on the injected knowledge and a naive rollout conditioned only on the original prompt. At each decoding step, MIXSD selects the expert token with probability 1 − λ and the naive token wit… view at source ↗
Figure 2
Figure 2. Figure 2: Trade-off between training accuracy on KGFACT-SMALL and average general-domain OOD test accuracy across AIME2024, MATH500, GSM8K, HumanEval, and MMLU. Each point corresponds to a checkpoint at a different training step, with larger markers indicating later stages of training. The horizontal dashed lines denote the average OOD accuracy of the untrained base model. We observe a consistent trade-off between t… view at source ↗
Figure 3
Figure 3. Figure 3: Empirical CDFs of per-token negative log-likelihood (NLL) under the base model, evaluated [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Error-mode breakdown on AIME-2024 after fine-tuning on KGF [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Error-mode breakdown on MATH-500 after fine-tuning on KGF [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Error-mode breakdown on GSM8K after fine-tuning on KGF [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Error-mode breakdown on HumanEval after fine-tuning on [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Error-mode breakdown on MMLU after fine-tuning on [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Error-mode breakdown on AIME-2024 after fine-tuning on [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Error-mode breakdown on MATH-500 after fine-tuning on KGF [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Error-mode breakdown on GSM8K after fine-tuning on KGF [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Error-mode breakdown on HumanEval after fine-tuning on KGF [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Error-mode breakdown on MMLU after fine-tuning on KGF [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗
read the original abstract

Supervised fine-tuning (SFT) is widely used to inject new knowledge into language models, but it often degrades pretrained capabilities such as reasoning and general-domain performance. We argue this forgetting arises because fine-tuning targets from humans or external systems diverge from the model's autoregressive distribution, forcing the optimizer to imitate low-probability token sequences. To address this problem, we propose MixSD, a simple external-teacher-free method for distribution-aligned knowledge injection. Instead of training on fixed targets, MixSD constructs supervision dynamically by mixing tokens from two conditionals of the base model itself: an expert conditional that observes the injected fact in context, and a naive conditional that reflects the model's original prior. The resulting supervision sequences preserve the factual learning signal while remaining substantially closer to the base model's distribution. We evaluate MixSD on two synthetic corpora that we construct to study factual recall and arithmetic function acquisition in a controlled setting, together with established benchmarks for open-domain factual question answering and knowledge editing. Across multiple model scales and settings, MixSD consistently achieves a better memorization-retention trade-off compared to SFT and on-policy self distillation baselines, retaining up to 100% of the base model's held-out capability while maintaining near-perfect training accuracy, whereas standard SFT retains as little as 1%. We further show that MixSD produces substantially lower-NLL supervision targets under the base model and reduces harmful movement along Fisher-sensitive parameter directions. These results suggest that aligning supervision with the model's native generation distribution is a simple and effective principle for knowledge injection that mitigates catastrophic forgetting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The manuscript proposes MixSD, a teacher-free method for injecting new knowledge into language models via dynamic token-level mixing of supervision targets drawn from the base model's own expert conditional (fact provided in context) and naive conditional. The central claim is that this produces supervision sequences that retain the factual signal while remaining close to the base autoregressive distribution, yielding a superior memorization-retention trade-off versus SFT and on-policy self-distillation: up to 100% retention of held-out capability with near-perfect training accuracy, versus as little as 1% retention under SFT. Supporting measurements include lower NLL of the mixed targets under the base model and reduced movement along Fisher-sensitive directions, evaluated on two synthetic corpora plus open-domain QA and knowledge-editing benchmarks across model scales.

Significance. If the central claim holds under more detailed verification, the work offers a simple, external-teacher-free principle for mitigating catastrophic forgetting during knowledge injection. The controlled synthetic benchmarks for factual recall and arithmetic acquisition, together with the NLL and Fisher analyses, provide useful diagnostic tools. The approach is internally consistent with the base model's conditionals and does not rely on circular self-reference.

major comments (3)
  1. [§3] §3 (Mixing Procedure): The token-mixing construction is load-bearing for the memorization-retention advantage, yet the manuscript provides only a high-level description. It is unclear whether mixing occurs independently per position, what probability governs selection from the expert versus naive conditional, or whether any mechanism anchors or up-weights the critical factual tokens. Without this, it remains possible that the observed near-perfect training accuracy arises from an unstated property rather than the intended alignment, as the skeptic concern notes.
  2. [§4] §4 (Experiments) and Table 1/2: The strong quantitative claims (100% retention vs. 1% for SFT) are reported without error bars, number of random seeds, or statistical significance tests. Hyperparameter choices for the mixing ratio and any post-hoc selection criteria are also omitted, undermining confidence that the advantage is robust rather than sensitive to specific settings.
  3. [§5] §5 (Analysis): While lower NLL and reduced Fisher movement are shown, the manuscript does not demonstrate that these quantities directly mediate the retention gains (e.g., via correlation or ablation). This leaves open whether the reported mechanism explains the trade-off or whether other factors are at work.
minor comments (3)
  1. [§3] The distinction between 'expert conditional' and 'naive conditional' would benefit from an explicit equation or pseudocode block early in the method section.
  2. [§5] Figure legends for the NLL and Fisher plots should include axis scales and confidence intervals to improve readability.
  3. [§2] A brief comparison to related distribution-alignment techniques (e.g., references to prior self-distillation or KL-regularized fine-tuning) would help situate the contribution.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback and for recognizing the potential of MixSD as a teacher-free approach to knowledge injection. We address each major comment below with clarifications and revisions to improve the manuscript.

read point-by-point responses

  1. Referee: §3 (Mixing Procedure): The token-mixing construction is load-bearing for the memorization-retention advantage, yet the manuscript provides only a high-level description. It is unclear whether mixing occurs independently per position, what probability governs selection from the expert versus naive conditional, or whether any mechanism anchors or up-weights the critical factual tokens. Without this, it remains possible that the observed near-perfect training accuracy arises from an unstated property rather than the intended alignment, as the skeptic concern notes.

    Authors: We agree that the original description in §3 was high-level. The mixing is performed independently at each token position: with probability α we draw the target token from the expert conditional (fact in context) and with probability 1-α from the naive conditional. In the reported experiments α=0.7 was used, selected via a small validation sweep; no explicit up-weighting or anchoring of factual tokens is applied beyond the natural probability mass the expert conditional places on correct continuations. We have added a formal definition, pseudocode, and an ablation varying α to the revised §3 and Appendix A to address this concern directly. revision: yes

  2. Referee: §4 (Experiments) and Table 1/2: The strong quantitative claims (100% retention vs. 1% for SFT) are reported without error bars, number of random seeds, or statistical significance tests. Hyperparameter choices for the mixing ratio and any post-hoc selection criteria are also omitted, undermining confidence that the advantage is robust rather than sensitive to specific settings.

    Authors: We acknowledge the omission. In the revised manuscript we report means and standard deviations over five random seeds for all main results in Tables 1 and 2. We have added paired t-tests confirming statistical significance (p < 0.05) of the retention improvements versus SFT. The mixing ratio α was chosen by grid search on a held-out validation split; full details and the search range now appear in §4.1 and Appendix B. No post-hoc selection of results was performed. revision: yes

  3. Referee: §5 (Analysis): While lower NLL and reduced Fisher movement are shown, the manuscript does not demonstrate that these quantities directly mediate the retention gains (e.g., via correlation or ablation). This leaves open whether the reported mechanism explains the trade-off or whether other factors are at work.

    Authors: The original §5 presents NLL and Fisher metrics as supporting diagnostics rather than a full mediation study. We have added a controlled ablation in the revised §5 that varies the mixing ratio to modulate NLL while measuring retention; the results show a consistent negative correlation between target NLL and retention score across settings. A complete causal mediation analysis would require additional instrumentation and is noted as future work, but the new ablation provides direct evidence linking the alignment metrics to the observed trade-off. revision: partial

Circularity Check

0 steps flagged

No significant circularity; MixSD is a definitional method with external empirical validation

full rationale

The paper defines MixSD explicitly as a procedure that mixes tokens sampled from the base model's expert conditional (fact in context) and naive conditional to create supervision targets. This is presented as an algorithmic choice in the method, not as a derived result that reduces to fitted parameters or self-referential equations by construction. Claims of improved memorization-retention trade-off are supported by experiments on synthetic corpora and standard benchmarks, compared against SFT and on-policy self-distillation baselines, without load-bearing self-citations, uniqueness theorems, or ansatzes imported from prior author work. The use of the model's own conditionals is intentional and transparent rather than tautological, and results rely on external performance metrics rather than internal consistency alone.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on domain assumptions about model conditionals and the effectiveness of mixing; no new physical entities are postulated and no free parameters are explicitly fitted in the abstract description.

axioms (2)
  • domain assumption The expert conditional that observes the injected fact in context and the naive conditional reflecting the original prior can be meaningfully combined to preserve factual signal.
    Invoked in the construction of supervision sequences as described in the abstract.
  • domain assumption Supervision targets closer to the base model's autoregressive distribution mitigate catastrophic forgetting.
    Core premise motivating the departure from standard SFT.

pith-pipeline@v0.9.0 · 5829 in / 1379 out tokens · 47279 ms · 2026-05-22T10:07:44.498172+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

58 extracted references · 58 canonical work pages · 10 internal anchors

  1. [1]

    Measuring short-form factuality in large language models

    Measuring short-form factuality in large language models , author=. arXiv preprint arXiv:2411.04368 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2502.20377 , year=

    Phantomwiki: On-demand datasets for reasoning and retrieval evaluation , author=. arXiv preprint arXiv:2502.20377 , year=

    work page arXiv

  3. [3]

    American Invitational Mathematics Examination (AIME) 2024 , author=

    work page 2024

  4. [4]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Training Verifiers to Solve Math Word Problems

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Evaluating Large Language Models Trained on Code

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Measuring Massive Multitask Language Understanding

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  8. [8]

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author=. arXiv preprint arXiv:2601.18734 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    On-Policy Context Distillation for Language Models

    On-policy context distillation for language models , author=. arXiv preprint arXiv:2602.12275 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Qwen3 Technical Report

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

    Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? , author=. arXiv preprint arXiv:2603.24472 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    The Twelfth International Conference on Learning Representations , year=

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes , author=. The Twelfth International Conference on Learning Representations , year=

    work page

  13. [13]

    2021 , eprint=

    A General Language Assistant as a Laboratory for Alignment , author=. 2021 , eprint=

    work page 2021

  14. [14]

    Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages =

    Buciluundefined, Cristian and Caruana, Rich and Niculescu-Mizil, Alexandru , title =. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages =. 2006 , isbn =. doi:10.1145/1150402.1150464 , abstract =

    work page doi:10.1145/1150402.1150464 2006

  15. [15]

    2025 , url=

    Tianzhe Chu and Yuexiang Zhai and Jihan Yang and Shengbang Tong and Saining Xie and Dale Schuurmans and Quoc V Le and Sergey Levine and Yi Ma , booktitle=. 2025 , url=

    work page 2025

  16. [16]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    work page 2024

  17. [17]

    Proceedings of Thirty Fifth Conference on Learning Theory , pages =

    How catastrophic can catastrophic forgetting be in linear regression? , author =. Proceedings of Thirty Fifth Conference on Learning Theory , pages =. 2022 , editor =

    work page 2022

  18. [18]

    Proceedings of the 35th International Conference on Machine Learning , pages =

    Born Again Neural Networks , author =. Proceedings of the 35th International Conference on Machine Learning , pages =. 2018 , editor =

    work page 2018

  19. [19]

    Yuxian Gu and Li Dong and Furu Wei and Minlie Huang , booktitle=. Mini. 2024 , url=

    work page 2024

  20. [20]

    2015 , eprint=

    Distilling the Knowledge in a Neural Network , author=. 2015 , eprint=

    work page 2015

  21. [21]

    Note on the quadratic penalties in elastic weight consolidation , volume=

    Huszár, Ferenc , year=. Note on the quadratic penalties in elastic weight consolidation , volume=. Proceedings of the National Academy of Sciences , publisher=. doi:10.1073/pnas.1717042115 , number=

    work page doi:10.1073/pnas.1717042115

  22. [22]

    2024 , eprint=

    Scaling Laws for Forgetting When Fine-Tuning Large Language Models , author=. 2024 , eprint=

    work page 2024

  23. [23]

    Sequence-Level Knowledge Distillation

    Kim, Yoon and Rush, Alexander M. Sequence-Level Knowledge Distillation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016. doi:10.18653/v1/D16-1139

    work page doi:10.18653/v1/d16-1139 2016

  24. [24]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    Kirkpatrick, James and Pascanu, Razvan and Rabinowitz, Neil and Veness, Joel and Desjardins, Guillaume and Rusu, Andrei A. and Milan, Kieran and Quan, John and Ramalho, Tiago and Grabska-Barwinska, Agnieszka and Hassabis, Demis and Clopath, Claudia and Kumaran, Dharshan and Hadsell, Raia , year=. Overcoming catastrophic forgetting in neural networks , vol...

    work page doi:10.1073/pnas.1611835114

  25. [25]

    Efficient Knowledge Injection in

    Kalle Kujanp. Efficient Knowledge Injection in. Transactions on Machine Learning Research , issn=. 2025 , url=

    work page 2025

  26. [26]

    2025 , eprint=

    An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning , author=. 2025 , eprint=

    work page 2025

  27. [27]

    Journal of Machine Learning Research , year =

    James Martens , title =. Journal of Machine Learning Research , year =

    work page

  28. [28]

    1989 , issn =

    Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , editor =. 1989 , issn =. doi:https://doi.org/10.1016/S0079-7421(08)60536-8 , url =

    work page doi:10.1016/s0079-7421(08)60536-8 1989

  29. [29]

    Advances in Neural Information Processing Systems , editor=

    Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , editor=. 2022 , url=

    work page 2022

  30. [30]

    Propagating Knowledge Updates to

    Shankar Padmanabhan and Yasumasa Onoe and Michael JQ Zhang and Greg Durrett and Eunsol Choi , booktitle=. Propagating Knowledge Updates to. 2023 , url=

    work page 2023

  31. [31]

    Forty-second International Conference on Machine Learning , year=

    Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting , author=. Forty-second International Conference on Machine Learning , year=

    work page

  32. [32]

    2026 , url=

    Idan Shenfeld and Jyothish Pari and Pulkit Agrawal , booktitle=. 2026 , url=

    work page 2026

  33. [33]

    2022 , eprint=

    Learning by Distilling Context , author=. 2022 , eprint=

    work page 2022

  34. [34]

    Forty-second International Conference on Machine Learning , year=

    Overtrained Language Models Are Harder to Fine-Tune , author=. Forty-second International Conference on Machine Learning , year=

    work page

  35. [35]

    Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation , year=

    Zhang, Linfeng and Song, Jiebo and Gao, Anni and Chen, Jingwei and Bao, Chenglong and Ma, Kaisheng , booktitle=. Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation , year=

    work page

  36. [36]

    , title =

    Nesterov, Y. , title =

    work page

  37. [37]

    Thinking Machines Lab: Connectionism , year =

    Kevin Lu and Thinking Machines Lab , title =. Thinking Machines Lab: Connectionism , year =

    work page

  38. [38]

    2020 , eprint=

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter , author=. 2020 , eprint=

    work page 2020

  39. [39]

    2026 , eprint=

    Self-Distilled RLVR , author=. 2026 , eprint=

    work page 2026

  40. [40]

    Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLM s

    Ovadia, Oded and Brief, Menachem and Mishaeli, Moshik and Elisha, Oren. Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLM s. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.15

    work page doi:10.18653/v1/2024.emnlp-main.15 2024

  41. [41]

    Retrieval-augmented generation for knowledge-intensive NLP tasks , year =

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K\". Retrieval-augmented generation for knowledge-intensive NLP tasks , year =. Proceedings of the 34th International Conference on Neural Information Processing Systems , articleno =

    work page

  42. [42]

    2024 , eprint=

    Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning , author=. 2024 , eprint=

    work page 2024

  43. [43]

    More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLM s

    Liu, Chengyuan and Kang, Yangyang and Wang, Shihang and Qing, Lizhi and Zhao, Fubang and Wu, Chao and Sun, Changlong and Kuang, Kun and Wu, Fei. More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLM s. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.em...

    work page doi:10.18653/v1/2024.emnlp-main.429 2024

  44. [44]

    Model Editing at Scale leads to Gradual and Catastrophic Forgetting

    Gupta, Akshat and Rao, Anurag and Anumanchipalli, Gopala. Model Editing at Scale leads to Gradual and Catastrophic Forgetting. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.902

    work page doi:10.18653/v1/2024.findings-acl.902 2024

  45. [45]

    Locating and Editing Factual Associations in

    Kevin Meng and David Bau and Alex J Andonian and Yonatan Belinkov , booktitle=. Locating and Editing Factual Associations in. 2022 , url=

    work page 2022

  46. [46]

    The Eleventh International Conference on Learning Representations , year=

    Mass-Editing Memory in a Transformer , author=. The Eleventh International Conference on Learning Representations , year=

    work page

  47. [47]

    Zero-Shot Relation Extraction via Reading Comprehension

    Levy, Omer and Seo, Minjoon and Choi, Eunsol and Zettlemoyer, Luke. Zero-Shot Relation Extraction via Reading Comprehension. Proceedings of the 21st Conference on Computational Natural Language Learning ( C o NLL 2017). 2017. doi:10.18653/v1/K17-1034

    work page doi:10.18653/v1/k17-1034 2017

  48. [48]

    Editing Factual Knowledge in Language Models

    De Cao, Nicola and Aziz, Wilker and Titov, Ivan. Editing Factual Knowledge in Language Models. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. doi:10.18653/v1/2021.emnlp-main.522

    work page doi:10.18653/v1/2021.emnlp-main.522 2021

  49. [49]

    MQ u AKE : Assessing Knowledge Editing in Language Models via Multi-Hop Questions

    Zhong, Zexuan and Wu, Zhengxuan and Manning, Christopher and Potts, Christopher and Chen, Danqi. MQ u AKE : Assessing Knowledge Editing in Language Models via Multi-Hop Questions. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.971

    work page doi:10.18653/v1/2023.emnlp-main.971 2023

  50. [50]

    Transactions of the Association for Computational Linguistics , volume=

    Evaluating the ripple effects of knowledge editing in language models , author=. Transactions of the Association for Computational Linguistics , volume=. 2024 , publisher=

    work page 2024

  51. [51]

    DU n E : Dataset for Unified Editing

    Aky. DU n E : Dataset for Unified Editing. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.114

    work page doi:10.18653/v1/2023.emnlp-main.114 2023

  52. [52]

    2024 , eprint=

    A Comprehensive Study of Knowledge Editing for Large Language Models , author=. 2024 , eprint=

    work page 2024

  53. [53]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. Iclr , volume=

    work page

  54. [54]

    IEEE transactions on pattern analysis and machine intelligence , volume=

    Learning without forgetting , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2017 , publisher=

    work page 2017

  55. [55]

    Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal

    Huang, Jianheng and Cui, Leyang and Wang, Ante and Yang, Chengyi and Liao, Xinting and Song, Linfeng and Yao, Junfeng and Su, Jinsong. Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.186...

    work page doi:10.18653/v1/2024.acl-long.77 2024

  56. [56]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages=

    Easyedit: An easy-to-use knowledge editing framework for large language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages=

    work page

  57. [57]

    International Conference on Learning Representations , volume=

    Let's verify step by step , author=. International Conference on Learning Representations , volume=

    work page

  58. [58]

    Finetuned Language Models Are Zero-Shot Learners

    Finetuned language models are zero-shot learners , author=. arXiv preprint arXiv:2109.01652 , year=

    work page internal anchor Pith review Pith/arXiv arXiv