Pith. sign in

REVIEW 4 major objections 4 minor 31 references

The paper claims that a plug-and-play parametric memory, gated by a learned token-level router, can give large language models specialized domain expertise without the usual alignment tax.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:54 UTC pith:KZVUTXLO

load-bearing objection A plausible and well-reported recipe for domain specialization without touching the backbone; the no-alignment-tax claim is real but rests on a thinly evidenced router-generalization premise. the 4 major comments →

arxiv 2607.25614 v1 pith:KZVUTXLO submitted 2026-07-28 cs.LG cs.CL

MemSFT: Mitigating Alignment Tax with an External Parametric Memory

Jiarui Wang , Xiang Shi , Jiaqi Cao , Rubin Wei , Xiquan Wang , Hao Sun , Jingzhi Wang , Zhiqi Yang
show 3 more authors
Qipeng Guo Bowen Zhou Zhouhan Lin
This is my paper
classification cs.LG cs.CL
keywords external parametric memoryalignment taxcatastrophic forgettingtoken-level routerretrieval-based supervisiondomain specializationmemory reusesupervised fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the alignment tax — the drop in general ability that typically accompanies fine-tuning an LLM on a specialized domain — is not inevitable. Its method keeps the backbone frozen and attaches an external parametric memory: a second, smaller language model trained to imitate what a token-level retriever over the domain data would output. A lightweight router trained with both models frozen decides, at each decoding step, how much of the memory's next-token distribution to blend with the backbone's. The reported result is large domain gains on biology, geoscience, and law (e.g., a 36-point jump on the biology suite with Qwen3-14B) while the average of five general benchmarks moves by less than 0.6 points, where full SFT loses 13–31 points. If true, this decouples domain specialization from backbone retraining and makes one trained domain memory reusable across model sizes.

Core claim

MemSFT's central claim is that domain expertise can be stored outside the backbone and invoked selectively. The authors train a separate 8B memory decoder to match a retrieval-based teacher distribution built from the domain SFT corpus; then they freeze both backbone and memory and train a two-layer router to predict a token-level weight λ. The final next-token distribution is (1−λ)p_base + λ p_mem. On Biology-Instructions the memory raises the Qwen3-14B backbone from 6.64 to 42.92 and changes the general benchmark average by +0.40 (83.22 to 83.62); full SFT reaches 41.16 on the domain but drops the general average to 51.83. The same 8B memory is reused on Qwen3-8B, 14B, 32B, and 235B-A22B,

What carries the argument

The load-bearing mechanism is a two-module architecture: (1) a parametric memory — a full decoder LM trained with KL divergence plus cross-entropy to approximate a KNN-retriever teacher distribution over answer tokens of domain SFT data (Eqs. 3–6); and (2) a token-level router — a two-layer MLP taking hidden states and confidence/entropy features from both frozen models, predicting λ_t ∈ [0,1] with a signed regularizer that pushes memory use up on domain tokens and down on general tokens (Eqs. 7–10). The router's selective invocation is what converts a memory that would hurt if used uniformly into one that only fires on domain-bearing spans; the paper shows that fixed interpolation cannot ac

Load-bearing premise

The load-bearing assumption is that the router — trained on one epoch of 15.3K general instruction examples plus a few thousand domain examples (Appendix A.3) — will keep the memory nearly silent on unseen general tasks; if it ever routes memory heavily on such tasks, general averages fall toward the paper's own fixed-λ=0.7 row (44.34 general average, Table 10).

What would settle it

Take the trained Qwen3-14B + 8B-memory system and run it on a large held-out set of general instructions not used in router training, recording the router's mean memory weight λ per prompt. If the mean λ on those general prompts is not close to zero, or if the general benchmark average drops by more than a point, the claim of negligible alignment tax fails. A complementary check is to swap in a router trained without the signed regularizer and show that general retention disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, specializing a model to a new domain no longer requires any update to the backbone parameters; the domain knowledge lives in a separate memory that can be added or removed.
  • One domain memory can serve multiple backbone sizes within a tokenizer-compatible family, so scaling to larger models does not multiply adaptation compute; the paper reports 0.22x the full-SFT FLOPs for adapting four backbones.
  • Because the router can switch within a single response — near-zero λ for ordinary language and near-1 λ for numerical/domain spans — the approach predicts that selective fusion, not replay or weight interpolation, is the way to escape the specialization-retention trade-off.
  • Larger memories give stronger domain specialization while preserving general ability (BioIns 30.38/37.12/42.92 for 1.7B/4B/8B memories), so capacity in the external module, not the backbone, drives domain gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the router's selectivity generalizes beyond the five evaluation benchmarks, the design opens a path to accumulating many domain memories on one frozen backbone, each added without retraining the model — though the paper's own Limitations section notes that reuse currently requires a shared tokenizer.
  • Editorial inference: the approach implicitly assumes domain knowledge can be captured by an 8B decoder's distribution given the same tokenizer; for domains whose knowledge is less localized in token-level next-token probabilities (e.g., long-horizon reasoning), the memory-only mechanism may need a different interface.
  • Editorial inference: the strongest testable extension is to measure the learned router's mean λ on a broad, unseen general instruction suite. The paper's evidence for routing is case studies and a single ablation; a wide-coverage λ histogram would settle whether the near-zero general degradation is a property of the router or of the five chosen benchmarks.
  • Editorial inference: cross-family reuse (e.g., from the Qwen3 family to a model with a different tokenizer) is left to future work by the authors; a short continued-pretraining vocabulary alignment step is the natural next experiment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MemSFT, a method for domain specialization of LLMs that avoids the alignment tax by keeping the backbone frozen and training an external parametric memory (a full 8B decoder) to imitate a non-parametric retrieval teacher built from domain SFT data. A learned token-level router (Eq. 7) dynamically interpolates between the backbone and memory output distributions at each decoding step. The memory is trained once per domain and reused across Qwen3 backbones from 8B to 235B-A22B, with a separate lightweight router trained per backbone. Experiments on biology (BioIns), geoscience (OpenSWI), and law (LawBench) report large domain gains with general benchmark averages changing by less than 0.6 points, whereas full SFT and LoRA show severe general degradation. The paper also includes ablations on fixed vs. learned interpolation, memory scale, a BM25 RAG baseline, forgetting-mitigation baselines, and an analytical FLOPs comparison.

Significance. If the central claim holds, MemSFT is a practically valuable contribution: it demonstrates that domain knowledge can be externalized into a reusable parametric memory while preserving the frozen backbone's general capabilities, and that the same memory transfers across model scales. The paper's methodology has notable strengths: the memory training is grounded in a retrieval teacher with a clear objective; the router is trained on held-out general and domain tokens, so the design is not circular with respect to the general benchmarks; the paper reports per-task results, several thoughtful ablations (fixed λ, memory scale, BM25 RAG, MixTraining, Wise-FT), and a detailed FLOPs accounting. The reported domain gains are large and consistent across three domains. However, the signature claim of 'negligible degradation in general performance' rests heavily on the router's ability to keep the memory weight near zero on unseen general tasks, and the evidence for that generalization is currently thin: only five general benchmarks, no variance estimates on the reported means, an approximated evaluation protocol for one of those benchmarks, and qualitative rather than quantitative analysis of

major comments (4)
  1. [§5.1, Table 10 and §A.3] The entire 'negligible degradation' claim hinges on the token-level router keeping λ near zero on general decoding contexts. Table 10 shows that if λ were fixed at 0.7 on general tokens, the general average collapses from 83.22 to 44.34. Yet the router is trained on only 15,336 general examples from a single dataset (Nemotron-Post-Training-Dataset-v1) plus a few thousand domain examples (Appendix A.3), and the evidence that it generalizes to arbitrary general tasks consists of five benchmark averages and six qualitative case studies. This is a load-bearing gap. The authors should provide quantitative evidence of router behavior on held-out general tasks—e.g., mean λ, fraction of tokens with λ>0.5, and the resulting general-average impact—and ideally evaluate on a broader and more diverse set of general benchmarks (GPQA, BBH, HumanEval, MMLU-Pro, etc.) to test the router's ability to supp
  2. [§A.2 and Tables 1, 4, 5] The paper states that stochastic-decoding results are averaged over five runs, but no standard deviations or per-seed results are reported anywhere. Several deltas in the main tables are as small as 0.1 points and are labeled 'negligible' (gray), yet run-to-run variation on IFEval and MATH-500 can easily exceed that. In addition, the INCLUDE benchmark is evaluated with an explicitly approximated protocol because the reference report is underspecified; since INCLUDE contributes 20% of the general average, this introduces uncontrolled uncertainty. The authors should report error bars for all stochastic benchmarks and either obtain the exact INCLUDE configuration or provide a sensitivity analysis across plausible settings.
  3. [§2.3, Eq. (7)] The router is trained on teacher-forced gold prefixes (c_t = [q; y_<t]), but at inference the preceding tokens are produced by the model's own decoding under the fused distribution. This is a classic exposure-bias problem: the hidden states of the backbone and memory—and hence the router features—are computed from a different distribution at inference than at training. The paper does not analyze this mismatch. A concrete test would be to compare router λ values and task accuracy using gold prefixes versus self-generated prefixes on a held-out set of the same prompts. If the mismatch is large, scheduled sampling or on-policy router fine-tuning may be needed. At minimum, the paper should acknowledge and empirically bound this effect, since the router is the component that guarantees general-capability retention.
  4. [§1, §3, Tables 1 and 4] The claim of 'plug-and-play reuse across Qwen3 backbones from 8B to 235B-A22B' is tested only within the Qwen3 family, and the largest backbone (235B-A22B) has no SFT or LoRA baselines because of compute constraints. The scaling conclusion at the largest scale is therefore based on the absolute MemSFT result rather than a comparison with trained baselines. This is acknowledged in Section 3, but it limits the strength of the cross-scale generality claim. The paper should either add a smaller-scale baseline at that size (e.g., LoRA on 235B with a reduced epoch count) or soften the claim accordingly.
minor comments (4)
  1. [Abstract and §1] The abstract and introduction use 'alignment tax' consistently, but the phrase 'aligning tax' appears once in the title and elsewhere; please unify terminology.
  2. [§5.3, Figure 5] The text refers to 'the gray cell in (a)' marking an omitted middle span, but no gray cell is visible in the figure as rendered. Please ensure the figure is correctly displayed or adjust the referencing.
  3. [§A.2] The sentence 'The report does not fully specify the evaluation protocol for INCLUDE' should identify which report is meant (presumably the Qwen3 Technical Report). Also, the phrase 'approximate the setup as closely as possible' should be accompanied by the specific settings used (e.g., top-k, number of choices) rather than only the 5-shot likelihood configuration.
  4. [§C] The router description mentions outputting two log-weights log(1−λ) and log(λ) for distribution-level fusion, but Eq. (7) is written directly in terms of λ. Please clarify how the log-weights are transformed into the scalar λ used in Eq. (7).

Circularity Check

0 steps flagged

No significant circularity: memory and router are trained on external supervision and evaluated on held-out benchmarks; the only self-citations are contextual, not load-bearing.

full rationale

MemSFT's derivation chain is self-contained. The memory is trained to match a retrieval-based teacher distribution p_teacher (Eq. 3) built from a token-level datastore over domain SFT data, plus a CE term toward the gold answer (Eqs. 4-6); this supervision is external to the reported benchmarks. The router is trained on a mixture of domain examples and 15,336 general examples from Nemotron-Post-Training-Dataset-v1 (Appendix A.3), with both LMs frozen, optimizing the CE of the fused distribution plus a signed regularizer (Eqs. 7-10). Domain benchmarks (BioIns, OpenSWI, LawBench) and general benchmarks (MATH-500, C-Eval, IFEval, MMLU-Redux, INCLUDE) are not used for memory or router training, and the paper reports that training and evaluation splits are disjoint (Appendix A). The central claim of negligible general degradation therefore depends on router generalization to held-out benchmarks, not on a fitted parameter renamed as a prediction. The only self-citations are Memory Decoder (Cao et al., 2026a) and MLP Memory (Wei et al., 2025), cited as the source of the parametric-memory paradigm; the paper fully specifies the training losses and reports external evaluations, so these citations are contextual rather than load-bearing. Stated limitations—exclusion of OpenSWI-deep, approximation of the INCLUDE protocol, exclusion of one LawBench task, and tokenizer-compatible reuse only—narrow the scope but do not indicate circularity. Score 2 reflects the presence of minor self-citation without any circular derivation.

Axiom & Free-Parameter Ledger

6 free parameters · 3 axioms · 0 invented entities

No new physical or mathematical entities are postulated. The method's free parameters are the teacher temperature, the KL/CE balance, the router regularizer constants, the routing-data composition, and the LoRA baseline config. The load-bearing axiomatic choices are that kNN-LM supervision transfers specialization, that token-level distribution fusion composes well, and that the router's features suffice to detect when memory should act.

free parameters (6)
  • KNN temperature tau = 16.0
    Set to 16.0 in teacher construction (Appendix B); not derived from first principles, and the teacher distribution is the main training signal for the memory.
  • Memory-training beta = 0.1–0.3 per domain
    Balances KL vs CE in Eq. 6; chosen in preliminary runs 'within this range without further tuning' (Appendix C), i.e., selected by hand per domain.
  • Router regularizer strength alpha_s = 0.2
    Signed linear regularizer in Eq. 10; one fixed value across all experiments, but its choice directly shapes the claimed behavior.
  • Signed regularizer constants s_t = -32 for domain, +32 for general
    Arbitrary scale choice in Eq. 9; interacts with alpha_s and is not derived.
  • LoRA rank/alpha = r=8, alpha=16
    Fixed for all baselines; acceptable as a baseline configuration, but any single LoRA config can be argued to handicap the baseline.
  • Router data composition = 2,250 / 6,000 / 5,610 domain + 15,336 general examples
    These counts are chosen by hand (Appendix A.3) and the router behavior they induce is central to the no-forgetting claim.
axioms (3)
  • domain assumption The kNN-LM teacher distribution (Eq. 3) is a useful supervision signal for the memory.
    Invoked in Section 2.3 as the main memory objective; the paper provides no independent evidence that this teacher (rather than plain CE) is what creates the domain capability, except the one ablation in Table 10.
  • domain assumption Token-level interpolation of two softmax distributions (Eq. 7) can compose a strong specialist without harming general behavior.
    Core architectural premise of the whole method; supported mainly by empirical results, not derived.
  • ad hoc to paper A frozen backbone's hidden states and entropy/confidence features suffice for the router to detect domain-relevant tokens.
    Router input design in Section 2.3/Appendix C; the router generalizes only if these features carry the needed signal, and no controlled study of feature sufficiency is given.

pith-pipeline@v1.3.0-alltime-deepseek · 32070 in / 8146 out tokens · 66253 ms · 2026-08-01T01:54:39.536321+00:00 · methodology

0 comments
read the original abstract

Adapting Large Language Models (LLMs) to specialized domains often incurs an alignment tax, as fine-tuning on domain-specific tasks can cause catastrophic forgetting and substantially degrade performance on general tasks. We propose MemSFT, which mitigates the alignment tax by decoupling domain specialization from backbone parameter updates through a plug-and-play parametric memory. The memory is trained to imitate the behavior of a non-parametric retriever operating over domain data, thereby memorizing knowledge and patterns that would otherwise be accessed through retrieval. Once trained on a specific domain, the memory can be reused across LLMs of different sizes. During generation, a learned router dynamically fuses the output distributions of the memory and backbone at each decoding step, allowing domain expertise to be invoked selectively. Across biology, geoscience, and law, evaluations with models ranging from Qwen3-8B to Qwen3-235B-A22B show that MemSFT consistently improves domain performance with negligible degradation in general performance, whereas full SFT suffers severe forgetting on general tasks. Overall, our results demonstrate a practical path to decoupling general model capabilities from domain-specific knowledge at the parameter level, thereby equipping LLMs with new specialized capabilities without compromising their general capabilities.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 15 linked inside Pith

  1. [1]

    Nested learning: The illusion of deep learning architectures.arXiv preprint arXiv:2512.24695,

    Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. Nested learning: The illusion of deep learning architectures.arXiv preprint arXiv:2512.24695,

    arXiv

  2. [3]

    Both baselines for mitigating forgetting use the Qwen3-14B Biology-Instructions setting. MixTraining(1:1) follows the same LoRA configuration as the main Qwen3-14B Biology-Instructions LoRA baseline, but trains on a 1:1 mixture of 500K Biology- Instructions examples and 500K general instruction examples sampled from NVIDIA’s Nemotron- Post-Training-Datase...

    2025

  3. [5]

    On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258,

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258,

    Pith/arXiv arXiv

  4. [8]

    Conditional memory via scalable lookup: A new axis of sparsity for large language models.arXiv preprint arXiv:2601.07372,

    16 MemSFT: Mitigating Alignment Tax with an External Parametric Memory Xin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Zhewen Hao, Yukun Li, et al. Conditional memory via scalable lookup: A new axis of sparsity for large language models.arXiv preprint arXiv:2601.07372,

    Pith/arXiv arXiv

  5. [10]

    Lawbench: Benchmarking legal knowledge of large language models

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, et al. Lawbench: Benchmarking legal knowledge of large language models. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 7933–7962,

    2024

  6. [11]

    URLhttps://zenodo.org/records/12608602. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computationa...

    arXiv 2025

  7. [12]

    Biology-instructions: A dataset and benchmark for multi-omics sequence understanding capability of large language models.arXiv preprint arXiv:2412.19191,

    17 MemSFT: Mitigating Alignment Tax with an External Parametric Memory Haonan He, Yuchen Ren, Yining Tang, Ziyang Xu, Junxian Li, Minghao Yang, Di Zhang, Dong Yuan, Tao Chen, Shufei Zhang, et al. Biology-instructions: A dataset and benchmark for multi-omics sequence understanding capability of large language models.arXiv preprint arXiv:2412.19191,

    arXiv

  8. [13]

    Scaling laws for forgetting when fine-tuning large language models.arXiv preprint arXiv:2401.05605,

    Damjan Kalajdzievski. Scaling laws for forgetting when fine-tuning large language models.arXiv preprint arXiv:2401.05605,

    Pith/arXiv arXiv

  9. [14]

    Generalization through memorization: Nearest neighbor language models.arXiv preprint arXiv:1911.00172,

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models.arXiv preprint arXiv:1911.00172,

    Pith/arXiv arXiv 1911

  10. [15]

    Revisiting catastrophic forgetting in large language model tuning

    Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. Revisiting catastrophic forgetting in large language model tuning. InFindings of the association for computational linguistics: EMNLP 2024, pages 4297–4308,

    2024

  11. [16]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601,

    2024

  12. [17]

    More than catastrophic forgetting: Integrating general capabilities for domain-specific llms

    Chengyuan Liu, Yangyang Kang, Shihang Wang, Lizhi Qing, Fubang Zhao, Chao Wu, Changlong Sun, Kun Kuang, and Fei Wu. More than catastrophic forgetting: Integrating general capabilities for domain-specific llms. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7531–7548,

    2024

  13. [18]

    Openswi: a massive-scale benchmark dataset for surface wave dispersion curve inversion.Earth System Science Data Discussions, 2025:1–37,

    18 MemSFT: Mitigating Alignment Tax with an External Parametric Memory Feng Liu, Sijie Zhao, Xinyu Gu, Fenghua Ling, Peiqin Zhuang, Yaxing Li, Rui Su, Lihua Fang, Lianqing Zhou, Jianping Huang, et al. Openswi: a massive-scale benchmark dataset for surface wave dispersion curve inversion.Earth System Science Data Discussions, 2025:1–37,

    2025

  14. [21]

    K-adapter: Infusing knowledge into pre-trained models with adapters

    Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuan-Jing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405–1418,

    2021

  15. [22]

    Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652,

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652,

    Pith/arXiv arXiv

  16. [24]

    Memorizing transformers

    Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint arXiv:2203.08913,

    Pith/arXiv arXiv

  17. [25]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  18. [26]

    Memory3: Language modeling with explicit memory.arXiv preprint arXiv:2407.01178,

    Hongkang Yang, Zehao Lin, Wenjin Wang, Hao Wu, Zhiyu Li, Bo Tang, Wenqiang Wei, Jinbo Wang, Zeyun Tang, Shichao Song, et al. Memory3: Language modeling with explicit memory.arXiv preprint arXiv:2407.01178,

    Pith/arXiv arXiv

  19. [27]

    Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset

    Botao Yu, Frazier N Baker, Ziqi Chen, Xia Ning, and Huan Sun. Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset. arXiv preprint arXiv:2402.09391,

    Pith/arXiv arXiv

  20. [28]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Pith/arXiv arXiv

  21. [29]

    2-10 trigger_word_extraction

    A. Datasets and Evaluation Protocols Table 6 summarizes the role and adopted scope of each domain dataset. The same sampled domain corpus is used for full SFT, LoRA, and memory training within each domain unless otherwise noted, ensuring that adaptation methods differ in training mechanism rather than access to domain su- pervision. For BioIns and OpenSWI...

    2024

  22. [128]

    The router outputs two log-weights, log(1−𝜆 𝑡) and log(𝜆 𝑡), which are used for distribution-level fusion of the backbone and memory distributions. Across domains and backbone sizes, router training uses one epoch, AdamW with a linear learning-rate schedule, per-device batch size 1, gradient accumulation 8, maximum sequence length 2048, and 100 warmup opt...

    2048

  23. [2009]

    In- clude: Evaluating multilingual language understanding with regional knowledge

    Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed Haggag, Alfonso Amayuelas, et al. In- clude: Evaluating multilingual language understanding with regional knowledge. InInternational Conference on Learning Representations, volume 2025, pages 83291–83322,

    2025

  24. [2019]

    Compressive trans- formers for long-range sequence modelling.arXiv preprint arXiv:1911.05507,

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive trans- formers for long-range sequence modelling.arXiv preprint arXiv:1911.05507,

    Pith/arXiv arXiv 1911

  25. [2020]

    Memory decoder: A pretrained, plug-and-play memory for large language models.Advances in Neural Information Processing Systems, 38:115487–115510, 2026a

    Jiaqi Cao, Jiarui Wang, Rubin Wei, Qipeng Guo, Kai Chen, Bowen Zhou, and Zhouhan Lin. Memory decoder: A pretrained, plug-and-play memory for large language models.Advances in Neural Information Processing Systems, 38:115487–115510, 2026a. Maosong Cao, Kai Chen, Haodong Duan, Yixiao Fang, Tong Gao, Ge Jiaye, Mo Li, Hongwei Liu, Junnan Liu, Yuan Liu, et al....

    Pith/arXiv arXiv

  26. [2021]

    Mlp mem- ory: A retriever-pretrained memory for large language models.arXiv preprint arXiv:2508.01832,

    Rubin Wei, Jiaqi Cao, Jiarui Wang, Jushi Kai, Qipeng Guo, Bowen Zhou, and Zhouhan Lin. Mlp mem- ory: A retriever-pretrained memory for large language models.arXiv preprint arXiv:2508.01832,

    arXiv

  27. [2022]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  28. [2023]

    Improved supervised fine-tuning for large language models to mitigate catastrophic forgetting.arXiv preprint arXiv:2506.09428,

    Fei Ding and Baiqiao Wang. Improved supervised fine-tuning for large language models to mitigate catastrophic forgetting.arXiv preprint arXiv:2506.09428,

    Pith/arXiv arXiv

  29. [2024]

    Lora learns less and forgets less.arXiv preprint arXiv:2405.09673,

    Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less.arXiv preprint arXiv:2405.09673,

    Pith/arXiv arXiv

  30. [2025]

    Memory layers at scale.arXiv preprint arXiv:2412.09764,

    Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, and Gargi Ghosh. Memory layers at scale.arXiv preprint arXiv:2412.09764,

    Pith/arXiv arXiv

  31. [2026]

    Llama-nemotron: Efficient reasoning models

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949,

    arXiv