Pith. sign in

REVIEW 2 major objections 1 minor 7 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Sparse Memory Finetuning adapts models by updating only the most-read memory rows per batch, limiting drift on general capabilities.

2026-06-30 23:51 UTC pith:RWT45WHE

load-bearing objection SMF gives a modest MedMCQA gain with less drift on WikiText and TriviaQA than LoRA or full tuning, but the two probes are too narrow to back the low-forgetting claim. the 2 major comments →

arxiv 2605.03229 v2 pith:RWT45WHE submitted 2026-05-04 cs.CL cs.LG

Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning

Prakhar Gupta , Garv Shah , Satyam Goyal , Anirudh Kanchi This is my paper
classification cs.CL cs.LG
keywords sparse memory finetuningcatastrophic forgettinglanguage model adaptationMedMCQALoRAkey-value memoryparameter efficient finetuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Sparse Memory Finetuning as a method to adapt pretrained language models to new tasks while reducing catastrophic forgetting of prior abilities. It inserts key-value memory layers and, during training, modifies only the small subset of rows that receive the heaviest usage from each batch. Tested on the MedMCQA medical multiple-choice task with Qwen-2.5-0.5B-Instruct, the approach yields a 2.5 percentage point gain on the target task while keeping WikiText perplexity and TriviaQA accuracy within roughly one point of the base model. LoRA and full finetuning produce larger task improvements but also produce measurable drift on the same probes. Two row-selection heuristics, KL-divergence and TF-IDF, are compared and shown to trade off differently between the forgetting metrics.

Core claim

Sparse Memory Finetuning adds key-value memory layers to the model and updates only the small set of memory rows that the current batch reads most heavily. On MedMCQA this produces a 2.5 percentage point improvement while holding both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains accompanied by clear drift on both probes.

What carries the argument

Key-value memory layers updated sparsely per batch via row selection by KL-divergence or TF-IDF, restricting changes to the rows read most heavily.

Load-bearing premise

The chosen forgetting probes together with the row-selection heuristics supply a sufficient and unbiased measure of preserved general capabilities after adaptation.

What would settle it

An experiment on a new task or with additional forgetting measures that finds SMF producing comparable or larger drift than LoRA while still improving on the target task would falsify the central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • SMF can serve as a lower-forgetting alternative when adaptation must preserve performance on general text and knowledge probes.
  • KL-divergence and TF-IDF row selection produce different balances between the two forgetting metrics.
  • Larger gains on the target task come with greater drift under both LoRA and full finetuning.
  • The method achieves its results by limiting updates to a small active subset of memory rows each step.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach may apply to other domain shifts where retention of broad pretraining knowledge is a priority.
  • The overhead of the added memory layers could be tested on larger base models to check whether the forgetting benefit scales.
  • SMF might be combined with existing parameter-efficient methods to further tune the accuracy-forgetting trade-off.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes Sparse Memory Finetuning (SMF), which augments a pretrained LM with key-value memory layers and updates only the most heavily accessed rows per batch during adaptation. It re-implements SMF on Qwen-2.5-0.5B-Instruct and compares it to LoRA and full finetuning on the MedMCQA 4-choice medical QA task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. The central empirical claim is that SMF yields a +2.5 pp gain on MedMCQA while keeping both probes within ~1 point of the base model, whereas the baselines achieve larger task gains at the cost of clear drift; two row-selection heuristics (KL-divergence and TF-IDF) are also contrasted.

Significance. If the results hold under properly documented conditions, SMF would offer a concrete, memory-augmented alternative to standard parameter-efficient and full finetuning methods that demonstrably reduces measured forgetting on the chosen probes. This could be practically relevant for domain adaptation tasks where retaining broad capabilities matters, and the explicit comparison of row-selection rules provides a starting point for further tuning of the method.

major comments (2)
  1. [Abstract] Abstract: the reported numerical improvements (+2.5 pp on MedMCQA, drift bounded within ~1 point on the probes) are stated without any accompanying experimental protocol, baseline implementation details, hyperparameter settings, number of runs, or statistical tests. This absence is load-bearing because the central claim is an empirical comparison whose reliability cannot be assessed from the given information.
  2. [Forgetting probes and row-selection] Forgetting probes and row-selection section: the claim that SMF is a low-forgetting alternative depends on WikiText perplexity and TriviaQA accuracy being adequate, unbiased proxies for retained general capabilities after MedMCQA adaptation. No justification, ablation against other metrics (e.g., reasoning or non-medical recall tasks), or sensitivity analysis is provided, so the sufficiency of these two probes for the distribution shift remains untested.
minor comments (1)
  1. [Method] The description of how KL-divergence or TF-IDF is applied to select memory rows would benefit from an explicit equation or pseudocode to clarify the exact computation and any hyperparameters involved.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed feedback on the empirical presentation and evaluation methodology. We address each major comment below, indicating planned revisions where appropriate.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the reported numerical improvements (+2.5 pp on MedMCQA, drift bounded within ~1 point on the probes) are stated without any accompanying experimental protocol, baseline implementation details, hyperparameter settings, number of runs, or statistical tests. This absence is load-bearing because the central claim is an empirical comparison whose reliability cannot be assessed from the given information.

    Authors: We agree the abstract is too terse for a central empirical claim. In revision we will expand it to name the base model (Qwen-2.5-0.5B-Instruct), the adaptation task (MedMCQA), the two baselines, and the two forgetting probes, while directing readers to the experimental section for hyperparameters and implementation. The current results are from single runs; we will add an explicit limitation note on the absence of multiple seeds or statistical tests rather than fabricate new runs. revision: partial

  2. Referee: [Forgetting probes and row-selection] Forgetting probes and row-selection section: the claim that SMF is a low-forgetting alternative depends on WikiText perplexity and TriviaQA accuracy being adequate, unbiased proxies for retained general capabilities after MedMCQA adaptation. No justification, ablation against other metrics (e.g., reasoning or non-medical recall tasks), or sensitivity analysis is provided, so the sufficiency of these two probes for the distribution shift remains untested.

    Authors: We accept that the manuscript provides no explicit rationale for the probe selection. We will insert a short paragraph in the relevant section citing prior forgetting literature that uses language-modeling perplexity and factual-recall accuracy as standard proxies for general capability retention under domain shift. We will also note the limitation that no additional reasoning or non-medical tasks were evaluated. No new experiments are feasible within the current revision timeline, but the added discussion directly addresses the concern. revision: yes

Circularity Check

0 steps flagged

No derivation chain present; purely empirical comparison

full rationale

The paper reports an empirical re-implementation and comparison of Sparse Memory Finetuning (SMF) versus LoRA and full finetuning on the MedMCQA task, measuring task gains against WikiText perplexity and TriviaQA accuracy as forgetting probes. No equations, derivations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the described content. All results are direct experimental outcomes rather than reductions of any claimed first-principles chain to its own inputs, satisfying the default expectation of no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no equations, fitted constants, or background lemmas; the memory-layer construction is described at the level of a technique rather than an axiomatic system.

pith-pipeline@v0.9.1-grok · 5718 in / 1223 out tokens · 31020 ms · 2026-06-30T23:51:07.760832+00:00 · methodology

0 comments
read the original abstract

Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently.

Figures

Figures reproduced from arXiv: 2605.03229 by Anirudh Kanchi, Garv Shah, Prakhar Gupta, Satyam Goyal.

Figure 1
Figure 1. Figure 1: Method overview. Each Qwen-2.5 transformer block uses RMSNorm, grouped-query self-attention, and a SwiGLU MLP down proj(silu(gate proj(x)) ⊙ up proj(x)). We compare two ways of inserting a Hashing Memory Layer at selected layers: Replacement substitutes the MLP entirely (left); Additive keeps the MLP and adds a memory-scaled branch (right). The middle panel details the memory layer (Section 3.1): queries a… view at source ↗
Figure 2
Figure 2. Figure 2: Plasticity–stability frontier on MedMCQA. Each method appears as one point with cross-seed standard-deviation error bars. Color encodes the method family. For sparse methods, marker shape distinguishes the slot-selection rule (circle = KL, triangle = TF-IDF). Non-sparse baselines (Base Qwen, LoRA, Full finetune) appear as their own labeled points. Top-left of the left panel and top-right of the right panel… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

7 extracted references · 7 canonical work pages

  1. [1]

    Memory Layers at Scale

    Berges, V .-P., O˘guz, B., Haziza, D., Yih, W.-t., Zettlemoyer, L., and Ghosh, G. Memory layers at scale, 2024. URL https://arxiv.org/abs/2412.09764

    work page Pith review arXiv 2024

  2. [2]

    J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations, 2022

    work page 2022

  3. [3]

    S., and Zettlemoyer, L

    Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Trivi- aQA: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017. K¨opf, A., Kilcher, Y ., von R ¨utte, D., et al. OpenAssis- tant conversations – democratizing large language mode...

    work page 2017

  4. [4]

    Large memory layers with product keys

    Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., and J´egou, H. Large memory layers with product keys. InAdvances in Neural Information Processing Systems, 2019

    work page 2019

  5. [5]

    Lin, J. et al. Continual learning via sparse memory finetun- ing, 2025. URL https://arxiv.org/abs/2510. 15103

    work page 2025

  6. [6]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. InInternational Conference on Learning Representations, 2017

    work page 2017

  7. [7]

    Trained params

    Pal, A., Umapathi, L. K., and Sankarasubbu, M. MedM- CQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. InProceedings of the Conference on Health, Inference, and Learning, pp. 248–260. PMLR, 2022. Qwen Team. Qwen2.5 technical report, 2024. 5 Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full...

    work page 2022