Pith. sign in

REVIEW 3 major objections 4 minor 31 references

Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Visual instruction tuning makes an LLM see but forget text; this paper claims a neuron-level, training-free merge can keep both, beating five merging methods over 13 benchmarks.

desk verdict Useful neuron-level merging recipe for MLLMs, but 'consistent outperformance' is contradicted by the paper's own per-dataset tables; the gains are small and the claims need scaling back. read the letter →

arxiv 2505.16703 v1 pith:R6HNECFR submitted 2025-05-22 cs.CL

classification cs.CL
keywords catastrophicforgettingmodelmergingmultimodallargelanguagemodelsneuron-levelparameterfusionvisualinstructiontuningtaskvectorneuronlocalizationcontexthallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual instruction tuning gives a large language model new image-understanding skills but typically erodes part of its original language ability. This paper proposes a training-free fix: compare the tuned model's weights with the base model's, score each neuron by the total size of its weight change, keep the neurons that changed most as carriers of the visual capability, and suppress the many small changes that supposedly disturb language skills. On seven language and six visual benchmarks, with two open-source 7–8B base models, the resulting Neuron-Fusion method claims to beat all five existing merging methods on overall accuracy. If that holds, it offers a cheap post-processing step that recovers most of the base model's language ability without giving back the visual gains, and it also reduces refusal and hallucination errors in generated answers.

What carries the argument

The carrying object is the task vector $\Delta = \theta_{\text{tuned}} - \theta_{\text{base}}$, reorganized neuron by neuron. Each neuron $i$ gets a change score $C(i) = \sum_j |\Delta_{i,j}|$ summing absolute weight differences across its associated rows and columns in FFN and attention matrices. Neuron-Fusion then runs three steps: Neuron-Locate selects the top $M\%$ of neurons by $C(i)$; Neuron-Suppress keeps only $K\%$ of parameters in each module (via TIES, Breadcrumbs, or Task Arithmetic-style trimming); Neuron-Restore reinstates the selected neurons either by directly replacing their parameters (Neuron-Replace) or by rescaling the surviving parameters so the neuron's total change score is preserved (Neuron-Rescale). The rescale step is justified through the key-value memory view of FFN layers: the neuron's influence is a coefficient $c_i = \sum_j x_j \Delta_{i,j}$, and rescaling by $C(i)/C'(i)$ approximately restores $c_i$ when input $x$ is roughly uniform.

What would settle it

Run the same 13-benchmark evaluation with Neuron-Restore applied to a randomly selected $M\%$ of neurons instead of the top-$M\%$ by change score; if the random version matches the visual accuracy, the localization claim is not the operative mechanism. Separately, record actual FFN coefficients $c_i$ on real visual inputs before and after the Rescale step to check whether the uniform-input assumption preserves them.

Watch

Extended reading notes

Core claim

The paper's central claim is that catastrophic forgetting after visual instruction tuning can be mitigated by a training-free, neuron-level parameter fusion built on the task vector $\Delta = \theta_{\text{tuned}} - \theta_{\text{base}}$. Each neuron $i$ is assigned a change score $C(i)=\sum_j |\Delta_{i,j}|$; the method keeps the top $M\%$ of neurons as important paths for visual adaptation, sparsifies each module down to $K\%$ of parameters to mute small changes, then restores the selected neurons either by direct replacement or by rescaling survivors to preserve each neuron's coefficient. Across experiments on Llama3- and Mistral-based MLLMs, the paper reports that Neuron-Fusion configurations such as Neu-P-TaskA and Neu-P-Bread reach the highest combined language-visual ability among all compared methods, recovering a substantial share of the base LLM's language accuracy while keeping most visual performance. A generation analysis on ScienceQA adds that the suppression stage mostly fixes 'Not-Known' refusal errors, while the restore stage corrects 97.2% of the context-hallucination errors it observes in that stage.

Load-bearing premise

The load-bearing premise is that the neurons with the largest total weight change during visual instruction tuning actually store the new visual capability, while the many small weight changes are what damage language ability; the paper validates this only indirectly through benchmark ablations.

Editorial extensions

If this is right

  • A training-free merge can serve as a drop-in post-processing step: replace the tuned LLM inside an MLLM with Neuron-Fusion's merged weights and keep the vision encoder and connector untouched.
  • Parameter-level merging methods are not equally effective for multimodal forgetting; methods that rely on indiscriminate dropout, such as DARE and DELLA, can end up with language ability below even the degraded MLLM, so neuron-level structure matters.
  • The Locate-then-Merge view unifies Task Arithmetic, TIES, Breadcrumbs, DARE, and DELLA as different choices of locating and merging functions, giving a common design space for future merging methods.
  • The two stages address different failure modes: the suppress stage mainly fixes refusal-type 'Not-Known' errors, while the restore stage mainly corrects context-hallucination errors.
  • Merging can be tuned with two interpretable hyperparameters, $K\%$ (suppression density) and $M\%$ (neuron restoration ratio), and a moderate $M$ gives the best vision-language balance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the large-change-neuron hypothesis generalizes, the same locate-then-merge recipe could be applied to other modality adaptations such as audio or video, or to any fine-tuned-to-base model pair; the paper's own limitations section leaves this open.
  • The uniform-input assumption behind Neuron-Rescale is likely too strong for real hidden states, so directly measuring coefficient distributions before and after rescaling on actual visual inputs would show how much of the restoration is approximation error.
  • Because the method is training-free, it could be combined with data-free hyperparameter selection, choosing $K$ and $M$ by probing on small held-out sets, to avoid per-task tuning.
  • A stronger causal test would compare restoration of the top-$M\%$ neurons against restoration of a random $M\%$ of neurons with the same parameter budget; if random restoration recovers comparable visual accuracy, the localization story is not the operative mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses catastrophic forgetting of language ability after visual instruction tuning in multimodal LLMs. It proposes a Locate-then-Merge framework, in which merging is written as θ_merge = θ_base + F(Sub(Δ)), and instantiates it as Neuron-Fusion: neurons are scored by their summed absolute parameter change (Eq. 2), the top M% of neurons are selected for restoration, a magnitude-based suppression step keeps K% of parameters, and either Neuron-Replace or Neuron-Rescale is applied. The method is evaluated on Llama3-8B and Mistral-7B over 7 language and 6 visual benchmarks, compared with Task Arithmetic, TIES, Breadcrumbs, DARE, and DELLA. The paper claims consistent superiority, reports small aggregate overall-ability gains, and presents a ScienceQA generation analysis attributing improvements to reduced Not-Known and Context-Hallucination errors.

Significance. If the aggregate results hold, the method is practically valuable because it is training-free, uses only the base and tuned checkpoints, and can be implemented with standard merge toolkits. Table 1's unification of existing merging methods under Sub(·) and F(·) is a useful expository contribution, and evaluation on two base LLMs over 13 benchmarks is more extensive than in many merging papers. However, the headline claim of consistent superiority is not supported by the paper's own per-dataset tables, the mechanistic derivation of Neuron-Rescale is incomplete, and the two free hyperparameters K and M are not validated on a separate split. The method may still be of interest as a marginal-average improvement, but the manuscript currently overstates both the empirical and mechanistic evidence.

major comments (3)
  1. [Abstract; §5.2; Table 5] The claim that "Neuron-Fusion consistently outperforms existing model merging methods" is not supported by the per-dataset results. For Llama3, the best variant Neu-P-TaskA is worse than Task Arithmetic on four of the six visual benchmarks: MME 71.0 vs 71.8, MMBench-EN 81.0 vs 81.8, MMBench-CN 74.2 vs 74.6, and GQA 58.2 vs 59.8; it wins only on MMMU and ScienceQA. The aggregate overall-ability advantage is 0.25 points on Llama3 and 0.26 points on Mistral, and no variance or statistical significance is reported. The central claim should be reduced to a marginal average improvement, or it should be supported by significance testing and a clearly separated validation protocol.
  2. [§4.3, Eq. (4)] The Neuron-Rescale restoration argument is incomplete. For a neuron with mixed-sign changes, the rescaled coefficient is R·Σ_{j∈S} x_j Δ_{i,j} with R = Σ_j |Δ_{i,j}| / Σ_{j∈S} |Δ_{i,j}|. Even under exactly uniform inputs, this equals the original coefficient only if Σ_{j∈S} Δ_{i,j} / Σ_{j∈S} |Δ_{i,j}| equals Σ_j Δ_{i,j} / Σ_j |Δ_{i,j}|, which the K%-magnitude subset does not guarantee in general. A magnitude rescaling can therefore distort, or even flip the sign of, the neuron's contribution. The paper should either prove a bound under stated assumptions or drop the claim that this rescaling approximately restores the original coefficient.
  3. [§5.2; Figures 6–8; Tables 2–3] K% and M% are free hyperparameters, and the paper does not describe any separate validation set. Figures 6–8 vary K and M while monitoring the same 13 benchmarks that are later used to report final overall ability, which makes the reported gains optimistic if any selection occurred on these test sets. Please disclose the selection procedure (for example, pre-registered defaults, a validation split, or selection on a separate suite) and state the sensitivity of the final overall-ability scores to reasonable perturbations of K and M.
minor comments (4)
  1. [§4.2; Figures 3–4] The statement that "similar trends are observed across other modules and layers" is not accompanied by quantitative evidence; Figures 3 and 4 show only one FFN layer and one attention layer. Reporting aggregate statistics across layers would make the claim reproducible.
  2. [§5.1; Table 5; References] The text says Llama3-8B but Table 5's caption says Llama3-7B, and the reference for Mistral-7B is incorrectly given as a Master's thesis rather than the model's technical report or model card.
  3. [§5.2; Generation analysis] The claim that Neuron-Fusion "effectively reduces context hallucination" is based only on corrected cases on ScienceQA, with no denominators for the reported percentages (56.7% and 97.2%) and no comparison on a dedicated hallucination benchmark; please report counts and, ideally, a hallucination-specific evaluation.
  4. [Table 1; §5.1] The hyperparameters used for each baseline are not reported (for example, the α in Task Arithmetic and the density K for TIES, Breadcrumbs, DARE, and DELLA), so the comparisons are not fully reproducible as described.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Neuron-Fusion result is an external empirical comparison, and the self-citations are motivational rather than load-bearing.

full rationale

The derivation chain is self-contained. Neuron-Fusion is defined directly from parameter deltas (Eqs. 2-5): C(i) aggregates absolute changes, top-M% neurons are restored, and small changes are suppressed; no step defines the benchmark outcome in terms of the method's own definitions or in terms of the authors' prior results. The headline claim is an empirical comparison on 13 external benchmarks, and the hypothesis that large-change neurons encode visual capabilities is tested by ablation rather than assumed as an identity. The only self-citations (Yu and Ananiadou 2023, 2024) are motivational and are corroborated by independent work (Dai et al., Geva et al., Schwettmann et al., Nikankin et al.), so they are not load-bearing. A validation caveat exists: K% and M% are selected from ablations on the same benchmarks used for final comparisons, which weakens the strength of the 'consistently outperforms' claim as an out-of-sample prediction; however, this is a test-set-selection issue, not a definitional circularity. The Rescale coefficient-restoration derivation (Eq. 4) also relies on an unstated same-sign assumption, but that is an assumption gap rather than a circular reduction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method itself is a heuristic combination of sparse task-vector merging and neuron selection; it introduces no new physical entities. The main free choices are the two ratios K and M. The claim rests on a mechanistic hypothesis about neuron-level encoding that is not independently established.

free parameters (2)
  • K (suppression density) = 0.2 (used in ablations; not stated for main results)
    Controls the fraction of parameter deltas retained in each module during Neuron-Suppress. The main tables do not report which K produced them.
  • M (neuron restoration ratio) = 0.3 (used in ablations)
    Controls the top fraction of neurons restored in Neuron-Restore. The paper does not state how this was chosen for the headline results.
assumptions (4)
  • domain assumption Neurons with large parameter shifts during visual tuning encode newly acquired visual capabilities, while widespread small shifts degrade language ability.
    Stated in Section 4.3 and Figure 1; it is the central mechanistic premise, supported only by post-hoc ablations on the same benchmarks.
  • domain assumption Suppressing a large fraction of small parameter changes per module does not destroy the base model's ability to use the retained visual information.
    Neuron-Suppress retains only K% of deltas; the paper assumes the surviving deltas plus restoration are sufficient for visual capability.
  • ad hoc to paper For the Rescale variant, FFN input vectors x have roughly uniform values across dimensions, making the rescaling factor approximately restore the original coefficient.
    Section 4.3 states 'If the input vector x has roughly uniform values, this rescaling approximately restores c_i.' This is an unverified simplifying assumption.
  • standard math The key-value memory interpretation of FFN layers from Geva et al. (2020) applies to the merged model.
    Used as the mechanistic basis for why restoring coefficients preserves neuron influence; taken from cited literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs." pith.science (2026). https://pith.science/paper/R6HNECFR

@misc{pith2026250516703,
  author       = {Pith},
  title        = {Pith review of: Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R6HNECFR}},
  note         = {Machine review of arXiv:2505.16703}
}
read the original abstract

Although multimodal large language models (MLLMs) have achieved impressive performance, the multimodal instruction tuning stage often causes catastrophic forgetting of the base LLM's language ability, even in strong models like Llama3. To address this, we propose Locate-then-Merge, a training-free parameter fusion framework that first locates important parameters and then selectively merges them. We further introduce Neuron-Fusion, a neuron-level strategy that preserves the influence of neurons with large parameter shifts--neurons likely responsible for newly acquired visual capabilities--while attenuating the influence of neurons with smaller changes that likely encode general-purpose language skills. This design enables better retention of visual adaptation while mitigating language degradation. Experiments on 13 benchmarks across both language and visual tasks show that Neuron-Fusion consistently outperforms existing model merging methods. Further analysis reveals that our method effectively reduces context hallucination in generation.

Figures

Figures reproduced from arXiv: 2505.16703 by the authors.

Figure 1
Figure 1. Neuron-Fusion in MLLMs. After visual tun [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The structures of LLM and MLLM. the changes introduced during visual instruction tuning. After tuning, the MLLM acquires a visual ability Vtuned, but its language ability degrades to Ltuned (typically Ltuned < Lbase) due to catastrophic forgetting. This degradation occurs because some parameters responsible for language capabilities are inadvertently modified during visual instruction tuning. A straightforward way t… view at source ↗
Figure 3
Figure 3. Change of neurons in FFN up matrix. large-change neuron small-change neuron [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Change of neurons in attention query matrix. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Change of coefficients after Neuron-Fusion. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Visual ability under different density K. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 6
Figure 6. Figure 6: Language ability under different density K. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Results when restoring TopM (%) neurons. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Accuracy of Llama3 (left) and Mistral (right) [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 11
Figure 11. Figure 11: Language ability under different density K. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Visual ability under different density K. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 4 canonical work pages

  1. [7]

    MohammadReza Davari and Eugene Belilovsky

    Knowledge neu- rons in pretrained transformers.arXiv preprint arXiv:2104.08696. MohammadReza Davari and Eugene Belilovsky

  2. [8]

    arXiv preprint arXiv:2406.11617

    Della-merging: Reducing interference in model merging through magnitude-based sampling. arXiv preprint arXiv:2406.11617. Jonathan Frankle and Michael Carbin

  3. [9]

    Chaoyou Fu, Peixian Chen, and Xunyang Shen

    The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635. Chaoyou Fu, Peixian Chen, and Xunyang Shen

  4. [10]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Bider- man, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Sc...

  5. [12]

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz

    Transformer feed-forward layers are key- value memories.arXiv preprint arXiv:2012.14913. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz

  6. [13]

    InProceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing: Industry Track, pages 477–485

    Arcee’s mergekit: A toolkit for merging large lan- guage models. InProceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing: Industry Track, pages 477–485. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio

  7. [15]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts- man, Suchin Gururangan, Ludwig Schmidt, Han- naneh Hajishirzi, and Ali Farhadi

    Gpt-4o system card.arXiv preprint arXiv:2410.21276. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts- man, Suchin Gururangan, Ludwig Schmidt, Han- naneh Hajishirzi, and Ali Farhadi

  8. [16]

    Fengqing Jiang

    Edit- ing models with task arithmetic.arXiv preprint arXiv:2212.04089. Fengqing Jiang

Show all 31 references
  1. [18]

    Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu

    A mechanistic understanding of alignment al- gorithms: A case study on dpo and toxicity.arXiv preprint arXiv:2401.01967. Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu

  2. [19]

    Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu

    Benchmark evaluations, applications, and challenges of large vision language models: A survey.arXiv preprint arXiv:2501.02189. Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu

  3. [20]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal

    An empirical study of catastrophic forgetting in large language mod- els during continual fine-tuning.arXiv preprint arXiv:2308.08747. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal

  4. [21]

    Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov

    Can a suit of armor conduct elec- tricity? a new dataset for open book question answer- ing.arXiv preprint arXiv:1809.02789. Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov

  5. [22]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others

    Arithmetic without algo- rithms: Language models solve math with a bag of heuristics.arXiv preprint arXiv:2410.21272. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others

  6. [23]

    Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba

    Training-free mitigation of language reasoning degradation after multimodal in- struction tuning.arXiv preprint arXiv:2412.03467. Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba

  7. [24]

    arXiv preprint arXiv:2002.05202

    Glu variants improve transformer. arXiv preprint arXiv:2002.05202. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant

  8. [25]

    Gido M Ven, Nicholas Soures, and Dhireesha Ku- dithipudi

    Commonsenseqa: A question answering challenge targeting commonsense knowl- edge.arXiv preprint arXiv:1811.00937. Gido M Ven, Nicholas Soures, and Dhireesha Ku- dithipudi

  9. [26]

    Continual learning and catastrophic forgetting.arXiv preprint arXiv:2403.05175. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkoong, Ali Farhadi, Yair Car- mon, Simon Kornblith, and 1 others

  10. [27]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal

    Deepseek-vl2: Mixture-of-experts vision- language models for advanced multimodal under- standing.arXiv preprint arXiv:2412.10302. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal

  11. [28]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others

    Ties-merging: Re- solving interference when merging models.Ad- vances in Neural Information Processing Systems, 36:7093–7115. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024a. Qwen2. 5...

  12. [29]

    arXiv preprint arXiv:2312.12141

    Neuron- level knowledge attribution in large language models. arXiv preprint arXiv:2312.12141. Zeping Yu and Sophia Ananiadou

  13. [30]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others

    Interpret- ing arithmetic mechanism in large language models through comparative neuron analysis.arXiv preprint arXiv:2409.14144. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others

  14. [31]

    A Detailed Results on All Datasets A.1 Catastrophic Forgetting in MLLMs Figure 9: Accuracy of Llama3 (left) and Mistral (right) on language datasets after visual instruction tuning

    Model tailor: Mitigating catastrophic forgetting in multi-modal large language models.arXiv preprint arXiv:2402.12048. A Detailed Results on All Datasets A.1 Catastrophic Forgetting in MLLMs Figure 9: Accuracy of Llama3 (left) and Mistral (right) on language datasets after vis...

  15. [2013]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others

    An em- pirical investigation of catastrophic forgetting in gradient-based neural networks.arXiv preprint arXiv:1312.6211. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaug...

  16. [2017]

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Watten- berg, Jonathan K Kummerfeld, and Rada Mihalcea

    Race: Large-scale reading comprehension dataset from examinations.arXiv preprint arXiv:1704.04683. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Watten- berg, Jonathan K Kummerfeld, and Rada Mihalcea

  17. [2018]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others

    Think you have solved question an- swering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others

  18. [2020]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others

  19. [2021]

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei

  20. [2022]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy

    Transformer feed-forward layers build predictions by promoting concepts in the vo- cabulary space.arXiv preprint arXiv:2203.14680. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy

  21. [2023]

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh

    Gemini: a family of highly capable multi- modal models.arXiv preprint arXiv:2312.11805. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh

  22. [2024]

    Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, An- drew M Dai, Anja Hauth, Katie Millican, and 1 oth- ers

    Mitigating catastrophic forget- ting in language transfer via model merging.arXiv preprint arXiv:2407.08699. Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, An- drew M Dai, Anja Hauth, Katie Millican, and 1 oth- ers

  23. [2025]

    5-vl technical report.arXiv preprint arXiv:2502.13923

    Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.