Pith. sign in

REVIEW 2 major objections 4 minor 66 references

Weight-adjusted gradients flag a tiny set of LLM parameters whose masking collapses generation—failure modes that weight or gradient scores alone miss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 09:07 UTC pith:4CFLZ5XR

load-bearing objection Classical θ·∇ saliency product that, when extremes are masked, collapses generation far faster than weight or gradient alone; same score helps four LLM tasks, with the main caveat that rankings use eval-set gradients. the 2 major comments →

arxiv 2607.10803 v1 pith:4CFLZ5XR submitted 2026-07-12 cs.LG cs.AI

Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs

Shrestha Datta , Hongfu Liu , Anshuman Chhabra This is my paper
classification cs.LG cs.AI
keywords parameter importanceweight-adjusted gradientsmodel collapseLLM interpretabilitymixture-of-expertsmachine unlearningmixed-precision quantizationknowledge editing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that parameter importance in large language models cannot be read from weight magnitudes or from gradients alone. The product of a parameter’s value and its gradient—Weight-Adjusted Gradients, or WAG—isolates coordinates that dominate sensitivity to relative, multiplicative changes. Masking only a few thousand extreme-WAG parameters drives coding and math generation into repetition loops or empty function bodies, while the same number of largest-weight or largest-gradient parameters leaves the model fluent and correct. The authors show that WAG is exactly the gradient of the loss in log-parameter space and linearizes first-order loss change under multiplicative rescalings, framing trained transformers as scale-balanced systems whose fragile directions are the large-|WAG| coordinates. They then reuse the same scores for expert allocation, targeted unlearning, mixed-precision quantization, and layer choice in knowledge editing, often beating established baselines at low extra cost.

Core claim

WAG, defined as the negative product of each parameter and its first-order gradient, consistently isolates a sparse subset of coordinates whose targeted masking produces rapid collapse on free-form coding and math tasks—collapse that pure magnitude rankings and pure gradient rankings do not produce at the same sparsity. The metric equals the log-parameter gradient and exactly describes first-order loss change under multiplicative perturbations, so ranking by absolute WAG ranks the directions of greatest local scale sensitivity.

What carries the argument

Weight-Adjusted Gradients (WAG): WAGi = −θi ∂ℓ/∂θi. This is exactly the gradient of the loss with respect to ui = log|θi| and linearizes loss change under multiplicative rescalings θi → θi(1+εi). Ranking by |WAG| therefore ranks the most sensitive log-scale directions.

Load-bearing premise

The method assumes that gradients taken on the evaluation, forget, or proxy set, multiplied by the trained weights, correctly mark the parameters that would cause collapse or that should be preferentially updated or preserved—without showing those same coordinates stay critical under training-set gradients, random data, or after further fine-tuning.

What would settle it

Compute WAG on a held-out training split or on random tokens, mask the same number of highest-|WAG| coordinates, and check whether generation stays fluent and correct while evaluation-set WAG still collapses the model; or show that equal-size pure weight or pure gradient masks produce equally sharp collapse on the same tasks.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Masking a few thousand extreme-WAG parameters can collapse coding and math generation while equal-count weight or gradient masks do not.
  • Layers ranked by mean WAG can receive non-uniform expert budgets that beat fixed MoLA patterns and spectral baselines on zero-shot accuracy.
  • Restricting unlearning updates to the top 75% of WAG-scored layers can improve forget metrics and utility while cutting runtime.
  • Keeping the top 5–10% of WAG-scored submodules in full precision under FP8 quantization can match or exceed magnitude baselines.
  • Selecting a single MLP layer by mean |WAG| for locate-then-edit knowledge editing can raise overall edit scores relative to causal-mediation layer choice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If WAG tracks scale balance, the same sparse coordinates should remain critical after modest fine-tuning or under different data used for the gradient; that invariance is not yet shown.
  • The multiplicative view implies multiplicative noise or scale-aware regularizers may protect or attack these coordinates more effectively than additive noise of equal magnitude.
  • Collapse from masking a few thousand weights implies that redundancy measured by total parameter count can hide a sparse critical skeleton that standard pruning criteria systematically miss.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes Weight-Adjusted Gradients (WAG), defined as WAGi = −θi ∂ℓ/∂θi, as a parameter-importance score that multiplies trained weights by first-order gradients. Theorems 3.1–3.3 show that WAG is exactly the gradient in log-parameter coordinates and that it characterizes the first-order change in loss under multiplicative perturbations θ′i = θi(1 + εi). Empirically, masking a few thousand extreme-WAG coordinates (WAGLow, WAGHigh, or |WAG|) on LLaMA-3.2-3B, Qwen3-4B and Gemma3-12B produces rapid collapse on HumanEval/HumanEval+ and GSM8K, while magnitude- and gradient-only masks of the same size leave performance essentially intact (Figs. 1–4). The same ranking is then used for four applications—MoE expert allocation, targeted GradDiff unlearning, mixed-precision quantization, and R-ROME layer selection—where WAG-guided choices match or modestly outperform published baselines under mean±SEM over three seeds.

Significance. If the collapse phenomenon is robust, the work supplies a cheap, first-order diagnostic that existing magnitude and gradient saliency scores miss, together with a clean geometric interpretation (log-scale sensitivity) and four concrete control applications. The theorems are elementary but correctly proved; the generation examples make the failure mode vivid; and the application tables report proper multi-seed statistics. These strengths make the manuscript a useful contribution to LLM interpretability and efficient adaptation, provided the data-dependence of the ranking is clarified.

major comments (2)
  1. Sec. 3.3 and Apps. A–D compute all WAG scores from evaluation-set (or forget/proxy-set) gradients. The central claim that WAG reveals a model-intrinsic “tiny but critical subset” and a “fundamental structural property” therefore rests on an untested assumption: that the same coordinates remain extreme under training-set gradients, a disjoint calibration set, or random data. Without those ablations the collapse curves and the application gains could be artifacts of alignment with the particular loss surface used for ranking.
  2. Thm. 3.2 licenses only first-order multiplicative perturbations of size ε. The experiments instead hard-zero the selected parameters. Hard masking is a large, non-multiplicative change; the first-order theory therefore does not explain the observed collapse. Either a higher-order argument or an experiment that multiplies the extreme-WAG weights by (1+ε) for small ε is needed to close the gap between theory and the failure-mode claim.
minor comments (4)
  1. Figs. 1–2 lack error bars or multiple seeds, unlike the application tables; adding them would strengthen the collapse claim.
  2. The free parameters β (MoE power), ρ = 0.75 (unlearning), κ ∈ {5 %, 10 %} (quantization) and the 10 % proxy set are stated but never ablated; a short sensitivity paragraph would help.
  3. Notation for the three ranking criteria (WAGLow / WAGHigh / |WAG|) is introduced late and used inconsistently across sections; a single definition box would improve readability.
  4. Related-work discussion of SNIP, Optimal Brain Damage and influence-function layer scores could more explicitly contrast the multiplicative versus additive perturbation views.

Circularity Check

0 steps flagged

No significant circularity: WAG theorems are definitional rewrites of the ordinary gradient via chain rule/Taylor; collapse and application metrics are independent of the WAG formula. Only non-load-bearing methodological self-citations of prior layer protocols.

full rationale

The derivation chain is self-contained and non-circular. WAG is introduced by definition as WAGi = −θi ∂ℓ/∂θi (Eq. 1). Theorem 3.1 is the elementary chain-rule identity ∂ℓ/∂ui = θi ∂ℓ/∂θi under ui = log|θi|, so WAG equals the negative log-parameter gradient by construction; Theorem 3.2 is the first-order Taylor expansion of the loss under multiplicative perturbations θ′i = θi(1+εi), which immediately yields the same linear form; Theorem 3.3 is Cauchy–Schwarz applied to that expansion. None of these steps imports external results, fits free parameters later called predictions, or renames a known empirical pattern. The empirical collapse curves (Figs. 1–4) and all four applications measure free-form generation metrics (Pass@1, exact match, TOFU Model Utility/Forget TR/KS, EasyEdit Rewrite/Locality/Portability/Overall) that are independent of the WAG formula itself. Self-citations (LayerIF [3], Golden Layers [12], both sharing co-authors) appear only as “similar allocation procedure” or “proxy-set protocol” scaffolding; they do not justify the central claim that extreme-WAG coordinates cause collapse overlooked by magnitude/gradient baselines, nor do they supply a uniqueness theorem that forces the choice of WAG. Hard zeroing is not a small-ε multiplicative perturbation, but that is a correctness gap, not circularity. Score 1 reflects only the minor, non-load-bearing self-citations of methodological scaffolding.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

The central claims rest on ordinary calculus (chain rule, first-order Taylor) plus the modeling choice that multiplicative/log-scale sensitivity is the right notion of importance for trained transformers. A handful of application-specific free parameters (selection ratios, power exponents) are chosen by hand; no new physical or mathematical entities are postulated beyond the named metric itself.

free parameters (4)
  • β (power-transform exponent for MoE allocation)
    Controls dispersion of expert counts across layers; chosen by hand, not derived.
  • ρ = 0.75 (layer selection ratio for unlearning)
    Fraction of layers updated by GradDiff; fixed without ablation on other values.
  • κ ∈ {5 %, 10 %} (submodule preservation budget for quantization)
    Arbitrary full-precision budgets used to compare selection criteria.
  • proxy-set size 10 % (knowledge editing)
    Fraction of edit samples used to rank layers; not justified by theory.
axioms (3)
  • standard math First-order Taylor expansion of the loss under multiplicative parameter perturbations is a faithful local description of sensitivity.
    Used in Theorems 3.2–3.3; higher-order terms are discarded without remainder bounds for the large (masking) perturbations later applied.
  • domain assumption Trained transformers behave as scale-balanced systems whose fragility is captured by log-parameter gradients.
    Interpretive claim in §3.2.3 that motivates why WAG should locate collapse-inducing coordinates; not independently verified.
  • ad hoc to paper Gradients computed on the evaluation / forget / proxy set are sufficient to rank parameters for both collapse diagnosis and downstream control tasks.
    Operational choice throughout §3.3 and Applications A–D; no comparison to training-set or random-data gradients is supplied.
invented entities (2)
  • Weight-Adjusted Gradient (WAG) score no independent evidence
    purpose: Scalar importance metric combining weight magnitude and first-order gradient.
    Defined by Eq. (1); algebraically classical but elevated here to a primary object of study and a unified control signal.
  • scale-balanced system view of trained LLMs no independent evidence
    purpose: Interpretive frame claiming that optimization maintains a coordinated balance of parameter scales whose residual sensitivity is the WAG vector.
    Introduced in §3.2.3 to explain why high-|WAG| coordinates cause collapse; no external falsifiable prediction is given.

pith-pipeline@v1.1.0-grok45 · 27798 in / 3035 out tokens · 42125 ms · 2026-07-14T09:07:50.006519+00:00 · methodology

0 comments
read the original abstract

Understanding which parameters are influential in Large Language Models (LLMs) is central to improving their efficiency, reliability, and interpretability. We introduce Weight-Adjusted Gradients (WAG), a simple yet effective approach for estimating parameter importance that explicitly captures the interaction between model weights and first-order gradient information and identifies parameters that disproportionately influence model behavior, such as those responsible for collapse phenomena in LLMs. Across a range of models and settings, we show that WAG surfaces a tiny but critical subset of parameters whose modification leads to dramatic degradation in performance, a failure mode that existing importance metrics overlook. These findings reveal a previously underexplored interplay between weights and gradients, suggesting that parameter importance cannot be fully understood through either signal alone. The surprising effectiveness of WAG points to fundamental structural properties of trained networks and motivates new open questions about the role of zeroth-order and first-order information in deep learning. We demonstrate the practical utility of WAG across multiple applications, including expert allocation in mixture-of-expert architectures, parameter-specific unlearning, mixed-precision quantization, and layer selection for knowledge editing. Our results position WAG as a unified approach for analyzing, debugging, and controlling LLMs, and opens new directions for principled model-level interpretation.

Figures

Figures reproduced from arXiv: 2607.10803 by Anshuman Chhabra, Hongfu Liu, Shrestha Datta.

Figure 1
Figure 1. Figure 1: Visualization of performance degradation due to masking parameters across various models, LLaMA3.2-3B [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pass@1 and exact match accuracy evaluated on the HumanEval (A, B and C), HumanEval+ (D, E and F), and GSM8K (G, H and I) datasets for masked LLaMA3.2-3B (A, D and G), Qwen3-4B (B, E and H), and Gemma3-12B (C, F and I) models across different numbers of masked parameters, where parameters are selected based on having the largest-magnitude gradient → |Gradient|, weight → |Weight|, and WAG → |WAG| values. Mas… view at source ↗
Figure 3
Figure 3. Figure 3: Verbatim Qwen3-4B generations on HumanEval at 4,000 masked parameters. The WAGLow variant emits only empty docstrings; WAGHigh repeats the signature and docstring without a body; |WAG| loops on a single sentence. All six gradient- and weight-masked variants return the canonical solution. 4 Utilizing WAG in Diverse Applications We now demonstrate how WAG’s ability to characterize local sensitivity on loss u… view at source ↗
Figure 4
Figure 4. Figure 4: Verbatim Qwen3-4B generations on GSM8K at 8,000 masked parameters. WAGHigh and |WAG| masking produces pure token repetition (“1. . . ”, “Greg. . . ”); WAGLow masking stays fluent but miscounts. All six gradient- and weight-masked variants solve the problem. 4.1 Application A: WAG for Mixture-of-Experts Allocation We first apply WAG to the problem of allocating a fixed budget of N parallel experts across th… view at source ↗
Figure 5
Figure 5. Figure 5: Mixed-precision quantized model performance evaluated via [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 18 linked inside Pith

  1. [1]

    Memory Aware Synapses: Learning What (not) to Forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory Aware Synapses: Learning What (not) to Forget. InEuropean Conference on Computer Vision, 2018

    2018

  2. [2]

    Systematic Outliers in Large Language Models

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Systematic Outliers in Large Language Models. In International Conference on Learning Representations, 2025

    2025

  3. [3]

    LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions

    Hadi Askari, Shivanshu Gupta, Fei Wang, Anshuman Chhabra, and Muhao Chen. LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions. InAdvances in Neural Information Processing Systems, 2025

    2025

  4. [4]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization.arXiv preprint arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  5. [5]

    Exposing the Illusion of Erasure in Knowledge Editing for LLMs

    Advik Raj Basani and Anshuman Chhabra. Exposing the Illusion of Erasure in Knowledge Editing for LLMs. arXiv preprint arXiv:2606.23276, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, 2020

    2020

  7. [7]

    Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  8. [8]

    Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models

    Anshuman Chhabra, Bo Li, Jian Chen, Prasant Mohapatra, and Hongfu Liu. Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models. InInternational Conference on Machine Learning, 2025. 13 Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs

    2025

  9. [9]

    Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  10. [10]

    Evaluating the Ripple Effects of Knowledge Editing in Language Models.Transactions of the Association for Computational Linguistics, 2024

    Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the Ripple Effects of Knowledge Editing in Language Models.Transactions of the Association for Computational Linguistics, 2024

    2024

  11. [11]

    Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models.arXiv preprint arXiv:2311.04902, 2023

    Rocktim Jyoti Das, Mingjie Sun, Liqun Ma, and Zhiqiang Shen. Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models.arXiv preprint arXiv:2311.04902, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models via Layer Gradient Analysis.arXiv preprint arXiv:2602.20207, 2026

    Shrestha Datta, Hongfu Liu, and Anshuman Chhabra. Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models via Layer Gradient Analysis.arXiv preprint arXiv:2602.20207, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    Sharp Minima Can Generalize For Deep Nets

    Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp Minima Can Generalize For Deep Nets. InInternational Conference on Machine Learning, 2017

    2017

  14. [14]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. Automatically Constructing a Corpus of Sentential Paraphrases. In International Workshop on Paraphrasing, 2005

    2005

  15. [15]

    A Primer on the Inner Workings of Transformer-based Language Models.arXiv preprint arXiv:2405.00208, 2024

    Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-Jussà. A Primer on the Inner Workings of Transformer-based Language Models.arXiv preprint arXiv:2405.00208, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Sharpness-Aware Minimization for Efficiently Improving Generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-Aware Minimization for Efficiently Improving Generalization. InInternational Conference on Learning Representations, 2021

    2021

  17. [17]

    GPTQ: Accurate Post-Training Quantization for Generative Pretrained Transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pretrained Transformers. InInternational Conference on Learning Representations, 2023

    2023

  18. [18]

    The State of Sparsity in Deep Neural Networks.arXiv preprint arXiv:1902.09574, 2019

    Trevor Gale, Erich Elsen, and Sara Hooker. The State of Sparsity in Deep Neural Networks.arXiv preprint arXiv:1902.09574, 2019

    Pith/arXiv arXiv 1902

  19. [19]

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Ruibo Liu, Baochen Sun, Yawen Zhang, Daiyi Peng, Xiaoyuan Guo, and V . S. Subrahmanian. MoLA: MoE LoRA with Layer-wise Expert Allocation. InFindings of the North American Chapter of the Association for Computational Linguistics, 2025

    2025

  20. [20]

    The Language Model Evaluation Harness, 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The Languag...

    2024

  21. [21]

    Defying Catastrophic Forgetting via Influence Function.Artificial Intelligence, 2025

    Rui Gao and Weiwei Liu. Defying Catastrophic Forgetting via Influence Function.Artificial Intelligence, 2025

    2025

  22. [22]

    OLMES: A Standard for Language Model Evaluations

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A Standard for Language Model Evaluations. InFindings of the North American Chapter of the Association for Computational Linguistics, 2025

    2025

  23. [23]

    Rebuilding ROME: Resolving Model Collapse during Sequential Model Editing

    Akshat Gupta, Sidharth Baskaran, and Gopala Anumanchipalli. Rebuilding ROME: Resolving Model Collapse during Sequential Model Editing. InEmpirical Methods in Natural Language Processing, 2024

    2024

  24. [24]

    Learning both Weights and Connections for Efficient Neural Networks

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both Weights and Connections for Efficient Neural Networks. InAdvances in Neural Information Processing Systems, 2015

    2015

  25. [25]

    Stephen Jose Hanson and Lorien Y . Pratt. Comparing Biases for Minimal Network Construction with Back- Propagation. InAdvances in Neural Information Processing Systems, 1988

    1988

  26. [26]

    Aging with Grace: Lifelong Model Editing with Discrete K-Value Adaptors

    Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with Grace: Lifelong Model Editing with Discrete K-Value Adaptors. InAdvances in Neural Information Processing Systems, 2023

    2023

  27. [27]

    Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal Brain Surgeon. In Advances in Neural Information Processing Systems, 1992

    1992

  28. [28]

    Training Compute-Optimal Large Language Models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training Compute-Optimal Large Language Models. InAdvances in Neural Information Processing Systems, 2022

    2022

  29. [29]

    SliM-LLM: Salience-driven mixed-precision quantization for large language models

    Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. SliM-LLM: Salience-driven mixed-precision quantization for large language models. InInternational Conference on Machine Learning (ICML), 2025. 14 Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs

    2025

  30. [30]

    Scaling Laws for Neural Language Models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  31. [31]

    Overcoming Catastrophic Forgetting in Neural Networks.Proceedings of the National Academy of Sciences, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming Catastrophic Forgetting in Neural Networks.Proceedings of the National Academy of Sciences, 2017

    2017

  32. [32]

    Denker, and Sara A

    Yann LeCun, John S. Denker, and Sara A. Solla. Optimal Brain Damage. InAdvances in Neural Information Processing Systems, 1989

    1989

  33. [33]

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. SNIP: Single-shot Network Pruning based on Connection Sensitivity. InInternational Conference on Learning Representations, 2019

    2019

  34. [34]

    Zero-Shot Relation Extraction via Reading Comprehension

    Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-Shot Relation Extraction via Reading Comprehension. InConference on Computational Natural Language Learning, 2017

    2017

  35. [35]

    Continual Learning and Private Unlearning

    Bo Liu, Qiang Liu, and Peter Stone. Continual Learning and Private Unlearning. InConference on Lifelong Learning Agents, 2022

    2022

  36. [36]

    Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. InAdvances in Neural Information Processing Systems, 2023

    2023

  37. [37]

    Mahoney, and Yaoqing Yang

    Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, and Yaoqing Yang. AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models. In Advances in Neural Information Processing Systems, 2024

    2024

  38. [38]

    Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. InAdvances in Neural Information Processing Systems, 2022

    2022

  39. [39]

    Zico Kolter

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J. Zico Kolter. TOFU: A Task of Fictitious Unlearning for LLMs. InFirst Conference on Language Modeling, 2024

    2024

  40. [40]

    Locating and Editing Factual Associations in GPT

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. InAdvances in Neural Information Processing Systems, 2022

    2022

  41. [41]

    Large Language Models: A Survey.arXiv preprint arXiv:2402.06196, 2024

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large Language Models: A Survey.arXiv preprint arXiv:2402.06196, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Pruning Convolutional Neural Networks for Resource Efficient Inference

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning Convolutional Neural Networks for Resource Efficient Inference. InInternational Conference on Learning Representations, 2017

    2017

  43. [43]

    Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling.arXiv preprint arXiv:2604.25098, 2026

    Ocean Monjur, Shahriar Kabir Nahin, and Anshuman Chhabra. Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling.arXiv preprint arXiv:2604.25098, 2026

    Pith/arXiv arXiv 2026

  44. [44]

    GPT-4 Technical Report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. GPT-4 Technical Report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality

    Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush V osoughi. AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality. InEmpirical Methods in Natural Language Processing, 2024

    2024

  46. [46]

    How Many Parameters Does Your Task Really Need? Task Specific Pruning with LLM-Sieve.arXiv preprint arXiv:2505.18350, 2025

    Waleed Reda, Abhinav Jangda, and Krishna Chintalapudi. How Many Parameters Does Your Task Really Need? Task Specific Pruning with LLM-Sieve.arXiv preprint arXiv:2505.18350, 2025

    arXiv 2025

  47. [47]

    PB-LLM: Partially binarized large language models

    Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. PB-LLM: Partially binarized large language models. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  48. [48]

    Understanding Performance Collapse in Layer- Pruned Large Language Models via Decision Representation Transitions.arXiv preprint arXiv:2605.07271, 2026

    Boyu Shi, Chang Liu, ChuanBao Gao, Xu Yang, and Xin Geng. Understanding Performance Collapse in Layer- Pruned Large Language Models via Decision Representation Transitions.arXiv preprint arXiv:2605.07271, 2026

    Pith/arXiv arXiv 2026

  49. [49]

    The Curse of Recursion: Training on Generated Data Makes Models Forget.arXiv preprint arXiv:2305.17493, 2023

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The Curse of Recursion: Training on Generated Data Makes Models Forget.arXiv preprint arXiv:2305.17493, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A Simple and Effective Pruning Approach for Large Language Models. InInternational Conference on Learning Representations, 2024

    2024

  51. [51]

    Optimal Brain Apoptosis

    Mingyuan Sun, Zheng Fang, Jiaxu Wang, Junjie Jiang, Delei Kong, Chenming Hu, Yuetong Fang, and Renjing Xu. Optimal Brain Apoptosis. InInternational Conference on Learning Representations, 2025. 15 Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs

    2025

  52. [52]

    Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    LLaMA: Open and Efficient Foundation Language Models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation

    Dmytro Vitel and Anshuman Chhabra. First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation. InInternational Conference on Learning Representations, 2026

    2026

  55. [55]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A Multi-task Benchmark and Analysis Platform for Natural Language Understanding. InBlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2018

    2018

  56. [56]

    Why Language Models Collapse when Trained on Recursively Generated Text.arXiv preprint arXiv:2412, 2024

    Lecheng Wang, Xianjie Shi, Ge Li, Jia Li, Yihong Dong, Xuanming Zhang, Wenpin Jiao, and Hong Mei. Why Language Models Collapse when Trained on Recursively Generated Text.arXiv preprint arXiv:2412, 2024

    2024

  57. [57]

    EasyEdit: An Easy-to-Use Knowledge Editing Framework for Large Language Models

    Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, et al. EasyEdit: An Easy-to-Use Knowledge Editing Framework for Large Language Models. InAssociation for Computational Linguistics, 2024

    2024

  58. [58]

    Qwen3 Technical Report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen3 Technical Report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, et al

    Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M. Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, et al. Wanda++: Pruning Large Language Models via Regional Gradients. InFindings of the Association for Computational Linguistics, 2025

    2025

  60. [60]

    Editing Large Language Models: Problems, Methods, and Opportunities

    Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing Large Language Models: Problems, Methods, and Opportunities. InEmpirical Methods in Natural Language Processing, 2023

    2023

  61. [61]

    Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity

    Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity. InInternational Conference on Machine Learning, 2024

    2024

  62. [62]

    Continual Learning Through Synaptic Intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual Learning Through Synaptic Intelligence. In International Conference on Machine Learning, 2017

    2017

  63. [63]

    Boosting Large Language Models with Mask Fine-Tuning.arXiv preprint arXiv:2503.22764, 2025

    Mingyuan Zhang, Yue Bai, Huan Wang, Yizhou Wang, Qihua Dong, Yitian Zhang, and Yun Fu. Boosting Large Language Models with Mask Fine-Tuning.arXiv preprint arXiv:2503.22764, 2025

    arXiv 2025

  64. [64]

    A Comprehensive Study of Knowledge Editing for Large Language Models

    Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A Comprehensive Study of Knowledge Editing for Large Language Models. arXiv preprint arXiv:2401.01286, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    SparseSwaps: Tractable LLM Pruning Mask Refinement at Scale.arXiv preprint arXiv:2512.10922, 2025

    Max Zimmer, Christophe Roux, Moritz Wagner, Deborah Hendrych, and Sebastian Pokutta. SparseSwaps: Tractable LLM Pruning Mask Refinement at Scale.arXiv preprint arXiv:2512.10922, 2025. 16 Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs Appendix In the Appendix, we provide the proofs of the theoretical analysis in Appendix A....

    arXiv 2025

  66. [66]

    codebases, respectively. Our codebase for Application D was built upon theEasyEdit[ 57] framework and LGA codebase [12], and evaluations of the edited models were performed following their standard conventions. 21