Machine Learning · arXiv:2611.16317 v1 · submitted 12 Nov 2026 · 12 references
Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization
The paper says
“Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical.”
more of the abstract
We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight–activation saliency to construct task-conditioned sparse perturbation masks while keeping the pretrained model dense. We analyze support quality through gradient energy capture and relate it to gradient estimation and convergence. Across 16 benchmarks with LLaMA and Qwen models ranging from 3B to 32B parameters, WIS-MeZO achieves higher aggregate scores than MeZO and reaches comparable validation losses in fewer optimization steps. Comparisons with random and magnitude-based sparse perturbations further highlight the importance of informative support selection.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. WIS-MeZO is a sensible incremental combination of Wanda and MeZO; the empirical work is solid but the theoretical justification has a gap that the authors acknowledge but do not resolve. This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 12 references say.
The paper says, at Section 5.2, Figure 2, Appendix C
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The mechanistic claim that WIS-MeZO's gains come from superior gradient-energy capture is validated empirically only on gradients from an auxiliary AdamW run (Figure 2 caption; Appendix C: '2,000 gradient snapshots from an auxiliary AdamW training run'), while Theorem 5.8 requires a deterministic lower bound ρ_min on ||M⊙∇f(W_t)||²_F / ||∇f(W_t)||²_F along the actual ZO trajectory, stated as holding almost surely for 0 ≤ t < T.
more of objection 1
The paper discloses the provenance of the gradients, but it does not discuss whether the ρ ordering transfers to ZO dynamics, whose updates are noisier and follow a different trajectory; as it stands, the diagnostics show that Wanda masks align with AdamW gradients near initialization, not that the theory's ρ_min is instantiated during WIS-MeZO training. Please measure ρ on actual ZO trajectories (even intermittently, on a few checkpoints per task) or explicitly reframe the diagnostics as step-0, auxiliary-trajectory evidence and soften the claimed connection.
The paper says, at Lemma 5.3, Theorem 5.4
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The quantitative content of the proxy bound is governed by κ_{i,ξ} = (γ/ν)(w_min,i / w_max,i)(1-ξ)/(1+ξ), and Theorem 5.4's guarantee ρ_WIS ≥ κξ ρ* is vacuous whenever κξ ρ* ≤ r. The constants γ/ν from Assumption 5.1 are unknown, may vary across layers and tasks, and are never estimated, so the paper establishes a qualitative ordering (Wanda ranking approximates gradient-energy ranking) but no verified quantitative advantage.
more of objection 2
Consider estimating the ratio on a few layers of a small model by comparing E[G²_{i,j}] with S²_{i,j} on the calibration set (cheap with backprop) and reporting the implied κξ, so that the sufficient condition can be checked rather than assumed.
The paper says, at Appendix B, Section 6.1
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The SuperGLUE results in Table 2 come from a single run (stated in Appendix B), yet Section 6.1 reports that WIS-MeZO improves the average score over MeZO, Random-MeZO, and Sparse-MeZO in all four model-benchmark combinations, and that periodic recalibration in WISD-MeZO increases the average score in all four, without hedging that two of those four averages rest on one seed.
more of objection 3
Given that the per-task differences in question (e.g., CB 96.4 versus 98.2, WSC 68.2 versus 71.1) are comparable to the ±1-3 point seed variation visible in Table 1, these claims should either be replicated with multiple seeds or explicitly flagged as single-run evidence.
4 minor objections
The paper says, at Appendix B, Tables 11-12
Pith says: minor objection 1 of 4
LOZO appears in the hyperparameter tables (learning rate, ε, rank, interval), but no LOZO results are reported anywhere in the paper; either include the results or remove the rows.
The paper says, at Tables 2 and 3
Pith says: minor objection 2 of 4
Several numeric entries are run together without separators (e.g., Table 2 Sparse-MeZO row '67.879.762.5' on LLaMA and '73.375.680.1' on Qwen; Table 3 MeZO 32B row '69.878.7'); these need reformatting.
show 2 more
The paper says, at Figure 2 and Figures 3-5
Pith says: minor objection 3 of 4
Figure 2 uses legend labels Sparse-small and Sparse-large, while its caption and Appendix C refer to W-small and W-large; the naming should be unified across figures and text.
The paper says, at Section 8
Pith says: minor objection 4 of 4
The paper has no limitations discussion; a short paragraph acknowledging the auxiliary-run gradient diagnostics, the unestimated theory constants, and the single-run SuperGLUE results would make the scope of the claims clearer.
What would settle it
Recompute the gradient energy capture ratio ρ for WIS, random, and magnitude-based masks using gradient snapshots collected during an actual WIS-MeZO training run (not AdamW) on at least one task (e.g., RTE or MNLI) with LLaMA-3.2-3B. Compare the resulting ρ values and ordering to those reported in Figure 2. If the ordering changes or the advantage of WIS diminishes, the central claim is weakened.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Hey,
This paper combines Wanda importance scores with MeZO to pick sparse perturbation coordinates for ZO fine-tuning. The idea is straightforward: use Wanda's weight-activation saliency to decide which parameters to perturb, rather than random or magnitude-based selection. The empirical evaluation is the main strength—16 benchmarks, models up to 32B, comparisons against MeZO, Sparse-MeZO, and Random-MeZO. The gains are modest but consistent, and the validation-loss curves show faster convergence. The memory and speed numbers are useful for practitioners.
The theoretical analysis is where I have concerns. The central quantity is gradient energy capture ρ, which measures how much of the gradient's Frobenius norm lies in the selected support. The paper proves that Wanda-based masks capture more energy than random masks under Assumptions 5.1 and 5.2. But the empirical validation of ρ (Figure 2 and Appendix C) is done on gradient snapshots from an auxiliary AdamW run, not during actual ZO training. The authors acknowledge this in the caption and text, but they don't discuss why this matters or provide evidence that the ordering holds under ZO dynamics. This is a real gap: ZO gradients are noisier and the loss landscape evolves differently, so the energy capture advantage might not transfer. The stress-test note is correct on this point.
Other soft spots: the improvements over baselines are sometimes within standard deviation (e.g., Qwen2.5-3B on GLUE/SQuAD, WIS-MeZO vs Sparse-MeZO), and the scaling results in Table 3 lack error bars. The hyperparameter search is extensive but could favor WIS-MeZO if its hyperparameters were tuned more carefully (the grid includes density ratio r, which is a free parameter for all sparse methods). The paper doesn't discuss failure cases or limitations.
That said, the paper is not circular, the empirical work is reproducible, and the method is practical. The theoretical gap is real but doesn't invalidate the empirical findings—it just means the explanation for why WIS-MeZO works is incomplete.
Who this is for: researchers working on memory-efficient fine-tuning, especially for on-device adaptation. The paper is worth a serious referee, but the authors should be asked to address the gradient energy measurement issue and add error bars to scaling results.
Recommendation: accept with moderate revision.
The strongest objection
The central theoretical and empirical justification for WIS-MeZO is that Wanda-based masks capture a higher fraction of gradient energy (ρ) than random or magnitude-based masks. However, the gradient energy measurements in Figure 2 and Appendix C are computed on gradient snapshots from an auxiliary AdamW training run, not from the actual ZO optimization trajectory. The paper acknowledges this in the caption ('from an auxiliary AdamW run') and in the text ('common mini-batch gradient snapshots from an auxiliary AdamW run'). This is a critical mismatch: the gradient structure during ZO optimization may differ substantially from that of first-order AdamW training, especially because ZO updates are noisier and the loss landscape evolves differently. The theoretical analysis (Lemma 5.3, Theorem 5.4) relates Wanda scores to expected gradient energy at a fixed parameter value, but the empirical validation of the key quantity ρ is performed on a different optimization trajectory. Without measuring ρ during actual ZO training, the paper's central claim that WIS-MeZO's advantage stems from superior gradient energy capture remains unsubstantiated. The authors do not discuss this limitation or provide any evidence that the gradient energy ordering holds under ZO dynamics.
The reader's weakest_assumption focused on Assumption 5.1 and the theoretical constants, which is a valid concern. However, I identify a more concrete and empirically testable issue: the gradient energy measurements, which are the primary empirical support for the theoretical analysis, are conducted on an auxiliary AdamW run rather than during the actual ZO optimization. This is a methodological gap that directly undermines the paper's central narrative linking Wanda scores to improved ZO performance. The paper does not acknowledge this as a limitation. The reader noted that 'the gradient energy capture analysis uses gradients from an auxiliary AdamW run, which may not reflect the actual ZO optimization dynamics,' so there is partial agreement. However, I elevate this to the single most load-bearing concern because it is more specific and testable than the theoretical assumption about constants. The concrete test I propose would directly address this gap. I recommend the verdict remain CONDITIONAL, as the paper is otherwise solid but this issue needs resolution before full acceptance.
The referee's report
If the results hold, this is a useful practical contribution to memory-efficient LLM fine-tuning: the method is simple (one forward calibration pass), keeps the base model dense, adds only a bit-packed mask (under 3 GB overhead at 32B), and its advantage over MeZO is demonstrated across 16 benchmarks, two model families, four scales, and three sparse-support baselines. The empirical design is a strength: hyperparameter grids and selected configurations are fully documented (Tables 11-13), GLUE/SQuAD results are averaged over five seeds, and the comparison against random and magnitude-based masks isolates the contribution of saliency-based selection. The gradient-energy capture framework, with the explicit sufficient condition κξρ* > r, is conceptually useful, and the energy diagnostics show a large margin (over 40% versus 5% captured energy at density 0.05).
Pith's reading of the paper (a machine's summary, not the authors')
This is a placeholder pith summary that should be replaced.
What carries the argument. This is a placeholder key machinery that should be replaced.
Core claim. This is a placeholder core discovery that should be replaced.
The premise it rests on. This is a placeholder weakest assumption.
If the paper is right (Pith's inferences, not the authors')
This is a placeholder forward implication.
This is a placeholder pith inference.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical. We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight–activation saliency to construct task-conditioned sparse perturbation masks while keeping the pretrained model dense. We analyze support quality through gradient energy capture and relate it to gradient estimation and convergence. Across 16 benchmarks with LLaMA and Qwen models ranging from 3B to 32B parameters, WIS-MeZO achieves higher aggregate scores than MeZO and reaches comparable validation losses in fewer optimization steps. Comparisons with random and magnitude-based sparse perturbations further highlight the importance of informative support selection.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 1.00
“Across 16 benchmarks with LLaMA and Qwen models ranging from 3B to 32B parameters, WIS-MeZO achieves higher aggregate scores than MeZO and reaches comparable validation losses in fewer optimization steps.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.12 · human participants 0.02 · controlled comparison 0.94 · randomized assignment 0.02 · states quantitative result 0.82 · compares with prior results 0.98 · mentions code or data availability 0.02
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. | background 0.99 | 0.06 | 0.03 | asserted 0.89 |
| Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical. | background 0.97 | 0.07 | 0.03 | asserted 1.00 |
| We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight--activation saliency to construct task-conditioned sparse perturbation masks while keeping the pretrained model dense. | contribution 0.51 | 0.94 | 0.14 | asserted 1.00 |
| We analyze support quality through gradient energy capture and relate it to gradient estimation and convergence. | contribution 0.73 | 0.86 | 0.04 | asserted 0.92 |
| Across 16 benchmarks with LLaMA and Qwen models ranging from 3B to 32B parameters, WIS-MeZO achieves higher aggregate scores than MeZO and reaches comparable validation losses in fewer optimization steps. | result 1.00 | 0.96 | 0.76 | asserted 1.00 |
| Comparisons with random and magnitude-based sparse perturbations further highlight the importance of informative support selection. | result 0.66 | 0.86 | 0.10 | asserted 0.99 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.86
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 4,927 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
References (12)
- [1] For each linear layer,W∈R m×n denotes its weight matrix,Z∈R m×n denotes the random perturbation, andM∈ {0,1}m×n denotes the corresponding binary mask. Seed replay regenerates the perturbation for the two loss evaluations and the update, retaining the memory advantages of MeZO.... on Pith
- [7] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391, on Pith
- [8] Checkpoints are selected by validation performance. We use a sequence length of 2048, constant learning rates, and zero weight decay for ZO methods, following the settings in MeZO and Sparse-MeZO [Malladi et al., 2023, Liu et al., 2025]. Tables 11–13 report the hyperparameter ... on Pith
- [10] Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, and Zheng Zhang. Mazo: Masked zeroth-order optimization for multi-task fine-tuning of large language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Languag... on Pith
- [2013] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, on PithPith's reviewarXiv
- [2015] Wei Lin, Yining Jiang, Qingyu Song, Qiao Xiang, and Hong Xu. Agzo: Activation-guided zeroth-order optimization for llm fine-tuning.arXiv preprint arXiv:2601.17261, on PithPith's reviewarXiv
- [2018] Taejong Joo, Wenhan Xia, Cheolmin Kim, Ming Zhang, and Eugene Ie. On surprising effectiveness of masking updates in adaptive optimizers.arXiv preprint arXiv:2602.15322, on PitharXiv
- [2019] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me... on PithPith's reviewarXiv
- [2022] Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training.arXiv preprint arXiv:2310.02025, on PithPith's reviewarXiv
- [2023] Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models.arXiv preprint arXiv:2404.08080, on PithPith's reviewarXiv
- [2024] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP... on Pith
- [2025] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment