{"id":"8ac5b723-e620-46fc-9d8a-35a4b5bde192","arxiv_id":"2611.16317","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"WIS-MeZO uses Wanda saliency scores to construct task-conditioned sparse perturbation masks for zeroth-order optimization, achieving higher aggregate scores than MeZO across 16 benchmarks with LLaMA and Qwen models up to 32B parameters.","lead":"The paper proposes WIS-MeZO, a method that uses Wanda weight-activation saliency scores to select which parameters to perturb during zeroth-order fine-tuning of large language models. This approach aims to improve the efficiency of memory-constrained fine-tuning by focusing perturbations on informative coordinates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Gradient energy capture is measured on an auxiliary AdamW run, not during ZO training, so the claimed advantage may not hold under the actual optimization dynamics.","rationale":"The reader's weakest_assumption focused on Assumption 5.1 and the theoretical constants, which is a valid concern. However, I identify a more concrete and empirically testable issue: the gradient energy measurements, which are the primary empirical support for the theoretical analysis, are conducted on an auxiliary AdamW run rather than during the actual ZO optimization. This is a methodological gap that directly undermines the paper's central narrative linking Wanda scores to improved ZO performance. The paper does not acknowledge this as a limitation. The reader noted that 'the gradient energy capture analysis uses gradients from an auxiliary AdamW run, which may not reflect the actual ZO optimization dynamics,' so there is partial agreement. However, I elevate this to the single most load-bearing concern because it is more specific and testable than the theoretical assumption about constants. The concrete test I propose would directly address this gap. I recommend the verdict remain CONDITIONAL, as the paper is otherwise solid but this issue needs resolution before full acceptance.","tokens_in":22247,"tokens_out":634,"duration_ms":10241,"concrete_test":"Recompute the gradient energy capture ratio ρ for WIS, random, and magnitude-based masks using gradient snapshots collected during an actual WIS-MeZO training run (not AdamW) on at least one task (e.g., RTE or MNLI) with LLaMA-3.2-3B. Compare the resulting ρ values and ordering to those reported in Figure 2. If the ordering changes or the advantage of WIS diminishes, the central claim is weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central theoretical and empirical justification for WIS-MeZO is that Wanda-based masks capture a higher fraction of gradient energy (ρ) than random or magnitude-based masks. However, the gradient energy measurements in Figure 2 and Appendix C are computed on gradient snapshots from an auxiliary AdamW training run, not from the actual ZO optimization trajectory. The paper acknowledges this in the caption ('from an auxiliary AdamW run') and in the text ('common mini-batch gradient snapshots from an auxiliary AdamW run'). This is a critical mismatch: the gradient structure during ZO optimization may differ substantially from that of first-order AdamW training, especially because ZO updates are noisier and the loss landscape evolves differently. The theoretical analysis (Lemma 5.3, Theorem 5.4) relates Wanda scores to expected gradient energy at a fixed parameter value, but the empirical validation of the key quantity ρ is performed on a different optimization trajectory. Without measuring ρ during actual ZO training, the paper's central claim that WIS-MeZO's advantage stems from superior gradient energy capture remains unsubstantiated. The authors do not discuss this limitation or provide any evidence that the gradient energy ordering holds under ZO dynamics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes WIS-MeZO, a zeroth-order (ZO) fine-tuning method that constructs sparse perturbation masks from Wanda weight-activation saliency scores computed on a small downstream calibration set, while keeping the pretrained model dense. A dynamic variant, WISD-MeZO, periodically refreshes the mask during training. The contributions are (i) this task-conditioned sparsity scheme, (ii) a theoretical analysis relating Wanda scores to expected gradient energy (Lemma 5.3, Theorem 5.4), connecting captured gradient energy to estimator alignment (Lemma 5.5) and to nonconvex finite-difference convergence (Theorem 5.8, plus a PL-condition variant), and (iii) an empirical evaluation on five GLUE tasks, SQuAD, five SuperGLUE tasks, and five commonsense/QA benchmarks using LLaMA-3.2-3B and Qwen2.5-3B through 32B, with memory, throughput, and wall-clock measurements. The headline claim is that WIS-MeZO improves aggregate scores over MeZO in all reported model-benchmark combinations and reaches comparable validation loss in fewer optimization steps.","tokens_in":22534,"tokens_out":10727,"duration_ms":67966,"significance":"If the results hold, this is a useful practical contribution to memory-efficient LLM fine-tuning: the method is simple (one forward calibration pass), keeps the base model dense, adds only a bit-packed mask (under 3 GB overhead at 32B), and its advantage over MeZO is demonstrated across 16 benchmarks, two model families, four scales, and three sparse-support baselines. The empirical design is a strength: hyperparameter grids and selected configurations are fully documented (Tables 11-13), GLUE/SQuAD results are averaged over five seeds, and the comparison against random and magnitude-based masks isolates the contribution of saliency-based selection. The gradient-energy capture framework, with the explicit sufficient condition κξρ* > r, is conceptually useful, and the energy diagnostics show a large margin (over 40% versus 5% captured energy at density 0.05).","major_comments":[{"comment":"The mechanistic claim that WIS-MeZO's gains come from superior gradient-energy capture is validated empirically only on gradients from an auxiliary AdamW run (Figure 2 caption; Appendix C: '2,000 gradient snapshots from an auxiliary AdamW training run'), while Theorem 5.8 requires a deterministic lower bound ρ_min on ||M⊙∇f(W_t)||²_F / ||∇f(W_t)||²_F along the actual ZO trajectory, stated as holding almost surely for 0 ≤ t < T. The paper discloses the provenance of the gradients, but it does not discuss whether the ρ ordering transfers to ZO dynamics, whose updates are noisier and follow a different trajectory; as it stands, the diagnostics show that Wanda masks align with AdamW gradients near initialization, not that the theory's ρ_min is instantiated during WIS-MeZO training. Please measure ρ on actual ZO trajectories (even intermittently, on a few checkpoints per task) or explicitly reframe the diagnostics as step-0, auxiliary-trajectory evidence and soften the claimed connection.","section":"Section 5.2, Figure 2, Appendix C","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The quantitative content of the proxy bound is governed by κ_{i,ξ} = (γ/ν)(w_min,i / w_max,i)(1-ξ)/(1+ξ), and Theorem 5.4's guarantee ρ_WIS ≥ κξ ρ* is vacuous whenever κξ ρ* ≤ r. The constants γ/ν from Assumption 5.1 are unknown, may vary across layers and tasks, and are never estimated, so the paper establishes a qualitative ordering (Wanda ranking approximates gradient-energy ranking) but no verified quantitative advantage. Consider estimating the ratio on a few layers of a small model by comparing E[G²_{i,j}] with S²_{i,j} on the calibration set (cheap with backprop) and reporting the implied κξ, so that the sufficient condition can be checked rather than assumed.","section":"Lemma 5.3, Theorem 5.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The SuperGLUE results in Table 2 come from a single run (stated in Appendix B), yet Section 6.1 reports that WIS-MeZO improves the average score over MeZO, Random-MeZO, and Sparse-MeZO in all four model-benchmark combinations, and that periodic recalibration in WISD-MeZO increases the average score in all four, without hedging that two of those four averages rest on one seed. Given that the per-task differences in question (e.g., CB 96.4 versus 98.2, WSC 68.2 versus 71.1) are comparable to the ±1-3 point seed variation visible in Table 1, these claims should either be replicated with multiple seeds or explicitly flagged as single-run evidence.","section":"Appendix B, Section 6.1","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"LOZO appears in the hyperparameter tables (learning rate, ε, rank, interval), but no LOZO results are reported anywhere in the paper; either include the results or remove the rows.","section":"Appendix B, Tables 11-12","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Several numeric entries are run together without separators (e.g., Table 2 Sparse-MeZO row '67.879.762.5' on LLaMA and '73.375.680.1' on Qwen; Table 3 MeZO 32B row '69.878.7'); these need reformatting.","section":"Tables 2 and 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 2 uses legend labels Sparse-small and Sparse-large, while its caption and Appendix C refer to W-small and W-large; the naming should be unified across figures and text.","section":"Figure 2 and Figures 3-5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper has no limitations discussion; a short paragraph acknowledging the auxiliary-run gradient diagnostics, the unestimated theory constants, and the single-run SuperGLUE results would make the scope of the claims clearer.","section":"Section 8","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Hey,\n\nThis paper combines Wanda importance scores with MeZO to pick sparse perturbation coordinates for ZO fine-tuning. The idea is straightforward: use Wanda's weight-activation saliency to decide which parameters to perturb, rather than random or magnitude-based selection. The empirical evaluation is the main strength—16 benchmarks, models up to 32B, comparisons against MeZO, Sparse-MeZO, and Random-MeZO. The gains are modest but consistent, and the validation-loss curves show faster convergence. The memory and speed numbers are useful for practitioners.\n\nThe theoretical analysis is where I have concerns. The central quantity is gradient energy capture ρ, which measures how much of the gradient's Frobenius norm lies in the selected support. The paper proves that Wanda-based masks capture more energy than random masks under Assumptions 5.1 and 5.2. But the empirical validation of ρ (Figure 2 and Appendix C) is done on gradient snapshots from an auxiliary AdamW run, not during actual ZO training. The authors acknowledge this in the caption and text, but they don't discuss why this matters or provide evidence that the ordering holds under ZO dynamics. This is a real gap: ZO gradients are noisier and the loss landscape evolves differently, so the energy capture advantage might not transfer. The stress-test note is correct on this point.\n\nOther soft spots: the improvements over baselines are sometimes within standard deviation (e.g., Qwen2.5-3B on GLUE/SQuAD, WIS-MeZO vs Sparse-MeZO), and the scaling results in Table 3 lack error bars. The hyperparameter search is extensive but could favor WIS-MeZO if its hyperparameters were tuned more carefully (the grid includes density ratio r, which is a free parameter for all sparse methods). The paper doesn't discuss failure cases or limitations.\n\nThat said, the paper is not circular, the empirical work is reproducible, and the method is practical. The theoretical gap is real but doesn't invalidate the empirical findings—it just means the explanation for why WIS-MeZO works is incomplete.\n\nWho this is for: researchers working on memory-efficient fine-tuning, especially for on-device adaptation. The paper is worth a serious referee, but the authors should be asked to address the gradient energy measurement issue and add error bars to scaling results.\n\nRecommendation: accept with moderate revision.","headline":"WIS-MeZO is a sensible incremental combination of Wanda and MeZO; the empirical work is solid but the theoretical justification has a gap that the authors acknowledge but do not resolve.","tokens_in":23133,"tokens_out":589,"duration_ms":9790},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"This is a placeholder headline that should be replaced.","keywords":["placeholder"],"falsifier":"This is a placeholder falsifier that should be replaced.","tokens_in":23630,"feed_emoji":"","tokens_out":201,"duration_ms":2907,"temperature":0.7,"pith_summary":"This is a placeholder pith summary that should be replaced.","feed_headline":"Placeholder headline","feed_subtitle":"Placeholder subtitle","key_machinery":"This is a placeholder key machinery that should be replaced.","core_discovery":"This is a placeholder core discovery that should be replaced.","pith_inferences":["This is a placeholder pith inference."],"forward_implications":["This is a placeholder forward implication."],"supporting_citations":[],"fun_headline_variants":["Saliency-driven sparse masks boost zeroth-order LLM fine-tuning","WIS-MeZO: Smarter perturbation selection for ZO optimization","Wanda saliency cuts search dimension in ZO fine-tuning","Informative sparsity speeds ZO optimization for LLMs","Task-conditioned perturbation masks improve ZO efficiency"],"cache_read_input_tokens":0,"weakest_assumption_plain":"This is a placeholder weakest assumption.","fun_headline_variants_meta":{"raw":{"variants":["Saliency-driven sparse masks boost zeroth-order LLM fine-tuning","WIS-MeZO: Smarter perturbation selection for ZO optimization","Wanda saliency cuts search dimension in ZO fine-tuning","Informative sparsity speeds ZO optimization for LLMs","Task-conditioned perturbation masks improve ZO efficiency"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":6.4e-05,"raw_usage":{"cost":6.43552e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":6.43552e-05,"upstream_inference_prompt_cost":4.94312e-05,"upstream_inference_completions_cost":1.4924e-05},"total_tokens":830,"prompt_tokens":748,"completion_tokens":82,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":748,"tokens_out":82,"cost_source":"openrouter_usage","duration_ms":2549,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T06:41:23.010046+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"This is a placeholder falsifier that should be replaced.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}