{"id":"7ee14588-dd1a-45fe-8b26-0d1a41c9ee68","arxiv_id":"2411.10696","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"HELENE is a proposed memory-efficient zeroth-order optimizer with annealed gradients and layer-wise clipped diagonal Hessian, claimed to converge in O(max layer dimension) steps.","lead":"This paper proposes a new optimizer called HELENE that combines zeroth-order gradient estimates with a diagonal Hessian preconditioner and layer-wise clipping to fine-tune large language models with low memory. The authors claim up to 20x faster convergence than the MeZO baseline, but the theoretical proof and the Hessian estimator contain serious errors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's O(max_i d_i) bound is proven for an idealized deterministic Newton flow, not for Algorithm 1: Lemma 4 contains an e^{-t} vs 1-t contradiction, and A-GNB is squared gradient, not Hessian.","rationale":"The reader's rejection is supported. The A-GNB estimator concern is valid and important: the squared mini-batch gradient is the empirical Fisher, not the Hessian diagonal, so the proposed preconditioner is not genuinely second-order in general. I diverge by elevating the proof-to-algorithm gap as the most load-bearing issue, because even if A-GNB were replaced by an exact Hessian diagonal estimator, Theorem 1 still would not follow: the proof analyzes a deterministic continuous Newton flow with exact gradients and full Hessians, none of which appear in Algorithm 1. The internal contradiction in Lemma 4 (e^{-t} versus 1-t) is a concrete symptom that the proof is not merely missing details but is reasoning about a different object. The empirical claims may still indicate practical value, but the central theoretical claim of O(max_i d_i) convergence is unsupported as written, and the numerical speedup claims are not independently checkable without code or data. No change to the reader's REJECT verdict is warranted.","tokens_in":1063,"tokens_out":1326,"duration_ms":73428,"concrete_test":"Re-derive Lemma 10, specifically equation (24), from Algorithm 1's update rule: theta_{t+1,i} = theta_{t,i} - eta (beta1 m_{t-1} + alpha_t g_t^{SPSA}) / (gamma max(h_{t,i}, lambda_i) + epsilon), with h_t computed by A-GNB as in Algorithm 2. Substitute this actual update into the Taylor expansion of the loss instead of the idealized theta_{t+1,i} = theta_{t,i} - eta V^T clip(V Hessian^{-1} V^T gradient L, lambda_i). If the resulting inequality contains extra SPSA noise terms, EMA bias, or A-GNB estimation error and does not reduce to the clean quadratic descent of Lemma 10, then Theorem 1 does not apply to HELENE. A numerical companion check on a small strictly convex logistic regression problem: compare the full-batch A-GNB value with the exact diagonal Hessian; if they differ, the algorithm is not preconditioning by true curvature.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing concern is that Theorem 1's convergence proof is not a proof about the optimizer defined in Algorithm 1. Lemma 4 studies the continuous Newton ODE dtheta_i/dt = -(Hessian L(theta_i))^{-1} gradient L(theta_i), concludes that gradient L(theta_i(t)) = e^{-t} gradient L(theta_i), and then asserts that this aligns perfectly with gradient L(theta_i(t)) = (1-t) gradient L(theta_i). Since e^{-t} does not equal 1-t, Lemma 4 is internally inconsistent. Even setting that aside, Lemmas 5-11 and Theorem 2 bound an idealized full-Hessian, exact-gradient Newton update. Algorithm 1 instead updates each layer by theta_{t+1,i} = theta_{t,i} - eta m_{t,i} / (gamma max(h_{t,i}, lambda_i) + epsilon), where m_t = beta1 m_{t-1} + alpha_t g_t^{SPSA} and h_t is the A-GNB estimate. None of the SPSA perturbation noise, EMA, annealing schedule, diagonal-only restriction, or the max(h,lambda) clipping appears in the proof's descent analysis; Lemma 10 uses a different clip on V H^{-1} V^T gradient L, not the algorithm's update. Thus the claimed O(max_i d_i) bound has not been connected to the actual method. The A-GNB estimator independently severs the theorem's premise: Algorithm 2 returns B times the squared gradient, so as B approaches infinity it approaches the squared expected gradient, which is zero at any minimizer, not the diagonal of the true Hessian. For cross-entropy, the gradient outer product differs from the Hessian of the loss with respect to logits. Therefore, neither the preconditioner nor the proof supports the claimed Hessian-based convergence rate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes HELENE, a zeroth-order (ZO) optimizer for fine-tuning large language models. The method combines SPSA-based gradient estimates with an annealed exponential moving average, a diagonal \"Asymptotic Gauss-Newton-Bartlett (A-GNB)\" preconditioner, and per-layer clipping of the estimated Hessian diagonal. The central theoretical claim (Theorem 1/2) is that HELENE converges in O(max_i d_i) steps, where d_i is the dimension of layer i, instead of O(d) steps. The empirical section reports up to 20x speedup over MeZO on RoBERTa-large and OPT-1.3B, with an average accuracy improvement of 1.5%. The appendix contains the proofs and ablation studies.","tokens_in":22828,"tokens_out":8946,"duration_ms":84648,"significance":"A valid ZO optimizer with a curvature-aware preconditioner and a convergence rate depending on the largest layer rather than the full parameter dimension would be a solid contribution to memory-efficient LLM fine-tuning. The paper has useful experimental scaffolding: it follows the MeZO evaluation protocol, includes PEFT variants, compares against several ZO baselines, and provides ablations that isolate momentum, annealing, and clipping. Those strengths do not, however, carry the central claims: the convergence proof is not about Algorithm 1, the A-GNB estimator is not a Hessian estimator, and the headline speedup is stated inconsistently. Because the main advertised results depend on these points, the contribution is currently not established.","major_comments":[{"comment":"Lemma 4 assumes that ∇L(θ_i(t)) = (1-t)∇L(θ_i), but the proof of the same lemma derives from the Newton ODE that d/dt ∇L(θ_i(t)) = -∇L(θ_i(t)), hence ∇L(θ_i(t)) = e^{-t}∇L(θ_i), and then states that the two expressions \"align perfectly.\" Since e^{-t} is not equal to 1-t, the lemma is internally inconsistent. This is not a cosmetic issue: Lemma 5, Lemma 8, and the integration steps in Theorem 2 use the (1-t) factor, so the subsequent convergence argument is invalid.","section":"Appendix C, Lemma 4"},{"comment":"The descent analysis concerns the update θ_i^+ = θ_i - η V_i^T clip(V_i ∇²L(θ_i)^{-1} V_i^T ∇L(θ_i), λ_i), i.e., an exact-gradient Newton step with a full-Hessian eigendecomposition. Algorithm 1 instead updates θ_{t+1,i} = θ_{t,i} - η_t m_{t,i}/(γ max(h_{t,i},λ_i)+ε), where m_t is an annealed EMA of SPSA gradient estimates and h_t is the A-GNB diagonal estimate. None of the SPSA perturbation noise, the EMA/annealing, the diagonal restriction, or the max(h,λ) clipping appears in Lemmas 5-11. Therefore Theorem 1 does not provide a convergence guarantee for the algorithm that the paper actually implements and evaluates.","section":"Section 4 / Appendix C, Lemmas 10-11"},{"comment":"Algorithm 2 computes \\hat{g} = ∇(1/B Σ_b L(f(φ(θ,x_b),y_b))) and returns B·\\hat{g}⊙\\hat{g}. With individual gradients g_b, this equals (1/B)(Σ_b g_b)⊙(Σ_b g_b). For B>1 this is not the average outer product (1/B)Σ_b g_b⊙g_b, and as B→∞ it diverges whenever E[g]≠0; at a stationary point it tends to the variance of g, not to diag(∇²L). The asserted identity lim_{B→∞} B·\\hat{g}⊙\\hat{g} = ∇²_θ L(θ) is therefore false. Moreover, the Gauss-Newton matrix is J^T S J with S=∇²_t L, whereas the estimator is built from J^T (∇_t L)(∇_t L)^T J; these coincide only under special distributional assumptions that are not stated. Hence the claimed second-order preconditioner is not a Hessian diagonal.","section":"Section 3.4 / Algorithm 2"},{"comment":"The empirical speedup claim is inconsistent: the Abstract and Section 5.1 claim up to 20× speedup over MeZO, while the caption of Figure 3 and Section 5.2 state \"approximate 10× speedup\" for OPT-1.3B. Similarly, the claim of 1.5% average accuracy improvement is not visible from Table 1 on several datasets (e.g., SST-5: HELENE 44.7 vs MeZO 45.5). A headline quantitative claim should be computed and reported with a consistent definition of steps-to-convergence.","section":"Section 5"}],"minor_comments":[{"comment":"The notation for ε is confusing: the algorithm input lists \"ϵ, ϵ\" twice, line 14 uses ε for weight decay, and line 15 uses ϵ as the denominator constant, while Section 2.1 uses ε for the SPSA perturbation scale; please disambiguate these quantities.","section":"Algorithm 1"},{"comment":"There are several typos: \"Laywerwise\" in the Section 3.5 heading, \"Garlett\" for \"Bartlett\" in Section 3, and \"incured\" in Section 3.2.","section":"Section 3.4 heading and body"},{"comment":"In Phase 2 of the proof, the contraction factor with η=1/2 is (1-η(1-η)) = 3/4, so the number of steps is ln(...)/ln(4/3), not 2 ln(...) as stated; the simplification needs correction should the theorem be revisited.","section":"Appendix C, Theorem 2 proof"}],"recommendation":"reject","confidential_remarks":"Reject. The central theoretical claim is disconnected from Algorithm 1, the A-GNB estimator identity is false and even divergent under the stated scaling, and the headline 20× speedup is contradicted elsewhere in the paper by 10×. A revision would require a substantially new theory for the actual stochastic algorithm and a re-framing of the preconditioner as a heuristic rather than a Hessian, which is beyond a normal revision. I did not factor the \"Under Review\" banner into the assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Name],\n\nQuick take: the paper has a plausible heuristic for accelerating zeroth-order fine-tuning, but the theoretical analysis is invalid and the empirical speedup numbers are inconsistent. I would not rely on the convergence proof.\n\nWhat is actually new: the specific combination—SPSA gradient estimates, EMA with annealing, and layer-wise clipping of a diagonal curvature estimate—does not appear in MeZO or Sophia. That is a legitimate design choice. The ablation study in Appendix B is a reasonable attempt to isolate components, and the experiments cover RoBERTa-large and OPT-1.3B with full fine-tuning, LoRA, and prefix tuning. If the heuristic works in practice, that would be useful.\n\nThe soft spots are load-bearing, not cosmetic. First, the A-GNB estimator in Section 3.4 is the squared mini-batch gradient. The paper claims it converges to the diagonal of the true Hessian as B→∞. That is the empirical Fisher, not the Hessian diagonal, for general losses and fixed labels. The preconditioner is therefore not what the theory assumes. Second, Lemma 4 contains a direct contradiction: it derives ∇L(θ(t))=e^{-t}∇L(θ) and then says this aligns perfectly with (1−t)∇L(θ). Those are different functions, so the lemma is internally inconsistent. The rest of the proof chain (Lemmas 5–11, Theorem 2) builds on this. Third, the analyzed update in the proof is an idealized deterministic Newton flow using the full Hessian inverse and a different clipping rule. Algorithm 1 uses a diagonal estimate, max(h,λ) clipping, SPSA noise, EMA, and annealing; none of those appear in the descent analysis. So the claimed O(max_i d_i) bound has not been connected to the actual method. Fourth, the speedup is reported as up to 20× in the abstract and Section 5.1, but 10× in Section 5.2 and Figure 3. That is an unexplained numerical inconsistency.\n\nI do credit the authors with engaging the literature and attempting ablations. The motivation is sound. But the central claims rest on these errors, and the code is not released, so the experimental results cannot be independently checked.\n\nWho this is for: someone working on memory-efficient fine-tuning might want to test the heuristic, but not the theory. I would not cite this paper in its current form. As an editor, I would desk reject it, or at most send it back with a demand to either fix the theory or reframe it as an empirical study with consistent numbers and code.\n\nRecommendation: do not accept. A revised version that drops the false theory and presents honest empirical evidence could be worth another look.","headline":"A plausible ZO-fine-tuning heuristic undermined by an invalid convergence proof and inconsistent speedup numbers.","tokens_in":23392,"tokens_out":4299,"would_cite":false,"duration_ms":41884,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HELENE claims a zeroth-order optimizer that uses a label-sampling-free diagonal Hessian estimate, layer-wise clipping, and annealed gradient momentum to cut LLM fine-tuning convergence steps from $O(d)$ to $O(\\max_i d_i)$, with up to 20x…","keywords":["zeroth-order optimization","large language model fine-tuning","diagonal Hessian estimation","layer-wise clipping","gradient annealing","memory-efficient training","parameter-efficient fine-tuning","Gauss-Newton-Bartlett estimator"],"falsifier":"Take a fixed layer of OPT-1.3B during fine-tuning and compare A-GNB's $\\hat{h} = B\\,\\hat{g} \\odot \\hat{g}$ against the exact diagonal Hessian computed by automatic differentiation on the same mini-batch; if the two disagree on many coordinates, the preconditioner is not the Hessian. A second check is to run HELENE on a network with one very wide layer and many narrow layers: if wall-clock convergence tracks total parameter count rather than the widest layer's dimension, the claimed $\\mathcal{O}(\\max_i d_i)$ scaling fails.","tokens_in":1877,"feed_emoji":"⚡","tokens_out":2829,"duration_ms":88166,"temperature":0.7,"pith_summary":"The paper introduces HELENE, an optimizer for fine-tuning large language models without backpropagation. It claims that by estimating a diagonal Hessian from squared mini-batch gradients and clipping that estimate per layer, a zeroth-order method converges in $O(\\max_i d_i)$ steps instead of $O(d)$, where $d_i$ is the size of layer $i$. If true, memory-efficient fine-tuning would no longer slow down with total parameter count but only with the widest layer, giving MeZO-level memory savings with much faster convergence. The authors report up to 20x speedup over MeZO and average accuracy gains of about 1.5% on RoBERTa-large and OPT-1.3B.","feed_headline":"Zeroth-order LLM fine-tuning gets up to 20x faster","feed_subtitle":"HELENE makes convergence scale with the widest layer, not total parameters, and beats MeZO by 1.5%.","key_machinery":"The load-bearing object is the A-GNB estimator, defined as $B$ times the elementwise square of the mini-batch gradient: $\\hat{h} = B\\,\\hat{g} \\odot \\hat{g}$. The paper claims this quantity converges to the diagonal of the true Hessian as the batch size grows, so it can serve as a second-order preconditioner without backpropagation or label sampling. A layer-wise clipping function $\\mathrm{clip}(h_i) = \\max(h_i, \\lambda_i)$ then bounds extreme curvature values per layer, while an annealed exponential moving average $\\alpha = \\beta_1 + (1-\\beta_1)e^{-t/T}$ damps noise in the momentum gradient. The layer-wise thresholds $\\lambda_i$ are what let the convergence proof track the largest layer dimension rather than the full parameter dimension.","core_discovery":"HELENE's central claim is that a zeroth-order optimizer can be made curvature-aware at low memory cost by replacing label-sampled Gauss-Newton-Bartlett estimates with a label-sampling-free estimator, namely the squared mini-batch gradient scaled by batch size, which is asserted to converge asymptotically to the diagonal of the true Hessian. This diagonal is smoothed by an exponential moving average, clipped layerwise with per-layer thresholds, and used to precondition an annealed momentum gradient. The paper proves, under convexity and local Hessian-continuity assumptions, that this update reaches $\\epsilon$ loss in at most $\\max_i \\left[d_i (L(\\theta_{0,i}) - \\min L) + \\ln\\left(\\frac{\\mu_i R_i^2}{32 d_i \\epsilon}\\right)\\right]$ steps, i.e. $\\mathcal{O}(\\max_i d_i)$, compared with the $\\mathcal{O}(d)$ bound it attributes to Sophia. Empirically, on RoBERTa-large and OPT-1.3B across classification, multiple-choice, and generation tasks, HELENE is reported to converge up to 20x faster than MeZO while improving average accuracy by about 1.5%.","pith_inferences":["A-GNB is mathematically the empirical Fisher information; equating it with the Hessian holds for negative log-likelihood losses at the model's own output distribution, not for arbitrary fine-tuning losses on fixed labels. A fair reader should treat the $O(\\max_i d_i)$ theorem as a statement about the idealized preconditioner unless the estimator's convergence to the Hessian is verified empirically","A natural testable extension is to replace A-GNB with the exact diagonal Hessian, computed by a few backward passes on a small model: if HELENE's gains persist, the speedup comes from the layerwise clipping and annealing; if they vanish, the estimator itself is the active ingredient.","The layer-wise clipping idea could transfer to first-order optimizers: applying per-layer thresholds to Adam's second moment instead of a global epsilon would give a cheap, architecture-aware preconditioner that does not require zeroth-order gradients at all.","The paper's claim that global clipping distorts gradient information is a testable hypothesis: compare Sophia-style clipping of the update with HELENE's Hessian-side clipping while holding all other components equal."],"forward_implications":["Convergence time for a given model is governed by its widest layer, not its total parameter count, so architectures with many narrow layers, such as typical attention stacks, should fine-tune much faster under HELENE than under MeZO.","The method remains compatible with full fine-tuning, LoRA, and prefix-tuning, so the speedup can be applied to parameter-efficient setups without changing the tuning paradigm.","Because the proof assumes the preconditioner is the true Hessian diagonal, the practical speedup depends on how well A-GNB estimates curvature in finite-batch, non-convex settings.","Memory cost is about three times MeZO's (14GB vs 4GB on OPT-1.3B per the paper), so the 20x speedup trades some memory savings for much faster wall-clock convergence."],"supporting_citations":[{"why":"Defines MeZO, the zeroth-order fine-tuning method that HELENE extends and the empirical baseline it must beat.","marker":"Malladi et al. (2023)"},{"why":"Supplies Sophia, the second-order preconditioning baseline with global clipping that HELENE contrasts and whose O(d) convergence bound is compared.","marker":"Liu et al. (2023)"},{"why":"Origin of the Gauss-Newton-Bartlett estimator whose label sampling A-GNB removes.","marker":"Martens (2020)"},{"why":"Provides fast curvature matrix-vector products underpinning the GNB line of estimators.","marker":"Schraudolph (2002)"},{"why":"Defines the SPSA gradient estimator used in HELENE's update loop.","marker":"Spall (1992)"},{"why":"Benchmark of zeroth-order optimizers that HELENE compares against in Section 5.3.","marker":"Zhang et al. (2024)"},{"why":"Evidence of heterogeneous loss curvature in neural networks, motivating per-layer rather than global clipping.","marker":"Ghorbani et al. (2019)"}],"fun_headline_variants":["HELENE: up to 20x faster zeroth-order LLM fine-tuning","Zeroth-order LLM tuning 20x faster with HELENE","HELENE cuts zeroth-order fine-tuning steps by 20x","HELENE: Hessian-aware zeroth-order, 20x speedup","LLM fine-tuning without backprop: HELENE speeds up 20x"],"cache_read_input_tokens":25344,"weakest_assumption_plain":"The speed guarantee rests on treating $B$ times the squared mini-batch gradient as the true diagonal Hessian; if that estimator is not the actual curvature, the $\\mathcal{O}(\\max_i d_i)$ convergence proof does not apply to the algorithm as run.","fun_headline_variants_meta":{"raw":{"variants":["HELENE: up to 20x faster zeroth-order LLM fine-tuning","Zeroth-order LLM tuning 20x faster with HELENE","HELENE cuts zeroth-order fine-tuning steps by 20x","HELENE: Hessian-aware zeroth-order, 20x speedup","LLM fine-tuning without backprop: HELENE speeds up 20x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000949,"raw_usage":{"total_tokens":4102,"prompt_tokens":1050,"completion_tokens":3052,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":2950}},"tokens_in":666,"tokens_out":3052,"duration_ms":22809,"temperature":1.0,"reasoning_tokens":2950,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:24:29.736394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed layer of OPT-1.3B during fine-tuning and compare A-GNB's $\\hat{h} = B\\,\\hat{g} \\odot \\hat{g}$ against the exact diagonal Hessian computed by automatic differentiation on the same mini-batch; if the two disagree on many coordinates, the preconditioner is not the Hessian. A second check is to run HELENE on a network with one very wide layer and many narrow layers: if wall-clock convergence tracks total parameter count rather than the widest layer's dimension, the claimed $\\mathcal{O}(\\max_i d_i)$ scaling fails.","supporting_citations":[{"cited_title":"Multivariate stochastic approximation using a simultaneous perturbation gradient approximation","cited_arxiv_id":null,"evidence_quote":"Defines the SPSA gradient estimator used in HELENE's update loop."},{"cited_title":"Fine-tuning language models with just forward passes","cited_arxiv_id":null,"evidence_quote":"Defines MeZO, the zeroth-order fine-tuning method that HELENE extends and the empirical baseline it must beat."},{"cited_title":"An investigation into neural net optimization via hessian eigenvalue density","cited_arxiv_id":null,"evidence_quote":"Evidence of heterogeneous loss curvature in neural networks, motivating per-layer rather than global clipping."}],"review_version":1}