{"id":"9a3e676e-6b9a-4c1e-b1cd-5e27010875ee","arxiv_id":"2506.09034","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"FZOO claims Adam-like zeroth-order fine-tuning via loss-std normalization and batched forward passes, but the paper's algorithm perturbs activations rather than parameters, breaking the link to its own theory.","lead":"FZOO is a zeroth-order optimizer that replaces backpropagation with batched forward passes and normalizes updates by the spread of losses, claiming Adam-like fine-tuning speed at inference-level memory. The paper's theory and empirical claims are undermined by an inconsistency: the algorithm as written perturbs activations, not parameters, so the analyzed gradient estimator is not what the code implements.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's batched forward pass computes diagonal/activation-space perturbations, not L(theta + epsilon u), so the convergence theory does not apply to the implemented update.","rationale":"The paper's central claim is that FZOO achieves Adam-scale convergence speed with inference-level memory, and the theoretical support is Proposition 3.2 and Theorem 3.6. Those results analyze the parameter-space estimator in Eq. (2), built from losses L(theta + epsilon u_i). The only implementation of parallel FZOO is Algorithm 1, and its recurrence for perturbed activations corresponds to diagonal weight perturbations rather than to adding epsilon u to all parameters. BatchUpdateParameter nevertheless updates every parameter, so the losses used to estimate the gradient are not losses of the function whose parameters are being updated. This is a mechanical, text-level mismatch. Algorithm 3 confirms the authors know how to implement the true parameter perturbation, which makes the discrepancy in Algorithm 1 a substantive gap rather than a notation issue. The missing 1/epsilon factor identified by the reader is less decisive by itself, since sigma_t is O(epsilon) and the factor could be absorbed into the learning rate; the core problem is the perturbation mismatch. The empirical tables do not resolve this because no code is provided to show that the actual implementation matches Eq. (2). I therefore agree with the reader's rejection: as written, the convergence guarantees do not apply to the described accelerated update, and the central claim is unsupported.","tokens_in":22664,"tokens_out":8242,"duration_ms":87589,"concrete_test":"On a small transformer or MLP, run Algorithm 1's BatchPerturbParameters for a fixed RNG seed and record the returned losses; then, using the identical seed, compute L(theta + epsilon u) by directly adding the full parameter perturbation u to every parameter. If the two loss vectors differ for any nonzero input, Algorithm 1 does not implement Eq. (2). Equivalently, compare Algorithm 1's losses with Algorithm 3's losses under the same seeds; a mismatch would show that the parallel variant is not equivalent to the non-parallel variant whose theory is proven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the estimator in Eq. (2), which is defined through losses L(theta + epsilon u_i). Algorithm 1's BatchPerturbParameters (lines 11-19) does not compute those losses. For the first layer it forms Y_i^(1) = W^(1)X + epsilon(u_i (circle) X), which equals the output of W^(1) + epsilon diag(u_i), a diagonal perturbation of the weight matrix rather than a perturbation of all parameters. For later layers it propagates Y^(j) = W^(j)Y^(j-1) + epsilon(u (circle) Y^(j-1)), again equivalent to a diagonal weight perturbation at each layer. Meanwhile BatchUpdateParameter (lines 22-30) updates every coordinate of theta, sampling a full sign vector u per parameter. The loss differences that feed the gradient estimate are therefore differences of a different, diagonally perturbed network, and Proposition 3.2 and Theorem 3.6, which analyze L(theta + epsilon u_i), do not govern the implemented update. This is an internal inconsistency in the pseudocode, not a disagreement with an external consensus. It is telling that Algorithm 3, the non-parallel FZOO variant, does compute L(theta + epsilon u_i) correctly, which isolates the discrepancy to the 'accelerated' batched implementation that the experiments claim to use.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FZOO, a zeroth-order optimizer for fine-tuning large language models. FZOO uses batched Rademacher perturbations, a one-sided loss-difference estimator, and a step size normalized by the standard deviation of batch losses. The authors claim that FZOO is equivalent to a normalized-SGD update (Section 3.4), provide a convergence guarantee (Theorem 3.6), and report experiments on RoBERTa-large, OPT, Phi-2, and Llama3 showing that FZOO outperforms MeZO in accuracy while using far fewer forward passes, approaching Adam-scale convergence speed. The theoretical analysis is built on the estimator g_t in Eq. (2), defined through losses L(θ_t + εu_i; B_t), while the accelerated implementation in Algorithm 1 perturbs activations rather than parameters. This discrepancy is the central issue addressed in the major comments.","tokens_in":22908,"tokens_out":10398,"duration_ms":99989,"significance":"If the claims were correct, FZOO would be a practically valuable contribution: it would show that zeroth-order fine-tuning can come close to first-order methods in step count while keeping inference-level memory, and the batched Rademacher implementation would be a useful engineering trick for ZO optimization. The paper has genuine strengths: a broad empirical study spanning models from 350M to 66B parameters, experiments with non-differentiable objectives, memory profiling, and a non-parallel control variant (Algorithm 3) that helps isolate the effect of batching. However, the load-bearing inconsistency between the objective analyzed in Section 3.4 and the algorithm implemented in Section 3.3 means that the theoretical guarantees do not apply to the executed update, and the headline empirical speedup numbers conflate step-count speedup with wall-clock speedup. The central claim is therefore not supported as stated.","major_comments":[{"comment":"The batched forward pass in Algorithm 1 computes Y^(j) = F^(j) + ε(u ⊙ Y^(j-1)), which is equivalent to adding ε diag(u) to the weight matrix of each layer, i.e., a diagonal weight perturbation. It does not compute L(θ + εu_i; B), the quantity used to define g_t in Eq. (2) and analyzed in Proposition 3.2 and Theorem 3.6. For a dense weight matrix W^(j), a full parameter perturbation (W^(j) + εU)X differs from W^(j)X + ε(u ⊙ (W^(j)X)) unless U is diagonal, so the gradient estimated by Algorithm 1 is not the one-sided gradient of the true parameter-perturbed loss. Algorithm 3, the non-parallel variant, correctly perturbs θ, which confirms that the discrepancy is introduced specifically by the 'accelerated' implementation used in the experiments. This is an internal inconsistency between the theory and the implemented algorithm, and it undermines the convergence claims for the method as executed.","section":"§3.3, Algorithm 1 (lines 11–19) vs. §3.4, Eq. (2)"},{"comment":"The claimed 'formal equivalence to normalized-SGD' is not established. Equations (6) and (7) only relate E[||g_t||^2] to E[σ_t^2] up to O(ε) remainder terms. The actual update uses the ratio g_t/σ_t, where σ_t is a random variable, and the proof does not show that g_t/σ_t is close to g_t/||g_t|| in any relevant sense. This is a heuristic scaling relation, not an equivalence of update rules. Since the introduction lists 'a formal proof of its equivalence to a normalized-SGD update rule' as a contribution, this claim should either be proven or substantially softened.","section":"§3.4, Remark 3.3 and Proposition 3.2"},{"comment":"The proof requires the step-size condition η_t = η/σ_t ≤ N/(16dL) for every t, but the theorem's proposed choice η = sqrt((L(θ_1)-L(θ*))/(4dLV^2 Σ_k σ_k^{-2})) does not enforce this bound; σ_t is random and can be arbitrarily small. Without an additional condition or a high-probability lower bound on σ_t, the descent inequality used to telescope the sum is not justified, so the convergence guarantee as stated is incomplete.","section":"§3.5, Theorem 3.6 and Appendix B.4"},{"comment":"Algorithm 1 computes projected_grad = (ℓ - L(θ; B))/(N·std), whereas Eq. (2) defines g_t = (1/(εN)) Σ_i (l_i - l0) u_i. The factor 1/ε is missing from the implemented update. Unless ε = 1 or the learning rate is rescaled to absorb this factor, the update executed by Algorithm 1 is not the update analyzed in Theorem 3.6. The pseudocode should be aligned with Eq. (2), or the learning-rate rescaling should be stated explicitly.","section":"Algorithm 1, line 4 vs. Eq. (2)"}],"minor_comments":[{"comment":"The heading 'Movtivation of FZOO' contains a typo and should read 'Motivation of FZOO'.","section":"§3.2.1 heading"},{"comment":"The pseudocode uses L(θ; B) in line 4, but BatchPerturbParameters does not return or compute the unperturbed loss; the algorithm should specify how l0 is obtained.","section":"Algorithm 1, line 4"},{"comment":"Reference [2] is cited for normalized-SGD, but the cited Bernstein et al. paper is about signSGD and sign-based compression, not normalized-SGD; an appropriate reference for normalized gradient descent should be provided.","section":"References"},{"comment":"The table header says 'Roberts-Large' instead of 'RoBERTa-large'.","section":"Table 7 header"},{"comment":"The speedup metrics are inconsistent across the abstract, Figure 1, and Table 6: '18× reduction in forward passes', '18× speedup', and '20× speed-up in total steps' are different quantities, and Table 5 shows that FZOO's per-step wall-clock time is actually higher than MeZO's. The paper should state explicitly which quantity each headline number refers to and should distinguish step-count speedup from wall-clock speedup.","section":"Abstract, Figure 1, and Table 6"},{"comment":"The text introduces U = diag(u_1, ..., u_N) as a block-diagonal sign matrix, but the subsequent line P^(j) = ε(U ⊙ Y^(j-1)) suggests elementwise broadcasting; the shape and broadcasting convention for U should be clarified.","section":"§3.3, notation for U"}],"recommendation":"reject","confidential_remarks":"I am not questioning the reported experimental numbers; the issue is that the paper's own theory does not cover the algorithm actually implemented. The diagonal-activation-perturbation trick is the main source of the claimed parallel speedup, and replacing it with true parameter perturbation would likely remove that speedup, while analyzing the diagonal estimator would require a substantial new theoretical development. For these reasons I do not see a minor revision that would resolve the central inconsistency, though a resubmission with a correct analysis of the implemented estimator could be viable. The self-citation of HiZOO [52] by overlapping authors is not in itself a problem, but the novelty relative to that work should be stated more carefully in a future version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — here's the honest read.\n\nThe paper's central claim is that FZOO reaches Adam-scale convergence speed with inference-level memory by combining one-sided Rademacher zeroth-order estimates with loss-standard-deviation normalization and a batched parallel forward pass. The specific combination is new, and the empirical work is broad: RoBERTa-large, OPT up to 66B, Phi-2, Llama3, 11 tasks, plus non-differentiable objectives. The accuracy gains over MeZO are consistent and often large. That is real, and it deserves credit.\n\nThe problem is that the theory does not analyze the implemented algorithm. Section 3.4 and Theorem 3.6 analyze the estimator in Eq. (2), which queries L(theta + epsilon u). Algorithm 1 computes something else: it perturbs activations, Y^(j) = F^(j) + epsilon(u ⊙ Y^(j-1)). For a linear layer this is equivalent to a diagonal perturbation of W^(j), not a full-parameter perturbation. The paper never shows these coincide for transformer weights, and they generally don't. The stress-test note is correct: Algorithm 3, the non-parallel variant, does compute L(theta + epsilon u) properly, isolating the discrepancy to the batched 'accelerated' version that the headline experiments claim to use. On top of that, Algorithm 1's projected_grad misses the 1/epsilon factor that appears in the theory's gt. The convergence guarantees therefore do not apply to the update that was actually run. This is not a minor fix; it's a load-bearing mismatch.\n\nThe wall-clock claim also deserves a closer look. Table 5 shows FZOO's per-step time is 2.8x MeZO on OPT-125M and 1.4x on RoBERTa-large. The '18x speedup' is a reduction in forward passes, not in wall-clock time; the paper never reports total end-to-end time. With the missing factor and the mismatch, the empirical tables cannot currently be interpreted as evidence for the analyzed method.\n\nTheoretical details: the proofs in Appendix B are mostly standard Taylor-expansion bounds, and the normalized-SGD equivalence in Remark 3.3 is basically a consistency check (sigma is an epsilon-scaled gradient norm by construction), not an independent result. The self-citation of HiZOO is minor and not a problem.\n\nWho it's for: the engineering insight—batched Rademacher perturbations via cheap sign flips—is genuinely interesting, and the empirical gains over MeZO might survive a corrected analysis. A serious referee should see this. But as written, the paper needs either a corrected algorithm that matches the theory, or a theory for the activation-perturbation estimator, plus total wall-clock numbers and code. I'd reject in current form and ask for those specific revisions.","headline":"The batched forward pass in Algorithm 1 does not compute the estimator that the theory analyzes, so the convergence guarantees apply to a different algorithm than the one being tested.","tokens_in":23488,"tokens_out":4213,"would_cite":false,"duration_ms":39588,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C56"],"pacs":[],"model":"deepseek-v4-flash","headline":"FZOO, a forward-only zeroth-order optimizer, claims Adam-scale fine-tuning speed at inference-level memory, reporting 18× fewer forward passes than MeZO on RoBERTa-large.","keywords":["zeroth-order optimization","LLM fine-tuning","normalized SGD","Rademacher perturbation","batched forward passes","inference-level memory","forward-only fine-tuning"],"falsifier":"Fine-tune the same model and task twice with identical seeds and hyperparameters: once with the batched activation-perturbation forward of Algorithm 1, and once with a genuine parameter-perturbation forward that evaluates $L(\\theta + \\epsilon u)$ for the same Rademacher vectors (the paper's own Algorithm 3 implements exactly this). If the per-step loss estimates and converged accuracies differ systematically, the batched forward is not computing the perturbed-parameter loss, and the normalized-SGD analysis of Proposition 3.2 does not govern the update that actually runs.","tokens_in":22418,"feed_emoji":"⚡","tokens_out":18231,"duration_ms":178928,"temperature":0.7,"pith_summary":"Fine-tuning large language models with Adam multiplies memory cost by more than ten because the backward pass stores activations; zeroth-order optimizers avoid that by using only forward passes, but they have been much slower to converge. FZOO is a zeroth-order optimizer that tries to remove this speed penalty. It estimates the gradient from a batch of one-sided function-value differences along Rademacher ($\\pm1$) directions and divides the estimate by the standard deviation of the batch losses, a step the paper proves to be, up to higher-order terms, a normalized stochastic gradient in the sense of normalized-SGD. The $\\pm1$ directions also let the $N$ perturbed forward passes be fused into single parallel kernels. The paper reports 3\\% average accuracy gain over MeZO with 3$\\times$ fewer forward passes, an 18$\\times$ reduction on RoBERTa-large with convergence comparable to Adam, and inference-level memory throughout. If right, this makes single-GPU full-parameter fine-tuning of models like OPT-30B practical today.","feed_headline":"18x fewer passes: forward-only tuning reaches Adam-scale speed","feed_subtitle":"FZOO batches ±1 perturbations and sizes each step by loss variance, staying at inference-level memory.","key_machinery":"Three mechanisms carry the argument. The first is the batched one-sided estimator with Rademacher directions: because $u_i \\in \\{+1,-1\\}^d$, the layer-wise perturbation $P^{(j)} = \\epsilon (u \\odot Y^{(j-1)})$ is an element-wise sign flip and addition rather than a second matrix multiplication, so all $N$ perturbed forwards can be concatenated along the batch axis and executed in fused CUDA kernels instead of sequential passes. The second is the variance-normalized update rule $g_t/\\sigma_t$: Proposition 3.2 and Remark 3.3 identify the ratio $\\mathbb{E}[\\|g_t\\|^2]/\\mathbb{E}[\\sigma_t^2]$ with the constant $\\frac{N+d-1}{N}\\epsilon^{-2}$, which is what licenses reading the update as normalized stochastic gradient descent, with the $\\sigma_t$ denominator supplying Adam-like adaptivity at no memory cost. The third is the seed-replay device inherited from MeZO: perturbation vectors are regenerated from stored random seeds during the parameter update, so the full $N \\times d$ perturbation matrix never has to reside in GPU memory.","core_discovery":"The paper's claim is that the apparent speed/memory trade-off of zeroth-order fine-tuning is not fundamental. FZOO uses the update $\\theta_{t+1} = \\theta_t - \\eta_t g_t/\\sigma_t$, where $g_t = \\frac{1}{\\epsilon N}\\sum_{i=1}^N (L(\\theta_t + \\epsilon u_i; B_t) - L(\\theta_t; B_t)) u_i$ is a batched one-sided gradient estimate along $N$ i.i.d. Rademacher ($\\pm 1$) directions $u_i$, and $\\sigma_t$ is the sample standard deviation of the $N$ perturbed losses. Proposition 3.2 computes $\\mathbb{E}[\\|g_t\\|^2] = \\frac{N+d-1}{N}\\|\\nabla L(\\theta_t, B_t)\\|^2 + O(\\epsilon)$ and $\\mathbb{E}[\\sigma_t^2] = \\epsilon^2 \\|\\nabla L(\\theta_t, B_t)\\|^2 + O(\\epsilon^3)$, so up to higher-order terms the ratio of the two expectations is the iteration-independent constant $\\frac{N+d-1}{N}\\epsilon^{-2}$; Remark 3.3 uses this to regard $g_t/\\sigma_t$ as a normalized stochastic gradient. The paper argues this gives Adam-style adaptivity, larger steps on flat loss regions and smaller steps on steep ones, without Adam's momentum memory, and Theorem 3.6 supplies a convergence bound for smooth objectives with bounded variance. Empirically, FZOO is reported to beat MeZO by 3\\% average accuracy with 3$\\times$ fewer forward passes across models up to OPT-66B, to converge comparably to Adam on RoBERTa-large with an 18$\\times$ reduction in forward passes, to hold memory at the inference level, to optimize non-differentiable objectives directly, and to compose with parameter-efficient fine-tuning.","pith_inferences":["The theory and the implementation are joined by an unproved equivalence: the batched forward pass (Algorithm 1) adds $\\epsilon(u \\odot Y)$ to each layer's activations, which the paper treats as producing the same losses as perturbing every model parameter by $\\epsilon u$; a head-to-head comparison of the batched path against genuine parameter perturbation (the paper's own Algorithm 3) would show w","The $\\sigma_t$ normalization ties step size to the spread of losses across the perturbation batch; a natural extension would test whether the same normalization accelerates other zeroth-order estimators, or whether it functions mainly as an adaptive learning-rate schedule.","The wall-clock advantage depends on kernel fusion being available for batched forwards; on runtimes without such fusion, the gain would shrink to the step-count reduction alone, which the paper's released non-parallel variant could quantify."],"forward_implications":["Zeroth-order fine-tuning can reach Adam-scale wall-clock convergence on models like RoBERTa-large, so the assumption that forward-only methods are inherently slow stops holding for well-engineered batched estimators.","Full-parameter fine-tuning of a model like OPT-30B becomes feasible on a single GPU, since FZOO keeps memory at inference level while converging in far fewer steps than MeZO.","FZOO optimizes non-differentiable objectives directly, because it only queries function values; the paper reports a 5.53\\% average F1 gain over MeZO on SQuAD across OPT scales.","FZOO is an update rule rather than a parameter-selection strategy, so it composes with PEFT methods such as prefix-tuning, yielding even larger memory savings.","Theorem 3.6's bound shows the variance-normalized zeroth-order update is provably convergent for smooth objectives under bounded variance, with the average squared gradient norm decreasing in the number of steps."],"supporting_citations":[{"why":"MeZO, the forward-only fine-tuning baseline: supplies the seed-replay memory mechanism FZOO inherits and defines the accuracy and speedup comparisons.","marker":"[28]"},{"why":"The normalized-SGD (signSGD) result whose update rule FZOO's variance-normalized step is claimed to realize in the zeroth-order regime.","marker":"[2]"},{"why":"Supplies the estimate that one backward pass costs about three forward passes, used to convert Adam's wall-clock into forward-pass equivalents for the speed comparisons.","marker":"[1]"},{"why":"Adam, the first-order optimizer whose convergence speed FZOO aims to match and whose backward-pass memory footprint motivates the work.","marker":"[20]"},{"why":"HiZOO, the Hessian-informed zeroth-order baseline: FZOO claims comparable accuracy while avoiding HiZOO's doubled memory.","marker":"[52]"},{"why":"The zeroth-order benchmark that catalogs the estimator variants used for the comparison in Section 4.5.","marker":"[49]"},{"why":"Prefix-tuning, the parameter-efficient method used to demonstrate that FZOO composes with PEFT.","marker":"[22]"},{"why":"SPSA, the classical simultaneous-perturbation estimator defining the zeroth-order estimation paradigm that FZOO's batched one-sided estimate extends.","marker":"[33]"}],"fun_headline_variants":["18x fewer passes: forward-only tuning hits Adam speed","FZOO: 3x fewer passes, 3% better than MeZO","Adam-scale speed with inference-level memory via FZOO","Loss-variance step sizing gives Adam-like adaptivity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's speedups are measured with a batched forward pass that nudges each layer's activations by a $\\pm1$ pattern instead of nudging the model's weights themselves; the paper assumes these two procedures give the same losses, and that unproved equivalence is the only bridge between the gradient theory and the update that actually runs.","fun_headline_variants_meta":{"raw":{"variants":["18x fewer passes: forward-only tuning hits Adam speed","FZOO: 3x fewer passes, 3% better than MeZO","Adam-scale speed with inference-level memory via FZOO","Loss-variance step sizing gives Adam-like adaptivity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002088,"raw_usage":{"total_tokens":8297,"prompt_tokens":1302,"completion_tokens":6995,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":918,"completion_tokens_details":{"reasoning_tokens":6923}},"tokens_in":918,"tokens_out":6995,"duration_ms":53536,"temperature":1.0,"reasoning_tokens":6923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:57:14.291885+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same model and task twice with identical seeds and hyperparameters: once with the batched activation-perturbation forward of Algorithm 1, and once with a genuine parameter-perturbation forward that evaluates $L(\\theta + \\epsilon u)$ for the same Rademacher vectors (the paper's own Algorithm 3 implements exactly this). If the per-step loss estimates and converged accuracies differ systematically, the batched forward is not computing the perturbed-parameter loss, and the normalized-SGD analysis of Proposition 3.2 does not govern the update that actually runs.","supporting_citations":[{"cited_title":"signsgd: Compressed optimisation for non-convex problems","cited_arxiv_id":null,"evidence_quote":"The normalized-SGD (signSGD) result whose update rule FZOO's variance-normalized step is claimed to realize in the zeroth-order regime."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HiZOO, the Hessian-informed zeroth-order baseline: FZOO claims comparable accuracy while avoiding HiZOO's doubled memory."},{"cited_title":"Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen","cited_arxiv_id":null,"evidence_quote":"The zeroth-order benchmark that catalogs the estimator variants used for the comparison in Section 4.5."},{"cited_title":"Prefix-tuning: Optimizing continuous prompts for generation","cited_arxiv_id":null,"evidence_quote":"Prefix-tuning, the parameter-efficient method used to demonstrate that FZOO composes with PEFT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SPSA, the classical simultaneous-perturbation estimator defining the zeroth-order estimation paradigm that FZOO's batched one-sided estimate extends."}],"review_version":1}