Pith. sign in

REVIEW 2 major objections 44 references

GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning

T0 review · 2 major / 0 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GRZO reduces variance in zeroth-order LLM fine-tuning by normalizing losses across batch perturbations.

desk verdict GRZO's claim of per-example perturbations at no extra forward cost conflicts with how global ZO perturbations work, so the variance reduction and unbiasedness results do not follow. read the letter →

arxiv 2606.02857 v1 pith:FANGPAH4 submitted 2026-06-01 cs.LG cs.AI

Liyan Tan , Yequan Zhao , Yifan Yang , Ruijie Zhang , Xinling Yu , Zheng Zhang This is my paper · ORCID
classification cs.LGcs.AI
keywords zeroth-orderoptimizationlargelanguagemodelfine-tuninggradientestimationmemory-efficientnonconvexconvergencegroupnormalizationvariancereduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes GRZO to overcome the high variance problem in zeroth-order optimization for fine-tuning large language models. By drawing a single pseudo-independent perturbation for each example in a mini-batch and normalizing the resulting losses in a group-relative manner, the method effectively multiplies the number of gradient directions by the batch size. This is achieved without any extra forward passes or memory overhead beyond standard inference. The authors prove the estimator remains directionally unbiased while its variance decreases with larger batches, leading to improved convergence guarantees over previous methods like MeZO. Empirical results show consistent accuracy gains across several model sizes and tasks.

What carries the argument

Group-relative normalization applied to per-example zeroth-order loss estimates from individual perturbations

What would settle it

An experiment that measures the variance of GRZO gradient estimates and finds it does not shrink as batch size grows would disprove the variance reduction and tighter convergence claims.

Watch

Extended reading notes

Core claim

GRZO draws one pseudo-independent perturbation per mini-batch example and aggregates the per-example losses through group-relative normalization, raising the effective gradient-direction count from one to the batch size at no additional forward cost while preserving inference-level memory. GRZO is directionally unbiased with variance shrinking proportionally to the batch size, yielding a tighter nonconvex convergence bound than MeZO.

Load-bearing premise

Drawing one pseudo-independent perturbation per mini-batch example and aggregating losses via group-relative normalization raises the effective gradient-direction count from one to the batch size at no additional forward cost while preserving the memory advantage.

Editorial extensions

If this is right

  • GRZO yields a tighter nonconvex convergence bound than MeZO because variance shrinks with batch size.
  • GRZO raises average accuracy by 3.0 points over MeZO on Llama3-8B while cutting peak GPU memory by 23 percent.
  • GRZO improves sparse, low-rank, and quantized ZO variants by 6.0 points on average when substituted for the MeZO core.
  • GRZO preserves inference-level memory use on RoBERTa-large, Llama3-8B, and OPT-13B across multiple tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The group-relative normalization step could apply to variance reduction in other single-forward-pass estimators outside language models.
  • Larger batches in GRZO may continue to tighten convergence in practice beyond the tested sizes.
  • Pairing GRZO with additional compression methods might allow fine-tuning of models larger than those evaluated here under fixed hardware limits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper proposes GRZO, a Group-Relative Zeroth-Order optimizer for memory-efficient LLM fine-tuning. It claims to draw one pseudo-independent perturbation per mini-batch example, aggregate losses via group-relative normalization to increase effective gradient directions to the batch size at no extra forward cost, prove directional unbiasedness with variance shrinking proportionally to batch size (yielding tighter nonconvex convergence than MeZO), and report empirical gains: +3.0 average accuracy on Llama3-8B over MeZO at 23% lower peak memory, plus +6.0 average lift to other ZO variants across RoBERTa-large, Llama3-8B, and OPT-13B.

Significance. If the implementation and proof hold, GRZO would meaningfully advance ZO fine-tuning by addressing high variance while preserving inference-level memory, potentially broadening applicability of memory-efficient methods to larger models.

major comments (2)
  1. [Abstract] Abstract: The central claim that distinct pseudo-independent perturbations can be assigned per mini-batch example 'at no additional forward cost while preserving inference-level memory' requires explicit justification. Standard ZO (including MeZO) applies one global perturbation vector to all parameters for a single batched forward pass; realizing B distinct perturbed parameter sets would appear to necessitate either B forward passes or an equivalent mechanism that increases cost or memory, directly affecting the variance-reduction and convergence claims.
  2. The directional-unbiasedness proof and variance-shrinkage argument (stated in the abstract) rest on the per-example perturbation mechanism; without a concrete description of how this is realized in a single forward pass (e.g., via a specific section or algorithm), the proof cannot be verified as load-bearing for the tighter bound versus MeZO.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thorough review and for identifying areas where the manuscript's description of the per-example perturbation mechanism requires greater clarity. We address the two major comments below and will revise the paper accordingly to strengthen the presentation of the implementation and supporting analysis.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that distinct pseudo-independent perturbations can be assigned per mini-batch example 'at no additional forward cost while preserving inference-level memory' requires explicit justification. Standard ZO (including MeZO) applies one global perturbation vector to all parameters for a single batched forward pass; realizing B distinct perturbed parameter sets would appear to necessitate either B forward passes or an equivalent mechanism that increases cost or memory, directly affecting the variance-reduction and convergence claims.

    Authors: We agree that the abstract is too concise on this point and does not supply the requested justification. The full manuscript (Section 3.2 and Algorithm 1) outlines the group-relative normalization applied to per-example losses, but the concrete realization of distinct perturbations within a single forward pass is not described with sufficient algorithmic detail. We will revise by expanding Section 3 with a dedicated subsection and updated pseudocode that explicitly shows the computation (including how the batch is processed to achieve the claimed memory and forward-pass cost). revision: yes

  2. Referee: [—] The directional-unbiasedness proof and variance-shrinkage argument (stated in the abstract) rest on the per-example perturbation mechanism; without a concrete description of how this is realized in a single forward pass (e.g., via a specific section or algorithm), the proof cannot be verified as load-bearing for the tighter bound versus MeZO.

    Authors: The referee is correct that the proof in Section 4 relies on the per-example mechanism introduced in Section 3. Because the current description of the single-forward-pass implementation is not sufficiently concrete, the load-bearing relationship between the mechanism and the tighter convergence bound cannot be fully verified from the text. We will revise the manuscript to insert the missing algorithmic details (expanded Algorithm 1 and a short illustrative example) so that the proof can be checked directly against the implementation. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity in derivation chain

full rationale

The paper claims an independent proof that GRZO is directionally unbiased with batch-size-dependent variance reduction, yielding a tighter convergence bound than MeZO. No equations, self-citations, fitted parameters renamed as predictions, or ansatzes smuggled via prior work are exhibited in the abstract or claims. The central derivation is presented as self-contained mathematical analysis building on but distinct from MeZO, without reduction to its own inputs by construction. This is the most common honest non-finding for papers with external proofs.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

Based on the abstract alone, no explicit free parameters or invented entities are introduced; the work relies on standard nonconvex optimization theory for the convergence claim.

assumptions (1)
  • standard math Standard assumptions underlying nonconvex convergence bounds in stochastic optimization
    Invoked to support the claim of a tighter convergence bound than MeZO.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning." pith.science (2026). https://pith.science/paper/FANGPAH4

@misc{pith2026260602857,
  author       = {Pith},
  title        = {Pith review of: GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FANGPAH4}},
  note         = {Machine review of arXiv:2606.02857}
}
abstract

Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation. We propose GRZO, a Group-Relative Zeroth-Order optimizer that draws one pseudo-independent perturbation per mini-batch example and aggregates the per-example losses through group-relative normalization, raising the effective gradient-direction count from one to the batch size at no additional forward cost while preserving inference-level memory. We prove that GRZO is directionally unbiased with variance shrinking proportionally to the batch size, yielding a tighter nonconvex convergence bound than MeZO. Across RoBERTa-large, Llama3-8B, and OPT-13B over multiple tasks, GRZO improves average accuracy on Llama3-8B by $+3.0$ over MeZO at $23\%$ lower peak GPU memory; as a drop-in replacement for the MeZO core, it lifts sparse, low-rank, and quantized ZO variants by $+6.0$ on average.

Figures

Figures reproduced from arXiv: 2606.02857 by the authors.

Figure 1
Figure 1. GRZO at a glance on RTE (Llama3-8B). Left: lowest peak memory (16.0 GB), highest accuracy (81.6%), and MeZO-comparable per-step time. Right: fastest convergence in both training steps and wall-clock time. of a step, even though the loss is evaluated per ex￾ample. Drawing B pseudo-independent directions instead—one per example—would, by standard Monte Carlo, reduce the SPSA (simultaneous per￾turbation stochastic appr… view at source ↗
Figure 2
Figure 2. Side-by-side pipeline comparison of MeZO (left) and GRZO (right). By constructing pseudo-independent [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training-loss curves on Llama3-8B (RTE, MultiRC) and OPT-13B (SQuAD, DROP) plotted against [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Peak GPU memory (GB) vs. model size for OPT (1.3B–30B). GRZO matches the inference foot￾print, consuming even less memory than MeZO. 5.2 Memory and Time Analysis GRZO offers a strong memory advantage at a mod￾est per-step time cost [PITH_FULL_IMAGE:figures/full_fig_p0…
Figure 5
Figure 5. Figure 5: Production profile on Llama3-8B (RTE, fp16, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Training loss curves on Llama3-8B comparing vanilla GRZO with the three GRZO-combined variants. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Left: GRZO components on SST-2. Right: Perturbation ablation on DROP. robustness. We default to Rademacher for its lower memory overhead. Batch Size Sensitivity [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Per-variant GRZO+X vs MeZO+X training-loss curves on Llama3-8B. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: shows training loss curves for GRZO across batch sizes B ∈ {4, 8, 16, 32} on two tasks: SST-2 (Llama3-8B) and COPA (OPT-13B). The results corroborate the theoretical prediction in Sec￾tion 5.4: the group-relative normalizer requires a stable within-batch loss standard …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 3 canonical work pages

  1. [1]

    International Conference on Learning Representations , volume=

    Enhancing Zeroth-Order Fine-Tuning for Language Models with Low-Rank Structures , author=. International Conference on Learning Representations , volume=

  2. [2]

    Advances in Neural Information Processing Systems , volume=

    Fine-tuning language models with just forward passes , author=. Advances in Neural Information Processing Systems , volume=

  3. [3]

    Neurocomputing , volume=

    Backpropagation and stochastic gradient descent method , author=. Neurocomputing , volume=. 1993 , publisher=

    1993

  4. [4]

    International Conference on Learning Representations , year=

    Decoupled weight decay regularization , author=. International Conference on Learning Representations , year=

  5. [5]

    Proceedings of the 41st International Conference on Machine Learning , series=

    Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models , author=. Proceedings of the 41st International Conference on Machine Learning , series=. 2024 , publisher=

    2024

  6. [6]

    , booktitle=

    Zhao, Yanjun and Dang, Sizhe and Ye, Haishan and Dai, Guang and Qian, Yi and Tsang, Ivor W. , booktitle=. Second-Order Fine-Tuning without Pain for

  7. [7]

    Yang, Yifan and Zhen, Kai and Banijamali, Ershad and Mouchtaris, Athanasios and Zhang, Zheng , booktitle=

  8. [8]

    Zhang, Zhen and Yang, Yifan and Zhen, Kai and Susanj, Nathan and Mouchtaris, Athanasios and Kunzmann, Siegfried and Zhang, Zheng , booktitle=

Show all 44 references
  1. [9]

    2025 , doi=

    Zhou, Jiajun and Yang, Yifan and Zhen, Kai and Liu, Ziyue and Zhao, Yequan and Banijamali, Ershad and Mouchtaris, Athanasios and Wong, Ngai and Zhang, Zheng , booktitle=. 2025 , doi=

    2025

  2. [10]

    Liu, Yong and Zhu, Zirui and Gong, Chaoyu and Cheng, Minhao and Hsieh, Cho-Jui and You, Yang , booktitle=. Sparse

  3. [11]

    International Conference on Learning Representations , year=

    Flipout: Efficient pseudo-independent weight perturbations on mini-batches , author=. International Conference on Learning Representations , year=

  4. [12]

    and Ba, Jimmy , booktitle=

    Kingma, Diederik P. and Ba, Jimmy , booktitle=

  5. [13]

    Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle=

  6. [14]

    , booktitle=

    Dang, Sizhe and Guo, Yangyang and Zhao, Yanjun and Ye, Haishan and Zheng, Xiaodong and Dai, Guang and Tsang, Ivor W. , booktitle=

  7. [15]

    Wang, Alex and Pruksachatkun, Yada and Nangia, Nikita and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel , journal=

  8. [16]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and others , journal=. The

  9. [17]

    Zhang, Susan and Roller, Stephen and Goyal, Naman and Artetxe, Mikel and Chen, Moya and Chen, Shuohui and Dewan, Christopher and Diab, Mona and Li, Xian and Lin, Xi Victoria and others , journal=

  10. [18]

    Zhao, Jiawei and Zhang, Zhenyu and Chen, Beidi and Wang, Zhangyang and Anandkumar, Anima and Tian, Yuandong , booktitle=

  11. [19]

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , booktitle=

  12. [20]

    Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin , journal=

  13. [21]

    arXiv preprint arXiv:2602.17155 , year=

    Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization , author=. arXiv preprint arXiv:2602.17155 , year=

  14. [22]

    arXiv preprint arXiv:2510.19975 , year=

    Revisiting zeroth-order optimization: Minimum-variance two-point estimators and directionally aligned perturbations , author=. arXiv preprint arXiv:2510.19975 , year=

  15. [23]

    IEEE Transactions on Information Theory , volume=

    Optimal rates for zero-order convex optimization: The power of two function evaluations , author=. IEEE Transactions on Information Theory , volume=. 2015 , publisher=

    2015

  16. [24]

    SIAM journal on optimization , volume=

    Stochastic first- and zeroth-order methods for nonconvex stochastic programming , author=. SIAM journal on optimization , volume=. 2013 , publisher=

    2013

  17. [25]

    IEEE transactions on automatic control , volume=

    Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , author=. IEEE transactions on automatic control , volume=. 2002 , publisher=

    2002

  18. [26]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, YK and Wu, Yang and others , journal=

  19. [27]

    Poor man's training on

    Zhao, Yequan and Li, Hai and Young, Ian and Zhang, Zheng , journal=. Poor man's training on. 2025 , publisher=

    2025

  20. [28]

    IEEE Signal Processing Magazine , volume=

    A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications , author=. IEEE Signal Processing Magazine , volume=. 2020 , publisher=

    2020

  21. [29]

    Foundations of Computational Mathematics , volume=

    Random gradient-free minimization of convex functions , author=. Foundations of Computational Mathematics , volume=. 2017 , publisher=

    2017

  22. [30]

    International Conference on Machine Learning , pages=

    Generalizing gaussian smoothing for random search , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  23. [31]

    International Conference on Machine Learning , pages=

    A kernel-based view of language model fine-tuning , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  24. [32]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning , author=. Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long paper...

  25. [33]

    Foundations of Computational Mathematics , volume=

    A theoretical and empirical comparison of gradient approximations in derivative-free optimization , author=. Foundations of Computational Mathematics , volume=. 2022 , publisher=

    2022

  26. [34]

    Chen, Aochuan and Zhang, Yimeng and Jia, Jinghan and Diffenderfer, James and Parasyris, Konstantinos and Liu, Jiancheng and Zhang, Yihua and Zhang, Zheng and Kailkhura, Bhavya and Liu, Sijia , booktitle=

  27. [35]

    arXiv preprint arXiv:2308.09858 , year=

    Tensor-compressed back-propagation-free training for (physics-informed) neural networks , author=. arXiv preprint arXiv:2308.09858 , year=

  28. [36]

    Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel , booktitle=

  29. [37]

    Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy , booktitle=

  30. [38]

    Dua, Dheeru and Wang, Yizhong and Dasigi, Pradeep and Stanovsky, Gabriel and Singh, Sameer and Gardner, Matt , booktitle=

  31. [39]

    Liu, Ziyue and Zhang, Ruijie and Wang, Zhengyang and Yan, Mingsong and Yang, Zi and Hovland, Paul D and Nicolae, Bogdan and Cappello, Franck and Tang, Sui and Zhang, Zheng , booktitle=

  32. [40]

    Zhang, Ruijie Ray and Liu, Ziyue Alvin and Wang, Zhengyang and Zhang, Zheng , journal=

  33. [41]

    Yang, Yifan and Zhang, Zhen and Swaminathan, Rupak Vignesh and Liu, Jing and Susanj, Nathan and Zhang, Zheng , journal=

  34. [42]

    Prefix-tuning: Optimizing continuous prompts for generation , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

  35. [43]

    Parameter-efficient transfer learning for

    Houlsby, Neil and Giurgiu, Andrei and Jastrzebski, Stanislaw and Morrone, Bruna and De Laroussilhe, Quentin and Gesmundo, Andrea and Attariyan, Mona and Gelly, Sylvain , booktitle=. Parameter-efficient transfer learning for. 2019 , organization=

    2019

  36. [44]

    Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    The power of scale for parameter-efficient prompt tuning , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    2021

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.