REVIEW 4 major objections 6 minor 1 cited by
DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read DistZO2 claims that zeroth-order fine-tuning of a 175B-parameter language model can run on four GPUs with under 20GB each and triple the throughput of single-GPU ZO2 by running the two perturbed forward passes concurrently and…
desk verdict Useful multi-GPU extension of ZO2 with real throughput gains, but missing accuracy validation and an underspecified RNG mechanism keep it from being fully convincing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the scalar projected gradient $g = (L(\theta+\epsilon z) - L(\theta-\epsilon z))/(2\epsilon)$, which replaces the high-dimensional gradient tensor of first-order training. Because $g$ is a scalar, all-reducing it costs almost nothing and every replica can update locally as $\theta \leftarrow \theta - \eta g z$, provided every device draws the same $z$. Around this the paper builds two parallel axes: Perturbation Parallelism (PertP), which assigns the $+\epsilon$ and $-\epsilon$ passes to two GPUs; and an adapted Distributed Data Parallelism, which averages per-batch scalars. The 2D design organizes $n = n_b \times 2$ GPUs with PertP on the inner axis and DDP on the outer axis, and a hardware-aware transfer scheme slices each parameter block into $n$ pieces so PCIe carries only one slice per GPU while NVLink handles the rest; offloading is likewise sliced because all replicas hold identical parameters.
What would settle it
Run identical fine-tuning of the same model on one GPU with ZO2 and on two GPUs with PertP using the same seed, hyperparameters, and data order, then compare projected gradients and final validation accuracy. If the two GPUs' RNG states diverge despite the shared seed, per-iteration gradients will differ by more than floating-point noise and replica parameters will drift; any measurable divergence, or an accuracy gap beyond ordinary seed noise, would falsify the claim that PertP preserves ZO2 behavior.
Extended reading notes
Core claim
On its own terms, DistZO2's central claim is that the two forward passes zeroth-order fine-tuning already needs—one with parameters shifted by $+\epsilon z$ and one by $-\epsilon z$—are logically independent, so they can run concurrently on different GPUs instead of sequentially on one. The paper shows this perturbation parallelism plus an adapted distributed data parallelism, in which devices each process a different batch and synchronize a single scalar projected gradient, can be combined into a 2D mesh where each GPU does exactly one forward pass. With a communication scheme that slices each CPU-resident transformer block and moves one slice per GPU over PCIe while the rest travel over NVLink, the framework reports a 3x throughput gain over ZO2 on OPT-175B, restoring MeZO-level throughput while using under 20GB of GPU memory per device.
Load-bearing premise
The design assumes that broadcasting one random seed makes every GPU generate exactly the same Gaussian perturbation vector $z$, so the two perturbed losses estimate the same directional derivative and all replicas stay synchronized; the paper states that a seed is broadcast but does not specify the sampling mechanism or verify that trained models match single-GPU ZO2 quality.
Editorial extensions
If this is right
- If the reported numbers hold, a 175B-parameter model can be zeroth-order fine-tuned on four GPUs with roughly 19GB of memory per device at about 508 tokens per second, three times the 166 tokens per second of single-GPU ZO2.
- Perturbation Parallelism alone should give close to a 2x throughput gain across model scales, since the two perturbed forward passes are the dominant compute and run concurrently.
- Adapted DDP should scale nearly linearly with GPU count, because synchronizing a single scalar per iteration adds negligible communication overhead.
- The hardware-aware slicing should cut per-GPU PCIe traffic by a factor equal to the GPU count and improve measured upload and offload bandwidth by up to about 4.6x.
- Because MeZO and ZO2 share the same dual-forward and scalar-gradient structure, the paper argues the same parallelization strategies transfer directly to MeZO as well.
Reading between the lines
- If the seed-synchronization assumption is made airtight, the same 2D recipe should extend beyond the paper's exact setup to other forward-only optimizers and to larger node counts, since each PertP pair can be replicated as its own DDP group.
- The offload-slicing shortcut depends on all replicas holding identical parameters after every step; a zeroth-order variant with per-parameter adaptive states, such as a ZO-Adam-style optimizer, would break that identity and need a different offload protocol.
- A natural but untested extension is more than two perturbation evaluations per step, which would replace the 2D mesh with a higher-dimensional grid and change the communication-versus-compute trade-off the paper measures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents DistZO2, a distributed extension of the ZO2 zeroth-order fine-tuning framework. It proposes three parallelization strategies: Perturbation Parallelism (PertP), which splits the two perturbed forward passes of the central-difference estimator across two GPUs; Distributed Data Parallelism (DDP), which synchronizes only scalar projected gradients; and a combined 2D parallelism that uses PertP as the inner dimension and DDP as the outer dimension. It also proposes a hardware-aware communication scheme that slices parameter blocks across GPUs to reduce PCIe pressure via NVLink. Experiments on OPT models from 1.3B to 175B report throughput, GPU memory usage, and communication bandwidth, with the headline result being a 3x throughput improvement over ZO2 on OPT-175B using 4 GPUs while keeping per-GPU memory around 18-20 GB. The text includes algorithm descriptions, figures, and tables but no downstream accuracy or loss-curve validation of the fine-tuning itself.
Significance. If the algorithmic and experimental gaps identified below are resolved, the contribution would be practically useful: it would show that memory-efficient zeroth-order fine-tuning of 100B+ parameter models can be accelerated through multi-GPU parallelism while retaining ZO2's low per-GPU memory footprint. The paper is also commendable for reporting concrete throughput and memory numbers on a range of OPT scales and for open-sourcing the implementation, which makes the claims independently checkable. The core ideas of PertP and scalar-gradient DDP are natural and likely to be of interest to the systems-for-LLM and ZO-optimization communities. However, the current manuscript leaves several load-bearing details about the distributed algorithm unspecified and provides no evidence that the reported throughput translates into valid fine-tuning behavior.
major comments (4)
- [§4.2, Step 4 and §5.1] The parameter-update step is described in a way that is either incomplete or incorrect. Step 4 states: "WDk ← WDk − η · g · z" and asserts that because g and z are synchronized, "the updates are consistent across devices." But in Step 2, GPU 1 computes its forward pass with parameters W+ = WH + εz and GPU 2 with W- = WH - εz. If WDk denotes the same tensor as the perturbed parameters used in the forward pass, then applying −ηgz to the two device copies leaves them at θ + (ε − ηg)z and θ − (ε + ηg)z, which are not equal. The paper never states that each GPU first restores its local parameters to the unperturbed θ (as ZO2's Algorithm 2 does in line 27) before applying the update. Without such a restore step, the replicas diverge and the claimed consistency is false; with a restore step, the paper must say so explicitly and explain how the perturbation is undone in the distributed setting.
- [§4.2, Step 1 and §5.1; Algorithm 2] The synchronization of the random perturbation vector z is underspecified in a way that directly threatens the central-difference estimate. The paper says only that a shared random seed is "broadcast" so that devices sample the same z, but it never states the mechanism. In particular, it does not say whether each device samples z from its own RNG after setting that seed, whether z is generated in a canonical parameter order independent of the block-wise offloading scheduler, whether the per-iteration random-state manager of ZO2 (Algorithm 2, lines 4-9 and 19-28) is replicated or synchronized across devices, or whether z is instead materialized on one device and broadcast as a tensor. The last option would incur a prohibitive bandwidth cost for 175B parameters, while the first option requires careful ordering guarantees that are not described. Without this specification, the loss difference (L+ - L-) on each GPU may use different z, which corrupts the gradient estimate, and DDP replicas may apply different updates and drift apart. This is exactly the kind of detail that needs to be pinned down for the paper's correctness claims.
- [§7 and Tables 1-5] The paper reports no validation that DistZO2 actually fine-tunes the model. All experiments measure throughput, GPU memory, and communication bandwidth; there are no training-loss curves, no SST-2 validation accuracies, and no comparison with the fine-tuning quality of MeZO or ZO2. Since the paper's central claim is "fine-tuning," not merely accelerated forward passes, it must show that the distributed updates optimize the objective and preserve model quality. This is especially important because the previous two comments identify possible sources of divergence: if z is inconsistent across devices, or if the update step does not first restore the unperturbed parameters, then the measured throughput would be irrelevant to actual fine-tuning. The authors should report at least one training curve and one downstream-task accuracy for a representative model (e.g., OPT-13B or OPT-30B) comparing DistZO2 against ZO2 and MeZO.
- [§7.1, Table 1] The reported "3x speedup over ZO2" on OPT-175B compares 4 GPUs (ZO2+PertP+DDP, 508 tokens/sec) against 1 GPU (ZO2, 166 tokens/sec). If the comparison is meant to demonstrate the benefit of distributed parallelism, the paper should report the speedup per GPU or the scaling efficiency, since a 4-GPU run should ideally be compared against a 1-GPU run with the same global batch size. The text also does not clarify whether the batch size is fixed per GPU or globally: Section 5.1 says "each GPU processes a different mini-batch" while Section 7 says "fixed batch size setting." If DDP increases the global batch size with the number of GPUs, part of the throughput gain comes from processing more data per iteration, not from pure computational speedup. The authors should state the global-batch-size convention and, if appropriate, report a balanced comparison with equal total work or equal per-GPU work.
minor comments (6)
- [Table 5] The column header "ZO2+PerP" contains a typo and should read "ZO2+PertP."
- [Figure 3 caption and §6.1] The caption says "NVLink transfers initiating only after PCIe transfers are complete," while the text says "PCIe and NVLink communication streams can be executed concurrently." These statements are contradictory and should be reconciled; the timing model T_comm = M/n (PCIe) + (n-1)M/n (NVLink) assumes at least partial overlap, so the caption should be corrected.
- [Table 2] The two columns "ZO2+DDP+PertP" and "ZO2+PertP+DDP" are said to correspond to "DDP(inner)+PertP(outer)" and "PertP(inner)+DDP(outer)" in the text, but the naming convention for which parallel strategy is inner versus outer is not defined anywhere. Please define the ordering in the table or caption.
- [§7] All throughput and memory measurements appear to be single runs with no variance information. For a systems paper whose central claim is throughput, at least a few repeated runs or error bars on the key configurations are needed to rule out run-to-run variability.
- [Table 1] For OPT-175B, the memory usage of ZO2+PertP+DDP is reported as 18713 MB, identical to ZO2, while ZO2+DDP is 19881 MB. Since the 2D configuration uses more GPUs and more communication, the identical memory number is surprising and should be explained.
- [§4.2 and §5.1] The paper would benefit from a pseudocode presentation of DistZO2's distributed iteration, analogous to Algorithm 2 for ZO2. Several ambiguities in Steps 2-4 of §4.2 would be resolved by such a specification.
Circularity Check
No circularity: the central throughput claims are empirical measurements against independent and open baselines, not derivations from fitted inputs.
full rationale
DistZO2's central claims are measured throughput and memory numbers, not derived predictions. The zeroth-order gradient estimator in Eqs. (1)-(2) is the standard central-difference estimator and is not fitted to the reported results. The 3x speedup over ZO2 is an empirical comparison against the authors' prior system, but it is also compared with the independent MeZO baseline, concrete numbers are reported, and the code is open-sourced, making the measurements externally checkable. The self-citation to ZO2 is contextual and serves as a baseline, not as load-bearing authority for the new claims; no uniqueness theorem or ansatz is imported from prior work to force the design choice. The underspecified RNG synchronization and the absence of fine-tuning quality validation are correctness or completeness concerns, not circularity: they do not make any reported throughput number equivalent to an input by construction.
Assumptions & free parameters
assumptions (4)
- standard math The central difference estimator g = (L(theta + eps z) - L(theta - eps z)) / (2 eps) is a usable gradient estimate for zeroth-order optimization.
- domain assumption Transformer blocks can be transferred and computed independently without changing the forward result.
- domain assumption A shared random seed produces identical perturbation vectors z on all devices.
- domain assumption After all-reduce, all GPUs hold identical updated parameters, so each can offload only its M/n slice.
Cite this review
Pith. "Pith review of DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing." pith.science (2026). https://pith.science/paper/JCHGLE46
@misc{pith2026250703211,
author = {Pith},
title = {Pith review of: DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing},
year = {2026},
howpublished = {\url{https://pith.science/paper/JCHGLE46}},
note = {Machine review of arXiv:2507.03211}
}
read the original abstract
Fine-tuning large language models (LLMs) remains resource-intensive due to their sheer scale. While zeroth-order (ZO) optimization provides a memory-efficient alternative by eliminating backward passes, its application to multi-hundred-billion-parameter models is constrained by GPU memory and compute throughput. The ZO2 framework addresses the memory bottleneck by offloading model parameters to CPU memory and overlapping transformer block transfer with dual forward computation on a single GPU. However, ZO2 remains limited by its single-device execution and achieves modest throughput. In this work, we present DistZO2, a high-throughput, memory-efficient framework for distributed zeroth-order fine-tuning of LLMs. DistZO2 introduces three parallel strategies: (1) Perturbation Parallelism (PertP), which parallelizes the two perturbed forward passes across devices; (2) Distributed Data Parallelism (DDP), adapted to the scalar-gradient nature of ZO training; and (3) a unified 2D Parallelism design that combines PertP and DDP. To further mitigate communication bottlenecks introduced by parameter offloading, we propose a hardware-aware communication strategy that slices parameter blocks and redistributes them across GPUs via high-speed interconnects such as NVLink. DistZO2 scales zeroth-order fine-tuning to modern multi-GPU systems, preserving ZO2's memory efficiency while substantially improving training throughput. In our experiments on OPT-175B, DistZO2 achieves a 3x speedup over ZO2 with distributed computing. DistZO2's code has been open-sourced in https://github.com/liangyuwang/zo2.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
Algorithmic Recourse of In-Context Learning for Tabular Data
The paper delivers the first theoretical analysis and practical zeroth-order framework for algorithmic recourse under in-context learning for tabular prediction.
Reference graph
Works this paper leans on
-
[7]
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,
-
[8]
An overview of gradient descent optimization algorithms
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747,
-
[10]
Megatron-lm: Training multi-billion parameter language models using model parallelism
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053,
arXiv 1909
-
[11]
Recursive deep models for semantic compositionality over a sentiment treebank
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642,
2013
-
[15]
have established theoretical guarantees under smoothness assumptions. ZO methods have found applications in black-box adversarial attacks (Chen et al., 2017), reinforcement learning (Salimans et al., 2017), and model-agnostic optimization. Despite scalability concerns due to dimension-dependent query complexity, techniques such as block-coordinate estimat...
work page 2017
-
[16]
is widely used in first-order training (Ruder, 2016; Loshchilov & Hutter, 2017), but require adaptation for ZO due to its scalar-gradient nature and dual forward passes. In particular, DDP must synchronize scalar gradients and coordinate consistent perturbation vectors across devices. These adaptations are non-trivial and introduce scheduling and communic...
work page 2016
-
[17]
and torch.compile (Ansel et al., 2024), which further emphasize the need for efficient communication in forward-only training. C More Experiments C.1 Additional Scaling Results with GPU Count Table 4: Throughput (Tokens/Second) Comparison Across Models and GPU Counts.Batch size is fixed at 4, using fp16 precision. Model Num GPUs MeZO+DDP ZO2+DDP OPT-1.3B ...
work page 2024
-
[2004]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
Show all 17 references
-
[2013]
Under review
11 Preprint. Under review. Liangyu Wang, Jie Ren, Hang Xu, Junxiao Wang, Huanyi Xie, David E Keyes, and Di Wang. Zo2: Scalable zeroth-order fine-tuning for extremely large language models with limited gpu memory. arXiv preprint arXiv:2503.12668,
-
[2016]
Evolution strategies as a scalable alternative to reinforcement learning
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864,
-
[2017]
Flashattention-2: Faster attention with better parallelism and work partitioning
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691,
-
[2019]
Pytorch distributed: Experi- ences on accelerating data parallel training
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experi- ences on accelerating data parallel training. arXiv preprint arXiv:2006.15704,
2006 arXiv
-
[2020]
Deepseek-v3 technical report
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,
-
[2022]
Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin- Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592,
-
[2023]
Online convex optimization in the bandit setting: gradient descent without a gradient
Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. arXiv preprint cs/0408007,
-
[2024]
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,
1901
-
[2025]
Opt: Open pre-trained transformer language models
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.