{"id":"66f55516-481f-4374-8a02-812e46dc619d","arxiv_id":"2507.03211","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"DistZO2 distributes ZO2's dual perturbed forward passes and scalar gradients across GPUs, achieving up to 3x throughput over ZO2 on OPT-175B while keeping per-GPU memory near 19GB.","lead":"This paper presents DistZO2, a system that runs memory-saving zeroth-order fine-tuning of huge language models across multiple GPUs. It reports up to 3x faster training than the single-GPU ZO2 baseline on OPT-175B while keeping each GPU's memory use near 19GB.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RNG synchronization is underspecified and fine-tuning quality is never validated; the 3x throughput claim rests on the unverified assumption that all GPUs sample identical perturbation vectors z.","rationale":"The reader's weakest assumption—shared-seed RNG consistency across GPUs—is indeed the most load-bearing technical condition in the paper. The paper repeatedly emphasizes \"random number synchronization\" as a challenge (Introduction, Sections 4 and 5) but never specifies the actual synchronization mechanism or verifies that replicas remain bit-identical after updates. This matters because the update rule θ←θ−ηgz is nonlinear in z: even a tiny mismatch in z across ranks makes the central-difference estimate invalid and causes replica drift, which would break the claimed fine-tuning behavior. I partially agree with the reader's framing: the throughput numbers themselves may be correct as measured, but the central claim that DistZO2 performs zeroth-order fine-tuning rather than just parallel forward passes is unverified without accuracy validation or RNG-consistency checks. The reader already assigned CONDITIONAL, which is the appropriate verdict; my concern strengthens that verdict's rationale but does not move it. I would keep CONDITIONAL and require the authors to (a) specify and validate the exact RNG synchronization protocol, and (b) report downstream accuracy or convergence loss for at least one representative DistZO2 run, preferably compared with ZO2 under matched total batch size.","tokens_in":13368,"tokens_out":4730,"duration_ms":62503,"concrete_test":"Instrument the released zo2 code: after the seed broadcast in §4.2 Step 1 and §5.1, log max|z_rank_i − z_rank_j| for the perturbation tensor used by PerturbParameters, and after each update log the max parameter difference across replicas, for OPT-175B on 4 H20 GPUs over 100 SST-2 steps. Also record final validation accuracy/loss and compare with single-GPU ZO2 at the same total batch size. If cross-rank z/parameter differences are nonzero, or if the accuracy comparison is missing or divergent, the RNG-synchronization assumption fails and the central fine-tuning claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—DistZO2 provides memory-efficient zeroth-order fine-tuning at 3x ZO2's throughput—requires that every replica uses the same Gaussian perturbation vector z: PertP needs the +ε and −ε forward passes to use the same z to form a valid central-difference estimate, and DDP needs all replicas to apply the same update θ−ηgz. Sections 4.2 (Step 1) and 5.1 say only that a shared random seed is \"broadcast\"; no mechanism is specified for producing a 175B-element z consistently across devices, nor is ZO2's RNG-state manager (Algorithm 2, which stores and restores per-iteration random states) extended to the multi-GPU setting. If z is regenerated per device, any divergence in device RNG streams, sampling order, or floating-point reduction corrupts the gradient estimate and causes replica drift; if z is instead broadcast as a tensor, the bandwidth cost for a model of this size is not accounted for in the throughput numbers. The paper also reports no loss curves or downstream task accuracy for any DistZO2 run, so the central claim that this is fine-tuning rather than merely faster forward passes is unverified. These are concrete correctness gaps, not style issues.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DistZO2, a distributed extension of the ZO2 zeroth-order fine-tuning framework. It proposes three parallelization strategies: Perturbation Parallelism (PertP), which splits the two perturbed forward passes of the central-difference estimator across two GPUs; Distributed Data Parallelism (DDP), which synchronizes only scalar projected gradients; and a combined 2D parallelism that uses PertP as the inner dimension and DDP as the outer dimension. It also proposes a hardware-aware communication scheme that slices parameter blocks across GPUs to reduce PCIe pressure via NVLink. Experiments on OPT models from 1.3B to 175B report throughput, GPU memory usage, and communication bandwidth, with the headline result being a 3x throughput improvement over ZO2 on OPT-175B using 4 GPUs while keeping per-GPU memory around 18-20 GB. The text includes algorithm descriptions, figures, and tables but no downstream accuracy or loss-curve validation of the fine-tuning itself.","tokens_in":13607,"tokens_out":7322,"duration_ms":91285,"significance":"If the algorithmic and experimental gaps identified below are resolved, the contribution would be practically useful: it would show that memory-efficient zeroth-order fine-tuning of 100B+ parameter models can be accelerated through multi-GPU parallelism while retaining ZO2's low per-GPU memory footprint. The paper is also commendable for reporting concrete throughput and memory numbers on a range of OPT scales and for open-sourcing the implementation, which makes the claims independently checkable. The core ideas of PertP and scalar-gradient DDP are natural and likely to be of interest to the systems-for-LLM and ZO-optimization communities. However, the current manuscript leaves several load-bearing details about the distributed algorithm unspecified and provides no evidence that the reported throughput translates into valid fine-tuning behavior.","major_comments":[{"comment":"The parameter-update step is described in a way that is either incomplete or incorrect. Step 4 states: \"WDk ← WDk − η · g · z\" and asserts that because g and z are synchronized, \"the updates are consistent across devices.\" But in Step 2, GPU 1 computes its forward pass with parameters W+ = WH + εz and GPU 2 with W- = WH - εz. If WDk denotes the same tensor as the perturbed parameters used in the forward pass, then applying −ηgz to the two device copies leaves them at θ + (ε − ηg)z and θ − (ε + ηg)z, which are not equal. The paper never states that each GPU first restores its local parameters to the unperturbed θ (as ZO2's Algorithm 2 does in line 27) before applying the update. Without such a restore step, the replicas diverge and the claimed consistency is false; with a restore step, the paper must say so explicitly and explain how the perturbation is undone in the distributed setting.","section":"§4.2, Step 4 and §5.1"},{"comment":"The synchronization of the random perturbation vector z is underspecified in a way that directly threatens the central-difference estimate. The paper says only that a shared random seed is \"broadcast\" so that devices sample the same z, but it never states the mechanism. In particular, it does not say whether each device samples z from its own RNG after setting that seed, whether z is generated in a canonical parameter order independent of the block-wise offloading scheduler, whether the per-iteration random-state manager of ZO2 (Algorithm 2, lines 4-9 and 19-28) is replicated or synchronized across devices, or whether z is instead materialized on one device and broadcast as a tensor. The last option would incur a prohibitive bandwidth cost for 175B parameters, while the first option requires careful ordering guarantees that are not described. Without this specification, the loss difference (L+ - L-) on each GPU may use different z, which corrupts the gradient estimate, and DDP replicas may apply different updates and drift apart. This is exactly the kind of detail that needs to be pinned down for the paper's correctness claims.","section":"§4.2, Step 1 and §5.1; Algorithm 2"},{"comment":"The paper reports no validation that DistZO2 actually fine-tunes the model. All experiments measure throughput, GPU memory, and communication bandwidth; there are no training-loss curves, no SST-2 validation accuracies, and no comparison with the fine-tuning quality of MeZO or ZO2. Since the paper's central claim is \"fine-tuning,\" not merely accelerated forward passes, it must show that the distributed updates optimize the objective and preserve model quality. This is especially important because the previous two comments identify possible sources of divergence: if z is inconsistent across devices, or if the update step does not first restore the unperturbed parameters, then the measured throughput would be irrelevant to actual fine-tuning. The authors should report at least one training curve and one downstream-task accuracy for a representative model (e.g., OPT-13B or OPT-30B) comparing DistZO2 against ZO2 and MeZO.","section":"§7 and Tables 1-5"},{"comment":"The reported \"3x speedup over ZO2\" on OPT-175B compares 4 GPUs (ZO2+PertP+DDP, 508 tokens/sec) against 1 GPU (ZO2, 166 tokens/sec). If the comparison is meant to demonstrate the benefit of distributed parallelism, the paper should report the speedup per GPU or the scaling efficiency, since a 4-GPU run should ideally be compared against a 1-GPU run with the same global batch size. The text also does not clarify whether the batch size is fixed per GPU or globally: Section 5.1 says \"each GPU processes a different mini-batch\" while Section 7 says \"fixed batch size setting.\" If DDP increases the global batch size with the number of GPUs, part of the throughput gain comes from processing more data per iteration, not from pure computational speedup. The authors should state the global-batch-size convention and, if appropriate, report a balanced comparison with equal total work or equal per-GPU work.","section":"§7.1, Table 1"}],"minor_comments":[{"comment":"The column header \"ZO2+PerP\" contains a typo and should read \"ZO2+PertP.\"","section":"Table 5"},{"comment":"The caption says \"NVLink transfers initiating only after PCIe transfers are complete,\" while the text says \"PCIe and NVLink communication streams can be executed concurrently.\" These statements are contradictory and should be reconciled; the timing model T_comm = M/n (PCIe) + (n-1)M/n (NVLink) assumes at least partial overlap, so the caption should be corrected.","section":"Figure 3 caption and §6.1"},{"comment":"The two columns \"ZO2+DDP+PertP\" and \"ZO2+PertP+DDP\" are said to correspond to \"DDP(inner)+PertP(outer)\" and \"PertP(inner)+DDP(outer)\" in the text, but the naming convention for which parallel strategy is inner versus outer is not defined anywhere. Please define the ordering in the table or caption.","section":"Table 2"},{"comment":"All throughput and memory measurements appear to be single runs with no variance information. For a systems paper whose central claim is throughput, at least a few repeated runs or error bars on the key configurations are needed to rule out run-to-run variability.","section":"§7"},{"comment":"For OPT-175B, the memory usage of ZO2+PertP+DDP is reported as 18713 MB, identical to ZO2, while ZO2+DDP is 19881 MB. Since the 2D configuration uses more GPUs and more communication, the identical memory number is surprising and should be explained.","section":"Table 1"},{"comment":"The paper would benefit from a pseudocode presentation of DistZO2's distributed iteration, analogous to Algorithm 2 for ZO2. Several ambiguities in Steps 2-4 of §4.2 would be resolved by such a specification.","section":"§4.2 and §5.1"}],"recommendation":"major_revision","confidential_remarks":"The authors are also the developers of the ZO2 baseline used throughout the paper, so the comparison is between the authors' new system and their own prior work. This is not itself a problem, but it increases the importance of independent validation of the RNG-synchronization behavior and the fine-tuning quality. The paper's central algorithmic claim currently rests on an underspecified update and RNG mechanism; the recommended revision should require a precise distributed pseudocode and at least one accuracy/loss validation. I would not recommend rejection because the direction is promising and the issues appear fixable within the scope of a systems paper, but the current version is not yet ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nDistZO2 is a useful engineering contribution: it extends ZO2's single-GPU, CPU-offloaded zeroth-order fine-tuning to multi-GPU systems by parallelizing the two perturbed forward passes (PertP), adapting DDP to scalar ZO gradients, and adding a hardware-aware communication scheme that slices parameter blocks across NVLink. The reported numbers are internally consistent, and the 3x throughput gain on OPT-175B is plausible—though it uses four GPUs against ZO2's one, so the speedup is a scaling result rather than an algorithmic leap.\n\nWhat is new is the system-level insight that ZO2's dual forward passes are independent and can run on separate devices, plus the scheduling and communication details that make that work with CPU offloading. The DDP adaptation itself is conceptually simple, but the RNG coordination across replicas is a real detail. The paper only says that a shared seed is broadcast; it does not specify how the 175B-element perturbation vector is sampled consistently across GPUs, nor how the RNG state manager from ZO2 extends to the distributed setting. This is under-specified but likely fixable.\n\nThe bigger problem is validation. The paper reports throughput and memory for OPT models up to 175B on SST-2, but no downstream accuracy, no loss curves, and no comparison of final model quality against ZO2 or MeZO. Since the entire premise is that this is fine-tuning, not just faster forward passes, this omission is significant. The measurements are also single runs with no variance, which is acceptable for a systems paper only if the effects are large and consistent, as they appear to be.\n\nNone of these are fatal. The engineering looks sound, the math is correct, and the paper is honest about building on ZO2 and prior distributed systems work. It deserves a serious referee, but it should be returned for major revision: add at least one accuracy table, specify the RNG mechanism precisely, and report multiple runs or at least per-GPU efficiency. I would not cite it as evidence that ZO fine-tuning quality is preserved until those gaps are closed, but I would keep it on the table for the systems contribution.","headline":"Useful multi-GPU extension of ZO2 with real throughput gains, but missing accuracy validation and an underspecified RNG mechanism keep it from being fully convincing.","tokens_in":14116,"tokens_out":3174,"would_cite":false,"duration_ms":37063,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DistZO2 claims that zeroth-order fine-tuning of a 175B-parameter language model can run on four GPUs with under 20GB each and triple the throughput of single-GPU ZO2 by running the two perturbed forward passes concurrently and…","keywords":["zeroth-order optimization","LLM fine-tuning","CPU offloading","distributed data parallelism","perturbation parallelism","memory-efficient training","communication optimization","multi-GPU training"],"falsifier":"Run identical fine-tuning of the same model on one GPU with ZO2 and on two GPUs with PertP using the same seed, hyperparameters, and data order, then compare projected gradients and final validation accuracy. If the two GPUs' RNG states diverge despite the shared seed, per-iteration gradients will differ by more than floating-point noise and replica parameters will drift; any measurable divergence, or an accuracy gap beyond ordinary seed noise, would falsify the claim that PertP preserves ZO2 behavior.","tokens_in":13190,"feed_emoji":"⚡","tokens_out":8486,"duration_ms":88612,"temperature":0.7,"pith_summary":"The paper claims that zeroth-order fine-tuning of hundred-billion-parameter language models can be made several times faster without sacrificing its tiny memory footprint by spreading the two perturbed forward passes and separate data batches across multiple GPUs. It packages this as DistZO2, built on the single-GPU ZO2 idea of keeping most parameters in CPU memory and loading one transformer block at a time. If the claims hold, a 175B model can be fine-tuned on four GPUs with about 19GB per device at roughly 508 tokens per second, triple the 166 tokens per second of its single-GPU predecessor. The central obstacles the framework must overcome are keeping every device's random perturbation identical and preventing communication from becoming the new bottleneck.","feed_headline":"No-backprop LLM fine-tuning gets 3x speedup on 4 GPUs","feed_subtitle":"A 175B-parameter model can be fine-tuned at 508 tokens/sec while using under 20GB per GPU.","key_machinery":"The load-bearing object is the scalar projected gradient $g = (L(\\theta+\\epsilon z) - L(\\theta-\\epsilon z))/(2\\epsilon)$, which replaces the high-dimensional gradient tensor of first-order training. Because $g$ is a scalar, all-reducing it costs almost nothing and every replica can update locally as $\\theta \\leftarrow \\theta - \\eta g z$, provided every device draws the same $z$. Around this the paper builds two parallel axes: Perturbation Parallelism (PertP), which assigns the $+\\epsilon$ and $-\\epsilon$ passes to two GPUs; and an adapted Distributed Data Parallelism, which averages per-batch scalars. The 2D design organizes $n = n_b \\times 2$ GPUs with PertP on the inner axis and DDP on the outer axis, and a hardware-aware transfer scheme slices each parameter block into $n$ pieces so PCIe carries only one slice per GPU while NVLink handles the rest; offloading is likewise sliced because all replicas hold identical parameters.","core_discovery":"On its own terms, DistZO2's central claim is that the two forward passes zeroth-order fine-tuning already needs—one with parameters shifted by $+\\epsilon z$ and one by $-\\epsilon z$—are logically independent, so they can run concurrently on different GPUs instead of sequentially on one. The paper shows this perturbation parallelism plus an adapted distributed data parallelism, in which devices each process a different batch and synchronize a single scalar projected gradient, can be combined into a 2D mesh where each GPU does exactly one forward pass. With a communication scheme that slices each CPU-resident transformer block and moves one slice per GPU over PCIe while the rest travel over NVLink, the framework reports a 3x throughput gain over ZO2 on OPT-175B, restoring MeZO-level throughput while using under 20GB of GPU memory per device.","pith_inferences":["If the seed-synchronization assumption is made airtight, the same 2D recipe should extend beyond the paper's exact setup to other forward-only optimizers and to larger node counts, since each PertP pair can be replicated as its own DDP group.","The offload-slicing shortcut depends on all replicas holding identical parameters after every step; a zeroth-order variant with per-parameter adaptive states, such as a ZO-Adam-style optimizer, would break that identity and need a different offload protocol.","A natural but untested extension is more than two perturbation evaluations per step, which would replace the 2D mesh with a higher-dimensional grid and change the communication-versus-compute trade-off the paper measures."],"forward_implications":["If the reported numbers hold, a 175B-parameter model can be zeroth-order fine-tuned on four GPUs with roughly 19GB of memory per device at about 508 tokens per second, three times the 166 tokens per second of single-GPU ZO2.","Perturbation Parallelism alone should give close to a 2x throughput gain across model scales, since the two perturbed forward passes are the dominant compute and run concurrently.","Adapted DDP should scale nearly linearly with GPU count, because synchronizing a single scalar per iteration adds negligible communication overhead.","The hardware-aware slicing should cut per-GPU PCIe traffic by a factor equal to the GPU count and improve measured upload and offload bandwidth by up to about 4.6x.","Because MeZO and ZO2 share the same dual-forward and scalar-gradient structure, the paper argues the same parallelization strategies transfer directly to MeZO as well."],"supporting_citations":[{"why":"ZO2, the single-GPU CPU-offloading framework whose dual-forward scheduler DistZO2 extends and whose throughput is the main baseline.","marker":"(Wang et al., 2025)"},{"why":"MeZO, which established forward-only zeroth-order fine-tuning and supplies the projected-gradient algorithm and the primary throughput comparison.","marker":"(Malladi et al., 2023)"},{"why":"PyTorch DDP, the data-parallel pattern that DistZO2 adapts to scalar zeroth-order gradients.","marker":"(Li et al., 2020)"},{"why":"The OPT model family, which provides the 1.3B-to-175B evaluation subjects for all experiments.","marker":"(Zhang et al., 2022)"},{"why":"SST-2, the dataset used for fine-tuning and throughput measurements throughout the paper.","marker":"(Socher et al., 2013)"},{"why":"Cited as prior art for tensor slicing and peer-to-peer communication patterns behind the hardware-aware communication optimization.","marker":"(Shoeybi et al., 2019)"}],"fun_headline_variants":["Distributed no-backprop fine-tuning hits 3x speedup on 175B","ZO fine-tuning goes parallel: two forward passes on separate GPUs","No-backprop LLM fine-tuning gets 3x faster with distributed GPUs","Memory-efficient zeroth-order fine-tuning scales to 175B with 3x gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design assumes that broadcasting one random seed makes every GPU generate exactly the same Gaussian perturbation vector $z$, so the two perturbed losses estimate the same directional derivative and all replicas stay synchronized; the paper states that a seed is broadcast but does not specify the sampling mechanism or verify that trained models match single-GPU ZO2 quality.","fun_headline_variants_meta":{"raw":{"variants":["Distributed no-backprop fine-tuning hits 3x speedup on 175B","ZO fine-tuning goes parallel: two forward passes on separate GPUs","No-backprop LLM fine-tuning gets 3x faster with distributed GPUs","Memory-efficient zeroth-order fine-tuning scales to 175B with 3x gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000368,"raw_usage":{"total_tokens":2021,"prompt_tokens":1037,"completion_tokens":984,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":898}},"tokens_in":653,"tokens_out":984,"duration_ms":10184,"temperature":1.0,"reasoning_tokens":898,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:15:46.877281+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run identical fine-tuning of the same model on one GPU with ZO2 and on two GPUs with PertP using the same seed, hyperparameters, and data order, then compare projected gradients and final validation accuracy. If the two GPUs' RNG states diverge despite the shared seed, per-iteration gradients will differ by more than floating-point noise and replica parameters will drift; any measurable divergence, or an accuracy gap beyond ordinary seed noise, would falsify the claim that PertP preserves ZO2 behavior.","supporting_citations":[],"review_version":1}