Pith. sign in

REVIEW 4 major objections 5 minor 64 references

ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ZenFlow makes offloaded LLM fine-tuning up to 5x faster by updating only important gradients on the GPU.

desk verdict A solid systems paper with a real locality observation, a broad evaluation, and an under-built convergence proof that should be fixable in revision. read the letter →

arxiv 2505.12242 v3 pith:OXNRGDLU submitted 2025-05-18 cs.DC cs.LG

Tingfeng Lan , Yusen Wu , Bin Ma , Zhaoyuan Su , Rui Yang , Tekin Bicer , Masahiro Tanaka , Olatunji Ruwase
show 2 more authors
Dong Li Yue Cheng
This is my paper · ORCID
classification cs.DCcs.LG
keywords offloadedtrainingasynchronousupdatesgradientimportancespatiallocalitytemporalGPUstallsCPU-GPUpipelineboundedstaleness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the main reason offloaded LLM fine-tuning is slow is that every parameter is treated alike: the full model is sent to the CPU for every update, leaving fast GPUs idle. ZenFlow instead splits parameters by learning importance, updating a small important subset in place on the GPU while a CPU process asynchronously accumulates the rest. Its core empirical claim is that important gradients cluster in a small, slowly changing set of input channels, making lightweight channel-level selection a faithful stand-in for expensive global top-k ranking. If this holds, offloaded fine-tuning can run 3.6--5x faster with about half the PCIe traffic, over 85 percent less GPU stall time, and the same accuracy. The paper backs the approach with a bounded-staleness convergence result, $O\big(\sqrt{(1+\rho S)/T}\big)$, which it says matches the ideal synchronous SGD rate in form.

What carries the argument

The load-bearing mechanism is a channel-level gradient importance proxy. Instead of gathering the full gradient matrix, which would cost tens of gigabytes per iteration in sharded training, each GPU computes per-column squared gradient norms, shares a compact vector, and caches a slowly refreshed top-10% set of important input channels; for a $4096 \times 4096$ weight matrix this cuts selection communication by over 4,000x. On top of that proxy, a selective GPU optimizer updates only the important channels in place, while a double-buffered CPU accumulator gathers unimportant gradients and applies them every $S$ iterations, fully overlapping CPU work with GPU compute. The convergence analysis couples these pieces through the bounded-staleness rate $O\big(\sqrt{(1+\rho S)/T}\big)$, where $\rho$ is the fraction of gradient-norm energy in delayed coordinates and $S$ is the accumulation interval.

What would settle it

Run the system on a fine-tuning task that switches domains mid-training, and record both the retention rate of the cached top-10% channel set and final accuracy; if retention of the top-1% gradients falls below about 90% for a sustained window while the update interval is 4, the promised accuracy preservation should measurably degrade, showing that the spatial-temporal locality premise is load-bearing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that in LLM fine-tuning the top 1% of gradients carry about 90% of the total gradient norm, and those high-magnitude gradients are not scattered: they persist in a narrow set of input channels across iterations, so a fixed top-10% channel set retains over 95% of the top-1% gradients over 100 steps. ZenFlow exploits this by performing in-place optimizer updates on the GPU for important channels and offloading the remaining gradients to the CPU, where they accumulate over a few iterations before an asynchronous update. The resulting partial staleness has a bounded cost: with the default accumulation interval $S=4$ and measured CPU-side gradient-energy fraction $\rho \approx 0.10$, the convergence penalty is a factor of $\sqrt{1.4} \approx 1.18$ relative to ideal synchronous SGD, and a short synchronous warm-up reduces the effective penalty to about 0.12. Across models from 350M to 13B parameters, the paper reports 3.6--5x end-to-end speedups over the state-of-the-art offloading baseline, roughly 2x lower I/O traffic, and accuracy matching or exceeding the baseline.

Load-bearing premise

The design assumes that a small, slowly changing set of input channels keeps carrying the truly important gradients for the whole fine-tuning run, so a cached top-10% channel set acts as a faithful proxy for global top-k importance.

Editorial extensions

If this is right

  • Offloaded fine-tuning of 7B--13B models would run 3.6--5x faster than current offloading systems on the same hardware, because GPU stalls are the dominant overhead.
  • PCIe traffic per iteration would drop from two full model copies to about 1.125 model copies for $S=4$ and a 10% important ratio, easing the communication bottleneck that limits transfer-bound training.
  • GPU stalls would fall by over 80% even when the CPU is heavily under-provisioned, e.g., with only 8 CPU threads, making shared GPU clusters more predictable for offloaded training.
  • The convergence guarantee would remain within about 18% of ideal synchronous SGD in default settings, and within about 12% with a 5% warm-up period, so accuracy preservation is argued as a formal property rather than only an empirical one.
  • The approach would scale to at least 13B parameters across four GPUs with model sizes comparable to conventional offloading, since the per-column proxy avoids global synchronization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to let the same stable channel set drive other memory-saving decisions, such as selective quantization, pruning, or gradient compression, since the paper only applies it to update scheduling and offloading.
  • The bounded-staleness bound assumes the CPU-side gradient-energy fraction $\rho$ stays near 0.1; if a later training phase shifts importance to new channels faster than the cached set refreshes, the effective penalty could exceed the reported 18%, so an adaptive refresh rate may be needed for tasks with sharp distribution shifts.
  • The fixed $S=4$ interval is a compromise: the paper's own auto-tuned variant suggests that starting with $S=1$--$2$ and relaxing later would make the speedup-accuracy tradeoff more robust for short fine-tuning runs.
  • The locality claim was measured on instruction-following and benchmark tasks; a quick test would be to monitor retention of the cached channel set on a multi-task or domain-switching fine-tuning run, where the assumption is most likely to fail.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. ZenFlow is an offloading framework for fine-tuning large language models that partitions parameters into an important subset updated in-place on the GPU and a less-important subset asynchronously accumulated and updated on the CPU. It uses a channel-level gradient-norm proxy to identify important parameters without global all-gather, exploits claimed spatial and temporal locality of important gradients, and overlaps CPU optimizer work with GPU computation using double buffering. The paper claims up to 5x end-to-end speedup, 2x lower PCIe traffic, over 85% reduction in GPU stalls, and accuracy preservation, supported by a bounded-staleness convergence rate of O(sqrt((1+rho*S)/T)) that is stated as matching ideal synchronous SGD up to a small constant.

Significance. If the performance and accuracy claims hold, ZenFlow addresses a practical and important bottleneck: GPU idle time caused by CPU-side optimizer updates in offloaded LLM fine-tuning. The evaluation is broad and internally consistent, covering five model families, two GPU platforms, and three CPU configurations, and the ZenFlow vs ZenFlow* ablation cleanly isolates the contribution of the pipelining design. The spatio-temporal locality observation for important gradients is a useful, falsifiable empirical finding. However, the theoretical convergence guarantee is asserted rather than derived, the key staleness parameter rho is not measured over training, and the reported I/O reduction is weaker than the abstract claims; these issues currently limit the strength of the central claims.

major comments (4)
  1. [§3.4 (bounded-staleness result)] The convergence bound O(sqrt((1+rho*S)/T)) is stated without a theorem statement, proof, or a citation to a result that implies this exact formula. The update rule for theta^(c), which is updated only every S iterations using alpha_t/S times the sum of gradients computed at theta_i for i in [t-S+1, t], is a partial-delay scheme not directly covered by the cited async-SGD analyses [11,22,24,54]. The text says the result follows from 'common assumptions' but does not specify the Lipschitz constant, the variance bound, or the step-size schedule, so the claim that ZenFlow achieves 'the same as the ideal rate of synchronous SGD' is not checkable as written. Please provide a complete theorem and proof, or explicitly derive the bound from a cited theorem, and state clearly that the proof is for SGD while the system uses AdamW.
  2. [§3.4 and §5.4 (rho dynamics)] The small penalty factor sqrt(1+rho*S) depends on rho staying near 0.10 for the whole training run, but the paper never reports rho(t), the fraction of gradient energy in the delayed coordinates, over time. Figure 6b reports channel retention rate and Figure 9 reports a cumulative distribution of top-k gradients, neither of which is rho(t). As the GPU-updated important coordinates converge, their gradient energy can shrink, so the delayed coordinates' energy fraction can grow late in training, exactly when final validation accuracy is determined. The authors should report rho(t) for the evaluated workloads and either bound rho in the convergence theorem via a schedule-dependent argument or add a mechanism that provably keeps rho small.
  3. [§3.2 and abstract (I/O traffic)] The paper's own I/O formula gives an average per-iteration traffic of (S+1)*(1-k)*M/S; with the stated representative values S=4 and k=0.1 this is 5*0.9*M/4 = 1.125M, compared to 2M for ZeRO-Offload. That is a 1.78x reduction, not the '2x lower PCIe traffic' claimed in the abstract. Please either qualify the abstract and §3.2, report measured byte counts, or state that 2x is approached only for larger S.
  4. [§5.2 and Fig. 10 (accuracy preservation)] The accuracy-preservation claim is not fully supported by the reported experiments. Section 5.2 explicitly states that OPT-350M accuracy is slightly lower under the fixed update interval S=4, which is in tension with the abstract's 'all while preserving accuracy.' In addition, Fig. 10 reports no error bars or seeded repeats for the GLUE results, so differences of 0.005-0.02 in accuracy cannot be distinguished from stochastic noise. Please report mean and standard deviation over at least three seeds, and show whether Zen-auto recovers the OPT-350M accuracy loss in the main accuracy figure.
minor comments (5)
  1. [Abstract/author list] The abstract contains the typo 'fulling overlapping' instead of 'fully overlapping', and the name 'Tekin Bicer' appears twice in the author list.
  2. [§2.3 and Fig. 4] The text says 'the top 1% of gradients account for over 90% of the total gradient norm', while Fig. 4 reports 88.9% for the Qwen2.5-0.5B run; please make the numbers consistent or add the qualifier 'approximately'.
  3. [§3.4 (warm-up penalty)] The warm-up penalty example does not evaluate to the stated 0.12. With beta=0.6, tau/T=0.05, rho=0.1, and S=4, the displayed formula gives sqrt(1 + 0.4*(1 - 0.05^0.4)) ≈ 1.131, i.e., a 13.1% penalty, not 0.12; please correct the arithmetic or the formula.
  4. [§5.1 and Fig. 10] The paper says fine-tuning is evaluated on the GLUE benchmark, but only four tasks (MNLI, QNLI, QQP, SST-2) appear in Fig. 10; please state explicitly that this is a subset of GLUE or add the remaining tasks.
  5. [Fig. 15(a)] The 'Zen-Auto' marker in Fig. 15(a) is not explained in the text; please describe how the auto-tuned configuration is obtained and how it differs from the fixed S=4 configuration in Fig. 10.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence bound in §3.4 is asserted rather than derived, but it is parameterized by a measured quantity (ρ) rather than fitted to the reported accuracy or speedup, and the central accuracy and performance claims are empirically validated on GLUE and measured system runs.

full rationale

I found no step in which a claimed result reduces by construction to a fitted input or to a load-bearing self-citation. The central theoretical statement in §3.4 is the bound O(sqrt((1+ρS)/T)), which the paper introduces with 'We prove' but then states without a derivation. This is an omitted proof and a rigor concern, but it is not circularity: the bound is conditional on ρ, which is defined as the fraction of gradient-norm energy in the delayed coordinates and is reported as an empirically measured value (ρ≈0.10), not fitted to match the reported accuracy or speedup. The reported '18% slowdown' and '0.12x penalty' are arithmetic consequences of the chosen S=4 and the measured ρ, not predictions that are statistically forced by those inputs. The paper's main accuracy claims are tested on GLUE benchmarks against ZeRO-Offload and other baselines (Figures 10 and 14), and the speedup and stall-reduction numbers are measured system results (Figures 11 and 13). The I/O traffic reduction is an analytic identity: average traffic (S+1)(1−k)M/S, which is not circular because it directly follows from the stated accumulation interval and top-k ratio. The paper also includes honest limitation statements, e.g., §5.2 reports slightly lower accuracy for OPT-350M under fixed S=4, and §5.5 reports a 0.02 accuracy drop at S=16; these concessions further indicate that the empirical claims are not constructed to match the theory. Self-citations are present, such as [48] for model popularity and [7] for asynchronous training, but they are not used to justify the central convergence or accuracy claims. Consequently, the derivation chain does not exhibit the kind of self-definitional or fitted-input circularity that would raise the score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

ZenFlow introduces no new physical entities, forces, or particles. Its components, such as the selective optimizer and the segment mapping table, are engineering artifacts rather than new postulated natural entities. The ledger entries above capture the hand-selected hyperparameters and the empirical or asserted assumptions on which the central performance and convergence claims rest.

free parameters (4)
  • topk_ratio = 10% (default 0.10)
    Chosen based on retention-rate analysis in Sections 2.3 and 5.5; it controls which channels and parameters receive GPU-side updates. Sensitivity to 1%, 5%, and 10% is explored.
  • update interval S = 4
    Set to 4 accumulation steps to hide CPU update latency and bound staleness, as discussed in Sections 3.2 and 3.4. The paper also evaluates S=2, 8, and 16.
  • rho (delayed gradient energy fraction) = ≈0.10
    Measured empirically on Qwen2.5-0.5B fine-tuning on Alpaca52K (Section 3.4). It determines the numerical staleness penalty in the convergence bound, so the 0.12x penalty claim depends on this measured value.
  • warm-up steps tau = 7,500 (5% of T=150,000 in the analysis example)
    Used in the closed-form warm-up penalty calculation in Section 3.4. Whether the implemented system actually applies this warm-up phase is not clearly stated in the evaluation.
assumptions (5)
  • standard math Unbiased gradients, bounded variance, and L-smoothness for SGD analysis
    Invoked without proof in Section 3.4 as common assumptions for the convergence bound; these are standard in the cited distributed SGD literature.
  • ad hoc to paper Bounded-staleness SGD convergence theorem with penalty factor sqrt(1+rho S)
    The paper states 'we prove' an O(1/sqrt(T)) bound with a staleness factor, but gives only the final display equation and no proof steps or explicit citation for this exact form.
  • domain assumption Channel locality: a fixed top-10% channel set retains over 90-95% of top-1% gradients across iterations
    This is an empirical observation from Qwen2.5-0.5B on Alpaca52K (Figures 5, 6, 9), generalized to all evaluated models and tasks. It is load-bearing for the selection proxy and for the rho≈0.10 assumption.
  • domain assumption Gradient energy decays as E[||grad L(theta_t)||^2] ~ t^{-beta} with beta between 0.4 and 0.6
    Imported from scaling-law references [14, 19, 28] in Section 3.4 and used to compute the warm-up penalty reduction.
  • domain assumption Adam and AdamW mitigate gradient staleness via momentum and adaptive learning rates
    Asserted in Section 3.4 without proof, and relied on to argue that the actual AdamW-based implementation has even smaller penalty than the SGD analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates." pith.science (2026). https://pith.science/paper/OXNRGDLU

@misc{pith2026250512242,
  author       = {Pith},
  title        = {Pith review of: ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OXNRGDLU}},
  note         = {Machine review of arXiv:2505.12242}
}
read the original abstract

Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs sit idle waiting for slow CPU updates and limited-bandwidth PCIe transfers. We present ZenFlow, a new offloading framework that prioritizes important parameters and decouples updates between GPU and CPU. ZenFlow performs in-place updates of important gradients on GPU, while asynchronously offloading and accumulating less important ones on CPU, fully overlapping CPU work with GPU computation. To scale across GPUs, ZenFlow introduces a lightweight gradient selection method that exploits a novel spatial and temporal locality property of important gradients, avoiding costly global synchronization. ZenFlow achieves up to 5x end-to-end speedup, 2x lower PCIe traffic, and reduces GPU stalls by over 85 percent, all while preserving accuracy.

Figures

Figures reproduced from arXiv: 2505.12242 by the authors.

Figure 1
Figure 1. GPU utilization of ZeRO-Offload and ZenFlow for fine￾tuning Llama2-7B on 4× A100. Unfortunately, this comes at the cost of substantial train￾ing overhead, as CPU-side updates are orders of magnitude slower than GPU computation, and communication over PCIe is constrained by limited bandwidth. For example, on fine-tuning a Llama-2-7B [45] model with 4 A100 GPUs, one training step time experiences a dramatic increase f… view at source ↗
Figure 2
Figure 2. Offloading strategy comparison. (a) ZeRO-Offload [37] sequentially executes FP and BP on GPU, then offloads gradients and performs UP on CPU, leaving the GPU idle. (b) StrongHold [43] overlaps CPU updates with GPU backward computation and gradient offloading by offloading and updating gradients layer by layer. However, CPU-side update is still too slow to hide, causing GPU stalls. (c) Example: iteration with accumul… view at source ↗
Figure 3
Figure 3. Per-iteration time breakdown under DeepSpeed ZeRO-Offload when training Qwen2.5-{1.5B,3B} and Llama2- {7B, 13B} models with 4 A100 40GB GPUs and a AMD EPYC processor with 64 CPU cores (128 threads, SMT enabled). Gra￾dient offloading (GO) represents the time spent transferring gra￾dients from GPU to CPU. 0.0 0.5 1.0 Proportion of Gradients 0.00 0.25 0.50 0.75 1.00 Cum. Grad. Norm² Top 1% 88.9% Norm² [PITH_FULL_IMAGE… view at source ↗
Figures from the paper (8 more)
Figure 6
Figure 6. Figure 6: Temporal locality of important gradients. (a) Input chan￾nels tracked over 1,000 iterations. The Y-axis shows input channel index, with the top-1% important channels highlighted in yellow. (b) Retention rate of top-1% gradients over time when tracking a fixed set of to…
Figure 7
Figure 7. Figure 7: Zero-stall pipeline with double buffering. overlap without additional delay. Empirically, we set the ac￾cumulation interval to 4 steps (𝑆 = 4), which is sufficient to hide CPU update latency in most cases while bounding staleness (see §3.4 for theoretical analysis). To…
Figure 9
Figure 9. Figure 9: CDF of top-𝑘 elements and gradient norm. The channel index is sorted by the number of top-𝑘 elements. both communication and top-𝑘 selection complexity from 𝑂(𝑛𝑚) to 𝑂(𝑚) while preserving gradient magnitude infor￾mation and omitting only directional components. For exa…
Figure 10
Figure 10. Figure 10: Accuracy vs. per-iteration speedup on GLUE tasks for various models and baselines. enhance scalability and performance—for example, by com￾bining with gradient compression techniques or enabling deeper offloading with computational storage devices [17]. ZenFlow Varian…
Figure 11
Figure 11. Figure 11: Throughput compar￾ison across different model and GPU count configurations. 1 GPU 2-GPU 4-GPU 0 5 10 15 20 Model size (B) ZenFlow* ZenFlow ZeRO-Offload [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 13
Figure 13. Figure 13: Training time breakdown and speedup across hardware configurations and models. Bars correspond to the left Y-axis. Lines (with markers) show relative speedup over ZeRO-Offload (ZO), referenced on the right Y-axis. Model Scale. We compare the maximum trainable model si…
Figure 14
Figure 14. Figure 14: Convergence on GLUE with OPT-350M. ZenFlow matches ZeRO-Offload in both loss and accuracy across tasks. terms of accuracy, ZenFlow matches or slightly outperforms ZeRO-Offload throughout training (e.g., QNLI), converging at a similar rate in terms of iterations, but a…
Figure 15
Figure 15. Figure 15: Sensitivity analysis of key hyperparameters. 5.5 Sensitivity of Hyperparameters We investigate the impact of hyperparameters in ZenFlow, specifically the top-𝑘 ratio and the update interval for CPU￾side gradient handling. By assigning longer update intervals to less i…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 27 canonical work pages

  1. [1]

    So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V

    Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a Human-like Open-Domain Chatbot. arXiv:2001.09977 [cs.CL]https://arxiv.org/ abs/2001.09977

    arXiv 2020

  2. [2]

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrin- sic Dimensionality Explains the Effectiveness of Language Model Fine- Tuning. arXiv:2012.13255 [cs.LG]https://arxiv.org/abs/2012.13255

    arXiv 2020

  3. [3]

    Alham Fikri Aji and Kenneth Heafield. 2017. Sparse communication for distributed gradient descent.arXiv preprint arXiv:1704.05021(2017)

    arXiv 2017

  4. [4]

    Olivier Beaumont, Lionel Eyraud-Dubois, and Alena Shilova. 2021. Efficient Combination of Rematerialization and Offloading for Training DNNs. InAdvances in Neural Information Processing Sys- tems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 23844– 23857.https://proceedings.neurips.cc...

    work page 2021

  5. [5]

    Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. InProceedings of COMPSTAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers. Springer, 177–186

    2010

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    2020

  7. [7]

    Zheng Chai, Yujing Chen, Ali Anwar, Liang Zhao, Yue Cheng, and Huzefa Rangwala. 2021. FedAT: A high-performance and communication-efficient federated learning system with asynchro- nous tiers. InProceedings of the international conference for high per- formance computing, networking, storage and analysis. 1–16

    work page 2021

  8. [8]

    Siyuan Chen, Zhuofeng Wang, Zelong Guan, Yudong Liu, and Phillip B Gibbons. 2025. Practical Offloading for Fine-Tuning LLM on Com- modity GPU via Learned Sparse Projectors. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 23614–23622

    work page 2025

Show all 64 references
  1. [9]

    Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, Tamay Besiroglu, and David Owen. 2025. The rising costs of training frontier AI models. arXiv:2405.21015 [cs.CY]https://arxiv.org/abs/2405.21015

    2025 arXiv

  2. [10]

    Ganger, Phillip B

    Henggang Cui, James Cipar, Qirong Ho, Jin Kyu Kim, Seunghak Lee, Abhimanu Kumar, Jinliang Wei, Wei Dai, Gregory R. Ganger, Phillip B. Gibbons, Garth A. Gibson, and Eric P. Xing. 2014. Exploiting bounded staleness to speed up big data analytics. InProceedings of the 2014 USENIX...

    2014

  3. [11]

    Wei Dai, Yi Zhou, Nanqing Dong, Hao Zhang, and Eric P Xing. 2018. Toward understanding the impact of staleness in distributed machine learning.arXiv preprint arXiv:1810.03264(2018)

    2018 arXiv

  4. [12]

    DeepSpeed Team. 2025. DeepSpeed Flops Profiler.https://www. deepspeed.ai/tutorials/flops-profiler/. Accessed: 2025-05-16

    2025

  5. [13]

    Nikoli Dryden, Tim Moon, Sam Ade Jacobs, and Brian Van Essen

  6. [14]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al . 2022. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556(2022)

    2022 arXiv

  7. [15]

    Chien-Chin Huang, Gu Jin, and Jinyang Li. 2020. SwapAdvisor: Push- ing Deep Learning Beyond the GPU Memory Limit via Smart Swap- ping. InProceedings of the Twenty-Fifth International Conference on Ar- chitectural Support for Programming Languages and Operating Systems (Lausann...

    2020

  8. [16]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. InAdvances in Neural Informa- tion Processi...

    2019

  9. [17]

    Hongsun Jang, Jaeyong Song, Jaewon Jung, Jaeyoung Park, Young- sok Kim, and Jinho Lee. 2024. Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System. arXiv:2403.06664 [cs.AR]https://arxiv.org/abs/2403.06664

    2024 arXiv

  10. [18]

    Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, and Jonni Kanerva

  11. [19]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361(2020)

    2020 arXiv

  12. [20]

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2004.04906 [cs.CL]https://arxiv.org/abs/2004.04906

    2020 arXiv

  13. [21]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Sto- chastic Optimization. arXiv:1412.6980 [cs.LG]https://arxiv.org/abs/ 1412.6980

    2017 arXiv

  14. [22]

    Anastasiia Koloskova, Sebastian U Stich, and Martin Jaggi. 2022. Sharper convergence guarantees for asynchronous SGD for distributed and federated learning.Advances in Neural Information Processing Systems35 (2022), 17202–17215

    2022

  15. [23]

    Andersen, Jun Woo Park, Alexander J

    Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, and Bor-Yiing Su. 2014. Scaling distributed machine learning with the parameter server. InProceedings of the 11th USENIX Conference on Operating Sys- tems De...

    2014

  16. [24]

    Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. 2015. Asyn- chronous parallel stochastic gradient for nonconvex optimization. Advances in neural information processing systems28 (2015)

    2015

  17. [25]

    Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. 2017. Deep gradient compression: Reducing the communication bandwidth for distributed training.arXiv preprint arXiv:1712.01887(2017)

    2017 arXiv

  18. [26]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101(2017)

    2017 arXiv

  19. [27]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. InInternational Conference on Learning Representations. https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  20. [28]

    Kairong Luo, Haodong Wen, Shengding Hu, Zhenbo Sun, Zhiyuan Liu, Maosong Sun, Kaifeng Lyu, and Wenguang Chen. 2025. A Multi- Power Law for Loss Curve Prediction Across Learning Rate Schedules. arXiv preprint arXiv:2503.12811(2025)

    2025 arXiv

  21. [29]

    Mustafa Rafique, Franck Cappello, and Bogdan Nicolae

    Avinash Maurya, Jie Ye, M. Mustafa Rafique, Franck Cappello, and Bogdan Nicolae. 2024. Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers. InProceedings of the 14th Workshop on AI and Scientific Computing at Sca...

    2024

  22. [30]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Se- shadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019. PipeDream: generalized pipeline parallelism for DNN training. InProceedings of the 27th ACM Symposium on Op- erating Systems P...

    2019

  23. [31]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chil- amkurthy, Benoit Steiner, Lu F...

  24. [32]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019)

    2019

  25. [33]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He

  26. [34]

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learning. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis(St....

    2021

  27. [35]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He

  28. [36]

    Jie Ren, Jiaolin Luo, Kai Wu, Minjia Zhang, Hyeran Jeon, and Dong Li

  29. [37]

    2021.{Zero-offload}: Democratizing{billion-scale} model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021.{Zero-offload}: Democratizing{billion-scale} model training. In2021 USENIX Annual Technical Conference (USENIX ATC 21). 551– 564

    2021

  30. [38]

    Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. InProceedings of the 35th Interna- tional Conference on Machine Learning

    2018

  31. [39]

    Li Shen, Yan Sun, Zhiyuan Yu, Liang Ding, Xinmei Tian, and Dacheng Tao. 2023. On Efficient Training of Large-Scale Deep Learning Models: A Literature Review. arXiv:2304.03589 [cs.LG]https://arxiv.org/abs/ 2304.03589

    2023 arXiv

  32. [40]

    InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 3505–3506

  33. [41]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053(2019)

    2019 arXiv

  34. [42]

    In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)

    Sentinel: Efficient Tensor Migration and Allocation on Heteroge- neous Memory Systems for Deep Learning. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 598–

  35. [43]

    Xiaoyang Sun, Wei Wang, Shenghao Qiu, Renyu Yang, Songfang Huang, Jie Xu, and Zheng Wang. 2022. Stronghold: fast and afford- able billion-scale deep learning model training. InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–17

    2022

  36. [44]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation L...

    2023 arXiv

  37. [45]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 arXiv

  38. [46]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A Multi-Task Bench- mark and Analysis Platform for Natural Language Understanding. arXiv:1804.07461 [cs.CL]https://arxiv.org/abs/1804.07461

    2019 arXiv

  39. [47]

    Sheng Shen, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. 2020. Powernorm: Rethinking batch normalization in trans- formers. InInternational conference on machine learning. PMLR, 8741– 8751

    2020

  40. [48]

    Zirui Wang, Tingfeng Lan, Zhaoyuan Su, Juncheng Yang, and Yue Cheng. 2025. Towards Efficient LLM Storage Reduction via Tensor Deduplication and Delta Compression. arXiv:2505.06252 [cs.DB]https: //arxiv.org/abs/2505.06252

    2025

  41. [49]

    Nikko Ström. 2015. Scalable distributed DNN training using commod- ity GPU cloud computing. (2015)

    2015

  42. [50]

    Hanfei Yu, Hao Wang, Devesh Tiwari, Jian Li, and Seung-Jong Park

  43. [51]

    Wei Zhang, Suyog Gupta, Xiangru Lian, and Ji Liu. 2015. Staleness- aware async-sgd for distributed deep learning.arXiv preprint arXiv:1511.05950(2015)

    2015 arXiv

  44. [52]

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, An- ima Anandkumar, and Yuandong Tian. 2024. GaLore: Memory- Efficient LLM Training by Gradient Low-Rank Projection. InPro- ceedings of the 41st International Conference on Machine Learning (Pro- ceedings of Machine Learni...

    2024

  45. [53]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al

  46. [54]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Mod- els for Code Understanding and Generation. arXiv:2109.00859 [cs.CL] https://arxiv.org/abs/2109.00859

    2021 arXiv

  47. [56]

    Eric P Xing, Qirong Ho, Wei Dai, Jin-Kyu Kim, Jinliang Wei, Seunghak Lee, Xun Zheng, Pengtao Xie, Abhimanu Kumar, and Yaoliang Yu

  48. [64]

    Yi Zhou, Yaoliang Yu, Wei Dai, Yingbin Liang, and Eric Xing. 2016. On convergence of model parallel proximal gradient algorithm for stale synchronous parallel system. InArtificial Intelligence and Statistics. PMLR, 713–722. 15

    2016

  49. [611]

    doi:10.1109/HPCA51647.2021.00057

    2021

  50. [2015]

    InProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

    Petuum: A new platform for distributed machine learning on big data. InProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1335–1344

  51. [2016]

    In2016 2nd Workshop on machine learning in hpc environments (MLHPC)

    Communication quantization for data-parallel training of deep neural networks. In2016 2nd Workshop on machine learning in hpc environments (MLHPC). IEEE, 1–8

  52. [2019]

    arXiv:1912.01703 [cs.LG]https://arxiv.org/abs/1912.01703

    PyTorch: An Imperative Style, High-Performance Deep Learning Library. arXiv:1912.01703 [cs.LG]https://arxiv.org/abs/1912.01703

    1912 arXiv

  53. [2020]

    InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis

    Zero: Memory optimizations toward training trillion param- eter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16

  54. [2021]

    Sparse is enough in scaling transformers.Advances in Neural Information Processing Systems34 (2021), 9895–9907

    2021

  55. [2023]

    arXiv preprint arXiv:2304.11277(2023)

    Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277(2023)

    2023 arXiv

  56. [2024]

    InSC24: International Conference for High Performance Computing, Networking, Storage and Analysis

    Stellaris: Staleness-Aware Distributed Reinforcement Learning with Serverless Computing. InSC24: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–17

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.