Pith. sign in

REVIEW 4 major objections 7 minor 82 references

DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Network-first parallel training claims 1.3–2.8× throughput gains for cloud-edge-end model training.

desk verdict A plausible system with a coherent design, but the headline throughput range is not supported by its own per-baseline numbers, and the time-to-accuracy comparisons are incomplete. read the letter →

arxiv 2505.15536 v2 pith:TNI7TDNW submitted 2025-05-21 eess.SY cs.DCcs.SY

Jinquan Wang , Xiaojian Liao , Xuzhao Liu , Jiashun Suo , Zhisheng Huo , Chenhao Zhang , Xiangrong Xu , Runnan Shen
show 2 more authors
Xilong Xie Limin Xiao
This is my paper · ORCID
classification eess.SYcs.DCcs.SY
keywords geo-distributedtrainingcloud-edge-endcomputingpipelineparallelismheterogeneousGPUsnetworkfluctuationautomaticparallelstrategyzero-bubblewide-area
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DeepCEE is a distributed-training system aimed at a specific but increasingly realistic setting: training a model across cloud data centers, edge servers, and end devices that are connected by slow, unstable wide-area links and equipped with very different GPUs. The paper's central claim is that in this cloud-edge-end environment the network, not compute, is the binding constraint, so parallel training strategies should be generated network-first. DeepCEE groups devices first by measured network quality, then by compute speed; places a compact zero-bubble pipeline across the slow network groups; and, at runtime, shrinks micro-batch sizes at stages that are waiting on a congested link. On a simulated ten-server CEE testbed the authors report 1.3–2.8× higher training throughput than widely used and state-of-the-art systems and 55–73.5% less time to reach target accuracy.

What carries the argument

The load-bearing mechanism is the network-centric two-level device group paired with a compact zero-bubble pipeline. Devices connected by fast links are clustered into first-level network groups so that cross-group communication happens only over slow wide-area links; within each group, compute-homogeneous second-level groups are formed. The pipeline across network groups is 'compact zero-bubble': weight updates are decoupled from gradient computation and packed into idle scheduling slots, so slow transfers do not force GPUs to sit idle. At runtime the adapter monitors transmission latency and shrinks per-stage micro-batch sizes when a delay is detected, letting the next stage start earlier.

What would settle it

A controlled reproduction on a live wide-area testbed, using the official implementations of the two closest baseline planners under real background traffic and with inter-region links set to equal bandwidth, would settle the claim: if the reported 1.3–2.8× advantage over widely used systems and the 1.04–1.52× advantage over the closest planners shrink to near parity, the network-centric grouping gains are specific to the simulated bandwidth hierarchy.

Watch

Extended reading notes

Core claim

DeepCEE claims that efficient cross-region training is achievable when the parallelization plan is organized by network rather than by compute. Its Heterogeneous Devices Profiler performs hierarchical clustering on measured bandwidth and latency to form first-level network device groups, then clusters within each such group by compute capability. Its Parallel Strategy Planner runs a compact zero-bubble pipeline across the network groups, decoupling weight computation from gradient computation and inserting weight work into bubbles that slow wide-area links would otherwise create, while a cost model and beam search choose the model split and a second level of asymmetric data, tensor, and pipeline parallelism. Its Dynamic Environment Adapter detects transmission delays and reduces the micro-batch size of the sending stage so downstream stages start sooner. On a simulated ten-server cloud-edge-end testbed the paper reports 1.3–2.8× higher training throughput than widely used and state-of-the-art systems, 1.5–1.7× higher throughput under induced network fluctuations, and 55–73.5% less time to reach the target accuracy.

Load-bearing premise

The central claim rests on the assumption that the simulated ten-server testbed, with manually fixed link bandwidths and the authors' reimplementations of the two closest auto-parallel baselines, fairly represents how those systems perform on real cloud-edge-end networks.

Editorial extensions

If this is right

  • Cross-region training becomes a practical way to absorb idle edge and end-device GPUs instead of waiting for spare cloud capacity.
  • The slowest wide-area link no longer sets the pipeline pace, because weight updates are decoupled and packed into slots that would otherwise be bubbles.
  • Automatic parallel planners for heterogeneous clusters should treat network topology and live bandwidth as first-class inputs, not just compute speed.
  • Network fluctuations can be absorbed by per-stage micro-batch resizing, avoiding pipeline restarts and checkpointing.
  • Larger batch sizes are disproportionately helpful when links are slow, because longer compute intervals give cross-region communication more room to overlap; DeepCEE measures 6.8–16.2% throughput gains from doubling batch size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the throughput gains hold on real WANs, the same network-first grouping could also improve other geo-distributed workloads such as federated fine-tuning and split inference, where wide-area latency dominates.
  • The 6.4–8.2% extra iterations the paper attributes to out-of-order weight updates point to a tunable knob: a less aggressive schedule would trade some throughput for better convergence, and a staleness-aware scheduler might recover both.
  • The adapter's micro-batch shrinking is tested against steady 40–60% bandwidth reductions; bursty contention from mixed workloads is a natural next stress test before deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents DeepCEE, a distributed training system for cloud-edge-end (CEE) environments with heterogeneous GPUs and unstable, hierarchical networks. The system has three components: a Heterogeneous Devices Profiler that groups devices by network and compute characteristics, a Parallel Strategy Planner that combines compact zero-bubble pipeline parallelism across network-defined groups with 3D parallelism inside groups, and a Dynamic Environment Adapter that changes per-stage micro-batch sizes in response to network fluctuations. The authors report throughput and time-to-accuracy experiments on a 10-server simulated CEE testbed against DDP, FSDP, DeepSpeed, GPipe, Alpa, Metis, HetPipe, and Asteroid, claiming 1.3–2.8x training throughput gains and 55–73.5% time-to-accuracy reductions. The central claims are measured from actual runs, but the reported per-baseline gains and the paper's own convergence analysis leave important gaps between the headline claims and the evidence presented.

Significance. If the claims hold, DeepCEE would be a useful contribution to geo-distributed training: the two-level grouping is a sensible response to hierarchical topologies, the compact zero-bubble pipeline is a plausible extension of zero-bubble schedules to bandwidth-constrained settings, and the dynamic micro-batch adapter addresses a real problem. The implementation effort is substantial (about 15K lines of PyTorch code) and the evaluation covers many baselines and models. However, the paper's headline throughput range is not supported by the body's minimum per-baseline gains, and the paper explicitly admits an extra 6.4–8.2% iteration cost from out-of-order weight updates; for the closest baselines (Asteroid, DeepSpeed Pipeline) this cost can exceed the reported 1.04x throughput advantage, so the end-to-end time-to-accuracy claims are not established for those systems. The evaluation also relies on reimplemented SOTA baselines, a simulated network environment, and single-run numbers without error bars, which limits confidence in the quantitative comparisons.

major comments (4)
  1. [Abstract; §4.2; §4.3.1] The abstract states that DeepCEE achieves 1.3–2.8x higher training throughput, but the lowest reported per-baseline gains in the body are 1.04x versus DeepSpeed Pipeline (§4.2) and 1.04–1.05x versus Asteroid (§4.3.1). The lower bound of the headline range is therefore unsupported by the presented measurements. Please either correct the abstract to reflect the actual per-baseline range (about 1.04–2.85x) or provide an explicit justification for excluding Asteroid and DeepSpeed Pipeline from the lower-bound calculation.
  2. [§4.2, Time Overhead paragraph] The paper states that the compact zero-bubble pipeline's out-of-order weight updates require 6.4–8.2% more training iterations to reach target accuracy. For the closest baselines, where throughput gain is only 1.04–1.05x, this iteration overhead is larger than the throughput advantage, so DeepCEE could be slower in wall-clock time to target accuracy than Asteroid or DeepSpeed Pipeline. Figure 10 reports time-to-accuracy only against DDP, FSDP, DeepSpeed, and GPipe; it does not include Asteroid, Metis, or HetPipe. To support the end-to-end claim, please report wall-clock time to target accuracy (or equivalent epochs-to-target with measured per-epoch time) for all compared systems, and discuss how the 6.4–8.2% iteration increase interacts with each baseline's throughput ratio.
  3. [§4.1, Compared Methods; §4.2] Metis and Asteroid are reimplementations ('we faithfully reimplement their partition search algorithms') rather than the original systems, and no code is released. The 1.04–1.05x margin over Asteroid is close enough to the uncertainty that a reimplementation could plausibly introduce. Please release the code and detailed configuration files for all baselines, and report repeated-run statistics (mean and standard deviation, or confidence intervals) for the throughput and time-to-accuracy numbers. This is important for the central comparison because the headline claims depend on small margins as well as large ones.
  4. [§3.3.2, Eq. (1)] The cost model in Eq. (1) is the basis for strategy selection in the Planner, but the paper never validates its predictions against measured execution times. Without a validation plot (predicted vs observed per-stage time, or predicted strategy ranking vs measured ranking), the claim of 'automatically deriving optimal parallel strategies' rests on an unverified model. Please add a cost-model validation experiment or an ablation comparing the searched strategy against simple heuristic alternatives.
minor comments (7)
  1. [Figures throughout] Several figure labels in the provided text render as unicode-escape sequences (e.g., '/uni00000014/uni00000015/...' in Figures 9–12), making the axis labels unreadable. Please regenerate the figures with plain-text labels.
  2. [§2.2] 'suddlenly' should be 'suddenly'.
  3. [Title page footnote] 'Corresponed author' should be 'Corresponding author'.
  4. [§4.4] The component is called 'Adapter' elsewhere but 'Adaptor' in several places in §4.4; please make the spelling consistent.
  5. [§4.5] 'GPipe's 1F1B pipeline structure' is inaccurate: GPipe uses a fill-flush schedule, while 1F1B is used by PipeDream and DeepSpeed Pipeline. Please correct this characterization.
  6. [§4.2] 'PyTorch FDSP' should be 'PyTorch FSDP'.
  7. [§4.2] The sentence 'our compact zero-bubble pipeline ... improves average throughput by 68.1% (Figure 9)' does not specify how this average is computed across models and settings; please state the aggregation method.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DeepCEE's throughput claims are measured on a simulated testbed, the cost model is an analytic search heuristic rather than a fitted predictor, and no self-citation is load-bearing.

full rationale

DeepCEE's central claims are empirical measurements obtained by comparing systems on a simulated CEE testbed, not outputs of a fitted model that are then relabeled as predictions. The cost model in Equation 1 is an analytic planning heuristic with hand-specified terms (B·t_c, t_l = t_comm − t_lap, AL = V/min(bandwidth)); no parameter is fitted to the reported throughput numbers and then reused to generate those same numbers. The device grouping thresholds are set by hand, and the Dynamic Environment Adapter is evaluated by direct on/off comparison (Figure 12b), so its benefit is measured rather than derived from its own assumptions. The paper contains no self-citations: the zero-bubble pipeline design is credited to independent prior work [43], and the SOTA baselines are either existing open-source frameworks or described as faithful reimplementations of published algorithms, so there is no load-bearing self-citation chain. The admitted 6.4–8.2% iteration overhead (Section 4.2) and the absence of time-to-accuracy curves for Metis, HetPipe, and Asteroid are evaluation limitations, not circularity: the throughput advantage claimed is a measured quantity, the convergence cost is disclosed, and no derived result is equivalent by construction to an input. The paper's own explanation of why larger batch sizes improve throughput is also a post-hoc interpretation of measured trends rather than a circular derivation. Overall, the derivation chain is self-contained with respect to the empirical claims, and the internal cost model does not conflate fit with prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its components (Profiler, Planner, Adapter, caches, monitors) are software artifacts described within the paper and do not require independent falsifiable handles. The free parameters and assumptions above are the hand-set knobs and unstated premises the central claims rely on.

free parameters (4)
  • Task-specific weights w_i in computing capability formula
    Equation pc = Σ(w_i / t_i) in §3.2.1 weights different benchmark tasks when scoring GPU compute capability; values are not given and affect device grouping and load balancing.
  • First-level network grouping threshold
    In §3.2.2, groups are merged if the difference in network capability is below a 'predefined threshold'; no value or sensitivity analysis is provided.
  • Adapter bandwidth threshold
    In §3.4, the Forward/Backward Monitor decides to aggregate or re-split tensors based on whether bandwidth changes cross a 'predefined threshold'; the value is unspecified.
  • Beam search beam width l and max_iter
    Algorithm 1 uses beam width l and max_iter to bound search; values are not reported, affecting the quality of the generated parallel strategy.
assumptions (4)
  • domain assumption Pipeline parallelism imposes the least network bandwidth demand among DP/TP/PP
    Invoked in §2.1 and §3.1 to justify PP across first-level network groups. If TP or DP were cheaper under CEE bandwidth, the design choice would be suboptimal.
  • domain assumption Network performance is the primary factor limiting training efficiency in CEE environments
    Stated in §1 and §3.1. The whole architecture prioritizes network-centric grouping; if compute heterogeneity dominates, the approach may be misdirected.
  • ad hoc to paper The cost model in Equation 1 captures pipeline execution time with residual latency t_l = t_comm - t_lap and collective communication AL
    No calibration of the cost model against measured runtime is reported; the search algorithm's near-optimality depends on this model being accurate.
  • ad hoc to paper Weight updates can be decoupled from gradient computation and reordered into pipeline bubbles without unacceptable convergence loss
    Section 3.3.1 assumes this to justify compact zero-bubble scheduling. The paper's own §4.2 admits it 'disrupts the conventional weight update sequence' and requires 6.4-8.2% more iterations, partially undermining the assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks." pith.science (2026). https://pith.science/paper/TNI7TDNW

@misc{pith2026250515536,
  author       = {Pith},
  title        = {Pith review of: DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNI7TDNW}},
  note         = {Machine review of arXiv:2505.15536}
}
read the original abstract

Most existing training systems focus on a single region. In contrast, we envision that cross-region training offers more flexible GPU resource allocation and yields significant potential. However, the hierarchical cluster topology and unstable networks in the cloud-edge-end (CEE) environment, a typical cross-region scenario, pose substantial challenges to building an efficient and autonomous model training system. We propose DeepCEE, a geo-distributed model training system tailored for heterogeneous GPUs and networks in CEE environments. DeepCEE adopts a communication-centric design philosophy to tackle challenges arising from slow and unstable inter-region networks. It begins with a heterogeneous device profiler that identifies and groups devices based on both network and compute characteristics. Leveraging device groups, DeepCEE implements compact, zero-bubble pipeline parallelism, automatically deriving optimal parallel strategies. To further adapt to runtime variability, DeepCEE integrates a dynamic environment adapter that reacts to network fluctuations. Extensive evaluations demonstrate that DeepCEE achieves 1.3-2.8x higher training throughput compared to widely used and SOTA training systems.

Figures

Figures reproduced from arXiv: 2505.15536 by the authors.

Figure 1
Figure 1. The CEE environment and its workload status. cloud data center. However, we envision that cross-region distributed training [26, 76] will become a key direction for reducing both the barriers and costs associated with model training [1, 30, 61, 64], for the following reasons. First, as model sizes continue to grow, the demand for GPUs is also increasing [15, 52, 77]. Training tasks cannot be executed immediately [50… view at source ↗
Figure 2
Figure 2. Performance comparison of different methods in a real CEE environment. computing capabilities remain significantly underutilized, with average GPU utilization rates as low as 28% [17, 64]. This presents a promising opportunity to harness idle GPU resources at the edge for model training. Third, with the continuous advancement of networking technologies, the bandwidth between different regions has significantly incre… view at source ↗
Figure 3
Figure 3. In the CEE environment, current distributed training strategies face two challenges. The hierarchical cluster topology fundamentally constrains training throughput in the CEE environment, while network fluctuations frequently triggers abrupt performance degradation. Profiler (§3.2) Planner (§3.3) Adapter (§3.4) Device Profiling Device Grouping Model Bench Network Bench 1 2 3 1 2 3 Latency Memory Comp. BW Warm up Run… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The main working components and workflow of DeepCEE. DeepCEE includes the pre-run performance evaluation component Heterogeneous Devices Profiler, the pre-run parallel planning component Parallel Strategy Planner, and the runtime environment adaptation component Dynami…
Figure 5
Figure 5. Figure 5: An example of heterogeneous device and network grouping in the CEE environment. (FG: first-level network device group; SG: second-level computing device group) 0 0 0 0 0 1 1 0 1 1 1 2 2 3 3 2 0 2 3 3 1 0 0 0 2 2 3 3 1 1 2 3 0 0 1 2 3 0 1 2 0 3 1 2 3 2 3 1 0 2 1 3 4 1 1…
Figure 6
Figure 6. Figure 6: The comparison of original and compact zero￾bubble pipeline. (FG: first-level network device group) the Planner, which implements a compact zero-bubble pipeline architecture that decouples weight updates from gradient computation and proactively inserts the weight upda…
Figure 7
Figure 7. Figure 7: Examples of asymmetric multi-dimensional paral￾lelism. (Group: second-level device group) by randomly permuting first-level network device groups and generating 𝑙 candidate split set A through computing aware model segmentation (Line 3). Through iterative re￾finement (…
Figure 8
Figure 8. Figure 8: The overview and example of the Dynamic Envi￾ronment Adapter. the time of the filling stage, the micro-batch size of each first￾level device group is reduced when the network capabilities are poor, thus completing the filling as soon as possible. Second, to reduce the …
Figure 9
Figure 9. Figure 9: Comparison of throughput for model training using different methods under 2 heterogeneous CEE environment. 0 1000 2000 3000 4000 Time (s) 0 20 40 60 80 Accuracy (%) DDP FSDP Gpipe Pipeline DeepCEE (a) Comparison of resnet101. 0 200 400 600 Time (s) 0 20 40 60 80 Accura…
Figure 10
Figure 10. Figure 10: Comparison of model accuracy for model training using different methods under Setting 1. FDSP, DeepSpeed Pipeline and Gpipe, resulting in worker im￾balance and inefficient synchronization. We further observe an intriguing phenomenon in the CEE: larger batch sizes cons…
Figure 11
Figure 11. Figure 11: Comparison of different parallel strategy genera￾tion methods. while neglecting network diversity and hierarchy. Metis’ sole optimization objective of minimizing inter-cluster commu￾nication consequently yields limited performance gains. Comparison against HetPipe. De…
Figure 13
Figure 13. Figure 13: Throughput results for varying numbers of de￾vices. and 1.49× for ResNet101 and GPT-2 during bandwidth fluctu￾ations. This acceleration from its dynamic network-aware op￾timization: (1) decoupling micro-batch sizes across pipeline stages, and (2) independently adjusti…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 44 canonical work pages

  1. [1]

    Romil Bhardwaj, Zhengxu Xia, Ganesh Ananthanarayanan, Junchen Jiang, Yuanchao Shu, Nikolaos Karianakis, Kevin Hsieh, Paramvir Bahl, and Ion Stoica. 2022. Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers. In19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). 119–135

    2022

  2. [2]

    Zhengda Bian, Qifan Xu, Boxiang Wang, and Yang You. 2021. Maxi- mizing Parallelism in Distributed Training for Huge Neural Networks. https://doi.org/10.48550/arXiv.2105.14450arXiv:2105.14450 [cs]

    arXiv doi:10.48550/arxiv.2105.14450 2021

  3. [3]

    Imane Cheikh, Rachid Aouami, Essaid Sabir, Mohamed Sadik, and Sébastien Roy. 2022. Multi-Layered Energy Efficiency in LoRa-WAN Networks: A Tutorial.IEEE Access10 (2022), 9198–9231

    2022

  4. [4]

    Yuhao Chen, Qianqian Yang, Shibo He, Zhiguo Shi, Jiming Chen, and Mohsen Guizani. 2024. FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Approach for Heterogeneous Edge Devices.IEEE Transactions on Mobile Computing23, 4 (April 2024), 3200–3212.https: //doi.org/10.1109/TMC.2023.3272567

    arXiv 2024

  5. [5]

    Naga Srinivasarao Chilamkurthy, Om Jee Pandey, Anirban Ghosh, Linga Reddy Cenkeramaddi, and Hong-Ning Dai. 2022. Low-Power Wide-Area Networks: A Broad Overview of Its Different Aspects.IEEE Access10 (2022), 81926–81959

    work page 2022

  6. [6]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: At- tentive Language Models Beyond a Fixed-Length Context. https://arxiv.org/abs/1901.02860v3

    arXiv 2019

  7. [7]

    Runliang Dou, Guiyu Zhuang, Xin Liu, Yanchao Hou, and Jing Sun

    work page

  8. [8]

    Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu, and Wei Lin. 2021. DAPPLE: A Pipelined Data Parallel Approach for Training Large Models. InProceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP ’21). Association fo...

    arXiv 2021

Show all 82 references
  1. [9]

    MindSpore Group. 2025. MindSpore Official Site | MindSpore. https://www.mindspore.cn/en

    2025

  2. [10]

    Liangzhe Han, Bowen Du, Leilei Sun, Yanjie Fu, Yisheng Lv, and Hui Xiong. 2021. Dynamic and Multi-faceted Spatio-temporal Deep Learn- ing for Traffic Speed Forecasting. InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. ACM, Virtual Event Sin...

    2021 doi

  3. [11]

    Pengzhan Hao and Yifan Zhang. 2021. EDDL: A Distributed Deep Learning System for Resource-limited Edge Computing Environment. In2021 IEEE/ACM Symposium on Edge Computing (SEC). 1–13.https: //doi.org/10.1145/3453142.3491286

    2021

  4. [12]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778

    2016

  5. [13]

    Wein- berger

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Wein- berger. 2017. Densely Connected Convolutional Networks. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700–4708

    2017

  6. [14]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism.https://doi.org/10. 48550/arXiv.1811.069...

    2019 doi arXiv

  7. [15]

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi...

    2024 arXiv

  8. [16]

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer.https://doi.org/10.48550/arXiv.2001.04451 arXiv:2001.04451 [cs]

    2020 doi arXiv

  9. [17]

    Kwon, Rui Li, Stylianos Venieris, Jagmohan Chauhan, Nicholas Donald Lane, and Cecilia Mascolo

    Young D. Kwon, Rui Li, Stylianos Venieris, Jagmohan Chauhan, Nicholas Donald Lane, and Cecilia Mascolo. 2024. TinyTrain: Resource- Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge. InForty-First International Conference on Machine Learning

    2024

  10. [18]

    Zhiquan Lai, Shengwei Li, Xudong Tang, Keshi Ge, Weijie Liu, Yabo Duan, Linbo Qiao, and Dongsheng Li. 2023. Merak: An Efficient Distributed DNN Training Framework with Automated 3D Paral- lelism for Giant Foundation Models.IEEE Transactions on Paral- lel and Distributed System...

    2023

  11. [19]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.https://doi.org/10.48550/arXiv. 13 Conference acronym ’XX...

    2020 doi arXiv

  12. [20]

    Dacheng Li, Hongyi Wang, Eric Xing, and Hao Zhang. 2022. Amp: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness.Advances in Neural Information Processing Systems35 (2022), 6630–6639

    2022

  13. [21]

    Fanxin Li, Shixiong Zhao, Yuhao Qing, Xusheng Chen, Xiuxian Guan, Sen Wang, Gong Zhang, and Heming Cui. 2023. Fold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models.IEEE Transactions on Parallel and Distributed Systems3...

    2023

  14. [22]

    Ruihan Li, Fangdan Ye, Yifei Yuan, Ruizhen Yang, Bingchuan Tian, Tianchen Guo, Hao Wu, Xiaobo Zhu, Zhongyu Guan, and Qing Ma

  15. [23]

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. 2020. PyTorch Distributed: Experiences on Acceler- ating Data Parallel Training.Proceedings of the VLDB Endowment13, 12 (A...

    2020

  16. [24]

    In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)

    Reasoning about Network Traffic Load Property at Production Scale. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 1063–1082

  17. [25]

    Zhiqi Lin, Youshan Miao, Quanlu Zhang, Fan Yang, Yi Zhu, Cheng Li, Saeed Maleki, Xu Cao, Ning Shang, Yilei Yang, Weijiang Xu, Mao Yang, Lintao Zhang, and Lidong Zhou. 2024. nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training. In18th USENIX Sy...

    2024

  18. [26]

    Yanan Li, Haitao Yuan, Zhe Fu, Xiao Ma, Mengwei Xu, and Shangguang Wang. 2023. ELASTIC: Edge Workload Forecasting Based on Collabo- rative Cloud-Edge Deep Learning. InProceedings of the ACM Web Con- ference 2023 (WWW ’23). Association for Computing Machinery, New York, NY, USA...

    2023

  19. [27]

    Luo Mai, Guo Li, Marcel Wagenländer, Konstantinos Fertakis, Andrei- Octavian Brabete, and Peter Pietzuch. [n. d.]. KungFu: Making Training in Distributed Machine Learning Adaptive

  20. [28]

    Xin Liu, Yaran Chen, Haoran Li, Boyu Li, and Dongbin Zhao

  21. [29]

    Alessio Meloni, Paolo Attilio Pegoraro, Luigi Atzori, Andrea Benigni, and Sara Sulis. 2018. Cloud-Based IoT Solution for State Estimation in Smart Grids: Exploiting Virtualization and Edge-Intelligence Tech- nologies.Computer Networks130 (2018), 156–165

    2018

  22. [30]

    Hao Miao, Yan Zhao, Chenjuan Guo, Bin Yang, Kai Zheng, Feiteng Huang, Jiandong Xie, and Christian S. Jensen. 2024. A Unified Replay- Based Continuous Learning Framework for Spatio-Temporal Predic- tion on Streaming Data. In2024 IEEE 40th International Conference on Data Engine...

    2024

  23. [31]

    Nixon, Christopher Krieger, and Yiran Chen

    Jiachen Mao, Xiang Chen, Kent W. Nixon, Christopher Krieger, and Yiran Chen. 2017. MoDNN: Local Distributed Mobile Computing Sys- tem for Deep Neural Network. InDesign, Automation & Test in Europe Conference & Exhibition (DATE), 2017. IEEE, Lausanne, Switzerland, 1396–1401.htt...

    2017

  24. [32]

    Microsoft. 2024. Microsoft/DeepSpeed: DeepSpeed Is a Deep Learning Optimization Library That Makes Dis- tributed Training and Inference Easy, Efficient, and Effective. https://github.com/microsoft/DeepSpeed

    2024

  25. [33]

    Zizhao Mo, Huanle Xu, and Chengzhong Xu. 2024. Heet: Accelerating Elastic Training in Heterogeneous Deep Learning Clusters. InProceed- ings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24, ...

    2024

  26. [34]

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism.Proceedings of the VLDB Endowment16, 3 (Nov. 2022), 470–479.https://doi.org/10. 14778/357069...

    2022

  27. [35]

    Jayashree Mohan, Amar Phanishayee, Ashish Raniwala, and Vijay Chidambaram. 2021. Analyzing and Mitigating Data Stalls in DNN Training. InVldb 2021

    2021

  28. [36]

    Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. 2024. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention.https://doi.org/10.48550/arXiv.2404.07143 arXiv:2404.07143 [cs]

    2024 doi arXiv

  29. [37]

    Jayashree Mohan, Amar Phanishayee, and Vijay Chidambaram. 2021. CheckFreq: Frequent, Fine-Grained DNN Checkpointing. InUsenix Fast 2021

    2021

  30. [38]

    NVIDIA. [n. d.]. GeForce RTX 4090 Graphics Cards for Gaming | NVIDIA. https://www.nvidia.com/en-us/geforce/graphics-cards/40- series/rtx-4090/

  31. [39]

    NVIDIA. 2024. NVIDIA GeForce RTX 30 Series GPUs Powered by Am- pere Architecture. https://www.nvidia.com/en-us/geforce/graphics- cards/30-series/

    2024

  32. [40]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized Pipeline Parallelism for DNN Training. InProceedings of the 27th ACM Symposium on Operating Systems Princ...

    2019

  33. [41]

    OpenAI. 2024. Multi-Datacenter Training: OpenAI’s Ambitious Plan To Beat Google’s Infrastructure

    2024

  34. [42]

    Park, Gyeongchan Yun, Chang M

    Jay H. Park, Gyeongchan Yun, Chang M. Yi, Nguyen T. Nguyen, Seung- min Lee, Jaesik Choi, Sam H. Noh, and Young-ri Choi. 2020. {HetPipe}: Enabling Large {DNN} Training on (Whimpy) Heterogeneous {GPU} Clusters through Integration of Pipelined Model Parallelism and Data Paralleli...

    2020

  35. [43]

    NVIDIA. 2025. TESLA P4 GPU ACCELERATOR. https://www.nvidia.com/content/dam/en-zz/Solutions/design- visualization/solutions/resources/documents1/Tesla-P4-Product- Brief.pdf

    2025

  36. [44]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory Optimizations Toward Training Tril- lion Parameter Models.https://doi.org/10.48550/arXiv.1910.02054 arXiv:1910.02054 [cs]

    2020 doi arXiv

  37. [45]

    J. Redmon. 2016. You Only Look Once: Unified, Real-Time Object Detection. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    2016

  38. [46]

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. 2023. Zero Bubble Pipeline Parallelism.https://doi.org/10.48550/arXiv.2401.10241 arXiv:2401.10241 [cs]

    2023 doi arXiv

  39. [47]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. https://doi.org/10.48550/arXiv.1909.08053arXiv:1909.08053 [cs]

    2020 doi arXiv

  40. [48]

    Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGres- ley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prab- humoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Mi...

    2018 doi arXiv

  41. [49]

    Zhaoyan Shen, Qingxiang Tang, Tianren Zhou, Yuhao Zhang, Zhiping Jia, Dongxiao Yu, Zhiyong Zhang, and Bingzhe Li. 2024. ASHL: An Adaptive Multi-Stage Distributed Deep Learning Training Scheme for Heterogeneous Environments.IEEE Trans. Comput.73, 1 (2024), 30–43. https://doi.or...

    2024

  42. [50]

    Kahou Tam, Chunlin Tian, Li Li, Haikai Zhao, and ChengZhong Xu

  43. [51]

    Jiabin Tang, Wei Wei, Lianghao Xia, and Chao Huang. 2024. EasyST: A Simple Framework for Spatio-Temporal Prediction. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management. ACM, Boise ID USA, 2220–2229.https://doi.org/10.1145/ 3627673.3679749

    2024

  44. [52]

    Foteini Strati, Michal Friedman, and Ana Klimovic. 2025. PCcheck: Persistent Concurrent Checkpointing for ML. InProceedings of the 30th ACM International Conference on Architectural Support for Program- ming Languages and Operating Systems, Volume 1. ACM, Rotterdam Netherlands...

    2025

  45. [53]

    Chunlin Tian, Li Li, Kahou Tam, Yebo Wu, and Cheng-Zhong Xu. 2024. Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting.IEEE Transactions on Parallel and Distributed Systems (2024)

    2024

  46. [54]

    InProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems

    FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management. InProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems. ACM, Hangzhou China, 394– 408.https://doi.org/10.1145/3666025.3699346

  47. [55]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs]

    2023 arXiv

  48. [56]

    Gemini Team. 2024. Gemini 1.5: Unlocking Multimodal Understanding across Millions of Tokens of Context.https://doi.org/10.48550/arXiv. 2403.05530arXiv:2403.05530 [cs]

    2024 doi arXiv

  49. [57]

    Yousuke Watanabe, Kenya Sato, and Hiroaki Takada. 2020. Dy- namicMap 2.0: A Traffic Data Management Platform Leveraging Clouds, Edges and Embedded Systems.International Journal of In- telligent Transportation Systems Research18, 1 (2020), 77–89.https: //doi.org/10.1007/s13177-...

    2020 doi

  50. [58]

    Taegeon Um, Byungsoo Oh, Minyoung Kang, Woo-Yeon Lee, Goeun Kim, Dongseob Kim, Youngtaek Kim, Mohd Muzzammil, and Myeong- jae Jeon. 2024. Metis: Fast Automatic Distributed Training on Hetero- geneous {GPUs}. In2024 USENIX Annual Technical Conference (USENIX ATC 24). 563–578

    2024

  51. [59]

    wikipedia. 2025. Local Area Network.Wikipedia(2025)

    2025

  52. [60]

    Kun Wang, Jiani Cao, Zimu Zhou, and Zhenjiang Li. 2024. SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget.IEEE Transactions on Mobile Computing(2024)

    2024

  53. [61]

    Daliang Xu, Mengwei Xu, Chiheng Lou, Li Zhang, Gang Huang, Xin Jin, and Xuanzhe Liu. 2024. SoCFlow: Efficient and Scalable DNN Training on SoC-Clustered Edge Servers. InProceedings of the 29th ACM International Conference on Architectural Support for Program- ming Languages an...

    2024

  54. [62]

    Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. 2022. {MLaaS} in the Wild: Workload Analysis and Scheduling in {Large-Scale} Het- erogeneous {GPU} Clusters. In19th USENIX Symposium on Networked Systems Design a...

    2022

  55. [63]

    Lei Yang, Yingqi Gan, Jinru Chen, and Jiannong Cao. 2024. AutoSF: Adaptive Distributed Model Training in Dynamic Edge Computing. IEEE Transactions on Mobile Computing23, 6 (2024), 6549–6562.https: //doi.org/10.1109/TMC.2023.3323456

    2024

  56. [64]

    Tengxi Xia, Ju Ren, Wei Rao, Qin Zu, Wenjie Wang, Shuai Chen, and Yaoxue Zhang. 2024. Aerorec: An Efficient on-Device Recommendation Framework Using Federated Self-Supervised Knowledge Distillation. InIEEE INFOCOM 2024-IEEE Conference on Computer Communications. IEEE, 121–130

    2024

  57. [65]

    Liekang Zeng, Xu Chen, Zhi Zhou, Lei Yang, and Junshan Zhang. 2021. CoEdge: Cooperative DNN Inference With Adaptive Workload Parti- tioning Over Heterogeneous Edge Devices.IEEE/ACM Transactions on Networking29, 2 (April 2021), 595–608.https://doi.org/10.1109/TNET. 2020.3042320

    2021

  58. [66]

    Enyue Yang, Weike Pan, Qiang Yang, and Zhong Ming. 2024. Discrete Federated Multi-behavior Recommendation for Privacy-Preserving Heterogeneous One-Class Collaborative Filtering.ACM Transactions on Information Systems42, 5 (2024), 1–50

    2024

  59. [67]

    Haotong Zhang, Weiwei Lin, Rong Xie, Shenghai Li, Zhiyan Dai, and James Z. Wang. 2024. An Optimal Container Update Method for Edge- cloud Collaboration.Software: Practice and Experience54, 4 (2024), 617–634.https://doi.org/10.1002/spe.3232

    2024 doi

  60. [68]

    Shengyuan Ye, Liekang Zeng, Xiaowen Chu, Guoliang Xing, and Xu Chen. 2024. Asteroid: Resource-Efficient Hybrid Pipeline Par- allelism for Collaborative DNN Training on Heterogeneous Edge Devices. InProceedings of the 30th Annual International Conference on Mobile Computing and...

    2024

  61. [69]

    Shijin Zhang, Zidong Du, Lei Zhang, Huiying Lan, Shaoli Liu, Ling Li, Qi Guo, Tianshi Chen, and Yunji Chen. 2016. Cambricon-X: An Accelerator for Sparse Neural Networks. In2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 1–12

    2016

  62. [70]

    Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Ji- dong Zhai. 2023. {SmartMoE}: Efficiently Training {Sparsely-Activated} Models through Combining Offline and Online Parallelization. In2023 USENIX Annual Technical Conference (USENIX ATC 23). 961–975

    2023

  63. [71]

    Weigang Zhang, Biyu Zhou, Xuehai Tang, Zhaoxing Wang, and Songlin Hu. 2023. MixPipe: Efficient Bidirectional Pipeline Paral- lelism for Training Large-Scale Models. In2023 60th ACM/IEEE Design Automation Conference (DAC). 1–6.https://doi.org/10.1109/DAC56929. 2023.10247730

    2023

  64. [72]

    Lizhi Zhang, Kai Lu, Zhiquan Lai, Yongquan Fu, Yu Tang, and Dong- sheng Li. 2023. Accelerating GNN Training by Adapting Large Graphs to Distributed Heterogeneous Architectures.IEEE Trans. Comput.72, 12 (2023), 3473–3488.https://doi.org/10.1109/TC.2023.3305077

    2023

  65. [73]

    Zhuoran Zhao, Kamyar Mirzazad Barijough, and Andreas Gerstlauer

  66. [74]

    Siyao Zhang, Daocheng Fu, Wenzhe Liang, Zhao Zhang, Bin Yu, Pin- long Cai, and Baozhen Yao. 2024. Trafficgpt: Viewing, Processing and Interacting with Traffic Foundation Models.Transport Policy150 (2024), 95–105

    2024

  67. [75]

    Ruiqi Zheng, Liang Qu, Tong Chen, Kai Zheng, Yuhui Shi, and Hongzhi Yin. 2024. Personalized Elastic Embedding Learning for On-Device Rec- ommendation.IEEE Transactions on Knowledge and Data Engineering (2024)

    2024

  68. [76]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences...

    2023

  69. [77]

    Jiahang Zhou, Yanyu Chen, Zicong Hong, Wuhui Chen, Yue Yu, Tao Zhang, Hui Wang, Chuanfu Zhang, and Zibin Zheng. 2024. Training and Serving System of Foundation Models: A Comprehensive Survey. arXiv:2401.02643 [cs] Received 20 February 2007; revised 12 March 2009; accepted 5 Ju...

    2024 arXiv

  70. [79]

    Xing, Joseph E

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Au- tomating Inter- and {Intra-Operator} Parallelism for Distributed Deep Learning. In16th US...

    2022

  71. [81]

    Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, and Daxin Jiang. 2025. StreamRL: 15 Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Wang et al. Scalable, Heteroge...

    2025 doi arXiv

  72. [2018]

    DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems37, 11 (2018), 2348–2359.https://doi.org/10.1109/TCAD.2018.2858384

    2018

  73. [2024]

    Potential of AI for Service Performance of Manufacturers: Ana- lytical and Empirical Insights.Advanced Engineering Informatics60 (2024), 102383

    2024

  74. [2025]

    Cross-Domain Random Pre-training with Prototypes for Re- inforcement Learning.https://doi.org/10.48550/arXiv.2302.05614 arXiv:2302.05614 [cs]

    doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.