pith. sign in

arxiv: 2303.05330 · v1 · submitted 2023-03-09 · 💻 cs.DC · cs.AI

Cloudless-Training: A Framework to Improve Efficiency of Geo-Distributed ML Training

Wenting Tan , Xiao Shi1 , Cunchi Lv , Xiaofang Zhao This is my paper

Pith reviewed 2026-05-24 09:13 UTC · model grok-4.3

classification 💻 cs.DC cs.AI
keywords geo-distributed machine learningserverless computingelastic schedulingasynchronous SGDmodel averagingwide area networkcloud resource managementparameter server
0
0 comments X

The pith

Cloudless-Training framework enables efficient geo-distributed ML training with elastic scheduling and specialized synchronization over wide-area networks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Cloudless-Training as a way to make machine learning training efficient when data and compute are spread across different cloud regions. It tackles poor resource use and slow communication over wide area networks by introducing a two-layer serverless setup for flexible scheduling and two specialized synchronization techniques. These allow training jobs to adjust dynamically to available resources and data locations while handling communication delays better than standard methods. If correct, this means organizations can use cloud resources more cost-effectively and complete training faster without sacrificing the quality of the resulting model.

Core claim

The authors claim that their Cloudless-Training framework, built on a two-layer architecture with control and physical training planes, supports elastic scheduling of training workflows based on cloud resource heterogeneity and pre-existing dataset distribution. It further introduces asynchronous SGD with gradient accumulation and inter-PS model averaging as synchronization strategies for training partitions among clouds. When implemented and tested, this results in geo-distributed ML training that reduces costs by 9.2 to 24 percent and achieves up to 1.7 times speedup compared to baseline, all with guarantees that the model correctness is maintained.

What carries the argument

A two-layer serverless architecture separating control and physical planes, together with ASGD-GA and inter-PS model averaging for handling synchronization across regions.

If this is right

  • Training can adaptively deploy across heterogeneous multi-regional cloud resources.
  • Communication overhead on fluctuating WAN links is mitigated through the new sync methods.
  • Overall resource utilization increases, translating to lower training costs.
  • Synchronization efficiency improves, yielding faster overall training times.
  • Model correctness is preserved without algorithm modifications.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the framework scales, it could enable more widespread use of geo-distributed training for very large models that exceed single-region capacity.
  • The methods might apply to other distributed systems facing similar bandwidth constraints beyond machine learning.
  • A potential extension would involve integrating with privacy mechanisms to support federated learning use cases.

Load-bearing premise

That the proposed synchronization strategies maintain model correctness despite WAN bandwidth fluctuations and resource heterogeneity in real deployments.

What would settle it

Running a controlled experiment where WAN conditions are varied and checking if the final model accuracy matches that of a non-geo-distributed baseline within acceptable bounds.

Figures

Figures reproduced from arXiv: 2303.05330 by Cunchi Lv, Wenting Tan, Xiaofang Zhao, Xiao Shi1.

Figure 1
Figure 1. Figure 1: The architecture of geo-distributed ML training method However, it is non-trivial to acquire efficient geo￾distributed training for users due to challenges in managing load balancing and communication in the complex environment. First, efficient elastic scheduling of multi￾regional cloud resources is usually missing, resulting in load imbalance. It decreases resource utilization, speed, and performance of … view at source ↗
Figure 2
Figure 2. Figure 2: The time proportion of training LeNet with various kinds of heterogeneous resource allocations and uneven data distributions in Shanghai and Chongqing regions of Tencent Cloud Load imbalance across multi-regional clouds. It is usually empirical or rough (e.g., with greedy strategy) to set resourcing plans for training tasks in each cloud. Considering the condition of heterogeneous resources and uneven data… view at source ↗
Figure 3
Figure 3. Figure 3: The time proportion of training ResNet18 with CPU or GPU in 2 Shanghai and Chongqing regions of Tencent Cloud In geo-distributed ML training, synchronization of intermediate results of models on WAN leads to much more overhead than on LAN, affecting (e.g., pausing, delaying) training processes periodically in running time. The overhead is nonnegligible. Synchronization overhead on WAN. In ML training, each… view at source ↗
Figure 4
Figure 4. Figure 4: Architecture of Cloudless-Training workflows in each cloud. Then, the global communicator function waits for PS function in each cloud to be ready, and assigns communication addresses for each PS communicator mapping their serverless identities with <IP, Port> on WAN. While the preparation is done, each cloud-level training partition deploys corresponding training functions and executes them locally, inclu… view at source ↗
Figure 5
Figure 5. Figure 5: ASGD-GA synchronization process [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Inter-PS MA synchronization process Asynchronous SGD with gradient accumulation (ASGD-GA). ASGD-GA focuses on synchronizing gradients, as shown in [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Accuracy and loss comparison of training LeNet, ResNet, DeepFM with Cloudless-Training in Shanghai and Chongqing regions of Tencent Cloud and trivial PS ML training in Shanghai region of Tencent Cloud III. EVALUATION Cloudless-Training is evaluated for usability, performance of elastic scheduling and communication with 3 AI models in Tencent Cloud, a typical public cloud provider in China, showing that Clo… view at source ↗
Figure 8
Figure 8. Figure 8: Training time and cost comparison with and without elastic scheduling in 3 cases with various data distribution (1:1, 1:2, 2:1) and CPU cores allocations in Shanghai and Chongqing regions of Tencent Cloud [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Accuracy convergence comparison with and without elastic scheduling in 3 cases with various data distribution (1:1, 1:2, 2:1) and CPU cores allocations in Shanghai and Chongqing regions of Tencent Cloud C. Performance of Synchronization Specific settings. To evaluate the effect of synchronization optimization, including asynchronous SGD with gradient accumulation (ASGD-GA) and inter-PS model averaging (MA)… view at source ↗
Figure 10
Figure 10. Figure 10: Training time and accuracy convergence comparison of ResNet with 3 kinds of model synchronization strategies (ASGD, ASGD-GA, AMA) in Shanghai and Chongqing regions of Tencent Cloud [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The training time and accuracy convergence comparison of ResNet with 4 kinds of model synchronization strategies (ASGD, ASGD-GA, AMA, SMA) in a self-hosted environment IV. RELATED WORK Geo-distributed ML training. Geo-distributed ML training has attracted wide research interest recently [8][9][10]. It can be seen as a high-level variant of distributed ML training, like Horovod[30] and Kungfu[31]. Some cha… view at source ↗
read the original abstract

Geo-distributed ML training can benefit many emerging ML scenarios (e.g., large model training, federated learning) with multi-regional cloud resources and wide area network. However, its efficiency is limited due to 2 challenges. First, efficient elastic scheduling of multi-regional cloud resources is usually missing, affecting resource utilization and performance of training. Second, training communication on WAN is still the main overhead, easily subjected to low bandwidth and high fluctuations of WAN. In this paper, we propose a framework, Cloudless-Training, to realize efficient PS-based geo-distributed ML training in 3 aspects. First, it uses a two-layer architecture with control and physical training planes to support elastic scheduling and communication for multi-regional clouds in a serverless maner.Second, it provides an elastic scheduling strategy that can deploy training workflows adaptively according to the heterogeneity of available cloud resources and distribution of pre-existing training datasets. Third, it provides 2 new synchronization strategies for training partitions among clouds, including asynchronous SGD with gradient accumulation (ASGD-GA) and inter-PS model averaging (MA). It is implemented with OpenFaaS and evaluated on Tencent Cloud. Experiments show that Cloudless-Training can support general ML training in a geo-distributed way, greatly improve resource utilization (e.g., 9.2%-24.0% training cost reduction) and synchronization efficiency (e.g., 1.7x training speedup over baseline at most) with model correctness guarantees.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes Cloudless-Training, a serverless framework for parameter-server-based geo-distributed ML training. It introduces a two-layer (control and physical) architecture to enable elastic scheduling of multi-regional cloud resources, an adaptive scheduling strategy that accounts for resource heterogeneity and pre-existing dataset distribution, and two new synchronization primitives (asynchronous SGD with gradient accumulation and inter-PS model averaging). Experiments on Tencent Cloud are reported to achieve 9.2–24.0% training cost reduction and up to 1.7× speedup over a baseline while preserving model correctness.

Significance. If the reported efficiency gains and correctness guarantees are reproducible under realistic WAN conditions, the work would offer a practical serverless approach to geo-distributed training that improves resource utilization and reduces communication overhead, with potential relevance to large-model and federated-learning scenarios.

major comments (2)
  1. [Abstract] Abstract: the stated experimental results (9.2–24.0% cost reduction, 1.7× speedup) are presented without any description of the baselines, models, datasets, number of runs, statistical significance tests, or error bars, which directly affects the verifiability of the central efficiency claims.
  2. [Synchronization strategies] The section describing the synchronization strategies: the assertion that ASGD-GA and inter-PS model averaging provide 'model correctness guarantees' under WAN bandwidth fluctuations and resource heterogeneity is made without convergence analysis, staleness bounds, or explicit handling of data heterogeneity, which is load-bearing for the claim that the strategies require no changes to the training algorithm.
minor comments (2)
  1. Several acronyms (PS, WAN, ASGD-GA, MA) are used before being defined; a glossary or first-use expansion would improve readability.
  2. [Abstract] The abstract mentions 'general ML training' but does not list the concrete models or tasks used in the evaluation; adding this information would clarify the scope of the claimed generality.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their insightful comments, which help improve the clarity and rigor of our paper. We address each major comment point by point below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the stated experimental results (9.2–24.0% cost reduction, 1.7× speedup) are presented without any description of the baselines, models, datasets, number of runs, statistical significance tests, or error bars, which directly affects the verifiability of the central efficiency claims.

    Authors: We agree with this observation. The abstract summarizes the key results but omits details on the experimental configuration. In the revised manuscript, we will expand the abstract to include brief descriptions of the baselines used (standard PS-based training), the models and datasets evaluated, the number of runs, and note that error bars are reported in the main text. This will improve verifiability without exceeding abstract length constraints. revision: yes

  2. Referee: [Synchronization strategies] The section describing the synchronization strategies: the assertion that ASGD-GA and inter-PS model averaging provide 'model correctness guarantees' under WAN bandwidth fluctuations and resource heterogeneity is made without convergence analysis, staleness bounds, or explicit handling of data heterogeneity, which is load-bearing for the claim that the strategies require no changes to the training algorithm.

    Authors: The ASGD-GA and inter-PS model averaging strategies are designed as extensions that preserve the semantics of the original synchronous training algorithms, requiring no modifications to the model or loss function. While we do not provide a new convergence proof, the strategies build on established properties of asynchronous SGD and model averaging, with gradient accumulation mitigating staleness effects. Experiments across heterogeneous WAN conditions validate model correctness. We will revise the section to explicitly discuss these design choices and reference relevant prior analyses on similar techniques. A full theoretical treatment of data heterogeneity in this context is beyond the current scope but could be noted as future work. revision: partial

Circularity Check

0 steps flagged

No significant circularity; framework claims rest on implementation and experiments

full rationale

The paper introduces a two-layer serverless architecture, elastic scheduling, and two synchronization strategies (ASGD-GA and inter-PS model averaging). These are presented as engineering contributions whose correctness and performance are supported by implementation on OpenFaaS and empirical measurements on Tencent Cloud (cost reductions, speedups). No equations, parameter-fitting steps, or derivations appear in the provided text. No self-citations are invoked as load-bearing uniqueness theorems, and no ansatz or renaming of known results is used to substitute for independent justification. The central claims therefore do not reduce to their own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The framework rests on standard domain assumptions about cloud resource heterogeneity and WAN limitations; no free parameters, new mathematical axioms, or invented physical entities are introduced.

axioms (2)
  • domain assumption Efficient elastic scheduling of multi-regional cloud resources is usually missing in geo-distributed ML training.
    Stated as the first challenge in the abstract.
  • domain assumption Training communication on WAN is the main overhead due to low bandwidth and high fluctuations.
    Stated as the second challenge in the abstract.

pith-pipeline@v0.9.0 · 5807 in / 1340 out tokens · 42565 ms · 2026-05-24T09:13:25.131593+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages · 4 internal anchors

  1. [1]

    Language models are few - shot learners

    T. Brown, B. Mann, N. Ryder, M. Subbiah , J. Kaplan, P. Dhariwal, et al. "Language models are few - shot learners." Advances in neural information processing systems, 33: 1877 - 1901,

    work page 1901

  2. [2]

    Stoic: Serverless teleoperable hybrid cloud for machine learning applica tions on edge device

    M. Zhang, C. Krintz and R. Wolski. "Stoic: Serverless teleoperable hybrid cloud for machine learning applica tions on edge device." 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). IEEE,

    work page 2020

  3. [3]

    [Online.] Available: https://cloud.tencent.com/?mobile&lang=en

    (2022) Tencent Cloud. [Online.] Available: https://cloud.tencent.com/?mobile&lang=en

    work page 2022

  4. [4]

    K. He, X. Zhang, S. Ren and J. Sun. (2016). Deep residual learning for image recognition. IEEE

    work page 2016

  5. [5]

    OpenFaaS - Serverless Functions Made Simple

    (2019). OpenFaaS - Serverless Functions Made Simple. https://github . com/openfaas/faas

    work page 2019

  6. [6]

    Nebula - I: A Gen eral Framework for Collaboratively Training Deep Learning Models on Low - Bandwidth Cloud Clusters

    Y. Xiang, Z. Wu, W. Gong, S. Ding, X. Mo, Y. Liu. "Nebula - I: A Gen eral Framework for Collaboratively Training Deep Learning Models on Low - Bandwidth Cloud Clusters." arXiv preprint arXiv:2205.09470

    work page arXiv

  7. [7]

    Towards scalable distributed training of deep learning on public cloud clusters

    Shi, S., X. Zhou, S. Song, X. Y. Wang, Z.L. Hu, X. Huang, et al. "Towards scalable distributed training of deep learning on public cloud clusters." Proceedings of Machine Learning and Systems 3 (2021): 401 -

    work page 2021

  8. [8]

    Deep gradient compression: Reducing the communication bandwidth for distributed training

    Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally. "Deep gradient compression: Reducing the communication bandwidth for distributed training." arXiv preprint arXiv:1712.01887 (2017)

    work page arXiv 2017

  9. [9]

    Poseidon: An efficient communication architecture for distributed deep learning on {GPU} clusters

    H. Zhang , Z. Zheng , S. Z. Xu, W. Dai, Q. R. Ho, X. D. Liang, et al. "Poseidon: An efficient communication architecture for distributed deep learning on {GPU} clusters." 2017 USENIX Annual Technical Conference (USENIX ATC 17)

    work page 2017

  10. [10]

    Mesh - tensorflow: Deep learning for supercomputers

    N. Shazeer, Y. L. Cheng, N. Parmar, D. Tran, A. Vaswani, P. Koanantakool, et al. "Mesh - tensorflow: Deep learning for supercomputers." A dvances in neural information processing systems 31 ( 2018 )

    work page 2018

  11. [11]

    Efficient processing of deep neural networks: A tutorial and survey

    V. Sze, Y. J. Chen, H. T. Yang and J. S. Emer . "Efficient processing of deep neural networks: A tutorial and survey." Proceedings of the IEEE 105.12: 2295 - 2329 , 2017

    work page 2017

  12. [12]

    Moving to the edge - cloud - of - things: recent advances and future research directions

    B., Hind, S. Rakrak, S. Raghay, a nd B. Buhnova. "Moving to the edge - cloud - of - things: recent advances and future research directions." Electronics , 7(11):309, 2018

    work page 2018

  13. [13]

    [Online.] Available: https://marketrealist.com/p/where - are - tiktok - servers - located

    (2022) TikTok Server Locations in Question, FCC Requests App Store Removal. [Online.] Available: https://marketrealist.com/p/where - are - tiktok - servers - located

    work page 2022

  14. [14]

    (2022) Amazon EC2 instance network bandwidth [Online.] Available: https://do cs.aws.amazon.com/zh_cn/AWSEC2/latest/UserGuide/ec2 - instance - network - bandwidth.html

    work page 2022

  15. [15]

    Serverless computing: Design, implementation, and performance

    G. McGrath and P. R. Brenner. "Serverless computing: Design, implementation, and performance." 2017 IEEE 37th International Conference on Distributed Computing Systems Wor kshops (ICDCSW). IEEE,

    work page 2017

  16. [16]

    An overview of gradient descent optimization algorithms

    S. Ruder. "An overview of gradient descent optimization algorithms." arXiv preprint arXiv:1609.04747 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Alex and G

    K. Alex and G. Hinton. Learning multiple layers of features from tiny images (Technical Report). University of Toronto , 2009

    work page 2009

  18. [18]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. "Megatron - lm: Training multi - billion parameter language models using model parallelism." arXiv preprint arXiv:1909.08053 (2019)

    work page internal anchor Pith review Pith/arXiv arXiv 1909

  19. [19]

    Horovod: fast and easy distributed deep learning in TensorFlow

    S. Alexander and M. D. Balso. "Horovod: fast and easy distributed deep learning in TensorFlow." arXiv preprint arXiv:1802.05799

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    (2017) Baidu - allreduce [ Online.] Available: https://github.com/baidu - research/baidu - allreduce

    work page 2017

  21. [21]

    Sparse communication for distributed gradient descent

    A. A. Fikri and K. Heafield. "Sparse communication for distributed gradient descent." arXiv preprint arXiv:1704.05021

    work page arXiv

  22. [22]

    [Online.] Available: https: //kubeless.io

    (2019) Kubeless: The Kubernetes Native Serverless Framework. [Online.] Available: https: //kubeless.io

    work page 2019

  23. [23]

    FedLess: secure and scalable federated learning using serve rless computing

    A. Grafberger, M. Chadha, A. Jindal, J. Gu and M. Gerndt. "FedLess: secure and scalable federated learning using serve rless computing." 2021 IEEE International Conference on Big Data (Big Data). IEEE,

    work page 2021

  24. [24]

    Cirrus: A Serverless Framework for End - to - end ML Workflows

    J. Carreira, P. Fonseca, A. Tumanov, A. Zhang and R. K. "Cirrus: A Serverless Framework for End - to - end ML Workflows." The 2019 ACM Symposium on Cloud Computing (SoCC ’ 19), 2019

    work page 2019

  25. [25]

    Towards Demystifying Serverless Machine Learning Training

    J. Jiang, S. Gan, Y. Liu, F. Wang, G. Alonse, A. Klimovic et al. "Towards Demystifying Serverless Machine Learning Training." The 2021 ACM International Conference on Management of Data (SIGMOD ’ 21),

    work page 2021

  26. [26]

    [Online.] Avai lable: http://openwhisk.org/

    (2017) Apache OpenWhisk. [Online.] Avai lable: http://openwhisk.org/

    work page 2017

  27. [27]

    [Online.] Available: https://aws.amazon.com/lambda/

    (2022) Lambda. [Online.] Available: https://aws.amazon.com/lambda/

    work page 2022

  28. [28]

    Enabling compute - communication overlap in distributed deep learning training platforms

    S. Rashidi, M. Denton, S. Sridharan, S. Srinivasan, A. Suresh, Jade Nie et al. "Enabling compute - communication overlap in distributed deep learning training platforms." 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA). IEEE,

    work page 2021

  29. [29]

    DeepFM: A Factorization-Machine based Neural Network for CTR Prediction

    H. Guo, R. Tang, Y. Ye, Z. Li and X. He. "DeepFM: a factorization - machine based neural network for CTR prediction." arXiv preprint arXiv:1703.04247 201

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    Strategies and principles of distributed machine learning on big data

    E . P. Xing, Q . Ho, P . Xie, and D. Wei. " Strategies and principles of distributed machine learning on big data. " Engineering 2, 2 (2016), 179 –

    work page 2016

  31. [31]

    A survey on distributed machine learning

    J. V erbraeken, M . W olting , J . K atzy , J . K loppenburg, T. Verbelen and J. S. Reller meyer. "A survey on distributed machine learning." Acm computing surveys (csur) 53.2 (2020): 1 -

    work page 2020

  32. [32]

    Giraph unchained: Barrierless asynchronous parallel execution in pregel - like graph processing systems

    H . Minyang and K . Daudjee. "Giraph unchained: Barrierless asynchronous parallel execution in pregel - like graph processing systems." Proceedings of the VLDB Endowment 8.9 (2015): 950 -

    work page 2015