ScaleAcross: Designing Multi-Data-Center Infrastructure for Geo-Distributed AI Training
Pith reviewed 2026-06-27 05:39 UTC · model grok-4.3
The pith
An EVPN-VXLAN emulation framework allows study of geo-distributed AI training workloads over wide-area networks with commodity tools.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The authors present a framework that combines VXLAN overlays with EVPN-based inter-data-center connectivity, implemented using ContainerLab and FRRouting. It incorporates Equal-Cost Multi-Path routing, Bidirectional Forwarding Detection, and a queue-pair-aware traffic distribution mechanism to handle synchronization-intensive workloads such as AllReduce and Parameter Server communication under realistic wide-area emulation.
What carries the argument
EVPN-VXLAN overlay network with ECMP routing, BFD, and queue-pair-aware traffic distribution for multi-data-center AI workload connectivity.
If this is right
- Traffic distribution improves for synchronization-intensive AI workloads while using standard routing features.
- BFD provides faster failure detection across data-center boundaries.
- The setup remains compatible with commodity infrastructure without custom hardware.
- Reproducible emulation yields insights into traffic and resilience behavior for geo-distributed training.
Where Pith is reading between the lines
- The emulation approach could reduce experimental costs by substituting for physical multi-site deployments in early-stage infrastructure design.
- Insights on queue-pair-aware distribution might extend to optimizing other collective communication patterns in distributed systems.
- Integration with real AI training frameworks could measure end-to-end effects on convergence time under emulated WAN conditions.
Load-bearing premise
The ContainerLab plus FRRouting emulation accurately reproduces latency, jitter, and packet-loss behavior of real wide-area networks when running AllReduce and Parameter Server workloads at scale.
What would settle it
Direct comparison of the same AllReduce and Parameter Server workloads on a physical multi-data-center testbed showing substantially different traffic distribution, latency, or packet-loss patterns than observed in the emulation.
Figures
read the original abstract
The rapid growth of AI models and increasing data sovereignty requirements are driving the transition toward geo-distributed AI training across multiple data centers. Such deployments introduce system-level challenges arising from synchronization-intensive communication, cross-site data exchange, and wide-area latency constraints. This paper investigates EVPN--VXLAN as an infrastructure foundation for geo-distributed AI training environments and presents a scalable emulation framework for systematically studying distributed AI workloads under realistic wide-area conditions. The proposed framework combines VXLAN overlays with EVPN-based inter-data-center connectivity and is implemented using ContainerLab and FRRouting (FRR). The framework further incorporates Equal-Cost Multi-Path (ECMP) routing, Bidirectional Forwarding Detection (BFD), and a queue-pair-aware traffic distribution mechanism designed to improve communication behavior for synchronization-intensive AI workloads while preserving compatibility with commodity infrastructure. Using realistic WAN emulation, we characterize communication and system behavior under distributed training workloads employing AllReduce and Parameter Server communication patterns. Results provide insights into traffic distribution, resilience, and infrastructure behavior in geo-distributed AI environments, highlighting the potential of reproducible multi-data-center infrastructure frameworks for scalable distributed AI training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents ScaleAcross, a framework for multi-data-center infrastructure supporting geo-distributed AI training. It combines VXLAN overlays with EVPN-based inter-DC connectivity, implemented using ContainerLab and FRRouting (FRR), and incorporates ECMP routing, BFD, and a queue-pair-aware traffic distribution mechanism. The framework emulates realistic WAN conditions to characterize communication and system behavior under AllReduce and Parameter Server workloads, providing insights into traffic distribution, resilience, and infrastructure choices while maintaining compatibility with commodity hardware.
Significance. If the emulation accurately captures real WAN conditions and the reported behaviors hold, the work could supply a reproducible, commodity-compatible platform for systematically evaluating infrastructure decisions in geo-distributed AI training, an area of growing importance due to model scale and data sovereignty constraints. The emphasis on EVPN-VXLAN, ECMP/BFD, and queue-pair mechanisms offers a practical bridge between networking research and synchronization-heavy AI workloads.
major comments (2)
- [Abstract] Abstract: The central claim that the framework yields actionable insights on traffic distribution and resilience under AllReduce and Parameter Server patterns rests on 'realistic WAN emulation,' yet the abstract (and by extension the manuscript) supplies no quantitative metrics, error bars, baseline comparisons, or validation of emulated latency/jitter/loss distributions against production inter-DC traces or hardware testbeds.
- [Implementation/Evaluation (assumed §4–§5)] Implementation and evaluation sections: The assertion that ContainerLab plus FRRouting faithfully reproduces WAN latency, jitter, packet-loss, and correlation structure for scale AllReduce/PS workloads is load-bearing for all downstream claims about communication behavior and infrastructure conclusions, but no validation experiments (e.g., statistical comparison to real WAN traces) are described.
minor comments (2)
- [Abstract] The abstract is lengthy and could be condensed to foreground the specific quantitative findings once they are added.
- Notation for the queue-pair-aware distribution mechanism should be defined more explicitly if equations or pseudocode are introduced later in the manuscript.
Simulated Author's Rebuttal
We thank the referee for the constructive comments on the validation of the WAN emulation in ScaleAcross. We address each major comment below and will revise the manuscript accordingly.
read point-by-point responses
-
Referee: [Abstract] Abstract: The central claim that the framework yields actionable insights on traffic distribution and resilience under AllReduce and Parameter Server patterns rests on 'realistic WAN emulation,' yet the abstract (and by extension the manuscript) supplies no quantitative metrics, error bars, baseline comparisons, or validation of emulated latency/jitter/loss distributions against production inter-DC traces or hardware testbeds.
Authors: We agree that the abstract does not provide quantitative validation metrics or comparisons for the emulated WAN conditions. The framework implements configurable emulation of latency, jitter, and loss via ContainerLab and FRR parameters drawn from typical inter-DC values in the literature, but no statistical validation against production traces is present. We will revise the abstract to describe the emulation more precisely as 'configurable WAN emulation' rather than 'realistic,' qualify the resulting insights accordingly, and add a dedicated subsection in the evaluation section that documents the chosen parameters, their grounding, and the absence of direct trace-based validation as a limitation. revision: yes
-
Referee: [Implementation/Evaluation (assumed §4–§5)] Implementation and evaluation sections: The assertion that ContainerLab plus FRRouting faithfully reproduces WAN latency, jitter, packet-loss, and correlation structure for scale AllReduce/PS workloads is load-bearing for all downstream claims about communication behavior and infrastructure conclusions, but no validation experiments (e.g., statistical comparison to real WAN traces) are described.
Authors: We acknowledge that the manuscript asserts the use of realistic WAN emulation without including validation experiments or statistical comparisons to real traces. This is a substantive gap for claims about communication behavior. In revision we will expand the implementation and evaluation sections to include (where feasible) additional analysis or experiments that characterize emulation fidelity, or else add an explicit limitations discussion on the emulation approach and its grounding in standard tooling and literature parameters. revision: yes
Circularity Check
No circularity: engineering framework description with no derivations or fitted predictions
full rationale
The paper describes an implemented emulation framework (VXLAN+EVPN via ContainerLab+FRR, with ECMP/BFD and queue-pair mechanisms) and reports observed behavior under AllReduce/Parameter Server workloads. No equations, parameters fitted to data subsets, predictions, or self-citation chains appear in the provided text. The central claims rest on emulation results rather than any reduction to inputs by construction. The noted limitation (unvalidated WAN fidelity) is an external-validity concern, not circularity. This matches the default case of a self-contained systems paper.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Hedera:dynamicflowschedulingfordatacenternetworks., in: Nsdi, San Jose, USA
Al-Fares,M.,Radhakrishnan,S.,Raghavan,B.,Huang,N.,Vahdat,A.,etal.,2010. Hedera:dynamicflowschedulingfordatacenternetworks., in: Nsdi, San Jose, USA. pp. 89–92
2010
-
[2]
Conga: Distributed congestion-aware load balancing for datacenters, in: Proceedings of the 2014 ACM conference on SIGCOMM, pp
Alizadeh, M., Edsall, T., Dharmapurikar, S., Vaidyanathan, R., Chu, K., Fingerhut, A., Lam, V.T., Matus, F., Pan, R., Yadav, N., et al., 2014. Conga: Distributed congestion-aware load balancing for datacenters, in: Proceedings of the 2014 ACM conference on SIGCOMM, pp. 503– 514
2014
-
[3]
The evolution of the carrier cloud networking, in: 2013 IEEE Seventh International Symposium on Service- Oriented System Engineering, IEEE
Cai, D., Natarajan, S., 2013. The evolution of the carrier cloud networking, in: 2013 IEEE Seventh International Symposium on Service- Oriented System Engineering, IEEE. pp. 286–291
2013
-
[4]
Multiprotocol Extensions for BGP-4
Chandra, R., Bates, T.J., Rekhter, Y., Katz, D., 2007. Multiprotocol Extensions for BGP-4. RFC 4760. URL:https://www.rfc-editor. org/info/rfc4760, doi:10.17487/RFC4760. accessed: 2025-10-21
-
[5]
containerlab: Container-based networking lab framework.https://github.com/srl-labs/containerlab
Dodin, R., et al., 2020. containerlab: Container-based networking lab framework.https://github.com/srl-labs/containerlab. Accessed: 2025-06-10
2020
-
[6]
Beyondasingleaicluster:Asurveyofdecentralizedllmtraining
Dong,H.,Jiang,J.,Lu,R.,Luo,J.,Song,J.,Li,B.,Shen,Y.,Wang,Z.,2025. Beyondasingleaicluster:Asurveyofdecentralizedllmtraining. arXiv preprint arXiv:2503.11023
-
[7]
Elmadani, M., Sati, S.O., 2023. Data center lab using vxlan data plane and bgp-evpn control plane, in: 2023 4th International Conference on Data Analytics for Business and Industry (ICDABI), pp. 354–358. doi:10.1109/ICDABI60145.2023.10629438
-
[8]
Rdmaoverethernet for distributed training at meta scale, in: Proceedings of the ACM SIGCOMM 2024 Conference, pp
Gangidi,A.,Miao,R.,Zheng,S.,Bondu,S.J.,Goes,G.,Morsy,H.,Puri,R.,Riftadi,M.,Shetty,A.J.,Yang,J.,etal.,2024. Rdmaoverethernet for distributed training at meta scale, in: Proceedings of the ACM SIGCOMM 2024 Conference, pp. 57–70
2024
-
[9]
Scc: Synchronization congestion control for multi-tenant learning over geo-distributed clouds
Gao, C., Li, F., Ye, K., Wang, Y., Wang, P., Wang, X., Xu, C., 2025. Scc: Synchronization congestion control for multi-tenant learning over geo-distributed clouds. IEEE Transactions on Computers
2025
-
[10]
A survey of virtual private lan services (vpls): Past, present and future
Gaur, K., Kalla, A., Grover, J., Borhani, M., Gurtov, A., Liyanage, M., 2021. A survey of virtual private lan services (vpls): Past, present and future. Computer Networks 196, 108245
2021
-
[11]
Ge,H.,Feng,J.,Huang,Q.,Fu,F.,Nie,X.,Zuo,L.,Lin,H.,Cui,B.,Liu,X.,2025. Bytescale:Communication-efficientscalingofllmtraining witha2048kcontextlengthon16384gpus,in:ProceedingsoftheACMSIGCOMM2025Conference,AssociationforComputingMachinery, New York, NY, USA. p. 963–978. URL:https://doi.org/10.1145/3718958.3754352, doi:10.1145/3718958.3754352
-
[12]
Evolution of data center design to handle ai workloads, in: 2024 34th International Telecommunication Networks and Applications Conference (ITNAC), IEEE
Gupta, T., 2024. Evolution of data center design to handle ai workloads, in: 2024 34th International Telecommunication Networks and Applications Conference (ITNAC), IEEE. pp. 1–8
2024
-
[13]
Reproduciblenetworkexperimentsusingcontainer-basedemulation, in: Proceedings of the 8th international conference on Emerging networking experiments and technologies, pp
Handigol,N.,Heller,B.,Jeyakumar,V.,Lantz,B.,McKeown,N.,2012. Reproduciblenetworkexperimentsusingcontainer-basedemulation, in: Proceedings of the 8th international conference on Emerging networking experiments and technologies, pp. 253–264
2012
-
[14]
Hoefler,T.,Roweth,D.,Underwood,K.,Alverson,R.,Griswold,M.,Tabatabaee,V.,Kalkunte,M.,Anubolu,S.,Shen,S.,McLaren,M.,etal.,
-
[15]
Computer 56, 67–77
Data center ethernet and remote direct memory access: Issues at hyperscale. Computer 56, 67–77
-
[16]
Hoefler, T., Schramm, K., Spada, E., Underwood, K., Alexander, C., Alverson, B., Bottorff, P., Caulfield, A., Handley, M., Huang, C., et al.,
-
[17]
Ultra ethernet’s design principles and architectural innovations,
Ultra ethernet’s design principles and architectural innovations. arXiv preprint arXiv:2508.08906
-
[18]
Analysis of an equal-cost multi-path algorithm
Hopps, C., 2000. Analysis of an equal-cost multi-path algorithm. Technical Report
2000
-
[19]
L3dml: Facilitating geo-distributed machine learning in network layer
Hou, X., Gao, S., Liu, N., Yao, F., Lei, B., Zhang, H., Das, S., 2024. L3dml: Facilitating geo-distributed machine learning in network layer. IEEE Transactions on Network and Service Management PP, 1–1. doi:10.1109/TNSM.2024.3509031
-
[20]
Gaia: Geo-Distributed machine learning approachingLANspeeds,in:14thUSENIXSymposiumonNetworkedSystemsDesignandImplementation(NSDI17),USENIXAssociation, Boston, MA
Hsieh, K., Harlap, A., Vijaykumar, N., Konomis, D., Ganger, G.R., Gibbons, P.B., Mutlu, O., 2017. Gaia: Geo-Distributed machine learning approachingLANspeeds,in:14thUSENIXSymposiumonNetworkedSystemsDesignandImplementation(NSDI17),USENIXAssociation, Boston, MA. pp. 629–647. URL:https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/hsieh....
2017
-
[21]
Demystifyingnccl:Anin-depth analysis of gpu communication protocols and algorithms, in: 2025 IEEE Symposium on High-Performance Interconnects (HOTI), IEEE
Hu,Z.,Shen,S.,Bonato,T.,Jeaugey,S.,Alexander,C.,Spada,E.,Dinan,J.,Hammond,J.,Hoefler,T.,2025. Demystifyingnccl:Anin-depth analysis of gpu communication protocols and algorithms, in: 2025 IEEE Symposium on High-Performance Interconnects (HOTI), IEEE. pp. 48–59
2025
-
[22]
Jiang, Z., Lin, H., Zhong, Y., Huang, Q., Chen, Y., Zhang, Z., Peng, Y., Li, X., Xie, C., Nong, S., et al., 2024.{MegaScale}: Scaling large language model training to more than 10,000{GPUs}, in: 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pp. 745–760
2024
-
[23]
Bidirectional Forwarding Detection (BFD)
Katz, D., Ward, D., 2010. Bidirectional Forwarding Detection (BFD). RFC 5880. URL:https://www.rfc-editor.org/rfc/rfc5880. html, doi:10.17487/RFC5880. accessed: 2025-10-21
-
[24]
Parallelgradientcomputationandsynchronization:Enhancingtheefficiencyofdistributed training for llms
Li,H.,Jiang,H.,Wu,J.,Yang,G.,Zhang,J.,2025. Parallelgradientcomputationandsynchronization:Enhancingtheefficiencyofdistributed training for llms. IEEE Transactions on Network Science and Engineering
2025
-
[25]
Communicationefficientdistributedmachinelearningwiththeparameterserver
Li,M.,Andersen,D.G.,Smola,A.,Yu,K.,2014. Communicationefficientdistributedmachinelearningwiththeparameterserver. Advances in neural information processing systems 27
2014
-
[26]
arXiv preprint arXiv:2404.06114
Liang,F.,Zhang,Z.,Lu,H.,Leung,V.,Guo,Y.,Hu,X.,2024.Communication-efficientlarge-scaledistributeddeeplearning:Acomprehensive survey. arXiv preprint arXiv:2404.06114
-
[27]
Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, L., Sridhar, T., Bursell, M., Wright, C., 2014. Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks. RFC 7348. URL:https: //www.rfc-editor.org/info/rfc7348, doi:10.17487/RFC7348. accessed: 2025-10-21
-
[28]
Federated learning of deep networks using model averaging,
McMahan, H.B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A., 2023. Communication-efficient learning of deep networks from decentralized data. URL:https://arxiv.org/abs/1602.05629,arXiv:1602.05629
-
[29]
Collaborative deep learning across multiple data centers
Mi, H., Xu, K., Feng, D., Wang, H., Zhang, Y., Zheng, Z., Chen, C., Lan, X., 2020. Collaborative deep learning across multiple data centers. Science China Information Sciences 63, 182102
2020
-
[30]
Miao, X., Nie, X., Shao, Y., Yang, Z., Jiang, J., Ma, L., Cui, B., 2021. Heterogeneity-aware distributed machine learning training via partial reduce,in:Proceedingsofthe2021InternationalConferenceonManagementofData,AssociationforComputingMachinery,NewYork,NY, USA. p. 2262–2270. URL:https://doi.org/10.1145/3448016.3452773, doi:10.1145/3448016.3452773
-
[31]
Noghani, K.A., Kassler, A., Gopannan, P.S., 2018. Evpn/sdn assisted live vm migration between geo-distributed data centers, in: 2018 4th IEEE Conference on Network Softwarization and Workshops (NetSoft), pp. 105–113. doi:10.1109/NETSOFT.2018.8459946
-
[32]
accessed: 2025-06-08
NVIDIADeveloperBlog,2024.Turbochargellmtrainingacrosslong-hauldatacenternetworkswithnvidianemoframework.URL:https:// developer.nvidia.com/blog/turbocharge-llm-training-across-long-haul-data-center-networks-with-nvidia-nemo-framework/. accessed: 2025-06-08
2024
-
[33]
Datacenter optimization methods for softwarized network services
Pannocchi, L., Lahiri, S., Fichera, S., Artale, A., Cucinotta, T., 2024. Datacenter optimization methods for softwarized network services. Journal of Systems Architecture 156, 103270
2024
-
[34]
The ns-3 network simulator, in: Modeling and tools for network simulation
Riley, G.F., Henderson, T.R., 2010. The ns-3 network simulator, in: Modeling and tools for network simulation. Springer, pp. 15–34
2010
-
[35]
Acceleratingcollectivecommunicationindataparalleltrainingacrossdeeplearningframeworks,in:19thUSENIXSymposiumonNetworked Systems Design and Implementation (NSDI 22), pp
Romero, J., Yin, J., Laanait, N., Xie, B., Young, M.T., Treichler, S., Starchenko, V., Borisevich, A., Sergeev, A., Matheson, M., 2022. Acceleratingcollectivecommunicationindataparalleltrainingacrossdeeplearningframeworks,in:19thUSENIXSymposiumonNetworked Systems Design and Implementation (NSDI 22), pp. 1027–1040
2022
-
[36]
A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)
Sajassi, A., Drake, J., Bitar, N., Shekhar, R., Uttaro, J., Henderickx, W., 2018. A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN). RFC 8365. URL:https://www.rfc-editor.org/info/rfc8365, doi:10.17487/RFC8365. accessed: 2025-10-21
-
[37]
librxe-dev: Software rdma over ethernet (soft-roce) implementation.https://github.com/SoftRoCE/ librxe-dev
Soft-RoCE Development Team, . librxe-dev: Software rdma over ethernet (soft-roce) implementation.https://github.com/SoftRoCE/ librxe-dev. Accessed: 2026-05-20
2026
-
[38]
Hcec:Anefficientgeo-distributeddeeplearningtrainingstrategybasedonwait-free back-propagation
Song,Y.,Ai,Y.,Xiao,X.,Liu,Z.,Tang,Z.,Li,K.,2024. Hcec:Anefficientgeo-distributeddeeplearningtrainingstrategybasedonwait-free back-propagation. Journal of Systems Architecture 148, 103070
2024
-
[39]
Won,W.,Heo,T.,Rashidi,S.,Sridharan,S.,Srinivasan,S.,Krishna,T.,2023.Astra-sim2.0:Modelinghierarchicalnetworksanddisaggregated systemsforlarge-modeltrainingatscale,in:2023IEEEInternationalSymposiumonPerformanceAnalysisofSystemsandSoftware(ISPASS), IEEE. pp. 283–294
2023
-
[40]
Collaborative Deep Learning Across Multiple Data Centers
Xu, K., Mi, H., Feng, D., Wang, H., Chen, C., Zheng, Z., Lan, X., 2018. Collaborative deep learning across multiple data centers. URL: https://arxiv.org/abs/1810.06877,arXiv:1810.06877. :Preprint submitted to Elsevier Page 19 of 19
work page internal anchor Pith review Pith/arXiv arXiv 2018
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.