Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Periodic synchronization with a dual gradient-and-parameter reconciliation step can beat per-step gradient synchronization on wide-area links, both in peak accuracy and in wall-clock time.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:05 UTC pith:QE3WOA75

load-bearing objection Honest, well-run empirical study of a LocalSGD+SlowMo periodic-sync recipe whose WAN accuracy-time claim is real under fixed LR=0.1 but would likely flip under their own linear-scaling rule. the 3 major comments →

arxiv 2607.21224 v1 pith:QE3WOA75 submitted 2026-07-23 cs.DC

Controlled Periodic Synchronization for Efficient Data-Parallel Training

Imane Ettifouri , Mostapha Zbakh , Claude Tadonki This is my paper
classification cs.DC
keywords distributed deep learningdata-parallel trainingsynchronization frequencyperiodic synchronizationcommunication efficiencywide-area network trainingSlowMo momentumLocalSGD
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that synchronization frequency is a tunable systems parameter, not a fixed per-step ritual, for distributed training under communication constraints. It proposes CPDP, in which workers run several local updates and then reconcile by AllReduce-aggregating gradients, applying a synchronized optimizer step, and averaging parameters through a SlowMo momentum filter. In a cross-site wide-area setting with a fixed learning rate of 0.1, the paper reports that at a synchronization period of K=4, CPDP reaches a higher peak test accuracy than standard per-step synchronous data parallelism (by 2.44 percentage points) while reducing average wall-clock time by 13.8%. Direct profiling shows exposed synchronization time at K=4 is roughly half that of per-step synchronization, explaining the improved accuracy-time trade-off. A careful reader would care because slow, high-latency links are where the default 'synchronize every step' policy hurts most, and the paper offers a practical lever and a measurement method for choosing how often to synchronize.

Core claim

The central claim: under communication-constrained cross-site training, periodic dual-phase reconciliation — gradient AllReduce followed by momentum-stabilized parameter averaging — yields better accuracy and lower wall-clock time than every-step gradient synchronization. In the paper's main WAN experiment (two sites, ~16.6 ms round-trip latency, 8 workers, fixed LR=0.1, residual network on a 100-class benchmark), CPDP at K=4 reaches 76.28±0.34% mean peak test accuracy versus 73.84±0.23% for synchronous DDP and 74.51±0.22% for LocalSGD, while mean wall-clock time drops 13.8% (4,275 s vs 4,959 s). Direct profiling attributes this to event count: synchronization events fall from 48 to 12 per e

What carries the argument

The load-bearing mechanism is the dual-phase reconciliation step executed every K mini-batch steps. First, gradients from all workers are combined by AllReduce and a synchronized optimizer step is taken; second, model parameters are averaged across workers and the averaged parameter is passed through SlowMo momentum — a velocity-accumulated low-pass filter on inter-worker drift — before being copied back to every replica. This restores exact parameter equality at boundaries while leaving optimizer internal states local. A second piece of machinery is the exposed-synchronization profiler, which measures per-event and per-epoch synchronization time directly rather than inferring it from wall-c

Load-bearing premise

The headline CPDP advantage is measured under the authors' chosen fixed learning rate of 0.1; under the linearly scaled LR=0.4 diagnostic they also report, per-step synchronous training edges ahead, so the main claim rests on the LR protocol being the right one for the WAN setting.

What would settle it

Run the same cross-site WAN setup at K=4 under a linearly scaled learning rate (LR=0.4 for the 1,024 global batch size). If per-step synchronous training's mean peak accuracy meets or exceeds CPDP's, the paper's claim that periodic dual-phase synchronization improves the WAN accuracy-time trade-off would fail; the authors currently report only the K=2 diagnostic at LR=0.4, leaving K=4 untested.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • On wide-area links with a stable learning rate, synchronization frequency is a practical control: K=4 simultaneously raises peak accuracy and cuts wall-clock time versus per-step synchronization.
  • Reducing synchronization events by a factor of K does not automatically save time, because each reconciliation boundary costs about twice as much as one per-step AllReduce; savings appear only when event reduction dominates, so K must be chosen for the network.
  • At practical periods (K≤4), dual reconciliation improves accuracy over LocalSGD, whose parameter-averaging-only boundary lacks a synchronized gradient signal.
  • The best synchronization period is environment-dependent: K=4 is the best accuracy-time point measured both intra-site and cross-site, while K≥16 collapses accuracy despite continued throughput gains.
  • Learning rate and SlowMo coefficient are coupled to K: aggressive linear LR scaling degrades periodic methods at 16 GPUs, but reducing LR largely recovers the loss, and β=0.3 is stable whereas β=0.9 collapses training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's fixed-LR protocol: because the authors' own LR=0.4 diagnostic puts DDP marginally ahead, a K=4 run under linearly scaled LR would be a direct test of whether the claimed WAN advantage is robust to LR protocol; the paper does not report that cell.
  • Beyond the paper: since CPDP reconciles parameters but not optimizer states, a testable extension is to also synchronize SGD momentum or AdamW moments at boundaries, which may extend the useful K range beyond K=4.
  • Beyond the paper: the profiler-based rule 'increase K until exposed synchronization stops dominating the epoch, then stop before accuracy degrades' could be automated into an adaptive K scheduler that reacts to measured latency and drift.
  • Beyond the paper: the sharp accuracy collapse at K≥16 suggests an interpretable drift-capacity limit; coupling K to a drift estimate such as parameter divergence between boundaries might predict the collapse point without a full sweep.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes Controlled Periodic Data Parallelism (CPDP), a synchronization-frequency control strategy for PyTorch DDP: workers run K local steps and at every K-th step perform a gradient AllReduce plus a SlowMo-corrected parameter averaging. The authors compare CPDP with DDP and LocalSGD on Grid'5000 in intra-site and cross-site (Nancy–Sophia, 16.6 ms RTT) settings, using ResNet-50/CIFAR-100, ViT-S/CIFAR-100, and ResNet-50/TinyImageNet. The headline result is in the cross-site WAN setting under a fixed LR=0.1 protocol: CPDP at K=4 reaches 76.28±0.34% peak test accuracy versus 73.84±0.23% for DDP, while reducing total wall-clock time by 13.8% (4275s vs 4959s). The paper also reports a K-sweep, SlowMo coefficient sensitivity, and direct synchronization profiling. The authors are explicit that CPDP does not universally dominate DDP and that the LR=0.1 choice is a 'stable WAN setting' rather than a linearly scaled LR.

Significance. If the headline result were robust, the paper would offer a useful, implementation-friendly knob for distributed training under WAN constraints. The experimental reporting is careful in several respects: three seeds, standard deviations, Welch tests on peak accuracy, sensitivity analyses for LR, K, and β, and a distinction between an indirect backward-time proxy and direct profiling. The authors also state clear limitations (no convergence theorem, no optimizer-state synchronization, no comparison with gradient compression). However, the central accuracy–time claim is not yet robust: the only LR at which CPDP beats DDP is the fixed LR=0.1 protocol, and the paper's own LR=0.4 diagnostic shows DDP ahead. In addition, the direct profiler numbers in Table 8 do not reconcile with the wall-clock values in Table 5. If those two issues are resolved, the empirical contribution would be valuable.

major comments (3)
  1. [§5.2 / Table 5 / §4.3] The main WAN accuracy claim is conditional on a protocol choice that reverses under the paper's own scaling rule. Section 4.3 and Table 4 define linear LR scaling with base 0.1 at batch 256, which prescribes LR=0.4 at the cross-site global batch 1024. The main comparison in Table 5 instead fixes LR=0.1; Section 5.2 reports that at LR=0.4 DDP reaches 77.39±0.70% and CPDP(K=2) 76.65±0.93%, so DDP is marginally ahead. No LR=0.4 K=4 run is given. Since DDP's accuracy shifts by +3.55pp between LR=0.1 and 0.4 while the CPDP advantage over DDP is +2.44pp, the headline 'improved accuracy–time trade-off' is not robust to the learning-rate protocol. The authors disclose this, but the claim needs either an a priori justification for fixing LR=0.1 in the cross-site setting or a complete LR=0.4 comparison at K=4.
  2. [§6.2 / Tables 8 and 5] The direct profiler does not explain the wall-clock improvements. Table 8 lists exposed synchronization time 20.5s/epoch and compute floor 9.0s/epoch for CPDP K=4, implying about 29.5s/epoch; Table 5 shows 42.75s/epoch. For CPDP K=2 the implied time is 48.8s/epoch versus an actual 72.0s/epoch. DDP, by contrast, is consistent (48.9s implied vs 49.6s actual). Moreover, Table 7 reports CPDP K=2 backward time as 26.3s/epoch, which is inconsistent with the Table 8 compute floor of 8.8s for the same configuration. The profiler appears to measure a different execution path from the runs in Table 5; until the missing ~13s/epoch (or ~23s/epoch at K=2) is accounted for and the compute floor definition is clarified, the statement that halved exposed synchronization 'explains' the 13.8% time saving is unsupported.
  3. [§6.7 / Tables 10 and 16] The claimed advantage over LocalSGD is not cleanly attributed to the dual-phase mechanism. At g4, CPDP with β=0.0 (no SlowMo) is 78.74±0.17% (Table 16), essentially equal to LocalSGD's 78.70±0.22% (Table 10); CPDP's advantage appears only with β=0.3 (79.37±0.14%). The paper says the dual-phase gradient+parameter reconciliation 'provides stronger optimization trajectories than parameter averaging alone,' but the included β=0.0 comparison suggests SlowMo, not the gradient AllReduce, may be the active ingredient. A LocalSGD+SlowMo baseline or a CPDP β=0 run at the scales where the improvement is claimed is needed to support the attribution.
minor comments (5)
  1. [§5.5] Welch tests are computed on peak test accuracy, a statistic selected over the run. With three seeds, this selection can inflate the apparent significance; the authors interpret borderline p-values conservatively, but this should be stated explicitly as a limitation of the test.
  2. [Table 5] The column heading 'Time (s) vs DDP (pp)' is ambiguous: the last column is an accuracy difference in percentage points, not a time comparison. Please split the heading.
  3. [Table 8 / §4.4] The 'Compute floor' quantity is not defined in the instrumentation section. Specify whether it is forward+backward time with no communication and why it differs from the T_bwd values in Table 7 for the same configuration.
  4. [Data availability] The implementation is promised only after publication. For an empirical systems paper, providing the artifact and scripts at review time would substantially strengthen reproducibility.
  5. [§6.1 / Table 6] The intra-site to cross-site 'degradation' comparison conflates changes in LR and global batch size with the network effect. The text acknowledges this, but the table and surrounding discussion should avoid phrasing that suggests the Δ column isolates WAN latency.

Circularity Check

0 steps flagged

No circular derivation: the paper is an empirical systems study; its only notable weakness is disclosed protocol-sensitivity in the choice of LR=0.1, which is a correctness/robustness concern, not circularity.

full rationale

The paper does not present a formal derivation whose conclusion is built into its assumptions. CPDP is defined by an explicit update rule (Algorithm 1, Section 3.2-3.3) combining periodic gradient AllReduce, parameter averaging, and SlowMo correction; the reported accuracies and wall-clock times are measured outcomes, not quantities recovered from fitted parameters. The main comparison uses identical hyperparameters across DDP, LocalSGD, and CPDP, and the method is evaluated on additional workloads. The strongest concern is the choice of the fixed LR=0.1 cross-site protocol: Section 5.2 states that under LR=0.4 'the three methods produce close peak accuracies relative to the observed inter-seed variability, with DDP marginally ahead,' and that the authors 'adopt LR=0.1 as the main WAN comparison while reporting LR=0.4 as a sensitivity result.' This is post-hoc conditioning on a favorable protocol, and it weakens the robustness of the headline accuracy-time claim, but it is not circular in the derivation sense: the paper does not define CPDP's value in terms of the LR=0.1 outcome, nor does it fit any parameter to the test accuracies it then 'predicts.' There is no load-bearing self-citation, no imported uniqueness theorem, and no renaming of a known result presented as a derivation. The limitation is explicitly disclosed in Section 5.2 and in the Discussion ('We do not claim that CPDP consistently outperforms fully synchronous DDP...'). Accordingly, no specific circular step can be exhibited, and the correct circularity score is 0; the LR-sensitivity concern belongs to correctness risk, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central empirical claim rests on tuned hyperparameters (K, β, LR) and on framework semantics (DDP no_sync). The most consequential free choice is LR=0.1, which the paper's own sensitivity analysis shows is necessary for the claimed CPDP advantage.

free parameters (3)
  • Synchronization period K = 2, 4 (primary claims); sweep 1–64
    Central control parameter; the positive result is at K=4, selected after ablation.
  • SlowMo coefficient β = 0.3
    Chosen because Table 16 shows β=0.3 achieves the highest mean peak accuracy; β=0.9 collapses training.
  • Learning rate (cross-site WAN) = 0.1 (main), 0.4 (sensitivity)
    The claimed CPDP-vs-DDP advantage appears at LR=0.1 and disappears at LR=0.4, making it a protocol-fitting parameter.
axioms (4)
  • domain assumption The loss is smooth and stochastic gradients have bounded variance (Section 3.1 'Boundary Conditions').
    Invoked to place CPDP in the theoretical context of LocalSGD/SlowMo; no new convergence theorem is provided.
  • domain assumption PyTorch DDP's no_sync() context disables gradient reduction while preserving local gradient computation.
    The entire implementation relies on this framework contract (Section 4.2).
  • domain assumption The direct profiler's exposed-synchronization time measures the true communication cost of each sync event.
    Used to explain the wall-clock differences (Table 8); the unexplained gap vs Table 5 timings suggests this assumption is imperfect.
  • domain assumption Peak test accuracy over a run is an unbiased metric for model quality.
    Used as the primary metric; selection of the best epoch can inflate apparent differences, though applied equally to all methods.

pith-pipeline@v1.3.0-alltime-deepseek · 24815 in / 14987 out tokens · 146249 ms · 2026-08-01T08:05:15.144279+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Controlled Periodic Synchronization for Efficient Data-Parallel Training." pith.science (2026). https://pith.science/paper/QE3WOA75

@misc{pith2026260721224,
  author       = {Pith},
  title        = {Pith review of: Controlled Periodic Synchronization for Efficient Data-Parallel Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QE3WOA75}},
  note         = {Machine review of arXiv:2607.21224}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Data-parallel training relies on frequent gradient synchronization across workers. Standard DDP synchronizes gradients at every iteration, which is effective on fast local-area networks but increasingly sensitive to communication latency and network variability in geographically distributed environments. Periodic methods such as LocalSGD reduce synchronization frequency but rely mainly on parameter averaging, which may be insufficient when worker trajectories diverge. This paper studies synchronization frequency as a systems parameter for communication-constrained distributed training. We evaluate Controlled Periodic Data Parallelism (CPDP), a PyTorch-DDP-compatible strategy that alternates local updates with a reconciliation step combining gradient AllReduce and SlowMo parameter averaging. Experiments are conducted on Grid'5000 across intra-site clusters and a cross-site WAN deployment spanning Nancy and Sophia with 16.6 ms RTT. In the main fixed learning-rate setting of 0.1 on ResNet-50/CIFAR-100, CPDP achieves the highest peak test accuracy among the evaluated configurations. At K=2, CPDP improves over DDP by 2.28 percentage points but incurs additional wall-clock time. At K=4, CPDP improves over DDP by 2.44 percentage points while reducing average wall-clock time by 13.8%. Direct profiling shows that exposed synchronization time at K=4 is roughly half that of DDP, explaining the improved WAN accuracy-time trade-off. Additional experiments on ViT-S/CIFAR-100 and ResNet-50/TinyImageNet show that CPDP remains competitive with DDP and generally improves over LocalSGD. Overall, the results show that synchronization frequency is a practical control parameter for distributed training under communication constraints.

Figures

Figures reproduced from arXiv: 2607.21224 by Claude Tadonki, Imane Ettifouri, Mostapha Zbakh.

Figure 1
Figure 1. Figure 1: visualizes these peak test accuracy results un￾der the fixed LR= 0.1 cross-site configuration. To quantify the effect at the conservative K=2 periodic setting, [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Intra-site strong scaling under linear learning-rate scaling on ResNet-50/CIFAR-100. Accuracy reports mean peak test accu￾racy across 3 seeds over the 100-epoch run. DDP maintains the highest accuracy at larger scales, while CPDP consistently outper￾forms LocalSGD. vative learning-rate setting brings all three methods to within 0.31 percentage points of each other, with CPDP achieving the highest mean accu… view at source ↗
Figure 3
Figure 3. Figure 3: Effect of a more conservative learning rate on 16-GPU ResNet-50/CIFAR-100 training. All methods use their main config￾uration (CPDP at β = 0.3); reducing the learning rate from LR= 0.4 to LR= 0.2 improves every method [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of synchronization period K on training accu￾racy and speedup relative to DDP (ResNet-50/CIFAR-100, 8 GPUs, LR= 0.2, seed 42). Increasing K reduces synchronization frequency and improves throughput but increases inter-worker drift; the best accuracy–time trade-off is environment-dependent. ing point in this intra-site setting. At K=8, CPDP re￾tains 76.09% at a 1.18× speedup. At K=2, where synchro￾ni… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 2 canonical work pages

  1. [1]

    S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Dama- nia, S. Chintala, PyTorch distributed: experiences on accelerating data parallel training , Proceedings of the VLDB Endowment 13 (12) (2020) 3005–3018. doi:10.14778/3415478.3415530. URL https://dl.acm.org/doi/10.14778/3415478. 3415530

    arXiv 2020

  2. [2]

    J. Liu, Z. Wu, D. Feng, M. Zhang, X. Wu, X. Yao, D. Yu, Y. Ma, F. Zhao, D. Dou, HeterPS: Dis- tributed deep learning with reinforcement learning based scheduling in heterogeneous environments , Future Generation Computer Systems 148 (2023) 106–117. doi:10.1016/j.future.2023.05.032. URL https://www.sciencedirect.com/science/ article/pii/S0167739X23002157

    work page doi:10.1016/j.future.2023.05.032 2023

  3. [3]

    S. Wang, S. Chen, Y. Shi, GPARS: Graph pre- dictive algorithm for efficient resource scheduling in heterogeneous GPU clusters , Future Gener- ation Computer Systems 152 (2024) 127–137. doi:10.1016/j.future.2023.10.022. URL https://www.sciencedirect.com/science/ article/pii/S0167739X23003953

    work page doi:10.1016/j.future.2023.10.022 2024

  4. [4]

    J. Liu, Z. Duan, X. Hu, J. Zhong, Y. Yin, Detrack- ing autoencoding conditional generative adversarial network: Improved generative adversarial network method for tabular missing value imputation, En- tropy 26 (5) (2024) 402. doi:10.3390/e26050402

    doi:10.3390/e26050402 2024

  5. [5]

    Z. Hou, B. Wang, J. Liu, Y. He, Y. Yao, Physics-inspired time-frequency feature extraction and lightweight neural network for power quality dis- turbance classification, Frontiers in Physics 13 (2025) 1616367. doi:10.3389/fphy.2025.1616367. 16

    arXiv 2025

  6. [6]

    J. Liu, Z. Hou, Establishment of Second-Hand Sail- boats price prediction model based on Random Forest and exploration of influencing factors, in: 2023 IEEE 3rd International Conference on Data Science and Computer Application (ICDSCA), 2023, pp. 1337–

    2023

  7. [7]

    E. Yu, D. Dong, X. Liao, Communication opti- mization algorithms for distributed deep learning systems: A survey , IEEE Transactions on Parallel and Distributed Systems 34 (12) (2023) 3294–3308. doi:10.1109/TPDS.2023.3323282. URL https://ieeexplore.ieee.org/abstract/ document/10275049

    arXiv 2023

  8. [8]

    Y. Wei, T. Hu, C. Liang, Y. Cui, Communication optimization for distributed training: Architecture, advances, and opportunities , IEEE Network 39 (3) (2025) 241–248. doi:10.1109/MNET.2024.3449276. URL https://ieeexplore.ieee.org/abstract/ document/10663574

    arXiv 2025

  9. [9]

    W. Li, X. Liu, Y. Li, Y. Jin, H. Tian, Z. Zhong, G. Liu, Y. Zhang, K. Chen, Understanding communication characteristics of distributed training, in: Proceedings of the 8th Asia-Pacific Workshop on Networking, AP- Net ’24, Association for Computing Machinery, 2024, pp. 1–8. doi:10.1145/3663408.3663409. URL https://dl.acm.org/doi/10.1145/3663408. 3663409

    arXiv 2024

  10. [10]

    Z. Ye, W. Gao, Q. Hu, P. Sun, X. Wang, Y. Luo, T. Zhang, Y. Wen, Deep learning workload scheduling in GPU datacenters: A survey , ACM Comput. Surv. 56 (6) (2024) 146:1–146:38. doi:10.1145/3638757. URL https://dl.acm.org/doi/10.1145/3638757

    doi:10.1145/3638757 2024

  11. [11]

    Hwang, K

    C. Hwang, K. Park, R. Shu, X. Qu, P. Cheng, Y. Xiong, ARK: GPU-driven code execution for dis- tributed deep learning , in: 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023, pp. 87–101. URL https://www.usenix.org/conference/ nsdi23/presentation/hwang

    2023

  12. [12]

    C. Chen, X. Li, Q. Zhu, J. Duan, P. Sun, X. Zhang, C. Yang, Centauri: Enabling efficient scheduling for communication-computation overlap in large model training via communication partitioning , in: Proceed- ings of the 29th ACM International Conference on Ar- chitectural Support for Programming Languages and Operating Systems, Volume 3, Vol. 3 of ASPLOS ’...

    2024

  13. [13]

    J. Cao, Y. Guan, K. Qian, J. Gao, W. Xiao, J. Dong, B. Fu, D. Cai, E. Zhai, Crux: GPU-efficient com- munication scheduling for deep learning training , in: Proceedings of the ACM SIGCOMM 2024 Confer- ence, ACM SIGCOMM ’24, Association for Com- puting Machinery, 2024, pp. 1–15. doi:10.1145/ 3651890.3672239. URL https://dl.acm.org/doi/10.1145/3651890. 3672239

    arXiv 2024

  14. [14]

    S. U. Stich, Local SGD converges fast and communi- cates little (2019). arXiv:1805.09767[math], doi: 10.48550/arXiv.1805.09767. URL http://arxiv.org/abs/1805.09767

    Pith/arXiv arXiv doi:10.48550/arxiv.1805.09767 2019

  15. [15]

    T. Lin, S. U. Stich, K. K. Patel, M. Jaggi, Don’t use large mini-batches, use local SGD (2020). arXiv:1808.07217[cs], doi:10.48550/ arXiv.1808.07217. URL http://arxiv.org/abs/1808.07217

    Pith/arXiv arXiv doi:10.48550/arxiv.1808.07217 2020

  16. [16]

    J. Wang, V. Tantia, N. Ballas, M. Rabbat, SlowMo: Improving communication-efficient distributed SGD with slow momentum, in: International Conference on Learning Representations (ICLR), 2020, pp. 1–27

    2020

  17. [17]

    Balouek, A

    D. Balouek, A. Carpen Amarie, G. Charrier, F. Desprez, E. Jeannot, E. Jeanvoine, A. Lèbre, D. Margery, N. Niclausse, L. Nussbaum, O. Richard, C. Pérez, F. Quesnel, C. Rohr, L. Sarzyniec, Adding virtualization capabilities to the Grid’5000 testbed, in: I. I. Ivanov, M. van Sinderen, F. Leymann, T. Shan (Eds.), Cloud Computing and Services Science, Vol. 367...

    doi:10.1007/978-3-319-04519-1 2013

  18. [18]

    Romero, J

    J. Romero, J. Yin, N. Laanait, B. Xie, M. T. Young, S. Treichler, V. Starchenko, A. Borisevich, A. Sergeev, M. Matheson, Accelerating collective communication in data parallel training across deep learning frameworks , in: 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), 2022, pp. 1027–1040. URL https://www.usenix.org/confer...

    2022

  19. [19]

    Rashidi, M

    S. Rashidi, M. Denton, S. Sridharan, S. Srinivasan, A. Suresh, J. Ni, T. Krishna, Enabling compute- communication overlap in distributed deep learning training platforms, in: 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), 2021, pp. 540–553. arXiv:2007.00156[cs], doi:10.1109/ISCA52012.2021.00049. URL https://ieeexplore.i...

    Pith/arXiv arXiv 2021

  20. [20]

    Zhang, S

    L. Zhang, S. Shi, X. Chu, W. Wang, B. Li, C. Liu, DeAR: Accelerating distributed deep learning with fine-grained all-reduce pipelining , in: 2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS), 2023, pp. 142–153. 17 doi:10.1109/ICDCS57875.2023.00054. URL https://ieeexplore.ieee.org/abstract/ document/10272398

    arXiv 2023

  21. [21]

    Q. Duan, Z. Wang, Y. Xu, S. Liu, J. Wu, Mercury: A simple transport layer scheduler to accelerate distributed DNN training , in: IEEE INFOCOM 2022 –IEEE Conference on Computer Communications, 2022, pp. 350–359. doi:10.1109/INFOCOM48880.2022.9796820. URL https://ieeexplore.ieee.org/abstract/ document/9796820

    arXiv 2022

  22. [22]

    Y. Gao, B. Hu, M. B. Mashhadi, A.-L. Jin, P. Xiao, C. Wu, US-byte: An efficient communication frame- work for scheduling unequal-sized tensor blocks in distributed deep learning , IEEE Transactions on Parallel and Distributed Systems 35 (1) (2024) 123–139. doi:10.1109/TPDS.2023.3331372. URL https://ieeexplore.ieee.org/abstract/ document/10314018

    arXiv 2024

  23. [23]

    Y. Gao, B. Hu, M. B. Mashhadi, W. Wang, R. Tafazolli, M. Debbah, PipeDAP: An effi- cient communication framework for scheduling decoupled all-reduce primitives in distributed DNN training , IEEE Transactions on Emerging Topics in Computing 13 (3) (2025) 1170–1184. doi:10.1109/TETC.2025.3573522. URL https://ieeexplore.ieee.org/abstract/ document/11021340

    arXiv 2025

  24. [24]

    Y. Gao, B. Hu, M. B. Mashhadi, W. Wang, R. Tafa- zolli, M. Debbah, A dynamic sliding window based tensor communication scheduling framework for distributed deep learning , IEEE Transactions on Network Science and Engineering 12 (2) (2025) 1080–1095. doi:10.1109/TNSE.2024.3523320. URL https://ieeexplore.ieee.org/document/ 10816583/

    arXiv 2025

  25. [25]

    Cheng, S

    S. Cheng, S. Lin, L. Diao, H. Wu, S. Wang, C. Si, Z. Liu, X. Zhao, J. Du, W. Lin, Y. You, Concerto: Automatic communication optimization and schedul- ing for large-scale deep learning , in: ASPLOS ’25: 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ACM, 2025, pp. 198–213. doi:10.1145/ 3669940.37...

    arXiv 2025

  26. [26]

    G. Xu, Z. Le, Y. Chen, Z. Lin, Z. Jin, Y. Miao, C. Li, AutoCCL: Automated collective communication tuning for accelerating distributed and parallel DNN training , in: 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), 2025, pp. 667–683. URL https://www.usenix.org/conference/ nsdi25/presentation/xu-guanbin

    2025

  27. [27]

    T. Liu, C. Hei, F. Li, C. Gao, J. Cao, T. Wang, E. Zhai, X. Wang, ResCCL: Resource-efficient scheduling for collective communication , in: Proceed- ings of the ACM SIGCOMM 2025 Conference, SIG- COMM ’25, Association for Computing Machinery, 2025, pp. 55–70. doi:10.1145/3718958.3750514. URL https://dl.acm.org/doi/10.1145/3718958. 3750514

    arXiv 2025

  28. [28]

    H. B. McMahan, E. Moore, D. Ramage, S. Hamp- son, B. A. y. Arcas, Communication-efficient learn- ing of deep networks from decentralized data (2023). arXiv:1602.05629[cs], doi:10.48550/ arXiv.1602.05629. URL http://arxiv.org/abs/1602.05629

    Pith/arXiv arXiv doi:10.48550/arxiv.1602.05629 2023

  29. [29]

    Hsieh, A

    K. Hsieh, A. Harlap, N. Vijaykumar, D. Konomis, G. R. Ganger, P. B. Gibbons, O. Mutlu, Gaia: Geo-distributed machine learning approaching lan speeds, in: NSDI, 2017, pp. 629––647. URL https://www.usenix.org/conference/ nsdi17/technical-sessions/presentation/ hsieh

    2017

  30. [30]

    Douillard, Q

    A. Douillard, Q. Feng, A. A. Rusu, R. Chha- paria, Y. Donchev, A. Kuncoro, M. Ranzato, A. Szlam, J. Shen, DiLoCo: Distributed low- communication training of language models , arXiv preprint arXiv:2311.08105 (2023). doi:10.48550/ arXiv.2311.08105. URL https://arxiv.org/abs/2311.08105

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.08105 2023

  31. [31]

    M. G. Venkata, V. Petrov, S. Lebedev, D. Bu- reddy, F. Aderholdt, J. Ladd, G. Bloch, M. Dub- man, G. Shainer, Unified collective communi- cation (UCC): An unified library for CPU, GPU, and DPU collectives , in: 2024 IEEE Symposium on High-Performance Interconnects (HOTI), 2024, pp. 37–46, ISSN: 2332-5569. doi:10.1109/HOTI63208.2024.00018. URL https://ieee...

    arXiv 2024

  32. [32]

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition (2015). arXiv:1512. 03385[cs], doi:10.48550/arXiv.1512.03385. URL http://arxiv.org/abs/1512.03385

    Pith/arXiv arXiv doi:10.48550/arxiv.1512.03385 2015

  33. [33]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszko- reit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale (2021). arXiv:2010.11929[cs], doi:10.48550/ arXiv.2010.11929. URL http://arxiv.org/abs/2010.11929 18

    Pith/arXiv arXiv doi:10.48550/arxiv.2010.11929 2021

  34. [34]

    Vogels, S

    T. Vogels, S. P. Karimireddy, M. Jaggi, Pow- ersgd: Practical low-rank gradient compression for distributed optimization, in: Advances in Neural Information Processing Systems, 2019, pp. 14236– 14245

    2019

  35. [35]

    L. Tao, X. Tong, C. W. Tan, Learning to optimize by differentiable programming (2026). arXiv:2601. 16510, doi:10.48550/arXiv.2601.16510. URL https://arxiv.org/abs/2601.16510

    Pith/arXiv arXiv doi:10.48550/arxiv.2601.16510 2026

  36. [36]

    Blondel, V

    M. Blondel, V. Roulet, The elements of differen- tiable programming (2025). arXiv:2403.14606, doi: 10.48550/arXiv.2403.14606. URL https://arxiv.org/abs/2403.14606 Appendix A. Additional Experimental Results This appendix provides additional experimental results that complement the analysis presented in Section 6. The figures included here illustrate conve...

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.14606 2025

  37. [191]

    URL https://dl.acm.org/doi/10.1145/3620666

    doi:10.1145/3620666.3651379. URL https://dl.acm.org/doi/10.1145/3620666. 3651379

    arXiv

  38. [1342]

    doi:10.1109/ICDSCA59871.2023.10393007

    arXiv 2023