Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Cross-region Model Training with Communication-Computation Overlapping and Delay Compensation

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CoCoDC cuts cross-region LLM training to target perplexity in 21.0% fewer steps than Streaming DiLoCo, with a lower final perplexity.

desk verdict Plausible framework and a new combination of ideas, but the headline 21% step reduction is confounded by twice the synchronization frequency and the derivation has a sign error. read the letter →

arxiv 2504.17672 v1 pith:B7TVETSF submitted 2025-04-24 cs.DC

Ying Zhu , Yang Xu , Hongli Xu , Yunming Liao , Zhiwei Yao , Liusheng Huang This is my paper · ORCID
classification cs.DC
keywords distributedtrainingcross-regionLLMcommunication-computationoverlapdelaycompensationadaptivetransmissionstalenessTaylorexpansionDiLoCo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the two problems introduced by overlapping communication with computation in cross-region language-model training—stale global updates and partial model synchronization—can be explicitly corrected, and that doing so makes distributed training faster and more stable. It proposes CoCoDC, which adds a Taylor-expansion delay-compensation step that estimates what the global update would have become during the overlap delay, and an adaptive transmission scheduler that syncs fast-changing model fragments more often. In a simulated four-worker cross-region setup on a 150M-parameter LLaMA-style transformer trained on C4, CoCoDC reaches perplexity 20.0 in 10,292 steps, 21.0% fewer than Streaming DiLoCo and 4.9% fewer than DiLoCo, with a slightly lower final perplexity. If correct, this shows that the wall-clock cost of cross-region training can be lowered without sacrificing convergence quality.

What carries the argument

The carrying object is the delay-compensated update in Eq. (8): after receiving the stale global state $\theta^g_{p,t_p}$, the worker computes its observed change rate $g = (\theta^m_{p,t_p}-\theta^m_{p,t_l})/\tau$, corrects it to $g_{\mathrm{corr}} = g + \lambda (g \odot g) \odot (\Delta\theta/H)$ using the per-step divergence $\Delta\theta/H$ from the global state, and applies $\theta^m_{p,t_l} \leftarrow \theta^g_{p,t_p} + g_{\mathrm{corr}}\,\tau$. The $\odot$ term is a Fisher-information-motivated stand-in for the Hessian that would otherwise be intractable. The second mechanism is the selection rule $R_p = \|\Delta\theta^g_p\|_2 / I_p$: the next fragment to synchronize is the one with the largest average change rate since its last sync, with a hard rule that any fragment unsynced for $H$ steps is chosen first. Together they convert a stale, partially synchronized update into an approximation of the current ideal global state while using idle bandwidth for the fragments that need it most.

What would settle it

Run an experiment that matches synchronization counts: give Streaming DiLoCo the same number of fragment synchronizations per $H$ that CoCoDC uses (8 per 100 in the reported configuration) and give CoCoDC a round-robin schedule with that same count, then compare steps to reach perplexity 20. If the 21.0% gap disappears or shrinks drastically, the gain comes from extra communication, not from delay compensation and adaptive selection.

Watch

Extended reading notes

Core claim

The paper's central claim is that the staleness and inconsistency introduced by overlapping communication with computation can be corrected well enough that low-communication cross-region training converges at least as well as blocking synchronization while using less wall-clock time. CoCoDC's delay compensation estimates the ideal global update at the current local step by a first-order Taylor expansion around the received global state, with the intractable Hessian replaced by a scaled outer product of local parameter-change rates. Its adaptive transmission then synchronizes the fragment with the largest averaged global pseudo-gradient norm first, subject to a maximum staleness of $H$ steps. In the reported experiments this yields faster convergence to perplexity 20 and a better final validation perplexity than both DiLoCo and Streaming DiLoCo, which the paper reads as evidence that explicit staleness handling restores the quality lost by overlap.

Load-bearing premise

The central comparison credits the speedup to CoCoDC's two mechanisms even though the configuration synchronizes fragments roughly twice as often as Streaming DiLoCo, and the paper provides no ablation that isolates the mechanisms at a matched synchronization frequency.

Editorial extensions

If this is right

  • In the simulated 4-worker, 150M-parameter setting, CoCoDC reaches perplexity 20.0 in 10,292 steps, 21.0% fewer than Streaming DiLoCo (13,021) and 4.9% fewer than DiLoCo (10,821).
  • CoCoDC reaches a lower final validation perplexity (18.0357) than both DiLoCo (18.1608) and Streaming DiLoCo (18.6457), indicating the correction does not trade final quality for speed.
  • Because communication and computation overlap, CoCoDC avoids the blocking all-reduce wait of DiLoCo, so wall-clock time per step is lower even when the number of steps is similar.
  • The paper expects the benefits to grow under more aggressive cross-region conditions, where higher latency forces larger overlap depth and longer local training intervals and therefore stronger staleness and inconsistency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The head-to-head comparison is not mechanism-matched: with $\gamma=0.4$, Eq. (9) gives 8 fragment synchronizations per $H=100$ steps for CoCoDC while Streaming DiLoCo performs $K=4$, so the 21.0% step reduction could partly reflect a higher update frequency rather than the compensation and adaptive selection themselves.
  • The $\lambda$-scaled term $\lambda (g \odot g) \odot (\Delta\theta/H)$ is structurally a diagonal natural-gradient correction, so a natural extension is to connect $\lambda$ to the outer optimizer's momentum and learning rate and to test whether a time-varying $\lambda$ helps as the overlap depth grows.
  • Because fragment selection uses globally averaged pseudo-gradients, the no-coordination property assumes workers observe the same $\Delta\theta^g_p$ history; under heterogeneous data or stragglers, local $R_p$ estimates can diverge, weakening the deterministic selection argument.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CoCoDC, a distributed training framework for cross-region LLM training that extends Streaming DiLoCo with two mechanisms: a Taylor-expansion-based delay compensation step (Section III.A, Eqs. (4)-(8)) and an adaptive fragment transmission policy (Section III.B, Eqs. (9)-(12)). The experimental section simulates four datacenter workers on a single 4-GPU server using a 150M-parameter LLaMA-style model trained on the C4 dataset, comparing against DiLoCo and Streaming DiLoCo. The paper reports that CoCoDC reaches validation perplexity 20 in 10,292 steps versus 13,021 for Streaming DiLoCo (a 21.0% reduction) and 10,821 for DiLoCo, and achieves the lowest final perplexity (18.0357).

Significance. If the mechanisms worked as described, the paper would address a real problem: Streaming DiLoCo's communication-computation overlap introduces staleness and partial-synchronization inconsistency, and a low-overhead correction plus smarter fragment scheduling could improve convergence per step. The empirical validation uses an external benchmark (C4) rather than a self-referential or constructed target, which is a strength, and the paper identifies a genuine limitation of the Streaming DiLoCo schedule. However, the central contribution is currently compromised by a sign inconsistency in the delay-compensation derivation and by an experimental comparison that does not control synchronization frequency or measure wall-clock time. The free constants λ and γ are hand-set without sensitivity analysis. With corrected equations and matched ablations, the framework could be a useful contribution; in its current form, the paper's main claims are not fully supported.

major comments (4)
  1. [Section III.A, Eqs. (4)-(8)] The sign of the local change rate is internally inconsistent. Eq. (4) defines g_m^p,tp = (θ^m_p,tp − θ^m_p,tl)/τ. Since tl = tp + τ, this is the negative of the average local parameter change over the overlap interval, not the change rate in the direction of training. Eq. (8) then computes θ^m_p,tl ← θ^g_p,tp + g_corr_p,tp × τ, which moves the compensated state opposite to the local update direction. A Taylor extrapolation from tp to tl requires the positive rate (θ^m_p,tl − θ^m_p,tp)/τ, or equivalently subtracting the quantity defined in Eq. (4). Please correct the sign and verify that Algorithm 1 and the actual implementation follow the corrected formula; as written, the derivation does not support the compensation mechanism.
  2. [Section IV.A, Eq. (9), and Table I] The headline step-count comparison is confounded by synchronization frequency. Streaming DiLoCo with K=4 shards synchronizes each fragment once per H=100 steps, i.e., 4 fragment all-reduce rounds per H. CoCoDC is run with γ=0.4, which the paper states results in 8 synchronizations per H steps—twice the number of communication rounds and twice the pseudo-gradient volume. Since more frequent global updates can independently accelerate convergence, the 21.0% step reduction in Table I cannot be attributed specifically to delay compensation or adaptive selection. Please add a matched-frequency ablation, e.g., CoCoDC restricted to K synchronizations per H or Streaming DiLoCo allowed N=8 synchronizations per H, and report the results.
  3. [Section IV.A, Eqs. (7) and (9)] The reported results depend on hand-set hyperparameters λ=0.5 and γ=0.4, with no sensitivity analysis or ablations. λ scales the Hessian approximation in Eq. (7), so the compensation term is not parameter-free, and γ directly controls the number of synchronizations in Eq. (9). Without varying these values, or at least reporting a small grid, the observed improvements may reflect tuning to the chosen setting rather than the proposed mechanisms. Please provide sensitivity curves for λ and γ, and ideally multiple random seeds, since Figs. 1 and 2 show single-run curves without error bars.
  4. [Section IV.B and Contributions] The paper's third contribution states that CoCoDC significantly reduces wall-clock training time, but the evaluation reports only training steps and validation metrics. No wall-clock time, communication time, or bandwidth utilization measurements are given. This is especially important because CoCoDC performs more synchronization rounds than Streaming DiLoCo, so a step-count advantage does not automatically translate into a wall-clock advantage. Please add wall-clock comparisons and a communication-overhead breakdown to support the training-speed claim.
minor comments (5)
  1. [Algorithm 1, line 3] Line 3 of Algorithm 1 says the correction is applied to "global state θ^m_p,tp"; to match Eq. (8) and the algorithm input, this should be θ^g_p,tp.
  2. [Section III.A, Eq. (7)] The notation ⊙ is used both for the outer product of g with itself and then for the three-term product g ⊙ g ⊙ (Δθ/H). The dimensions are ambiguous; please define whether the Hessian action is a matrix-vector product and write it consistently, for example as λ g gᵀ (Δθ/H) or a diagonal approximation.
  3. [Section III.B] There is a typo in the text: "negelects" should be "neglects."
  4. [Section IV.A] The setup states overlap depth τ=5, but it does not explain how the simulated network latency/bandwidth maps to the measured values Tc and Ts used in Eq. (9). Reporting these values would make the stated N=8 synchronizations per H reproducible.
  5. [Section IV.A] The paper mentions possible non-IID data distributions across datacenters, but the experimental section does not describe how the C4 data is partitioned among workers. Please clarify whether the training data is shuffled globally or partitioned per worker.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CoCoDC's claims are empirical benchmarks against external baselines, not derivations equivalent to their inputs.

full rationale

The paper's central claims are empirical comparisons between CoCoDC, DiLoCo, and Streaming DiLoCo on a held-out C4 validation split (Table I, Figs. 1-2), so the headline 21.0% step reduction is measured, not derived from its own assumptions. The delay compensation formula (Eqs. 4-8) is presented as a stated heuristic: it uses a Taylor-expansion-inspired correction with a tunable strength lambda, and Algorithm 1 explicitly lists lambda as an input hyperparameter; the paper does not claim this formula is deduced from the target result. The adaptive transmission count (Eq. 9) is a scheduling estimate with hand-set gamma = 0.4, and Section IV.A discloses that this yields 8 synchronizations per H steps versus K = 4 for Streaming DiLoCo; this is a potential experimental confound but not a circular reduction, because the improvement claim is an empirical outcome rather than a logical consequence of the definitions. No load-bearing self-citation occurs: references to DiLoCo, Streaming DiLoCo, and asynchronous-SGD delay compensation are to independent external work, not to the present authors. The limitation passage in Section IV.B explicitly says the current experiments may not show substantial final-metric improvements and that stronger gains are expected only under more aggressive real-world conditions; this weighs as an honest limitation, not as a circular step. No circular step meeting the quoting requirement was found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two hand-tuned hyperparameters (lambda=0.5, gamma=0.4), a Taylor-expansion approximation that uses a worker's local change rate as a proxy for the ideal global change rate, and a rank-one Hessian approximation of unclear dimensionality. The WAN is simulated as a single overlap depth tau=5 without a network model. No new physical entities are introduced.

free parameters (3)
  • lambda (compensation strength) = 0.5
    Scales the rank-one Hessian approximation in Eq. 7; set to 0.5 without sensitivity analysis.
  • gamma (network utilization factor) = 0.4
    Controls target sync count N in Eq. 9; set to 0.4, yielding 8 synchronizations per H steps, twice Streaming DiLoCo's 4.
  • tau (overlap depth) = 5
    Simulated communication delay in steps; set to 5 for both CoCoDC and Streaming DiLoCo, not derived from a real network model.
assumptions (4)
  • ad hoc to paper The ideal global parameter evolution over the tau overlap steps can be approximated by a first-order Taylor expansion using the worker's local parameter change rate.
    Section III.A, Eqs. 4-5: the local change rate is biased by local data and the local model has diverged from the global state, yet it is used as a proxy for the ideal global change rate without justification.
  • ad hoc to paper The Hessian of the change rate can be approximated by lambda times g circled dot g, a rank-one outer product scaled by a fitted lambda.
    Section III.A, Eq. 7: justified only by a vague connection to the Fisher information matrix; the notation makes the dimensionality ambiguous and no derivation is given.
  • domain assumption The fragment importance metric R_p = ||Delta_theta^g_p||_2 / I_p reflects the benefit of more frequent synchronization.
    Section III.B, Eq. 11: assumes larger recent pseudo-gradient magnitude means more value from extra sync, which is plausible but untested.
  • domain assumption Ring all-reduce latency is uniform and captured by a single overlap depth tau for all fragments.
    Section II.A: the simulation sets tau=5 and treats it as the effective communication delay, ignoring bandwidth contention and network variability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-region Model Training with Communication-Computation Overlapping and Delay Compensation." pith.science (2026). https://pith.science/paper/B7TVETSF

@misc{pith2026250417672,
  author       = {Pith},
  title        = {Pith review of: Cross-region Model Training with Communication-Computation Overlapping and Delay Compensation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B7TVETSF}},
  note         = {Machine review of arXiv:2504.17672}
}
read the original abstract

Training large language models (LLMs) requires massive computational resources, often necessitating the aggregation of geographically distributed data centers (\ie, cross-region training). However, the high communication latency in wide-area networks severely degrades the efficiency of traditional distributed training. While methods like DiLoCo reduce communication frequency, they suffer from blocking synchronization. Streaming DiLoCo alleviates this issue via communication-computation overlapping but introduces update staleness and model inconsistency due to delayed global updates and partial synchronization. These factors impair convergence, especially when aggressive overlap is needed to mask high latency. We propose CoCoDC, a novel distributed training framework with communication-computation overlapping and delay compensation, to explicitly tackle these challenges. Within the CoCoDC framework, we specifically develop a novel Delay Compensation strategy based on Taylor expansion to effectively mitigate the staleness and an Adaptive Transmission strategy that dynamically schedules model fragment synchronization to optimize bandwidth usage and accelerate convergence. Extensive experiments highlight the superior performance of CoCoDC over both DiLoCo and Streaming DiLoCo regarding final accuracy and training speed. Specifically, CoCoDC reduces the training steps needed to reach a comparable perplexity by up to 21.0% compared to Streaming DiLoCo. Our work provides an effective solution for scalable and efficient cross-region LLM training.

Figures

Figures reproduced from arXiv: 2504.17672 by the authors.

Figure 1
Figure 1. Validation loss vs. training steps of different methods [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Validation perplexity vs. training steps of different methods [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 23 canonical work pages

  1. [20]

    Asynchronous Stochastic Gradient Descent with Delay Compensation,

    S. Zheng, Q. Meng, T. Wang, W. Chen, N. Yu, Z.-M. Ma, and T.-Y . Liu, “Asynchronous Stochastic Gradient Descent with Delay Compensation,” Feb. 2020

    work page 2020

  2. [22]

    Asynchronous Training Schemes in Distributed Learning with Time Delay,

    H. Wang, Z. Jiang, C. Liu, S. Sarkar, D. Jiang, and Y . M. Lee, “Asynchronous Training Schemes in Distributed Learning with Time Delay,” Transactions on Machine Learning Research , Nov. 2023

    work page 2023

  3. [1]

    Minaee, T

    S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao. Large Language Models: A Survey. [Online]. Available: http://arxiv.org/abs/2402.06196

    arXiv

  4. [2]

    Efficient Large Scale Language Modeling with Mixtures of Experts,

    M. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V . Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O’Horo, J. Wang, L. Zettlemoyer, M. Diab, Z. Kozareva, and V . Stoyanov, “Efficient Large Scale Language Modeling with Mixtures of Experts,” Oct. 2022

    work page 2022

  5. [3]

    Language Models Are Few-Shot Learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Am...

    work page 2020

  6. [4]

    Megatron-LM: Training Multi-Billion Parameter Language Mod- els Using Model Parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-LM: Training Multi-Billion Parameter Language Mod- els Using Model Parallelism,” Mar. 2020

    work page 2020

  7. [5]

    PaLM: Scaling Language Modeling with Pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y . Tay, N. Shazeer, V . Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghe- mawat, S. De...

    work page 2022

  8. [6]

    Multi-Datacenter Training: OpenAI’s Ambitious Plan to Beat Google’s Infrastructure,

    “Multi-Datacenter Training: OpenAI’s Ambitious Plan to Beat Google’s Infrastructure,” https://semianalysis.com/2024/09/04/multi-datacenter- training-openais/, Sep. 2024

    work page 2024

Show all 28 references
  1. [7]

    Communication-Efficient Learning of Deep Networks from Decentralized Data,

    H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” Jan. 2023

    2023

  2. [8]

    Cross-Silo Federated Learning across Divergent Domains with Iterative Parameter Alignment,

    M. Gorbett, H. Shirazi, and I. Ray, “Cross-Silo Federated Learning across Divergent Domains with Iterative Parameter Alignment,” Apr. 2024

    2024

  3. [9]

    Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey,

    F. Liang, Z. Zhang, H. Lu, V . C. M. Leung, Y . Guo, and X. Hu, “Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey,” Apr. 2024

    2024

  4. [10]

    Measuring the Effects of Data Parallelism on Neural Network Training,

    C. J. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl, “Measuring the Effects of Data Parallelism on Neural Network Training,” Jul. 2019

    2019

  5. [11]

    J. Chen, X. Pan, R. Monga, S. Bengio, and R. Jozefowicz. Revisiting Distributed Synchronous SGD. [Online]. Available: http: //arxiv.org/abs/1604.00981

    arXiv

  6. [12]

    Douillard, Q

    A. Douillard, Q. Feng, A. A. Rusu, R. Chhaparia, Y . Donchev, A. Kuncoro, M. Ranzato, A. Szlam, and J. Shen. DiLoCo: Distributed Low-Communication Training of Language Models. [Online]. Available: http://arxiv.org/abs/2311.08105

    arXiv

  7. [13]

    OpenDiLoCo: An Open- Source Framework for Globally Distributed Low-Communication Train- ing,

    S. Jaghouar, J. M. Ong, and J. Hagemann, “OpenDiLoCo: An Open- Source Framework for Globally Distributed Low-Communication Train- ing,” Jul. 2024

    2024

  8. [14]

    Douillard, Y

    A. Douillard, Y . Donchev, K. Rush, S. Kale, Z. Charles, Z. Garrett, G. Teston, D. Lacey, R. McIlroy, J. Shen, A. Ram ´e, A. Szlam, M. Ranzato, and P. Barham. Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch. [Online]. Available: http://arxiv.o...

    arXiv

  9. [15]

    PipeDream: Generalized Pipeline Parallelism for DNN Training,

    D. Narayanan, A. Harlap, A. Phanishayee, V . Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “PipeDream: Generalized Pipeline Parallelism for DNN Training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles . Huntsville Ontario Cana...

    2019

  10. [16]

    SAPipe: Staleness-Aware Pipeline for Data Parallel DNN Training,

    Y . Chen, C. Xie, M. Ma, J. Gu, Y . Peng, H. Lin, C. Wu, and Y . Zhu, “SAPipe: Staleness-Aware Pipeline for Data Parallel DNN Training,” Advances in Neural Information Processing Systems , vol. 35, pp. 17 981–17 993, Dec. 2022

    2022

  11. [17]

    Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM,” Aug. 2021

    2021

  12. [18]

    Stragglers are not disaster: A hybrid federated learning algorithm with delayed gradients,

    X. Li, Z. Qu, B. Tang, and Z. Lu, “Stragglers are not disaster: A hybrid federated learning algorithm with delayed gradients,” Feb. 2021

    2021

  13. [19]

    Structured second-and higher-order derivatives through univariate taylor series,

    C. Bischof, C. , G., and A. and Griewank, “Structured second-and higher-order derivatives through univariate taylor series,” Optimization Methods and Software , vol. 2, no. 3-4, pp. 211–232, Jan. 1993

    1993

  14. [21]

    DC-S3GD: Delay-Compensated Stale-Synchronous SGD for Large-Scale Decentralized Neural Network Training,

    A. Rigazzi, “DC-S3GD: Delay-Compensated Stale-Synchronous SGD for Large-Scale Decentralized Neural Network Training,” in 2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS). Denver, CO, USA: IEEE, Nov. 2019, pp. 62–68

    2019

  15. [23]

    The elements of statistical learning. vol. 1 Springer series in statistics,

    J. Friedman, T. Hastie, and R. Tibshirani, “The elements of statistical learning. vol. 1 Springer series in statistics,” New York, 2001

    2001

  16. [24]

    Natural Gradient Works Efficiently in Learning,

    S.-I. Amari, “Natural Gradient Works Efficiently in Learning,” Neural Computation, vol. 10, no. 2, pp. 251–276, 1998

    1998

  17. [25]

    Pytorch distributed: Experiences on accelerating data parallel training,

    S. Li, Y . Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, “Pytorch distributed: Experiences on accelerating data parallel training,” Jun. 2020

    2020

  18. [26]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” Feb. 2023

    2023

  19. [27]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Sep. 2023

    2023

  20. [28]

    A study of bfloat16 for deep learning training,

    D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. V ooturi, N. Jammalamadaka, J. Huang, H. Yuen, J. Yang, J. Park, A. Heinecke, E. Georganas, S. Srinivasan, A. Kundu, M. Smelyanskiy, B. Kaul, and P. Dubey, “A study of bfloat16 for deep learning tr...

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.