Pith. sign in

REVIEW 3 major objections 4 minor 54 references

Decentralized SGD that deliberately keeps workers slightly out of sync finds flatter minima and beats centralized SGD in test accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:11 UTC pith:F5JZATOE

load-bearing objection Strong empirical result showing adaptive consensus improves generalization, but the theory claiming Hessian-aligned disagreement is not established and needs major rework. the 3 major comments →

arxiv 2602.02899 v2 pith:F5JZATOE submitted 2026-02-02 cs.LG cs.DC

Decentralized SGD with Controlled Disagreement Finds Flatter Minima

Zesen Wang , Mikael Johansson This is my paper
classification cs.LG cs.DC
keywords decentralized SGDconsensus errorsimplicit regularizationflat minimaHessian alignmentgeneralizationsharpness-aware minimizationdistributed training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper challenges the long-held view that consensus errors in decentralized training are harmful and should be minimized. It introduces DSGD-AC, which intentionally preserves non-vanishing consensus errors by scaling the consensus step with a time-dependent factor. The authors argue that these errors are not random noise but align with the dominant Hessian subspace, acting as a structured perturbation that steers optimization toward flatter minima. They prove that this alignment creates a curvature-dependent penalty that implicitly regularizes sharp directions, and they show empirically that DSGD-AC consistently exceeds both standard DSGD and centralized SGD in test accuracy and solution flatness. If correct, this means decentralized training can outperform centralized training without extra gradient evaluations, turning a perceived weakness into a free regularizer.

Core claim

The paper claims that consensus errors in decentralized SGD are a useful implicit regularizer rather than a nuisance. By introducing an adaptive consensus scale γ(t) = (α(t)/α_max)^p, DSGD-AC maintains a controlled disagreement radius throughout training. A local stability analysis shows that high-curvature Hessian modes retain larger variance, so consensus errors concentrate along sharp directions; the average local loss decomposes into the central loss plus a Hessian-weighted disagreement penalty whose weights strictly increase with curvature. Under the assumption of Hessian-aligned minibatch noise, this yields an implicit spectral penalty that disproportionately penalizes large eigenvalue

What carries the argument

The central mechanism is the adaptive consensus factor γ(t) = (α(t)/α_max)^p, which weakens graph Laplacian damping as the learning rate shrinks, combined with an eigenmode analysis of the consensus-error recursion. This yields a stability condition α(t) < (2 + (λ_min(W)-1)γ(t))/λ_k(H) that lets low-curvature modes stabilize earlier while high-curvature modes retain variance, and a spectral penalty ω_t(λ) that is strictly increasing in the Hessian eigenvalue, producing the curvature tilt.

Load-bearing premise

The curvature-tilt conclusion rests on the assumption that minibatch SGD noise is approximately Hessian-aligned (Cov(g(x) − ∇F(x)) ≈ c_t L(x) H(x)) near the minimum, an assumption the paper cites from the literature but does not verify for the trained models; without that, the spectral penalty being increasing in curvature loses its grounding.

What would settle it

Compute the covariance of minibatch gradients on a trained WRN28-10 model and compare the top eigenvectors of that covariance with the top eigenvectors of the Hessian; if the alignment is weak, the claimed alignment of consensus errors with the dominant Hessian subspace is unsupported. Additionally, if DSGD-AC's test-accuracy advantage over centralized SGD disappears when the learning rate schedule is switched from cosine annealing to a constant schedule, the dependence on the late-training radius uptick would be suspect.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the curvature-tilt account is correct, consensus errors act as a free sharpness-aware regularizer: DSGD-AC improves generalization without the 2× gradient cost of explicit sharpness-aware methods.
  • Maintaining a controlled disagreement radius is necessary; p ≥ 2 keeps the radius non-vanishing as the step size decays, and p ≈ 3 with cosine schedules gives a mild late-training uptick that correlates with better test performance.
  • The benefit persists across worker counts (8–32), topologies (ring, exponential, complete), and models, and transfers to a decentralized Adam variant for machine translation.
  • DSGD-AC runs at 0.81–0.87× the wall-clock time of synchronous SGD, so the generalization gain does not require sacrificing the speed advantage of decentralized training.
  • The deployed model is the global average of workers; the curvature tilt arises from the disagreement around that average, not from any single worker's trajectory.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test would be to measure the gradient-noise covariance on the actually trained Wide ResNets and check whether its top eigenvectors match the Hessian's top eigenvectors; the curvature-tilt mechanism would be undercut if the dominant noise directions are unrelated to large-curvature directions.
  • The same adaptive-consensus principle could guide communication-compression design: compressors that spend more budget early in training, or that selectively preserve high-curvature disagreement, could strengthen the alignment and push generalization further.
  • If the effect is fundamentally spectral, DSGD-AC's gain should be most pronounced in overparameterized models with very ill-conditioned Hessians, and might be reproducible by any algorithm that injects curvature-aligned noise at a controlled radius, such as low-pass filtered SGD.
  • The paper's appendix observes that adaptive optimizers like Adam destroy the anisotropic noise structure that AC relies on; combining AC with optimizers that restore that structure could be a promising route for extending the benefits to adaptive methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes DSGD-AC, a decentralized SGD variant that scales the consensus-mixing term by γ(t) = (α(t)/α_max)^p, where α(t) is the scheduled learning rate. The method intentionally preserves non-vanishing consensus errors during training. The authors claim that these errors are not isotropic noise but systematically align with the dominant Hessian subspace, acting as structured perturbations that bias optimization toward flatter minima. The paper reports consistent test-accuracy and test-loss improvements over both standard DSGD and synchronous SGD across CIFAR-100 image-classification setups (WRN28-10/WRN16-8, 8/16/32 workers, ring/exponential/complete topologies), plus an ablation on CIFAR-10 and a Transformer/WMT14 experiment with a decentralized Adam variant. The theoretical part consists of Proposition 3.1 on the disagreement radius, Proposition 3.2 on stability and alignment of consensus-error modes, and an appendix analysis (Lemma A.1, Proposition A.2, Corollary A.3) that derives a Hessian-weighted 'curvature tilt' under a Hessian-aligned SGD-noise assumption.

Significance. If the empirical claim holds, the paper makes a valuable practical point: a simple per-step scaling of the consensus term can turn decentralized training from an approximation of centralized SGD into a method that generalizes better than centralized SGD, without extra gradient evaluations. The experimental design is strong: three seeds, eighteen configurations, consistent accuracy gains, lower top-1 Hessian eigenvalues, careful BatchNorm calibration, and runtime measurements. The ablation over p and E_start is informative. However, the theoretical mechanism — the central advertised contribution — is not currently established. Proposition 3.1 contains an internal asymptotic inconsistency, Proposition 3.2 does not prove the claimed variance ordering, and Corollary A.3 depends on an unverified Hessian-aligned noise assumption. The empirical results are credible and useful, but the manuscript's claim to have proven that consensus errors are structured rather than random is not supported by the current analysis.

major comments (3)
  1. [Prop. 3.1 and Appendix A.1 (Eq. (14) and following lower bound)] The stated Θ(1) behavior for p=2 is contradicted by the proof's own lower bound. Appendix A.1 derives E||z_i||² ≥ (||μ_i||²/(λ_i²g_0²))(α)^{2-2p} + (σ_i²/(2λ_i g_0))(α)^{2-p}. For p=2 the first term is O(α^{-2}) unless μ_i=0, and for p=3 both terms diverge as α→0. Thus the quasi-stationary radius diverges rather than remaining Θ(1) or merely 'non-vanishing'. The fixed-point m_i = -(α/γ)μ_i/λ_i used in the proof also blows up for p≥2. This undermines the 'controlled disagreement' claim and the motivation for choosing p=3. The authors need either a finite-horizon analysis, an additional vanishing condition on μ, or a corrected statement.
  2. [Prop. 3.2 and Appendix A.2 (Eqs. (21), (25), and condition (9))] The proof establishes only the stability threshold (9); the assertion that high-curvature modes 'retain higher variance under the same injected noise' is not derived. For the per-mode AR(1) recursion, the stationary variance is S_{k,j}=α²q_{k,j}/(1-(1-γλ_j(L)-αλ_k(H))²). For fixed graph mode j, this quantity is not monotone in λ_k(H): modes with very small λ_k have large variance because the denominator is O(γλ_j), modes near the stability boundary also have large variance, and intermediate modes can have smaller variance. Stability alone therefore does not imply alignment with the dominant Hessian directions. To obtain the claimed concentration, one must show that q_{k,j} increases with λ_k(H), which is precisely the assumption imported later in Eq. (28), not a consequence of Proposition 3.2.
  3. [Cor. A.3, Eq. (28)] The monotone spectral penalty ω_t(λ) and the 'curvature tilt' rest entirely on the assumption Cov(g_i(x)−∇f_i(x))≈c_tL(x)H(x). The cited literature supports this in some settings, but the paper does not verify it for the trained WRN models, nor does it report any diagnostic such as the cosine similarity between the top eigenspaces of the gradient-noise covariance and the Hessian. If the noise covariance is closer to isotropic, q_{k,j} is roughly constant and the weight ω_t(λ)/λ is not increasing; the claimed mechanism fails. Since the abstract and Section 1 present this alignment as a proven result, Eq. (28) is load-bearing and needs direct validation, or the theoretical claims must be weakened to a conditional statement.
minor comments (4)
  1. [Algorithm 1 / Prop. 3.1] Algorithm 1 defines γ(t)=(α(t)/α_max)^p, while the proof uses γ(t)=g_0(α(t))^p. State explicitly that g_0=α_max^{-p} to avoid confusion.
  2. [Figure 3] The random-direction comparison would be more convincing if the perturbation norm were specified. Please report the scale used for the random directions and how it is matched to the consensus-error radius.
  3. [Table 5 / Appendix A.6] The DAdam-AC experiment is interesting, but the main text should note that the theoretical analysis does not cover adaptive optimizers (as acknowledged in A.6). Also, the DAdam baseline uses different β1, β2, and α0 than the Adam baseline; clarify whether these were independently tuned for the decentralized setting.
  4. [References and numbering] The reference 'Adam, K. D. B. J. et al.' is not the standard citation for Adam; use Kingma and Ba. Additionally, Eq. (28) appears both as the Hessian-aligned noise assumption in Corollary A.3 and as the loss definition in Appendix A.5.3; renumber to avoid ambiguity.

Circularity Check

0 steps flagged

No significant circularity; the empirical benchmark results stand independently, and the theoretical alignment claim is conditional on an external Hessian-aligned noise assumption rather than on a fitted parameter or self-citation.

full rationale

The paper's main theoretical chain is: DSGD-AC update (Eq. 5) -> disagreement recursion (Eq. 8) -> radius scaling (Prop. 3.1) -> per-mode stability (Prop. 3.2) -> loss-envelope/curvature-tilt analysis (Appendix A.3). Each step is an internal derivation from the algorithm's update rule and Taylor expansion, not a fitted parameter renamed as a prediction. The algorithm hyperparameters p and E_start are explicitly tuned from sensitivity tables and then fixed for the final benchmarks; the paper does not present these choices as theory-derived predictions, so no fitted-input-called-prediction circularity arises. The empirical claim that DSGD-AC beats DSGD and centralized SGD is supported by direct, repeatable benchmark comparisons (CIFAR-10/100, WMT14), i.e., it is self-contained against external baselines. The main weakness is in the alignment claim: Proposition 3.2's concluding sentence (high-curvature modes 'retain higher variance under the same injected noise') does not follow from the stationary variance formula in its own proof, and the only route to the 'systematic alignment' conclusion is Corollary A.3's assumption (28) that mini-batch SGD noise is Hessian-aligned, cited from Ziyin et al. (2022), Wu et al. (2022), and Mori et al. (2022). This is a load-bearing unverified assumption and a proof gap, but it is not circularity: Eq. (28) is an external, independently published claim, not a self-citation or a parameter fitted in this paper, and Corollary A.3 states the condition explicitly as an assumption. No uniqueness theorem, self-citation chain, or ansatz-smuggling step is load-bearing. Therefore, under the circularity rubric, the appropriate finding is no significant circularity (score 0).

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central claims rest on two tuned hyperparameters (p, Estart) and on several unverified modeling assumptions: Hessian-aligned SGD noise, local quadratic approximation, quasi-stationarity, and a fixed mixing matrix. No new physical or algorithmic entities are postulated.

free parameters (2)
  • p (consensus exponent) = 3 (best p ∈ [2,4] in sensitivity analysis)
    Tuned on CIFAR-10 WRN28-10 (Table 2); the theory's 'p≥2 necessary' claim is rationalized after tuning, and p=3 is used in all main experiments.
  • Estart (activation epoch of AC) = 10 (8 workers), 100 (16), 150 (32)
    Tuned per worker count (Table 3 and Appendix A.4.5); performance varies strongly with this choice, so it is a fitted hyperparameter, not a predicted constant.
axioms (5)
  • domain assumption SGD minibatch gradient noise is Hessian-aligned: Cov(g_i − ∇f_i) ≈ c_t L(x) H(x)
    Invoked in Corollary A.3 (Eq. 28) to make the disagreement envelope increase with Hessian eigenvalues; cited from Ziyin et al. 2022, Wu et al. 2022, Mori et al. 2022, but not verified for the WRN models trained here.
  • domain assumption Local quadratic approximation: ∇f_i(x_i) ≈ H(x_i − x*) near a strongly convex minimizer
    Used in Propositions 3.2 and A.2; not justified for non-convex deep networks during training, where the iterates may be far from any local minimum.
  • domain assumption Quasi-stationary regime with bounded moments and independent innovations
    Used in the proof of Proposition 3.1 (Appendix A.1); not verified for cosine LR schedules with warmup.
  • domain assumption Communication matrix W is fixed, symmetric, doubly stochastic with eigenvalues in (−1,1] and connected graph
    Assumed in Propositions 3.1–3.2 and A.2; the main experiments use an alternating one-peer ring (time-varying W), so the analysis does not literally cover the deployed topology.
  • domain assumption Top-1 Hessian eigenvalue is a valid flatness/generalization surrogate
    Used to claim DSGD-AC finds flatter minima; the paper itself notes that the best sharpness metric is an open question.

pith-pipeline@v1.3.0-alltime-deepseek · 25061 in / 17589 out tokens · 174732 ms · 2026-08-03T05:11:18.024786+00:00 · methodology

0 comments
read the original abstract

Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization. This work challenges this view by introducing decentralized SGD with Adaptive Consensus (DSGD-AC), which uses a time-dependent scaling mechanism to maintain consensus errors throughout the training. We show that adaptive consensus changes the stationary variance of disagreement modes by balancing two effects: it preserves consensus-error magnitude through weaker graph damping while still allowing curvature-dependent damping to shape the disagreement directions. This balance can produce a stronger Hessian-weighted loss-envelope penalty around the deployed model, even when normalized Hessian alignment is weaker than in standard DSGD. Empirical results on image classification show that DSGD-AC reaches flatter solutions and higher test accuracy than standard DSGD and even centralized SGD. Together, these results support consensus errors as a useful implicit regularizer and open a new perspective on the design of decentralized learning algorithms.

Figures

Figures reproduced from arXiv: 2602.02899 by Mikael Johansson, Zesen Wang.

Figure 1
Figure 1. Figure 1: Decentralized training of WRN28-10 on CIFAR-10 with 8 workers and the one-peer ring topology. p = 3 for DSGD-AC. 3.3. Controlled consensus errors in DSGD-AC The motivation of DSGD-AC is to maintain non-diminishing consensus errors. Therefore, we multiply the weight of the consensus regularizer in Eq. (4) by an adaptive γ, which directly leads to the DSGD-AC algorithm. The per-step surrogate function of DSG… view at source ↗
Figure 2
Figure 2. Figure 2: Losses on the whole training dataset at local workers and global average. The losses are evaluated every 10 epochs [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training loss at epoch 180 along: (1) worker i: lines connecting global average and worker i, (2) gradient: the line that aligns with the full-batch gradient at the global average and crosses the global average, and (3) random: 500 lines that cross the global average and follow random directions generated as in (Bisla et al., 2022). The x-axis means the directional magnitude of the perturbation along these… view at source ↗
Figure 4
Figure 4. Figure 4: Average norm of consensus errors over epochs with varying p in the WRN28-10 on CIFAR-10 experiments with 8 workers and the one-peer ring topology. Estart = 10, which is after the learning rate warm-up phase. In setups with more workers or weaker connectivity in the communication topology, decentralization itself already in￾duces substantial consensus error in the early training phase. As shown in [PITH_FU… view at source ↗
Figure 5
Figure 5. Figure 5: Transformer (big) on WMT14 English-to-German. Left: Losses on training set. Right: BLEU scores on the test set. Algorithm BLEU score ↑ Test loss ↓ Train loss ↓ Adam 28.68 ± 0.07 2.9290 ± 0.0026 2.8310 ± 0.0019 DAdam 28.38 ± 0.22 2.9258 ± 0.0018 2.8195 ± 0.0008 DAdam-AC 28.89 ± 0.17 2.9205 ± 0.0020 2.8456 ± 0.0016 [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: WRN28-10 on CIFAR-100 with 8 workers. Left: Test accuracy on test set. For decentralized training, the accuracy is evaluated on the global average model. Right: Training and test losses [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: WRN28-10 on CIFAR-100 with 16 workers. Left: Test accuracy on test set. For decentralized training, the accuracy is evaluated on the global average model. Right: Training and test losses [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: WRN28-10 on CIFAR-100 with 32 workers. Left: Test accuracy on test set. For decentralized training, the accuracy is evaluated on the global average model. Right: Training and test losses. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: WRN16-8 on CIFAR-100 with 8 workers. Left: Test accuracy on test set. For decentralized training, the accuracy is evaluated on the global average model. Right: Training and test losses [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: WRN16-8 on CIFAR-100 with 16 workers. Left: Test accuracy on test set. For decentralized training, the accuracy is evaluated on the global average model. Right: Training and test losses [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: WRN16-8 on CIFAR-100 with 32 workers. Left: Test accuracy on test set. For decentralized training, the accuracy is evaluated on the global average model. Right: Training and test losses. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: DSGD(-AC) on WRN28-10 on CIFAR-10 with varying p. Left: Test accuracy on test set. For decentralized training, the accuracy is evaluated on the global average model. Right: Training and test losses. p Test Accuracy (%) ↑ Train Loss ↓ Test Loss ↓ 0 96.07 ± 0.13 0.002 ± 0.000 0.176 ± 0.005 1 96.26 ± 0.14 0.002 ± 0.000 0.159 ± 0.003 2 96.58 ± 0.18 0.003 ± 0.000 0.141 ± 0.006 3 96.77 ± 0.11 0.012 ± 0.000 0.12… view at source ↗
Figure 13
Figure 13. Figure 13: DSGD-AC on WRN28-10 on CIFAR-10 with varying Estart with 16 workers and one-peer ring topology. Left: Test accuracy on test set. For decentralized training, the accuracy is evaluated on the global average model. Right: Training and test losses. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: WRN28-10 on CIFAR100. The global batch size 128. Figures from top to bottom correspond to complete, exponential graph, and one-peer ring, respectively. The best test accuracy and the best test loss are highlighted by red marks. 0 25 50 75 100 125 150 175 200 Activate AC from Epoch 94 95 96 Test Accuracy (%) DSGD-AC Acc DSGD-AC Loss SGD Acc SGD Loss 0.12 0.14 0.16 0.18 0.20 Test Loss 0 25 50 75 100 125 150… view at source ↗
Figure 15
Figure 15. Figure 15: WRN28-10 on CIFAR10. The global batch size 128. Figures from top to bottom correspond to complete, exponential graph, and one-peer ring, respectively. The best test accuracy and the best test loss are highlighted by red marks. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: WRN16-8 on CIFAR10. The global batch size 128. Figures from top to bottom correspond to complete, exponential graph, and one-peer ring, respectively. The best test accuracy and the best test loss are highlighted by red marks. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 20 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Tensor F low: A system for large-scale machine learning

    Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. Tensor F low: A system for large-scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp.\ 265--283, 2016

    2016

  3. [3]

    Adam, K. D. B. J. et al. A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 1412 0 (6), 2014

    Pith/arXiv arXiv 2014

  4. [4]

    Alghunaim, S. A. and Yuan, K. A unified and refined convergence analysis for non-convex decentralized learning. IEEE Transactions on Signal Processing, 70: 0 3264--3279, 2022

    2022

  5. [5]

    Stochastic gradient push for distributed deep learning

    Assran, M., Loizou, N., Ballas, N., and Rabbat, M. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning, pp.\ 344--353. PMLR, 2019

    2019

  6. [6]

    Improved stability and generalization guarantees of the decentralized SGD algorithm

    Bellet, A., Tommasi, M., Scaman, K., Neglia, G., et al. Improved stability and generalization guarantees of the decentralized SGD algorithm. In Forty-first International Conference on Machine Learning, 2023

    2023

  7. [7]

    and Ventura, E

    Benedetti, M. and Ventura, E. Training neural networks with structured noise improves classification and generalization. Journal of Physics A: Mathematical and Theoretical, 57 0 (41): 0 415001, 2024

    2024

  8. [8]

    Low-pass filtering SGD for recovering flat optima in the deep learning optimization landscape

    Bisla, D., Wang, J., and Choromanska, A. Low-pass filtering SGD for recovering flat optima in the deep learning optimization landscape. In International Conference on Artificial Intelligence and Statistics, pp.\ 8299--8339. PMLR, 2022

    2022

  9. [9]

    Findings of the 2014 workshop on statistical machine translation

    Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pp.\ 12--58, 2014

    2014

  10. [10]

    The road less scheduled

    Defazio, A., Yang, X., Khaled, A., Mishchenko, K., Mehta, H., and Cutkosky, A. The road less scheduled. Advances in Neural Information Processing Systems, 37: 0 9974--10007, 2024

    2024

  11. [11]

    Decentralized learning made easy with decentralizepy

    Dhasade, A., Kermarrec, A.-M., Pires, R., Sharma, R., and Vujasinovic, M. Decentralized learning made easy with decentralizepy. In Proceedings of the 3rd Workshop on Machine Learning and Systems, pp.\ 34--41, 2023

    2023

  12. [12]

    Sharpness-aware minimization for efficiently improving generalization

    Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020

    Pith/arXiv arXiv 2010

  13. [13]

    and Seferoglu, H

    Gholami, P. and Seferoglu, H. Digest: Fast and communication efficient decentralized learning with local updates. IEEE Transactions on Machine Learning in Communications and Networking, 2: 0 1456--1474, 2024

    2024

  14. [14]

    S., Xiong, C., and Socher, R

    Gotmare, A., Keskar, N. S., Xiong, C., and Socher, R. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprint arXiv:1810.13243, 2018

    Pith/arXiv arXiv 2018

  15. [15]

    A., and Dyer, E

    Gur-Ari, G., Roberts, D. A., and Dyer, E. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018

    Pith/arXiv arXiv 2018

  16. [16]

    and Pu, S

    Huang, K. and Pu, S. Cedas: A compressed decentralized stochastic gradient method with improved convergence. IEEE Transactions on Automatic Control, 2024

    2024

  17. [17]

    Imfeld, M., Graldi, J., Giordano, M., Hofmann, T., Anagnostidis, S., and Singh, S. P. Transformer fusion with optimal transport. arXiv preprint arXiv:2310.05719, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Fantastic generalization measures and where to find them

    Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178, 2019

    Pith/arXiv arXiv 1912

  19. [19]

    Kalra, D. S. and Barkeshli, M. Why warmup the learning rate? underlying mechanisms and improvements. Advances in Neural Information Processing Systems, 37: 0 111760--111801, 2024

    2024

  20. [20]

    S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P

    Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016

    Pith/arXiv arXiv 2016

  21. [21]

    Decentralized stochastic optimization and gossip algorithms with compressed communication

    Koloskova, A., Stich, S., and Jaggi, M. Decentralized stochastic optimization and gossip algorithms with compressed communication. In International conference on machine learning, pp.\ 3478--3487. PMLR, 2019

    2019

  22. [22]

    Consensus control for decentralized deep learning

    Kong, L., Lin, T., Koloskova, A., Jaggi, M., and Stich, S. Consensus control for decentralized deep learning. In International Conference on Machine Learning, pp.\ 5686--5696. PMLR, 2021

    2021

  23. [23]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  24. [24]

    Kwon, J., Kim, J., Park, H., and Choi, I. K. ASAM : Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International conference on machine learning, pp.\ 5905--5914. PMLR, 2021

    2021

  25. [25]

    An iteration method for the solution of the eigenvalue problem of linear differential and integral operators

    Lanczos, C. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of research of the National Bureau of Standards, 45 0 (4): 0 255--282, 1950

    1950

  26. [26]

    Pytorch distributed: Experiences on accelerating data parallel training

    Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., et al. Pytorch distributed: Experiences on accelerating data parallel training . arXiv preprint arXiv:2006.15704, 2020

    Pith/arXiv arXiv 2006

  27. [27]

    Revisiting random weight perturbation for efficiently improving generalization

    Li, T., Tao, Q., Yan, W., Lei, Z., Wu, Y., Fang, K., He, M., and Huang, X. Revisiting random weight perturbation for efficiently improving generalization. arXiv preprint arXiv:2404.00357, 2024 a

    Pith/arXiv arXiv 2024

  28. [28]

    Friendly sharpness-aware minimization

    Li, T., Zhou, P., He, Z., Cheng, X., and Huang, X. Friendly sharpness-aware minimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 5631--5640, 2024 b

    2024

  29. [29]

    Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent

    Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., and Liu, J. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017

    2017

  30. [30]

    P., Stich, S

    Lin, T., Karimireddy, S. P., Stich, S. U., and Jaggi, M. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data . arXiv preprint arXiv:2102.04761, 2021

    Pith/arXiv arXiv 2021

  31. [31]

    Random sharpness-aware minimization

    Liu, Y., Mai, S., Cheng, M., Chen, X., Hsieh, C.-J., and You, Y. Random sharpness-aware minimization. Advances in neural information processing systems, 35: 0 24543--24556, 2022

    2022

  32. [32]

    and Hutter, F

    Loshchilov, I. and Hutter, F. SGDR : Stochastic gradient descent with warm restarts . arXiv preprint arXiv:1608.03983, 2016

    Pith/arXiv arXiv 2016

  33. [33]

    Explicit eigenvalue regularization improves sharpness-aware minimization

    Luo, H., Truong, T., Pham, T., Harandi, M., Phung, D., and Le, T. Explicit eigenvalue regularization improves sharpness-aware minimization. Advances in Neural Information Processing Systems, 37: 0 4424--4453, 2024

    2024

  34. [34]

    Power-law escape rate of SGD

    Mori, T., Ziyin, L., Liu, K., and Ueda, M. Power-law escape rate of SGD . In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.\ 15959--15975, 2022. URL https://proceedings.mlr.press/v162/mori22a.html

    2022

  35. [35]

    Bleu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

    2002

  36. [36]

    and Nedi \'c , A

    Pu, S. and Nedi \'c , A. Distributed stochastic gradient tracking methods. Mathematical Programming, 187 0 (1): 0 409--457, 2021

    2021

  37. [37]

    Singh, S. P. and Jaggi, M. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33: 0 22045--22055, 2020

    2020

  38. [38]

    Does SGD really happen in tiny subspaces? arXiv preprint arXiv:2405.16002, 2024

    Song, M., Ahn, K., and Yun, C. Does SGD really happen in tiny subspaces? arXiv preprint arXiv:2405.16002, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Momentum tracking: Momentum acceleration for decentralized deep learning on heterogeneous data

    Takezawa, Y., Bao, H., Niwa, K., Sato, R., and Yamada, M. Momentum tracking: Momentum acceleration for decentralized deep learning on heterogeneous data. arXiv preprint arXiv:2209.15505, 2022

    Pith/arXiv arXiv 2022

  40. [40]

    Beyond exponential graph: Communication-efficient topologies for decentralized learning via finite-time convergence

    Takezawa, Y., Sato, R., Bao, H., Niwa, K., and Yamada, M. Beyond exponential graph: Communication-efficient topologies for decentralized learning via finite-time convergence. Advances in Neural Information Processing Systems, 36: 0 76692--76717, 2023

    2023

  41. [41]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  42. [42]

    P., and Jaggi, M

    Vogels, T., Karimireddy, S. P., and Jaggi, M. Practical low-rank communication compression in decentralized deep learning. Advances in Neural Information Processing Systems, 33: 0 14171--14181, 2020

    2020

  43. [43]

    Slow M o: Improving communication-efficient distributed SGD with slow momentum

    Wang, J., Tantia, V., Ballas, N., and Rabbat, M. Slow M o: Improving communication-efficient distributed SGD with slow momentum . arXiv preprint arXiv:1910.00643, 2019

    Pith/arXiv arXiv 1910

  44. [44]

    From promise to practice: realizing high-performance decentralized training

    Wang, Z., Jiaojiao, Z., Xuyang, W., and Johansson, M. From promise to practice: realizing high-performance decentralized training. In The Thirteenth International Conference on Learning Representations. ICLR, 2025

    2025

  45. [45]

    Wu, L., Wang, M., and Su, W. J. The alignment property of SGD noise and how it helps select flat minima: A stability analysis. In Advances in Neural Information Processing Systems, volume 35, pp.\ 16843--16857, 2022. URL https://arxiv.org/abs/2207.02628

    Pith/arXiv arXiv 2022

  46. [46]

    Exponential graph is provably efficient for decentralized deep training

    Ying, B., Yuan, K., Chen, Y., Hu, H., Pan, P., and Yin, W. Exponential graph is provably efficient for decentralized deep training. Advances in Neural Information Processing Systems, 34: 0 13975--13987, 2021

    2021

  47. [47]

    DecentLaM: Decentralized momentum SGD for large-batch deep training

    Yuan, K., Chen, Y., Huang, X., Zhang, Y., Pan, P., Xu, Y., and Yin, W. DecentLaM: Decentralized momentum SGD for large-batch deep training . In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3029--3039, 2021

    2021

  48. [48]

    and Komodakis, N

    Zagoruyko, S. and Komodakis, N. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016

    Pith/arXiv arXiv 2016

  49. [49]

    R., Li, D., and Ju, H

    Zhang, H. R., Li, D., and Ju, H. Noise stability optimization for finding flat minima: A H essian-based regularization approach. arXiv preprint arXiv:2306.08553, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    P., Ye, Y., Luo, Z.-Q., and Sun, R

    Zhang, Y., Chen, C., Li, Z., Ding, T., Wu, C., Kingma, D. P., Ye, Y., Luo, Z.-Q., and Sun, R. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Towards theoretically understanding why sgd generalizes better than adam in deep learning

    Zhou, P., Feng, J., Ma, C., Xiong, C., Hoi, S., and Weinan, E. Towards theoretically understanding why sgd generalizes better than adam in deep learning. arxiv 2020. arXiv preprint arXiv:2010.05627, 2010

    Pith/arXiv arXiv 2020

  52. [52]

    Topology-aware generalization of decentralized SGD

    Zhu, T., He, F., Zhang, L., Niu, Z., Song, M., and Tao, D. Topology-aware generalization of decentralized SGD . In International Conference on Machine Learning, pp.\ 27479--27503. PMLR, 2022

    2022

  53. [53]

    Decentralized SGD and average-direction SAM are asymptotically equivalent

    Zhu, T., He, F., Chen, K., Song, M., and Tao, D. Decentralized SGD and average-direction SAM are asymptotically equivalent . In International Conference on Machine Learning, pp.\ 43005--43036. PMLR, 2023

    2023

  54. [54]

    Strength of minibatch noise in SGD

    Ziyin, L., Liu, K., Mori, T., and Ueda, M. Strength of minibatch noise in SGD . In International Conference on Learning Representations, 2022. URL https://arxiv.org/abs/2102.05375

    Pith/arXiv arXiv 2022