Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Gradient Correction in Federated Learning with Adaptive Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Federated Adam can be stabilized against client drift by correcting raw gradients before the first and second moment estimates are formed, not by correcting the final adaptive update direction.

desk verdict Pre-moment drift correction is a genuinely useful idea, but the advertised no-bounded-gradient theory doesn't cover the deployed algorithm; still worth serious review. read the letter →

arxiv 2502.02727 v3 pith:IGOWQIXA submitted 2025-02-04 cs.LG cs.AIcs.DC

Evan Chen , Shiqiang Wang , Jianing Zhang , Dong-Jun Han , Chaoyue Liu , Christopher Brinton This is my paper · ORCID
classification cs.LGcs.AIcs.DC
keywords federatedlearningclientdriftgradientcorrectionadaptiveoptimizationAdamnon-convexconvergencecommunicationefficiencydataheterogeneity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes FAdamGC, a federated learning algorithm that combines local Adam optimization with client-drift correction. The central claim is that drift compensation must be injected before the Adam moment estimates are formed, correcting raw gradients rather than the final adaptive update direction. The authors derive non-convex convergence rates with linear speedup, and show that this pre-estimation correction needs milder assumptions and removes a factor of K from the error bound compared with naively porting SCAFFOLD-style tracking into Adam. If the analysis is right, federated learners can get Adam's fast geometry-adaptive updates without the drift that heterogeneous data otherwise causes, at a communication cost reducible by updating control variates on only a subset of clients. Experiments on image and language tasks report fewer global rounds and lower simulated run time to target accuracy than baseline methods.

What carries the argument

The load-bearing mechanism is the pre-estimation corrected gradient $\hat{g}_i^{(t,k)} = g_i^{(t,k)} + y^{(t)} - y_i^{(t)}$ injected before Adam's first-moment $m$ and second-moment $v$ recurrences are updated. The control variates $y_i$ are refreshed as the client's average raw stochastic gradient over the local window, $y_i^{(t+1)} = \frac{1}{K}\sum_{k=1}^K g_i^{(t,k)}$, and the server maintains the global average $y^{(t)}$; a selective-tracking rule updates $y_i$ on only a subset of sampled clients each round. This pre-moment placement is what makes the global optimum a fixed point of the modified local update, because the exact correction $\nabla f(x^*) - \nabla f_i(x^*)$ zeroes the gradient fed into the moments; the analysis then bounds the local deviation term to obtain the convergence rate.

What would settle it

Run the $\beta_2=0$ case on a non-convex federated problem with heavy-tailed gradients whose norms have no finite bound, using the exact schedule $\beta_1 = \left(\frac{nK-2T}{2nK}\right)^{1/K}$ and a horizon satisfying $T < nK/2$; if the averaged gradient norm does not approach zero at the claimed rate, Theorem 5.2 is refuted. To test the transfer claim, take a problem where the $\beta_2=0$ variant converges and switch to $\beta_2=0.99$; divergence there would show that the practical method's guarantee still depends on the bounded-gradient assumption.

Watch

Extended reading notes

Core claim

The central discovery is that drift compensation in adaptive federated learning must operate on the gradient before it enters Adam's moment recursions. FAdamGC sets $\hat{g}_i^{(t,k)} = g_i^{(t,k)} + y^{(t)} - y_i^{(t)}$, where $y_i$ and $y$ track averaged raw gradients, and feeds this corrected gradient into both the first and second moment updates; this placement restores the global optimum as a fixed point of each client's update, while correcting the final adaptive direction (the naive FA-NT variant) does not. The paper proves that under L-smoothness and bounded gradient noise, FAdamGC converges to a stationary point at rate $$O\!\left(\sqrt{\frac{LF\$sigma^{2}$}{nKT}} + \frac{LF}{T} + \frac{$KG^{6}$}{\$epsilon^{2}$ T} + \frac{K(\$sigma^{2}$+(1+\$epsilon^{2}$)$G^{2}$)}{\$epsilon^{2}$ T}\right),$$ and that in the special case $\beta_2=\epsilon=0$ the same convergence holds without bounded gradients or bounded heterogeneity. It further shows that naive tracking carries an extra factor of $K$ in its lower-order term and requires stronger step-size restrictions, and reports that FAdamGC reaches target accuracy in fewer rounds and less simulated time than the comparison methods.

Load-bearing premise

The load-bearing premise is that the milder-assumption result transfers to the Adam configuration actually deployed: the proof that FAdamGC avoids bounded-gradient and bounded-heterogeneity assumptions holds only when Adam's second-moment decay is disabled ($\beta_2=\epsilon=0$), with the special momentum decay $\beta_1 = \left(\frac{nK-2T}{2nK}\right)^{1/K}$ and $T < nK/2$, while the algorithm tested in experiments uses standard second-moment decay ($\beta_2=0.99$, $\epsilon=10^{-8}$, $\beta_1=0.9$), where the only guarantee still assumes bounded gradients.

Editorial extensions

If this is right

  • If FAdamGC is correct, federated systems using local Adam no longer have to choose between adaptive speed and drift stability: the corrected gradient feed provides both.
  • The convergence rate for general Adam settings reaches the same $O(1/\sqrt{nKT})$ linear-speedup order as existing federated adaptive methods but with an explicit dependence on gradient variance $\sigma$, so batch size can be used to control convergence.
  • Compared with naive tracking, the corrected variant removes one factor of $K$ from the lower-order error term and relaxes local step-size constraints, so its advantage grows as the number of local steps $K$ grows.
  • Selective tracking means only a subset of clients needs to transmit correction updates; the experiments show total rounds degrade slowly as the tracking subset shrinks, reducing communication overhead.
  • Under $\beta_2=0$, convergence is guaranteed without bounded gradients or bounded data heterogeneity, matching the mild assumptions of SCAFFOLD-style correction and going beyond what FedAdam and FedAMS analyses provide.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the pre-estimation correction principle should apply to other diagonal adaptive optimizers such as RMSProp and AdaGrad, because the fix lives entirely in the gradient feed; testing the same federated protocol with those optimizers would reveal whether the moment structure is the only obstruction.
  • Beyond the paper: the selective-tracking experiments suggest control-variate communication can be reduced without losing drift correction, but the theory does not state how small the tracking subset can become before the $O(1/\sqrt{nKT})$ rate breaks; deriving that threshold is a natural next step.
  • Beyond the paper: the fixed-point argument implies drift correction helps most near optima where local gradients disagree, so one could construct a synthetic two-client problem with known gradients at the optimum and measure how far pre-estimation versus post-estimation correction moves the model away from the global optimum.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FAdamGC, a federated learning algorithm that combines adaptive client-side optimization (Adam) with client-drift correction. The correction is applied to the raw stochastic gradient before both the first- and second-moment updates, and a selective-tracking mechanism updates the correction terms on only a subset of clients. The theoretical section states a bounded-gradient-free convergence rate for the special case beta2=epsilon=0 (Theorem 5.2) and a bounded-gradient convergence rate for general beta2 (Theorem 5.4), comparing both with a naive tracking variant FA-NT. Experiments on CIFAR-10, CIFAR-100, TinyImageNet, and language-model fine-tuning tasks report lower total communication rounds and lower simulated runtime than several baselines.

Significance. If fully established, the central design idea of FAdamGC—correcting the gradient before Adam's moment accumulation rather than after—would be a useful and nontrivial contribution to federated adaptive optimization, and the selective-tracking mechanism addresses a real communication bottleneck of control-variate methods. The empirical study is broad, including image classification and LLM fine-tuning, and the comparison with a naive tracking baseline is informative. However, the current version outruns its proofs: the advertised 'milder assumptions' result is established only in a parameter regime that is neither used in Algorithm 1 nor in any experiment, and the general-parameter theorem has step-size conditions that are orders of magnitude away from the experimental settings. The theoretical significance of the present version is therefore substantially reduced, although the algorithm and empirical findings remain of interest.

major comments (4)
  1. [Theorem 5.2; Sections 1 and 6; Section 7]
  2. [Theorem 5.4; condition preceding Eq. (53); Table 4]
  3. [Appendix A, Lemma A.1, Eqs. (77)-(79)]
  4. [Table 1; Theorems 5.4, C.2, and C.3]
minor comments (5)
  1. [Section 4.2, Eq. (4)]
  2. [Section 5, footnote]
  3. [Section 6 and Appendix F]
  4. [Throughout]
  5. [Table 1]

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; FAdamGC's proof is self-contained, though the beta2=0 theorem does not cover the beta2=0.99 experiments.

full rationale

FAdamGC is defined in Algorithm 1 by a concrete update rule (corrected gradient g_hat = g + y(t) - y_i(t) fed into Adam moments), and the convergence proofs in Section 5 and Appendices A-B are independent analyses of that rule under stated assumptions (L-smoothness, bounded noise, and for Theorem 5.4 bounded gradients). No parameter of the algorithm is fitted to the experimental outcome, and the rate in Theorem 5.4 is derived from the descent lemma and moment-deviation lemmas rather than assumed. The comparison with FA-NT is a comparison with a baseline defined in the same paper; its worse K-factor and BDH requirement are derived in Theorems C.2-C.3, not baked into the definition, so the comparison is a mathematical conclusion rather than a circular one. The main caveat is a theory-experiment gap, which the paper itself flags: Theorem 5.2 is proved only for beta2=epsilon=0 with beta1 = ((KN-2T)/(2KN))^(1/K) and T<KN/2, while all experiments use beta2=0.99, epsilon=1e-8, beta1=0.9 (Section 6 and Table 4), and Theorem 5.4's step-size bound eta_l <= (1-beta1)beta1*epsilon/(40(G+epsilon)sqrt(T)L) is orders of magnitude below the used eta_l=1e-3. The paper acknowledges this: Section 7 states 'A limitation of our theoretical analysis, as in the existing adaptive federated optimization literature, is the reliance on a bounded gradient assumption for arbitrary Adam parameters,' and Appendix G concedes that 'empirical results consistently show improved performance when beta2>0' while only the beta2=0 case has the mild-assumption guarantee. This gap weakens the abstract's phrasing of the 'milder assumptions' contribution for the implemented configuration, but it is a validity/rigor limitation, not circular reasoning: the theorems do not assume their own conclusions. The self-citations [24,25] appear only as related-work references and are not load-bearing for the algorithm, proof, or experimental claims.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

FAdamGC introduces no new physical or mathematical entities; its control variates and moment buffers are standard optimization state. The free parameters are learning rates and tracking-subset size, plus a proof-only beta1 choice in Theorem 5.2 that is not used in experiments. The axioms are the standard smoothness, noise-variance, bounded-gradient, and bounded-below assumptions. Bounded data heterogeneity is deliberately avoided for FAdamGC, which is the key advertised advantage, but the only bounded-gradient-free theorem requires beta2=epsilon=0.

free parameters (4)
  • local learning rate eta_l = 3e-5 to 5e-3 across tasks
    Selected by grid search over [1e-4, 1e-1] per baseline and dataset; the convergence theorems constrain eta_l by G, L, epsilon, and T.
  • global learning rate eta_g = 0.1 to 1 across tasks
    Selected by grid search over [1e-3, 1]; the product eta_g*eta_l appears in all convergence-rate bounds.
  • tracking subset size eS = S/2 for image tasks, S for language tasks
    Design choice controlling how many clients transmit correction statistics; Section 6 and Figure 4 show the cost performance trade-off.
  • momentum decay beta1 in Theorem 5.2 = ((KN-2T)/(2KN))^(1/K)
    Proof-only choice that requires T < KN/2; experiments instead fix beta1=0.9, so the special-case theorem does not describe the tested configuration.
assumptions (5)
  • domain assumption Each local loss f_i is L-smooth (Assumption 5.1.1)
    Used throughout the descent inequalities in Appendices A and B to bound the difference between local and global gradients.
  • domain assumption Stochastic gradient noise has bounded second moment sigma^2 (Assumption 5.1.2)
    Needed for the noise-dependent speedup term in Theorems 5.2 and 5.4.
  • domain assumption Gradient norms are bounded by G (Assumption 5.3)
    Required by Theorem 5.4 to control second-moment deviation; the paper acknowledges this limitation, and it can be violated by unbounded loss landscapes.
  • standard math Global loss is bounded below with finite f*
    Implicit in defining F = E f(x(1)) - f* and used to telescope the descent inequality over T rounds.
  • domain assumption Client minibatches are unbiased samples from local distributions
    Used to define the noise term n_i^{(t,k)} and to take expectations over stochastic gradients; this is standard in FL analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gradient Correction in Federated Learning with Adaptive Optimization." pith.science (2026). https://pith.science/paper/IGOWQIXA

@misc{pith2026250202727,
  author       = {Pith},
  title        = {Pith review of: Gradient Correction in Federated Learning with Adaptive Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IGOWQIXA}},
  note         = {Machine review of arXiv:2502.02727}
}
read the original abstract

In federated learning (FL), model training performance is strongly impacted by data heterogeneity across clients. Client-drift compensation methods have recently emerged as a solution to this issue, introducing correction terms into local model updates. To date, these methods have only been considered under stochastic gradient descent (SGD)-based model training, while modern FL frameworks also employ adaptive optimizers (e.g., Adam) for improved convergence. However, due to the complex interplay between first and second moments found in most adaptive optimization methods, naively injecting correction terms can lead to performance degradation in heterogeneous settings. In this work, we propose {\tt FAdamGC}, the first algorithm to integrate drift compensation into adaptive federated optimization. The key idea of {\tt FAdamGC} is injecting a pre-estimation correction term that aligns with the moment structure of adaptive methods. We provide a rigorous convergence analysis of our algorithm under non-convex settings, showing that {\tt FAdamGC} results in better rate and milder assumptions than naively porting SGD-based correction algorithms into adaptive optimizers. Our experimental results demonstrate that {\tt FAdamGC} consistently outperform existing methods in total communication and computation cost across varying levels of data heterogeneity, showing the efficacy of correcting gradient information in federated adaptive optimization.

Figures

Figures reproduced from arXiv: 2502.02727 by the authors.

Figure 1
Figure 1. Visualization of the local update process under adaptive optimization with gradient correction. While [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of achieved accuracy over global [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the total cost of Adam-based [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Experimental results on CIFAR100 under different sample rate of clients and [PITH_FULL_IMAGE:figures/full_fig_p034_5.png]
Figure 6
Figure 6. Figure 6: Experimental results on CIFAR100 under different sample rate of clients and [PITH_FULL_IMAGE:figures/full_fig_p035_6.png]
Figure 7
Figure 7. Figure 7: Experimental results on CIFAR10 under different sample rate of clients and [PITH_FULL_IMAGE:figures/full_fig_p035_7.png]
Figure 8
Figure 8. Figure 8: Experimental results on CIFAR10 under different sample rate of clients and [PITH_FULL_IMAGE:figures/full_fig_p036_8.png]
Figure 9
Figure 9. Figure 9: Comparison of total cost to attain certain accuracy between different tracking sampling rate TinyImageNet [PITH_FULL_IMAGE:figures/full_fig_p036_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Global-aware coordinate trust modulation after corrected AdamW updates improves federated Transformer and LLM training under data heterogeneity over strong adaptive baselines.

Reference graph

Works this paper leans on

36 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    Federated learning: Challenges, methods, and future directions,

    T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020

    2020

  2. [2]

    Advances and open problems in federated learning,

    P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummingset al., “Advances and open problems in federated learning,” Foundations and Trends® in Machine Learning, vol. 14, no. 1–2, pp. 1–210, 2021

    work page 2021

  3. [3]

    Adaptive subgradient methods for online learning and stochastic optimization

    J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization.” Journal of machine learning research, vol. 12, no. 7, 2011

    work page 2011

  4. [4]

    Generating sequences with recurrent neural networks,

    A. Graves, “Generating sequences with recurrent neural networks,” arXiv preprint arXiv:1308.0850, 2013

    arXiv 2013

  5. [5]

    Adam: A method for stochastic optimization,

    D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014

    arXiv 2014

  6. [6]

    Towards building the federatedgpt: Federated instruction tuning,

    J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y. Chen, “Towards building the federatedgpt: Federated instruction tuning,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 6915–6919

    work page 2024

  7. [7]

    Momentum benefits non-iid federated learning simply and provably,

    Z. Cheng, X. Huang, P. Wu, and K. Yuan, “Momentum benefits non-iid federated learning simply and provably,”arXiv preprint arXiv:2306.16504, 2023

    arXiv 2023

  8. [8]

    Adaptive federated optimization,

    S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečn` y, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” arXiv preprint arXiv:2003.00295, 2020

    arXiv 2003

Show all 36 references
  1. [9]

    Communication-efficient adaptive federated learning,

    Y. Wang, L. Lin, and J. Chen, “Communication-efficient adaptive federated learning,” inInternational conference on machine learning. PMLR, 2022, pp. 22802–22838

    2022

  2. [10]

    Scaffold: Stochastic controlled averaging for federated learning,

    S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in International conference on machine learning. PMLR, 2020, pp. 5132–5143

    2020

  3. [11]

    Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally!

    K. Mishchenko, G. Malinovsky, S. Stich, and P. Richtárik, “Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally!” in International Conference on Machine Learning. PMLR, 2022, pp. 15750–15769. 11

    2022

  4. [12]

    Next: In-network nonconvex optimization,

    P. Di Lorenzo and G. Scutari, “Next: In-network nonconvex optimization,”IEEE Transactions on Signal and Information Processing over Networks, vol. 2, no. 2, pp. 120–136, 2016

    2016

  5. [13]

    Achieving geometric convergence for distributed optimization over time-varying graphs,

    A. Nedic, A. Olshevsky, and W. Shi, “Achieving geometric convergence for distributed optimization over time-varying graphs,”SIAM Journal on Optimization, vol. 27, no. 4, pp. 2597–2633, 2017

    2017

  6. [14]

    Asy-sonata: Achieving linear convergence in distributed asynchronous multiagent optimization,

    Y. Tian, Y. Sun, and G. Scutari, “Asy-sonata: Achieving linear convergence in distributed asynchronous multiagent optimization,” in 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 2018, pp. 543–551

    2018

  7. [15]

    An improved analysis of gradient tracking for decentralized machine learning,

    A. Koloskova, T. Lin, and S. U. Stich, “An improved analysis of gradient tracking for decentralized machine learning,” Neural Information Processing Systems, vol. 34, pp. 11422–11435, 2021

    2021

  8. [16]

    Gtadam: Gradient tracking with adaptive momentum for distributed online optimization,

    G. Carnevale, F. Farina, I. Notarnicola, and G. Notarstefano, “Gtadam: Gradient tracking with adaptive momentum for distributed online optimization,” IEEE Transactions on Control of Network Systems, vol. 10, no. 3, pp. 1436–1448, 2022

    2022

  9. [17]

    Momentum tracking: Momentum acceleration for decentralized deep learning on heterogeneous data,

    Y. Takezawa, H. Bao, K. Niwa, R. Sato, and M. Yamada, “Momentum tracking: Momentum acceleration for decentralized deep learning on heterogeneous data,” arXiv preprint arXiv:2209.15505, 2022

    2022 arXiv

  10. [18]

    Momentum-based distributed gradient tracking algorithms for distributed aggregative optimization over unbalanced directed graphs,

    Z. Wang, D. Wang, J. Lian, H. Ge, and W. Wang, “Momentum-based distributed gradient tracking algorithms for distributed aggregative optimization over unbalanced directed graphs,” Automatica, vol. 164, p. 111596, 2024

    2024

  11. [19]

    Towards optimal communication complexity in distributed non-convex optimization,

    K. K. Patel, L. Wang, B. E. Woodworth, B. Bullins, and N. Srebro, “Towards optimal communication complexity in distributed non-convex optimization,” Advances in Neural Information Processing Systems, vol. 35, pp. 13316–13328, 2022

    2022

  12. [20]

    Decentralized gradient tracking with local steps,

    Y. Liu, T. Lin, A. Koloskova, and S. U. Stich, “Decentralized gradient tracking with local steps,” arXiv preprint arXiv:2301.01313, 2023

    2023 arXiv

  13. [21]

    Gradient and variable tracking with multiple local SGD for decentralized non-convex learning,

    S. Ge and T.-H. Chang, “Gradient and variable tracking with multiple local SGD for decentralized non-convex learning,” arXiv preprint arXiv:2302.01537, 2023

    2023 arXiv

  14. [22]

    Balancing communication and computation in gradient tracking algorithms for decentralized optimization,

    A. S. Berahas, R. Bollapragada, and S. Gupta, “Balancing communication and computation in gradient tracking algorithms for decentralized optimization,” arXiv preprint arXiv:2303.14289, 2023

    2023 arXiv

  15. [23]

    Local exact-diffusion for decentralized optimization and learning,

    S. A. Alghunaim, “Local exact-diffusion for decentralized optimization and learning,”IEEE Transactions on Automatic Control, 2024

    2024

  16. [24]

    Taming subnet-drift in d2d-enabled fog learning: A hierarchicalgradienttrackingapproach,

    E. Chen, S. Wang, and C. G. Brinton, “Taming subnet-drift in d2d-enabled fog learning: A hierarchicalgradienttrackingapproach,” in IEEE INFOCOM 2024-IEEE Conference on Computer Communications. IEEE, 2024, pp. 2438–2447

    2024

  17. [25]

    Hierarchical federated learning with multi-timescale gradient correction,

    W. Fang, D.-J. Han, E. Chen, S. Wang, and C. Brinton, “Hierarchical federated learning with multi-timescale gradient correction,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  18. [26]

    Decoupled weight decay regularization,

    I. Loshchilov, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  19. [27]

    On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective,

    Z.Xie, Z.Xu, J.Zhang, I.Sato, andM.Sugiyama, “On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  20. [28]

    Local adaalter: Communication-efficient stochastic gradient descent with adaptive learning rates,

    C. Xie, O. Koyejo, I. Gupta, and H. Lin, “Local adaalter: Communication-efficient stochastic gradient descent with adaptive learning rates,” arXiv preprint arXiv:1911.09030, 2019. 12

    1911 arXiv

  21. [29]

    Efficient federated learning via local adaptive amended optimizer with linear speedup,

    Y. Sun, L. Shen, H. Sun, L. Ding, and D. Tao, “Efficient federated learning via local adaptive amended optimizer with linear speedup,”arXiv preprint arXiv:2308.00522, 2023

    2023 arXiv

  22. [30]

    A sufficient condition for convergences of adam and rmsprop,

    F. Zou, L. Shen, Z. Jie, W. Zhang, and W. Liu, “A sufficient condition for convergences of adam and rmsprop,” in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2019, pp. 11127–11135

    2019

  23. [31]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” Master’s thesis, University of Tront, 2009

    2009

  24. [32]

    Tiny imagenet visual recognition challenge,

    Y. Le and X. Yang, “Tiny imagenet visual recognition challenge,” CS 231N, vol. 7, no. 7, p. 3, 2015

    2015

  25. [33]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  26. [34]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  27. [35]

    Newsweeder: Learning to filter netnews,

    K. Lang, “Newsweeder: Learning to filter netnews,” in Machine learning proceedings 1995. Elsevier, 1995, pp. 331–339

    1995

  28. [36]

    Glue: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” arXiv preprint arXiv:1804.07461, 2018. 13 Appendix A Theoretical Analysis for FAdamGC (Theorem 5.4) 15 B Analsis of FAdamGC for Special Cases (Theorem. 5.2) 22 C The Algorithm and...

    2018 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.