Pith. sign in

REVIEW 3 major objections 6 minor 83 references

Even after global correction, federated AdamW still over-trusts many coordinates; reallocating step sizes by a joint local-global trust score fixes that mismatch and lifts Transformer accuracy most under strong non-IID data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 00:06 UTC pith:WFVHCD5U

load-bearing objection Solid, practical extension of federated AdamW: global-aware coordinate reweighting of a corrected direction, with broad empirical gains especially on Transformers under strong non-IID; theory is standard and the trust signal is heuristic but ablated. the 3 major comments →

arxiv 2607.03763 v1 pith:WFVHCD5U submitted 2026-07-04 cs.LG cs.AI

FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity

Shuai Li , Qinglin Wang , Ping Luo , Jiahuan Wang , Hongyang Hu , Haotian Mo , Yigui Feng , Ziang Liu
show 3 more authors
Qisong Xiao Jie Liu Tao Sun
This is my paper
classification cs.LG cs.AI
keywords federated learningAdamWcoordinate trustTransformerdata heterogeneityadaptive optimizationglobal-local correction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Federated training of Transformers now often uses local AdamW because its coordinate-wise adaptive scales make much faster progress than SGD. Under non-IID client data, however, even a globally corrected AdamW direction still has coordinates of very uneven reliability: some are jointly backed by the current gradient and the global signal, while others are mostly local noise. Existing federated adaptive methods correct at the client-update or round level but then still apply that direction densely and uniformly. FedACT adds a second, coordinate-level step: it forms the same corrected direction, scores each coordinate by the product of that direction and the local gradient, and reallocates magnitudes so high-trust coordinates take larger steps while the rest keep smaller but non-zero updates. The paper shows this consistently beats strong adaptive baselines on vision Transformers, CNNs, LLM pre-training and fine-tuning, with the biggest gains on Transformers under stronger heterogeneity, and that the reallocation also improves cross-client direction consistency.

Core claim

Coordinate trust mismatch is a real residual failure mode of federated AdamW: after round-level global-local correction, different coordinates of the corrected adaptive update remain highly uneven in reliability under heterogeneous data. FedACT shows that preserving that dense corrected direction while softly reallocating its magnitudes according to a global-aware trust score (product of corrected direction and current gradient) improves accuracy, communication efficiency and client-update consistency, especially for Transformers under strong non-IID partitions.

What carries the argument

Global-aware ACT: after forming the corrected AdamW direction u = (1-ρ) u_loc + ρ Δ_G, compute the coordinate trust score s = u ⊙ g, then apply a soft top-τ mask φ that multiplies high-score coordinates by α > 1 and the rest by γ ∈ (0,1), so the update becomes φ ⊙ u rather than uniform u.

Load-bearing premise

That the product of the corrected direction and the current stochastic gradient is a good enough trust signal to decide which coordinates deserve larger steps, and that a fixed soft top-τ reallocation works without adapting the trust ratio online.

What would settle it

On the same ViT or Swin CIFAR-100 Dir-0.1 runs, replace the trust score s = u ⊙ g with a random or purely local-gradient score while keeping every other hyper-parameter fixed; if the accuracy and direction-consistency gains over FedAdamW disappear, the claimed global-aware coordinate mechanism is not doing the work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Federated AdamW pipelines for Transformers should treat coordinate-level magnitude reallocation as a standard complement to round-level global correction, not as an optional extra.
  • Under strong Dirichlet heterogeneity the largest accuracy lifts appear precisely on vision Transformers and Swin variants, so practitioners can expect the biggest payoff on those architectures.
  • Soft attenuation of low-trust coordinates is preferable to hard masking; keeping small non-zero steps preserves useful local signal while still reducing over-trust of the unreliable tail.
  • The same mechanism improves both communication-round efficiency in LLM pre-training and objective metrics in LoRA-based federated instruction tuning and preference alignment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the trust score is reliable, online schedules that shrink the trusted fraction early and enlarge it later (or adapt τ from measured score concentration) could further reduce the stationary neighborhood in the analysis.
  • Block- or matrix-aware analogues of ACT would be needed before the same idea can safely sit on top of matrix-structured optimizers that do not treat coordinates independently.
  • The concentration of positive score mass observed even after FedAdamW suggests that other dense adaptive federated methods may silently over-trust long unreliable tails; a lightweight coordinate diagnostic could become a standard monitoring tool.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper identifies coordinate trust mismatch in federated AdamW training: after round-level global-local correction, coordinates of the corrected adaptive direction remain highly uneven in reliability under non-IID data. FedACT forms a globally corrected AdamW direction u = (1-ρ)u_loc + ρΔ_G and reallocates magnitudes via a coordinate-wise trust score s = u ⊙ g, amplifying the top-τ coordinates (factor α) while softly attenuating the rest (factor γ > 0). The method is evaluated on ViT-Tiny, Swin-Lite, ResNet-18, Llama2 pre-training (60M/130M/250M), and LoRA SFT/DPO fine-tuning, with ablations on FedACT-Local, τ, γ, and ρ, plus direction-consistency diagnostics. A nonconvex analysis recovers the standard O(1/√RK) rate under full participation, λ=0, and a predictable diagonal preconditioner.

Significance. If the empirical pattern holds, FedACT supplies a useful and complementary design axis for federated adaptive optimization of Transformers: coordinate-level trust allocation on top of existing round-level correction (FedAdamW/FedLADA-style). The experimental scope is a genuine strength—vision Transformers and CNNs under Dirichlet partitions, Chinchilla-matched LLM pre-training, and parameter-efficient SFT/DPO—together with mechanism plots (direction consistency, positive score ratio, top-p score mass) and ablations that separate local MGUP-style reweighting from global-aware selection. The contribution is primarily algorithmic and empirical rather than theoretical; the analysis does not establish a tighter neighborhood for ACT versus dense corrected AdamW, but the practical gains under strong heterogeneity would still matter for federated Transformer training.

major comments (3)
  1. Eq. (7)–(8) and the ACT construction: the central premise is that the instantaneous product s = u ⊙ g is a sufficiently reliable coordinate trust signal under non-IID stochastic gradients. Under client bias and high variance, this product can mark only transiently aligned coordinates. Table 4 already shows severe collapse at τ=0.3 and clear degradation at τ=0.7, so the fixed soft top-τ policy is load-bearing and brittle. The manuscript should either (i) add diagnostics that the selected coordinates remain stable across local steps/clients (e.g., Jaccard of top-τ sets, correlation of s with multi-step descent), or (ii) discuss failure modes and when s is expected to be noisy, rather than treating the default τ=0.5 as generally adequate.
  2. Theorem 1 / Appendix B: the analysis recovers the standard nonconvex O(1/√RK) rate under full participation, λ=0, and F_t-measurable H_t, with an ACT-dependent neighborhood term B_{ρ,τ} that does not improve on dense corrected AdamW. The claim that coordinate trust allocation “effectively complements” round-level correction is therefore empirical, not theoretical. The main text should state this limitation explicitly next to Theorem 1 (not only in Appendix F), and avoid language that suggests the rate or neighborhood analysis validates the ACT bias term.
  3. Table 3 and Figure 3 (FedACT-Local vs FedACT): the ablation is important and supports global-aware selection, but it is reported only on CIFAR-100 for ViT-Tiny/Swin-Lite. Given that the largest claimed gains are under Dir-0.1 and on Transformers/LLMs, the same Local vs full comparison should be extended at least to one strong-heterogeneity Transformer setting and one LLM setting so that the “global-aware” part of the claim is not under-supported relative to the accuracy tables.
minor comments (6)
  1. Default ACT scaling: the text says “MGUP-style default α=1/τ and γ=τ”, while Table 4 and the sensitivity discussion often use γ=0.5 with τ=0.5. Please state the exact default (α,γ,τ,ρ) used in all main tables in one place (e.g., Experimental Setup or Algorithm 1 caption).
  2. Figure 1 caption and panels (c)–(d): “Top-1 Score Mass” should be defined with the same p as Appendix A (Top-p); if p=0.01, say so in the figure caption to match Eq. (18).
  3. Algorithm 1 line 12: bias correction uses (1-β_2^t) with a global step t, while m uses k; a one-line note that t is the global Adam step counter would avoid confusion with local step k.
  4. Related Work: Magma (Joo et al. 2026) and MGUP are cited appropriately; a brief sentence on how FedACT differs from block-level Magma (coordinate vs block, federated global signal) would help readers place the method.
  5. Appendix F Limitations is candid (Muon, scale of LLMs, fixed hyperparameters, partial participation). Consider promoting a short paragraph of that content into the main Conclusion so the scope is visible without the appendix.
  6. Typos/consistency: abstract “globally corrected AdamW updatesmayremain” spacing; occasional missing spaces after commas in the motivation section; ensure Dir-0.6/0.3/0.1 notation is uniform in tables and figures.

Circularity Check

0 steps flagged

No significant circularity: FedACT is an independently defined algorithmic reweighting evaluated on external accuracy/PPL metrics; diagnostics motivate but do not force the claimed gains.

full rationale

The paper’s load-bearing chain is: (i) observe that even after round-level global–local correction, coordinate-wise products s = u ⊙ g remain highly non-uniform (motivation figures); (ii) define ACT reallocation of magnitudes of a dense corrected AdamW direction via a top-τ soft mask (Eqs. 7–9); (iii) report accuracy/perplexity gains vs. FedAdamW and other baselines on held-out vision and LLM tasks; (iv) show improved direction consistency as a post-hoc mechanism check; (v) prove a standard nonconvex O(1/√RK) rate under full participation and predictable preconditioner. None of these steps reduces a claimed prediction to its own inputs by construction. The trust score and ACT coefficients are design choices, not parameters fitted to the accuracy tables and then re-presented as predictions. Direction consistency, positive-score ratio, and top-p score mass are measured diagnostics, not objectives that the method is optimized to match. Self-citations to FedAdamW/FedLADA supply prior baselines and the global-correction scaffolding that FedACT extends; they are not uniqueness theorems that forbid alternatives or force the empirical margins. MGUP-style α=1/τ, γ=τ is an adopted centralized ansatz made global-aware, not a self-derived identity. Convergence recovers a classical rate and does not claim that the ACT bias term is uniquely optimal. The paper is therefore self-contained against external benchmarks with no circular derivation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on standard federated nonconvex assumptions plus three hand-chosen modulation hyperparameters and the modeling choice that coordinate-wise product trust is the right reliability signal. No new physical entities are postulated; 'coordinate trust mismatch' is a named diagnostic phenomenon. Theory further assumes full participation and a predictable diagonal preconditioner.

free parameters (4)
  • trust ratio τ = 0.5 (default)
    Fraction of coordinates receiving the amplified step; default 0.5 chosen by sensitivity table; τ=0.3 collapses accuracy, so the claim depends on this choice.
  • attenuation factor γ = 0.5 (default)
    Step scale for non-top coordinates; default 0.5 (or MGUP-style γ=τ); hard mask γ=0 is worse, so soft attenuation is a free design choice.
  • global guidance strength ρ = 0.5 (default)
    Interpolation weight between local AdamW direction and server correction; default 0.5; ρ→1 collapses performance, so moderate ρ is required.
  • amplification α = 1/τ
    Usually set to 1/τ following MGUP-style defaults rather than derived; controls how strongly high-trust coordinates are boosted.
axioms (4)
  • domain assumption L-smooth local and global objectives, unbiased stochastic gradients with bounded variance σ², bounded client heterogeneity ζ², and uniformly bounded stochastic gradients G.
    Standard nonconvex federated optimization assumptions used for Theorem 1 / Appendix B.
  • ad hoc to paper Full client participation and zero weight decay (λ=0) in the convergence analysis; predictable (F_t-measurable) diagonal AdamW preconditioner H_t.
    Simplifications stated explicitly for the proof; practical experiments use partial participation and weight decay.
  • domain assumption Dirichlet label partitioning adequately models the relevant non-IID regime for vision and language federated tasks.
    Used throughout the experimental section to generate Dir-0.6/0.3/0.1 splits.
  • domain assumption AdamW first- and second-moment updates with bias correction remain the correct local adaptive base under federated Transformer training.
    Inherited from FedAdamW / local AdamW literature; FedACT modulates rather than replaces this base.
invented entities (2)
  • coordinate trust mismatch no independent evidence
    purpose: Name the claimed fine-grained failure mode: residual uneven reliability across coordinates of a globally corrected AdamW update under non-IID data.
    Diagnostic framing supported by positive-score-ratio and top-score-mass plots; not an independent physical object, but a paper-specific concept that organizes the method.
  • global-aware coordinate trust score s = u ⊙ g no independent evidence
    purpose: Select which coordinates of the corrected direction receive larger magnitudes.
    Defined operationally from the corrected direction and local gradient; success is measured only by downstream accuracy and direction consistency inside this paper.

pith-pipeline@v1.1.0-grok45 · 34158 in / 3368 out tokens · 31693 ms · 2026-07-12T00:06:58.072819+00:00 · methodology

0 comments
read the original abstract

Federated Transformer training increasingly relies on local AdamW, whose adaptive updates can provide much stronger local progress than SGD-based training. However, under heterogeneous client data, even globally corrected AdamW updates may remain highly uneven in coordinate-wise reliability. We refer to this phenomenon as coordinate trust mismatch. Existing federated adaptive optimizers mainly address mismatch at the client-update or communication-round level, but still apply the corrected adaptive direction densely and uniformly across coordinates. In this paper, we propose FedACT, a global-aware coordinate trust modulation method for federated AdamW training. FedACT first forms a globally corrected adaptive direction and then reallocates update magnitudes according to a coordinate-wise trust score, assigning larger steps to coordinates jointly supported by local gradients and global correction, while preserving smaller non-zero updates on the remaining coordinates. Extensive experiments on federated vision Transformers, CNNs, LLM pre-training, and LLM fine-tuning show that FedACT consistently improves over strong federated adaptive baselines, with the largest gains on Transformer models under stronger data heterogeneity. Mechanism analyses further show that FedACT improves cross-client direction consistency, suggesting that coordinate-level trust allocation effectively complements round-level global-local correction. Code will be released.

Figures

Figures reproduced from arXiv: 2607.03763 by Haotian Mo, Hongyang Hu, Jiahuan Wang, Jie Liu, Ping Luo, Qinglin Wang, Qisong Xiao, Shuai Li, Tao Sun, Yigui Feng, Ziang Liu.

Figure 1
Figure 1. Figure 1: Motivation for coordinate-level trust in federated AdamW training. On CIFAR-100 with ViT-Tiny under 300 communication rounds, 100 clients, 10% participation, and 50 local steps, (a) Local AdamW substantially outperforms Local SGD, confirming the effectiveness of adaptive local optimization for federated Transformer training; (b) FedAdamW improves round-level direction consistency over Local AdamW; however,… view at source ↗
Figure 2
Figure 2. Figure 2: Test Top-1 accuracy over communication rounds on federated vision Transformer training. The top row shows results [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Direction Consistency. On CIFAR-100 with ViT-Tiny and Swin-Lite under 300 communication rounds, 100 clients, 10% participation, and 50 local steps, FedACT consistently achieves higher direction consistency than Local AdamW, FedAdamW, and FedACT-Local under both Dirichlet heterogeneity levels. Compared with FedAdamW, it reduces the validation per￾plexity from 35.64 to 32.57 on Llama2-60M, from 22.82 to 21.4… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of different methods for feder [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training dynamics of different methods for FedIT [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training dynamics of different methods for federated instruction tuning (FedIT) and federated value alignment (FedVA). [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Test Top-1 accuracy over communication rounds for federated Swin-Lite training. Results are reported on CIFAR￾10, CIFAR-100, and Tiny-ImageNet under Dirichlet heterogeneity levels Dir-0.6, Dir-0.3, and Dir-0.1. FedACT consistently improves convergence and final accuracy across datasets and heterogeneity levels [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Test Top-1 accuracy over communication rounds for federated ViT-Tiny training. Results are reported on CIFAR￾10, CIFAR-100, and Tiny-ImageNet under Dirichlet heterogeneity levels Dir-0.6, Dir-0.3, and Dir-0.1. FedACT also shows improved convergence and higher final accuracy [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Test Top-1 accuracy over communication rounds for ResNet-18. Results are reported on CIFAR-10, CIFAR-100, and Tiny-ImageNet under Dirichlet heterogeneity levels Dir-0.6, Dir-0.3, and Dir-0.1. FedACT remains effective on CNN training, but the improvements are generally smaller than those on vision Transformers, supporting our view that coordinate-level trust modulation is most beneficial when local adaptive… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

83 extracted references · 23 linked inside Pith

  1. [4]

    Chang, D.; and Yuan, G. 2025. MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems

    2025

  2. [6]

    Chen, X.; Liang, C.; Huang, D.; Real, E.; Wang, K.; Pham, H.; Dong, X.; Luong, T.; Hsieh, C.-J.; Lu, Y.; et al. 2023. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36: 49205--49233

    2023

  3. [7]

    E.; et al

    Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3): 6

    2023

  4. [10]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  5. [13]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2022. Lora: Low-rank adaptation of large language models. Iclr, 1(2): 3

    2022

  6. [15]

    Jordan, K.; Jin, Y.; Boza, V.; You, J.; Cesista, F.; Newhouse, L.; and Bernstein, J. 2024. Muon: An optimizer for hidden layers in neural networks

    2024

  7. [17]

    P.; Kale, S.; Mohri, M.; Reddi, S.; Stich, S.; and Suresh, A

    Karimireddy, S. P.; Kale, S.; Mohri, M.; Reddi, S.; Stich, S.; and Suresh, A. T. 2020 b . Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, 5132--5143. PMLR

    2020

  8. [18]

    Kim, G.; Kim, J.; and Han, B. 2024. Communication-efficient federated learning with accelerated client gradient. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12385--12394

    2024

  9. [20]

    Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

    2009

  10. [21]

    Le, Y.; Yang, X.; et al. 2015. Tiny imagenet visual recognition challenge. CS 231N, 7(7): 3

    2015

  11. [22]

    K.; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; and Smith, V

    Li, T.; Sahu, A. K.; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; and Smith, V. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2: 429--450

    2020

  12. [24]

    Liu, J.; Shang, F.; Liu, H.; Tian, Y.; Liu, Y.; Liu, J.; Zhu, K.; and Lin, Z. 2026. Fedadamw: A communication-efficient optimizer with convergence and generalization guarantees for federated large models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 23748--23756

    2026

  13. [26]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 10012--10022

    2021

  14. [28]

    McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; and y Arcas, B. A. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, 1273--1282. Pmlr

    2017

  15. [29]

    Ozfatura, E.; Ozfatura, K.; and G \"u nd \"u z, D. 2021. Fedadc: Accelerated federated learning with drift control. In 2021 IEEE International Symposium on Information Theory (ISIT), 467--472. IEEE

    2021

  16. [30]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1--67

    2020

  17. [32]

    Sun, Y.; Shen, L.; Sun, H.; Ding, L.; and Tao, D. 2023. Efficient federated learning via local adaptive amended optimizer with linear speedup. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12): 14453--14464

    2023

  18. [33]

    Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford alpaca: An instruction-following llama model

    2023

  19. [35]

    Wang, J.; Liu, Q.; Liang, H.; Joshi, G.; and Poor, H. V. 2020. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33: 7611--7623

    2020

  20. [37]

    A.; Khashabi, D.; and Hajishirzi, H

    Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), 13484--13508

    2023

  21. [38]

    Wang, Y.; Lin, L.; and Chen, J. 2022. Communication-efficient adaptive federated learning. In International conference on machine learning, 22802--22838. PMLR

    2022

  22. [40]

    Ye, R.; Ge, R.; Zhu, X.; Chai, J.; Du, Y.; Liu, Y.; Wang, Y.; and Chen, S. 2024 a . Fedllm-bench: Realistic benchmarks for federated learning of large language models. Advances in Neural Information Processing Systems, 37: 111106--111130

    2024

  23. [41]

    Ye, R.; Wang, W.; Chai, J.; Li, D.; Li, Z.; Xu, Y.; Du, Y.; Wang, Y.; and Chen, S. 2024 b . Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, 6137--6147

    2024

  24. [42]

    Zhang, X.; Hong, M.; Dhople, S.; Yin, W.; and Liu, Y. 2021. FedPD: A federated learning framework with adaptivity to non-IID data. IEEE Transactions on Signal Processing, 69: 6055--6070

    2021

  25. [43]

    Zhang, Y.; Chen, C.; Ding, T.; Li, Z.; Sun, R.; and Luo, Z.-Q. 2024 a . Why transformers need adam: A hessian perspective. Advances in neural information processing systems, 37: 131786--131823

    2024

  26. [45]

    Zhou, P.; Feng, J.; Ma, C.; Xiong, C.; Hoi, S. C. H.; et al. 2020. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33: 21285--21296

    2020

  27. [46]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Fedadamw: A communication-efficient optimizer with convergence and generalization guarantees for federated large models , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  28. [47]

    Advances in neural information processing systems , volume=

    Why transformers need adam: A hessian perspective , author=. Advances in neural information processing systems , volume=

  29. [48]

    Advances in Neural Information Processing Systems , volume=

    Towards theoretically understanding why sgd generalizes better than adam in deep learning , author=. Advances in Neural Information Processing Systems , volume=

  30. [49]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    Pith/arXiv arXiv

  31. [50]

    Advances in neural information processing systems , volume=

    Symbolic discovery of optimization algorithms , author=. Advances in neural information processing systems , volume=

  32. [51]

    2024 , url =

    Keller Jordan and Yuchen Jin and Vlado Boza and Jiacheng You and Franz Cesista and Laker Newhouse and Jeremy Bernstein , title =. 2024 , url =

    2024

  33. [52]

    arXiv preprint arXiv:2510.27403 , year=

    FedMuon: Accelerating Federated Learning with Matrix Orthogonalization , author=. arXiv preprint arXiv:2510.27403 , year=

    arXiv

  34. [53]

    arXiv preprint arXiv:2602.19271 , year=

    Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data , author=. arXiv preprint arXiv:2602.19271 , year=

    arXiv

  35. [54]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    Pith/arXiv arXiv

  36. [55]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

    Efficient federated learning via local adaptive amended optimizer with linear speedup , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=. 2023 , publisher=

    2023

  37. [56]

    arXiv preprint arXiv:2411.16085 , year=

    Cautious optimizers: Improving training with one line of code , author=. arXiv preprint arXiv:2411.16085 , year=

    arXiv

  38. [57]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  39. [58]

    2009 , publisher=

    Learning multiple layers of features from tiny images , author=. 2009 , publisher=

    2009

  40. [59]

    CS 231N , volume=

    Tiny imagenet visual recognition challenge , author=. CS 231N , volume=

  41. [60]

    Proceedings of the 2013 conference on empirical methods in natural language processing , pages=

    Recursive deep models for semantic compositionality over a sentiment treebank , author=. Proceedings of the 2013 conference on empirical methods in natural language processing , pages=

    2013

  42. [61]

    arXiv preprint arXiv:1909.06335 , year=

    Measuring the effects of non-identical data distribution for federated visual classification , author=. arXiv preprint arXiv:1909.06335 , year=

    Pith/arXiv arXiv 1909

  43. [62]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  44. [63]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Swin transformer: Hierarchical vision transformer using shifted windows , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  45. [64]

    arXiv preprint arXiv:2010.11929 , year=

    An image is worth 16x16 words: Transformers for image recognition at scale , author=. arXiv preprint arXiv:2010.11929 , year=

    Pith/arXiv arXiv 2010

  46. [65]

    arXiv preprint arXiv:1907.11692 , year=

    Roberta: A robustly optimized bert pretraining approach , author=. arXiv preprint arXiv:1907.11692 , year=

    Pith/arXiv arXiv 1907

  47. [66]

    Artificial intelligence and statistics , pages=

    Communication-efficient learning of deep networks from decentralized data , author=. Artificial intelligence and statistics , pages=. 2017 , organization=

    2017

  48. [67]

    International conference on machine learning , pages=

    Scaffold: Stochastic controlled averaging for federated learning , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  49. [68]

    arXiv preprint arXiv:2106.10874 , year=

    Fedcm: Federated learning with client-level momentum , author=. arXiv preprint arXiv:2106.10874 , year=

    Pith/arXiv arXiv

  50. [69]

    arXiv preprint arXiv:2003.00295 , year=

    Adaptive federated optimization , author=. arXiv preprint arXiv:2003.00295 , year=

    Pith/arXiv arXiv 2003

  51. [70]

    arXiv preprint arXiv:2502.02727 , year=

    Gradient Correction in Federated Learning with Adaptive Optimization , author=. arXiv preprint arXiv:2502.02727 , year=

    Pith/arXiv arXiv

  52. [71]

    arXiv preprint arXiv:2406.16793 , year=

    Adam-mini: Use fewer learning rates to gain more , author=. arXiv preprint arXiv:2406.16793 , year=

    Pith/arXiv arXiv

  53. [72]

    International conference on machine learning , pages=

    Communication-efficient adaptive federated learning , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  54. [73]

    Proceedings of Machine learning and systems , volume=

    Federated optimization in heterogeneous networks , author=. Proceedings of Machine learning and systems , volume=

  55. [74]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Communication-efficient federated learning with accelerated client gradient , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  56. [75]

    arXiv preprint arXiv:2509.26337 , year=

    FedMuon: Federated Learning with Bias-corrected LMO-based Optimization , author=. arXiv preprint arXiv:2509.26337 , year=

    arXiv

  57. [76]

    Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining , pages=

    Openfedllm: Training large language models on decentralized private data via federated learning , author=. Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining , pages=

  58. [77]

    Advances in Neural Information Processing Systems , volume=

    Fedllm-bench: Realistic benchmarks for federated learning of large language models , author=. Advances in Neural Information Processing Systems , volume=

  59. [78]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  60. [79]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. Iclr , volume=

  61. [80]

    2023 , publisher=

    Stanford alpaca: An instruction-following llama model , author=. 2023 , publisher=

    2023

  62. [81]

    arXiv preprint arXiv:2304.12244 , year=

    Wizardlm: Empowering large language models to follow complex instructions , author=. arXiv preprint arXiv:2304.12244 , year=

    Pith/arXiv arXiv

  63. [82]

    See https://vicuna

    Vicuna: An open-source chatbot impressing gpt-4 with 90\ author=. See https://vicuna. lmsys. org (accessed 14 April 2023) , volume=

    2023

  64. [83]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  65. [84]

    arXiv preprint arXiv:2203.15556 , volume=

    Training compute-optimal large language models , author=. arXiv preprint arXiv:2203.15556 , volume=

    Pith/arXiv arXiv

  66. [85]

    arXiv preprint arXiv:2101.11203 , year=

    Achieving linear speedup with partial worker participation in non-iid federated learning , author=. arXiv preprint arXiv:2101.11203 , year=

    Pith/arXiv arXiv

  67. [86]

    arXiv preprint arXiv:1907.02189 , year=

    On the convergence of fedavg on non-iid data , author=. arXiv preprint arXiv:1907.02189 , year=

    Pith/arXiv arXiv 1907

  68. [87]

    arXiv preprint arXiv:1808.07217 , year=

    Don't use large mini-batches, use local SGD , author=. arXiv preprint arXiv:1808.07217 , year=

    Pith/arXiv arXiv

  69. [88]

    Advances in neural information processing systems , volume=

    Accelerating stochastic gradient descent using predictive variance reduction , author=. Advances in neural information processing systems , volume=

  70. [89]

    Advances in neural information processing systems , volume=

    Tackling the objective inconsistency problem in heterogeneous federated optimization , author=. Advances in neural information processing systems , volume=

  71. [90]

    IEEE Transactions on Signal Processing , volume=

    FedPD: A federated learning framework with adaptivity to non-IID data , author=. IEEE Transactions on Signal Processing , volume=. 2021 , publisher=

    2021

  72. [91]

    arXiv preprint arXiv:2111.04263 , year=

    Federated learning based on dynamic regularization , author=. arXiv preprint arXiv:2111.04263 , year=

    Pith/arXiv arXiv

  73. [92]

    arXiv preprint arXiv:1910.00643 , year=

    Slowmo: Improving communication-efficient distributed sgd with slow momentum , author=. arXiv preprint arXiv:1910.00643 , year=

    Pith/arXiv arXiv 1910

  74. [93]

    2021 IEEE International Symposium on Information Theory (ISIT) , pages=

    Fedadc: Accelerated federated learning with drift control , author=. 2021 IEEE International Symposium on Information Theory (ISIT) , pages=. 2021 , organization=

    2021

  75. [94]

    arXiv preprint arXiv:1808.05671 , year=

    On the convergence of adaptive gradient methods for nonconvex optimization , author=. arXiv preprint arXiv:1808.05671 , year=

    Pith/arXiv arXiv

  76. [95]

    arXiv preprint arXiv:2106.02969 , year=

    FedNL: Making Newton-type methods applicable to federated learning , author=. arXiv preprint arXiv:2106.02969 , year=

    Pith/arXiv arXiv

  77. [96]

    International conference on machine learning , pages=

    Distributed second order methods with fast rates and compressed communication , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  78. [97]

    arXiv preprint arXiv:2602.15322 , year=

    On Surprising Effectiveness of Masking Updates in Adaptive Optimizers , author=. arXiv preprint arXiv:2602.15322 , year=

    arXiv

  79. [98]

    arXiv preprint arXiv:2008.03606 , year=

    Mime: Mimicking centralized stochastic algorithms in federated learning , author=. arXiv preprint arXiv:2008.03606 , year=

    Pith/arXiv arXiv 2008

  80. [99]

    Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Self-instruct: Aligning language models with self-generated instructions , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

Showing first 80 references.