FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models

Fanhua Shang; Hongying Liu; Jin Liu; Junkang Liu; Kewen Zhu; Yuanyuan Liu; Yuxuan Tian; Zhouchen Lin

arxiv: 2510.27486 · v3 · submitted 2025-10-31 · 💻 cs.LG · cs.AI

FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models

Junkang Liu , Fanhua Shang , Hongying Liu , Yuxuan Tian , Yuanyuan Liu , Jin Liu , Kewen Zhu , Zhouchen Lin This is my paper

Pith reviewed 2026-05-18 02:22 UTC · model grok-4.3

classification 💻 cs.LG cs.AI

keywords Federated learningAdamW optimizerConvergence guaranteesLarge modelsCommunication efficiencyTransformer modelsOptimization

0 comments

The pith

FedAdamW achieves linear speedup convergence without data homogeneity assumptions for federated large models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents FedAdamW as an adaptation of AdamW for federated learning to handle large-scale models. It tackles problems of variance in second-moment estimates, client drift from local overfitting, and slow convergence from moment reinitialization. Through a local correction mechanism, decoupled weight decay, and mean aggregation of second-moment estimates, the algorithm aligns local and global updates. The authors establish a convergence rate showing linear speedup without requiring data homogeneity, supported by PAC-Bayesian generalization bounds. Experiments confirm reduced communication rounds and higher test accuracy on language and vision Transformer models compared to baselines.

Core claim

FedAdamW uses a local correction mechanism and decoupled weight decay to align local updates with the global update, efficiently aggregates the mean of the second-moment estimates to reduce variance, and proves a linear speedup convergence rate of O(sqrt((L Δ σ_l²)/(S K R ε²)) + (L Δ)/R) without heterogeneity assumption, while improving test accuracy on Transformer models.

What carries the argument

Local correction mechanism with mean aggregation of second-moment estimates, which mitigates client drift and controls variance to enable convergence without homogeneity assumptions.

If this is right

The convergence bound holds independently of data heterogeneity, allowing application to diverse real-world datasets.
The method reduces communication rounds while improving accuracy on language and vision tasks.
Decoupled weight decay in local training provides generalization benefits as explained by PAC-Bayesian analysis.
Reinitializing moment estimates is avoided, speeding up the overall training process.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This could extend to other first-order optimizers in federated settings by similar correction techniques.
Testing on non-Transformer architectures might reveal if the benefits are specific to attention-based models.
The approach may inspire hybrid centralized-federated training protocols for even larger models.

Load-bearing premise

The local correction mechanism and mean aggregation of second-moment estimates sufficiently control client drift and variance to achieve the convergence bound without data homogeneity.

What would settle it

Observing that the empirical convergence rate fails to match the predicted linear speedup when client data distributions are highly heterogeneous would challenge the central claim.

read the original abstract

AdamW has become one of the most effective optimizers for training large-scale models. We have also observed its effectiveness in the context of federated learning (FL). However, directly applying AdamW in federated learning settings poses significant challenges: (1) due to data heterogeneity, AdamW often yields high variance in the second-moment estimate $\boldsymbol{v}$; (2) the local overfitting of AdamW may cause client drift; and (3) Reinitializing moment estimates ($\boldsymbol{v}$, $\boldsymbol{m}$) at each round slows down convergence. To address these challenges, we propose the first \underline{Fed}erated \underline{AdamW} algorithm, called \texttt{FedAdamW}, for training and fine-tuning various large models. \texttt{FedAdamW} aligns local updates with the global update using both a \textbf{local correction mechanism} and decoupled weight decay to mitigate local overfitting. \texttt{FedAdamW} efficiently aggregates the \texttt{mean} of the second-moment estimates to reduce their variance and reinitialize them. Theoretically, we prove that \texttt{FedAdamW} achieves a linear speedup convergence rate of $\mathcal{O}(\sqrt{(L \Delta \sigma_l^2)/(S K R \epsilon^2)}+(L \Delta)/R)$ without \textbf{heterogeneity assumption}, where $S$ is the number of participating clients per round, $K$ is the number of local iterations, and $R$ is the total number of communication rounds. We also employ PAC-Bayesian generalization analysis to explain the effectiveness of decoupled weight decay in local training. Empirically, we validate the effectiveness of \texttt{FedAdamW} on language and vision Transformer models. Compared to several baselines, \texttt{FedAdamW} significantly reduces communication rounds and improves test accuracy. The code is available in https://github.com/junkangLiu0/FedAdamW.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

FedAdamW claims a heterogeneity-free linear speedup rate for federated AdamW via local correction and mean second-moment aggregation, but the abstract leaves the supporting derivation unshown.

read the letter

The main point to know is that FedAdamW modifies AdamW for federated learning by adding a local correction to keep local updates aligned with the global one and by taking the mean of the second-moment estimates across clients instead of something else. This is supposed to let the method converge at a linear speedup rate that does not require any assumption on data heterogeneity across clients. The paper does a good job spelling out the three practical problems that arise when you just plug AdamW into FL: variance in the second moment due to heterogeneity, client drift from local overfitting, and the slowdown from resetting moments each round. The fixes are straightforward and the PAC-Bayesian part for explaining the decoupled weight decay is a nice touch. On the experimental side, they test on both language and vision transformer models and report that it cuts down the number of communication rounds while lifting test accuracy over several baselines. Having the code public is helpful too. Where it gets soft is in the theory. The abstract gives the rate O(sqrt((L Δ σ_l²)/(S K R ε²)) + (L Δ)/R) without heterogeneity, but there is no outline of the proof or list of assumptions. The local correction and mean aggregation are presented as the key to controlling drift and variance, yet without seeing how they bound those terms it's difficult to know if the no-heterogeneity claim is fully supported or if some condition is still needed in disguise. This is the kind of detail that matters for a result like this. The paper is aimed at people working on federated training or fine-tuning of large models where data is distributed and private. Someone looking for an optimizer that works well with transformers in that setting would find the algorithm and results useful. Even with the abstract-only view, the combination looks novel enough and the potential impact high enough that it should go to peer review for a full check on the math and the experimental controls.

Referee Report

1 major / 2 minor

Summary. The manuscript presents FedAdamW, an adaptation of the AdamW optimizer for federated learning. It introduces a local correction mechanism and mean aggregation of second-moment estimates to mitigate high variance in v, client drift from local overfitting, and issues with reinitializing moments. The paper claims a linear speedup convergence rate of O(sqrt((L Δ σ_l²)/(S K R ε²)) + (L Δ)/R) that holds without a heterogeneity assumption, supported by PAC-Bayesian generalization analysis for decoupled weight decay, and empirical results showing reduced communication rounds and improved test accuracy on language and vision Transformer models.

Significance. Should the theoretical guarantees be verified, this work could be significant in advancing federated optimization by removing the common heterogeneity assumption in convergence analyses for adaptive optimizers like AdamW. This would facilitate training large models in heterogeneous data settings. The empirical validation on Transformers and the provision of open-source code are additional strengths that enhance the paper's impact if the claims are substantiated.

major comments (1)

Abstract: The stated convergence rate O(sqrt((L Δ σ_l²)/(S K R ε²)) + (L Δ)/R) is claimed to hold without any heterogeneity assumption. However, the abstract provides no derivation, assumption list, or explanation of how the local correction mechanism combined with mean aggregation of second-moment estimates suffices to control client drift and bound the variance, which is central to eliminating the heterogeneity term typically present in FL convergence bounds.

minor comments (2)

Abstract: The abstract mentions 'we prove that FedAdamW achieves...' but does not specify the full set of notations (e.g., definitions of L, Δ, σ_l²) which are needed to fully interpret the rate.
Abstract: There is a minor formatting issue with underlined text in 'the first Fed erated AdamW' which appears to be LaTeX artifacts.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful review and constructive feedback on our manuscript. We address the major comment point by point below.

read point-by-point responses

Referee: [—] Abstract: The stated convergence rate O(sqrt((L Δ σ_l²)/(S K R ε²)) + (L Δ)/R) is claimed to hold without any heterogeneity assumption. However, the abstract provides no derivation, assumption list, or explanation of how the local correction mechanism combined with mean aggregation of second-moment estimates suffices to control client drift and bound the variance, which is central to eliminating the heterogeneity term typically present in FL convergence bounds.

Authors: We appreciate the referee's point. The abstract is intentionally concise and serves as a high-level summary of the contributions and results, following standard academic conventions where space is limited. The complete list of assumptions, the full derivation of the convergence rate, and the detailed explanation of how the local correction mechanism together with mean aggregation of second-moment estimates controls client drift and bounds variance (thereby removing the heterogeneity term) are provided in the theoretical analysis section of the full manuscript. Our proof establishes the stated linear speedup rate under standard smoothness and bounded-variance assumptions on the stochastic gradients. If the referee finds it helpful, we can add a short sentence to the abstract that explicitly references the main convergence theorem. revision: partial

Circularity Check

0 steps flagged

No circularity detected; convergence claim presented as independent theorem

full rationale

The abstract states that FedAdamW achieves the rate O(sqrt((L Δ σ_l²)/(S K R ε²)) + (L Δ)/R) without heterogeneity assumption via local correction and mean aggregation of second moments, but supplies no equations, proof steps, or self-referential definitions that reduce the claimed result to its inputs by construction. No fitted parameters are renamed as predictions, no self-citations load-bear the central theorem, and no ansatz or renaming of known results is invoked. The derivation is therefore treated as self-contained on the basis of the provided text.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review yields minimal ledger entries; the central mechanisms are treated as domain assumptions whose validity is asserted rather than derived from prior results.

axioms (1)

domain assumption Local correction and decoupled weight decay align client updates with the global model sufficiently to control drift
Invoked to address local overfitting and client drift in the absence of heterogeneity.

pith-pipeline@v0.9.0 · 5895 in / 1235 out tokens · 34113 ms · 2026-05-18T02:22:59.103115+00:00 · methodology

discussion (0)

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Fusion and Alignment Enhancement with Large Language Models for Tail-item Sequential Recommendation
cs.IR 2026-04 unverdicted novelty 7.0

FAERec fuses collaborative ID embeddings with LLM semantic embeddings using adaptive gating and dual-level alignment to enhance tail-item sequential recommendations.
FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning
cs.LG 2026-03 unverdicted novelty 7.0

FedBCGD reduces communication in federated learning by a factor of 1/N through block-wise parameter updates with accelerated convergence guarantees.
DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models
cs.LG 2026-02 unverdicted novelty 7.0

DP-FedAdamW delivers an unbiased second-moment estimator for AdamW in DPFL, proving linear convergence acceleration without heterogeneity assumptions and outperforming SOTA by 5.83% on Tiny-ImageNet with Swin-Base at ε=1.
From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneity
cs.LG 2026-04 unverdicted novelty 6.0

FEAT mitigates representation collapse and prediction bias in federated continual learning by aligning feature angular similarities to shared Equiangular Tight Frame prototypes and removing task-irrelevant directional...
Personalized Federated Learning for Gradient Alignment
cs.LG 2026-05 unverdicted novelty 5.0

pFLAlign uses two gradient alignment mechanisms derived from PAC-Bayesian analysis to reduce variance in local training and distortion in aggregation, yielding state-of-the-art personalization in federated learning.
FedNSAM:Consistency of Local and Global Flatness for Federated Learning
cs.LG 2026-02 unverdicted novelty 4.0

FedNSAM uses global Nesterov momentum to make local flatness consistent with global flatness in federated learning, yielding tighter convergence than FedSAM and better empirical performance.