FedEve uses a Kalman filter to combine server momentum (prediction) with client updates (observation) to offset period drift and client drift in cross-device federated learning.
Strength of Minibatch Noise in SGD
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
The noise in stochastic gradient descent (SGD), caused by minibatch sampling, is poorly understood despite its practical importance in deep learning. This work presents the first systematic study of the SGD noise and fluctuations close to a local minimum. We first analyze the SGD noise in linear regression in detail and then derive a general formula for approximating SGD noise in different types of minima. For application, our results (1) provide insight into the stability of training a neural network, (2) suggest that a large learning rate can help generalization by introducing an implicit regularization, (3) explain why the linear learning rate-batchsize scaling law fails at a large learning rate or at a small batchsize and (4) can provide an understanding of how discrete-time nature of SGD affects the recently discovered power-law phenomenon of SGD.
fields
cs.LG 1years
2025 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
FedEve: On Bridging the Client Drift and Period Drift for Cross-device Federated Learning
FedEve uses a Kalman filter to combine server momentum (prediction) with client updates (observation) to offset period drift and client drift in cross-device federated learning.