REVIEW 4 cited by
FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This paper establishes a mathematical foundation for the Adam optimizer, elucidating its connection to natural gradient descent through Riemannian and information geometry. We provide an accessible and detailed analysis of the diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all detailed approximations and advocating for the use of log probability functions as loss, which should be based on discrete distributions, due to the limitations of empirical FIM. Our analysis uncovers flaws in the original Adam algorithm, leading to proposed corrections such as enhanced momentum calculations, adjusted bias corrections, adaptive epsilon, and gradient clipping. We refine the weight decay term based on our theoretical framework. Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.
Forward citations
Cited by 4 Pith papers
-
How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks
Zapping the last layer during pretraining speeds a model's recovery after transfer, and Adam produces different learning and forgetting patterns than SGD in continual learning.
-
Beyond the LUMIR challenge: The pathway to foundational registration models
Deep learning registration models trained on 4,014 unlabeled brain MRIs generalized across sites, contrasts, and even macaque brains, outperforming optimization-based methods in most zero-shot tasks.
-
Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization
GEAR-SAM re-allocates SAM's fixed perturbation radius across network blocks in proportion to an EMA of squared block-gradient norms, improving generalization on CIFAR, transfer, and label-noise benchmarks.
-
Module-Aware Parameter-Efficient Machine Unlearning on Transformers
MAPE-Unlearn uses Fisher-information-based scores and greedy search to select important heads and filters, then applies sparse unlearning updates, claiming improved efficacy-fidelity trade-offs on Transformers.
Discussion (0). Sign in to comment.