Pith. sign in

REVIEW

On Higher-order Moments in Adam

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1910.06878 v1 pith:NXD4Q2AH submitted 2019-10-15 cs.LG stat.ML

Zhanhong Jiang , Aditya Balu , Sin Yong Tan , Young M Lee , Chinmay Hegde , Soumik Sarkar This is my paper · ORCID
classification cs.LGstat.ML
keywords momentsadamgradientstochasticanalysishadamhigher-ordersome
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we investigate the popular deep learning optimization routine, Adam, from the perspective of statistical moments. While Adam is an adaptive lower-order moment based (of the stochastic gradient) method, we propose an extension namely, HAdam, which uses higher order moments of the stochastic gradient. Our analysis and experiments reveal that certain higher-order moments of the stochastic gradient are able to achieve better performance compared to the vanilla Adam algorithm. We also provide some analysis of HAdam related to odd and even moments to explain some intriguing and seemingly non-intuitive empirical results.

Discussion (0). Continue with ORCID to comment.

Pith tools