DES-LOC synchronizes model parameters and Adam/ADOPT momentum states on separate schedules, matching Local Adam quality with about 2x less communication and 170x less than DDP in tests up to 1.7B parameters.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
DES-LOC synchronizes model parameters and Adam/ADOPT momentum states on separate schedules, matching Local Adam quality with about 2x less communication and 170x less than DDP in tests up to 1.7B parameters.