In budget-constrained 2.7B-parameter LLM pre-training, Lion is fastest, Sophia reaches the lowest loss, but AdamW wins on downstream benchmarks.
sign SGD with majority vote is communication efficient and fault tolerant
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Pre-Training LLMs on a budget: A comparison of three optimizers
In budget-constrained 2.7B-parameter LLM pre-training, Lion is fastest, Sophia reaches the lowest loss, but AdamW wins on downstream benchmarks.