REVIEW 4 major objections 4 minor 26 references
Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper proves Lion converges at the optimal stochastic rate $O(d^{1/2}T^{-1/4})$ without coerciveness, and that variance reduction and distributed sign compression improve or match these rates.
desk verdict The centralized Lion analysis is credible, but the two communication-efficient theorems are proved in the wrong norm and contain an unbounded-sign-oracle gap; the paper needs major revision before it can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pair of momentum estimators $v_t$ and $m_t$ and the sign-implicit update $x_{t+1}=x_t-\eta(\operatorname{sign}(v_t)+\lambda x_t)$. Lemma 1 shows this update is self-bounding: with $\|x_1\|_\infty \le \eta$ and $\lambda \le 1/(2\eta T)$, every iterate satisfies $\|x_t\|_\infty \le \eta t$ and $\|x_{t+1}-x_t\|^2 \le 4\eta^2 d$, which removes the coerciveness assumption. The proof then reduces the average $\ell^1$ gradient norm to the estimator error $\mathbb{E}\|v_t-\nabla f(x_t)\|^2$, controlled either by tuning $\beta_1, \beta_2$ or by a recursive variance-reduced momentum correction. In the communication-efficient theorems, the unbiased sign operation $S_R(v)$ — a coordinatewise coin flip that rounds each coordinate to $\pm 1$ so that the expected outcome is $v/R$ — carries the two-way compression argument, at the price of requiring $\|v\|_\infty \le R$.
What would settle it
On a scalar problem satisfying Assumptions 3 and 9 with gradient bound $G$, run the Lion-VR recursion with $\beta_2 = T^{-1/2}$ using zero true gradient and mean-zero noise $\xi_t \in \{-G,G\}$ with alternating signs. The STORM update $m_t = (1-\beta_2)m_{t-1} + \beta_2 \xi_t + (1-\beta_2)(\xi_t - \xi_{t-1})$ has a worst-case fixed point larger than $G$ for small $\beta_2$, so $\|m_t\|_\infty$ exceeds $G$ and $S_G(m_t)$ is undefined. Recording the maximum $\|m_t\|_\infty$ over such a run settles whether the compression oracle used in Theorem 7 actually exists for the stated inputs.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that the sign-momentum structure of Lion is analyzable under standard assumptions, provided the update $x_{t+1}=x_t-\eta(\operatorname{sign}(v_t)+\lambda x_t)$ is recognized as self-bounding. Theorems 1 and 2 give $O(d^{1/2}T^{-1/4})$ and $O(d^{1/2}T^{-1/3})$ for Lion and Lion-VR; Theorems 3 and 4 give $O(d^{1/2}(nT)^{-1/4})$ and $O(d^{1/2}(nT)^{-1/3})$ in heterogeneous distributed settings; Theorems 5--7 show that two-way sign compression with unbiased sign operations still drives the average $\ell^1$ gradient norm to zero at $O(\max\{d^{1/4}T^{-1/4}, d^{1/10}n^{-1/5}T^{-1/5}\})$, or $O(d^{1/4}T^{-1/4})$ with variance reduction. The object doing the work is the pair of momentum estimators $v_t, m_t$ plus the randomized sign oracle $S_R$, whose unbiasedness converts coordinatewise compression into a controllable error term.
Load-bearing premise
The proof of the two-way sign-compressed theorems assumes that every coordinate of the vector fed to the unbiased sign operator lies in $[-G,G]$; for the variance-reduced momentum the paper does not establish this, and with $\beta_2 = O(T^{-1/2})$ the momentum can in principle grow well past $G$.
Editorial extensions
If this is right
- Lion can be used in nonconvex stochastic problems under standard assumptions; the coerciveness condition in earlier analyses is unnecessary.
- The variance-reduced Lion-VR converges at $O(d^{1/2}T^{-1/3})$ under average smoothness, matching the known lower bound for that setting.
- Distributed Lion with $n$ nodes achieves centralized-style rates with iteration count replaced by $nT$, so adding nodes speeds convergence in the aggregate.
- With unbiased sign compression in both directions, the communication-efficient variant keeps the gradient norm driving to zero as $T$ grows rather than plateauing.
- The two-way compressed rates $O(\max\{d^{1/4}T^{-1/4}, d^{1/10}n^{-1/5}T^{-1/5}\})$ and $O(d^{1/4}T^{-1/4})$ give concrete bandwidth-versus-iteration trade-offs for sign-based distributed training.
Reading between the lines
- We infer that the proof template transfers to other sign-based optimizers: any update of the form sign(momentum) can inherit these rates if the iterate bound of Lemma 1 and the estimator-error recursion hold.
- We infer that the initialization condition $\|x_1\|_\infty \le \eta$ couples the initial scale to the learning rate, which could be restrictive in practice; rescaling the initialization or adding a projection is a natural extension not tested in the paper.
- We infer a concrete fix for the unbiased-sign domain issue: clip each $v_j^t$ coordinate to $[-G,G]$ before applying $S_G$, then check whether the rates survive; this is a testable modification of the two-way compressed algorithm.
- We infer that the $\ell^1$ stationarity metric used throughout is weaker than the more common $\ell^2$ metric, so the rates do not directly imply comparable $\ell^2$ gradient convergence.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes the Lion optimizer in centralized and distributed settings. In the centralized case, it proves an O(d^{1/2}T^{-1/4}) rate for the standard Lion update under L-smoothness and bounded noise, and an O(d^{1/2}T^{-1/3}) rate for a STORM-based variance-reduced variant under average smoothness. These results are extended to a heterogeneous distributed setting, with rates O(d^{1/2}(nT)^{-1/4}) and O(d^{1/2}(nT)^{-1/3}). The final part of the paper proposes communication-efficient variants that compress both the uplink and downlink by sign operations, including an 'unbiased sign' operation, and claims rates of order O(max{d^{1/4}/T^{1/4}, d^{1/10}/(n^{1/5}T^{1/5})}) and O(d^{1/4}/T^{1/4}). The centralized proofs are largely self-contained, but the communication-efficient theorems contain norm mismatches and a miscomputed variance, and the unbiased sign operation is applied to vectors whose boundedness is not established.
Significance. If all the claimed rates were established, the paper would be a useful contribution: it would provide standard-assumption convergence guarantees for Lion without coerciveness, extend them to heterogeneous distributed systems, and give two-way sign-compressed variants with explicit dimension and node dependence. The centralized Theorems 1 and 2 appear to be the strongest and most defensible parts, and the paper is honest about its assumptions and prior work. However, the communication-efficient results (Theorems 5-7) are advertised in the abstract and introduction as main contributions, and those proofs do not, as written, establish the stated l1-norm rates. Because the main new selling points are precisely the unbiased-sign compression rates, the paper cannot be accepted in its present form.
major comments (4)
- [Appendices G and H; Theorems 6 and 7] There is a norm mismatch between the proofs and the theorem statements. The descent inequality in Appendix G, Eq. (15), has a negative term proportional to E||grad f(x_t)||^2, and the final displayed bound is on (1/T) sum E||grad f(x_t)||, i.e., the Euclidean norm. Theorems 6 and 7, however, claim bounds on (1/T) sum E||grad f(x_t)||_1. Since ||g||_1 <= sqrt(d) ||g||_2, the proved l2 bound cannot be converted into the claimed l1 rate; doing so would multiply the dimension exponent by sqrt(d), turning d^{1/4}/T^{1/4} into d^{3/4}/T^{1/4} and d^{1/10}/(n^{1/5}T^{1/5}) into d^{3/5}/(n^{1/5}T^{1/5}) in Theorem 6, and d^{1/4}/T^{1/4} into d^{3/4}/T^{1/4} in Theorem 7. The proofs therefore do not support the advertised communication-efficient rates.
- [Definition 1; Theorems 6 and 7] The unbiased sign operation S_R is only defined for vectors v with ||v||_inf <= R, but Theorems 6 and 7 apply S_G to v_j^t. Assumption 9 only bounds the stochastic gradients themselves, not the momentum vectors. Under the STORM update used in the v2 variants, m_j^t = (1-beta2)(m_{j,t-1} - g_{j,t-1}) + g_{j,t} and v_j^t = (1-beta1)(m_{j,t-1} - g_{j,t-1}) + g_{j,t}; since m_{j,1} is a batch average rather than g_{j,1}, the difference m_{j,t-1} - g_{j,t-1} is not zero, so ||v_j^t||_inf can exceed G (for example, it can be as large as 3G in sup norm). No lemma in the paper proves ||v_j^t||_inf <= G for all t, so S_G may be applied to vectors outside its domain of definition.
- [Appendix F, Eq. (14)] The variance of the averaged unbiased sign vectors is miscalculated. After bounding E||(1/n) sum_j (S_G(v_j^t) - v_j^t/G)||, the proof upper-bounds the resulting square-root term by sqrt(d) sqrt(n). For n independent vectors whose coordinates have variance at most 1, E||(1/n) sum_j (S_G(v_j^t) - v_j^t/G)||^2 <= d/n, so the correct upper bound is sqrt(d/n), not sqrt(dn). As written, the proof yields a term of order d sqrt(n) in Theorem 5 rather than the claimed d/sqrt(n). The factor error is fixable without changing the stated rates, but the proof as written is internally inconsistent.
- [Theorem 6; parameter conditions] The momentum conditions beta2^2 <= beta1 <= sqrt(beta2) force beta2 <= 1. In Theorem 6, the stated choice beta2 = O(n^{1/3} eta^{2/3} d^{1/3}) can violate this when eta is in the second branch eta = O(n^{2/5}/(T^{3/5} d^{1/5})): for T = O(n) and d large, beta2 can be polynomially larger than 1. The theorem should state an explicit lower bound on T, or another restriction, that guarantees beta2 <= 1; as stated, the parameter regime is not fully specified.
minor comments (4)
- [Algorithms 2 and 3] The pseudocode indentation places the server-side update x_{t+1} = x_t - eta(sign(v_t) + lambda x_t) inside the loop over nodes j. This update should be performed once after all v_j^t have been collected and averaged.
- [Lemma 1 and Theorems 1-4] The initialization condition ||x_1||_inf <= eta is quite restrictive because eta is horizon-dependent (for example, eta = O(d^{-1/2}T^{-3/4}) in Theorem 1). This should be discussed explicitly, since it is not a standard assumption and it weakens the claim that the analysis avoids auxiliary conditions.
- [Appendix F] In the sentence following Eq. (14), 'forth inequality' should be 'fourth inequality', and the notation for the norm of the averaged sign vectors is inconsistent with the rest of the appendix.
- [Appendix H] In the bound for E||(1/n) sum_j v_j^{t+1} - grad f(x_{t+1})||^2, the term 8L^2 eta^2 d/beta1 appears without the factor 1/n that appears in the analogous derivation in Appendix E. This is likely a typo, but it should be corrected for consistency.
Circularity Check
No circularity found: the convergence proofs are self-contained and the stated rates follow from the stated assumptions by direct algebra; self-citations are background only.
full rationale
I walked the derivation chain of Theorems 1–7 and found no step in which a claimed conclusion is used as an input, nor any fitted parameter renamed as a prediction. The centralized analysis (Appendix B) reduces the l1 gradient norm to a momentum-error term, bounds that term by a self-contained recursion, and obtains O(d^{1/2}/T^{1/4}) by substituting the stated β2 and η; the variance-reduced version (Appendix C) repeats this structure with a STORM recursion. The distributed theorems (Appendices D and E) are derived from the same self-contained machinery, and the communication-efficient proofs (Appendices F–H) use the definition of the unbiased sign operation plus the paper's own momentum-error bounds. The self-citations to Jiang et al. 2024/2025 appear only as related-work context and as motivation for using momentum or variance reduction; no load-bearing theorem is imported from those papers to force the central rates. The paper is therefore not circular. I note, outside the circularity question, that Appendices G and H prove bounds on the Euclidean norm while Theorems 6 and 7 state l1-norm rates, and that the S_G oracle range is not proved for the STORM-corrected momentum in Theorem 7; these are correctness gaps, not circular reductions.
Assumptions & free parameters
free parameters (6)
- beta1 =
O(1) with beta2^2 <= beta1 <= sqrt(beta2)
- beta2 =
O(T^{-1/2}) in Theorem 1, O(T^{-2/3}) in Theorem 2, etc.
- eta =
O(d^{-1/2}T^{-3/4}) or O(d^{-1/2}T^{-2/3}) depending on theorem
- lambda =
<= 1/(2 eta T)
- B0 =
O(T^{1/3}) in Theorem 2, O(n^{-2/3}T^{1/3}) in Theorem 4
- G =
Assumed finite bound
assumptions (6)
- domain assumption L-smoothness of the objective and average smoothness of stochastic gradients (Assumptions 1, 2, 5, 6).
- domain assumption Unbiased stochastic gradients with bounded variance (Assumptions 3 and 7).
- domain assumption The objective is bounded below and the initial function gap is finite (Assumptions 4 and 8).
- domain assumption Bounded gradients in l_inf norm (Assumption 9).
- ad hoc to paper Lemma 1's initialization condition ||x1||_inf <= eta and lambda <= 1/(2 eta T).
- standard math The fresh-sample independence used to set cross terms to zero in the estimator error recursions.
Cite this review
Pith. "Pith review of Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings." pith.science (2026). https://pith.science/paper/IMDQZIJI
@misc{pith2026250812327,
author = {Pith},
title = {Pith review of: Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings},
year = {2026},
howpublished = {\url{https://pith.science/paper/IMDQZIJI}},
note = {Machine review of arXiv:2508.12327}
}
abstract
In this paper, we analyze the convergence properties of the Lion optimizer. First, we establish that the Lion optimizer attains a convergence rate of $\mathcal{O}(d^{1/2}T^{-1/4})$ under standard assumptions, where $d$ denotes the problem dimension and $T$ is the iteration number. To further improve this rate, we introduce the Lion optimizer with variance reduction, resulting in an enhanced convergence rate of $\mathcal{O}(d^{1/2}T^{-1/3})$. We then analyze in distributed settings, where the standard and variance reduced version of the distributed Lion can obtain the convergence rates of $\mathcal{O}(d^{1/2}(nT)^{-1/4})$ and $\mathcal{O}(d^{1/2}(nT)^{-1/3})$, with $n$ denoting the number of nodes. Furthermore, we investigate a communication-efficient variant of the distributed Lion that ensures sign compression in both communication directions. By employing the unbiased sign operations, the proposed Lion variant and its variance reduction counterpart, achieve convergence rates of $\mathcal{O}\left( \max \left\{\frac{d^{1/4}}{T^{1/4}}, \frac{d^{1/10}}{n^{1/5}T^{1/5}} \right\} \right)$ and $\mathcal{O}\left( \frac{d^{1/4}}{T^{1/4}} \right)$, respectively.
Reference graph
Works this paper leans on
-
[1]
Arjevani, Y
Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. E. Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199 0 (1-2): 0 165–214, 2023
2023
-
[2]
Bernstein, Y.-X
J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar. sign SGD : Compressed optimisation for non-convex problems. In Proceedings of the 35th International Conference on Machine Learning, pages 560--569, 2018
2018
-
[3]
Bernstein, J
J. Bernstein, J. Zhao, K. Azizzadenesheli, and A. Anandkumar. sign SGD with majority vote is communication efficient and fault tolerant. In International Conference on Learning Representations, 2019
2019
-
[4]
L. Chen, B. Liu, K. Liang, and qiang liu. Lion secretly solves a constrained optimization: As lyapunov predicts. In The Twelfth International Conference on Learning Representations, 2024
work page 2024
-
[5]
X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C.-J. Hsieh, Y. Lu, and Q. V. Le. Symbolic discovery of optimization algorithms. In Advances in Neural Information Processing Systems 37, 2023
work page 2023
-
[6]
Cutkosky and F
A. Cutkosky and F. Orabona. Momentum-based variance reduction in non-convex SGD . In Advances in Neural Information Processing Systems 32, pages 15210--15219, 2019
2019
-
[7]
Y. Dong, H. Li, and Z. Lin. Convergence rate analysis of lion. ArXiv e-prints, arXiv:2411.07724, 2024
arXiv 2024
-
[8]
C. Fang, C. J. Li, Z. Lin, and T. Zhang. SPIDER : Near-optimal non-convex optimization via stochastic path-integrated differential estimator. In Advances in Neural Information Processing Systems 31, pages 689--699, 2018
2018
Show all 26 references
-
[9]
Ghadimi and G
S. Ghadimi and G. Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23 0 (4): 0 2341--2368, 2013
2013
-
[10]
Ishikawa, T
S. Ishikawa, T. Ben-Nun, B. V. Essen, R. Yokota, and N. Dryden. Lion cub: Minimizing communication overhead in distributed lion. ArXiv e-prints, arXiv:2411.16462, 2025
2025 arXiv
-
[11]
Jiang, S
W. Jiang, S. Yang, W. Yang, and L. Zhang. Efficient sign-based optimization: Accelerating convergence via variance reduction. In Advances in Neural Information Processing Systems 38, pages 33891--33932, 2024
2024
-
[12]
Jiang, D
W. Jiang, D. Yu, S. Yang, W. Yang, and L. Zhang. Improved analysis for sign-based methods with momentum updates. ArXiv e-prints, arXiv:2507.12091, 2025
2025 arXiv
-
[13]
R. Jin, Y. Huang, X. He, H. Dai, and T. Wu. Stochastic- S ign SGD for federated learning with theoretical guarantees. ArXiv e-prints, arXiv:2002.10940, 2021
2002 arXiv
-
[14]
Johnson and T
R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems 26, pages 315--323, 2013
2013
-
[15]
K. Y. Levy, A. Kavis, and V. Cevher. STORM +: Fully adaptive SGD with recursive momentum for nonconvex optimization. In Advances in Neural Information Processing Systems 34, 2021
2021
-
[16]
Z. Li, H. Bao, X. Zhang, and P. Richtarik. Page: A simple and optimal probabilistic gradient estimator for nonconvex optimization. In Proceedings of the 38th International Conference on Machine Learning, pages 6286--6295, 2021
2021
-
[17]
B. Liu, L. Wu, L. Chen, K. Liang, J. Zhu, C. Liang, R. Krishnamoorthi, and qiang liu. Communication efficient distributed training with distributed lion. In Advances in Neural Information Processing Systems 38, 2024
2024
-
[18]
Y. Liu, Y. Gao, and W. Yin. An improved analysis of stochastic gradient descent with momentum. In Advances in Neural Information Processing Systems 34, 2020
2020
-
[19]
L. M. Nguyen, J. Liu, K. Scheinberg, and M. Tak a c . SARAH: A novel method for machine learning problems using stochastic recursive gradient. In Proceedings of the 34th International Conference on Machine Learning, pages 2613--2621, 2017
2017
-
[20]
N. L. Roux, M. Schmidt, and F. R. Bach. A stochastic gradient method with an exponential convergence rate for finite training sets. In Advances in Neural Information Processing Systems 25, pages 2672--2680, 2012
2012
-
[21]
Sfyraki and J.-K
M.-E. Sfyraki and J.-K. Wang. Lions and muons: Optimization via stochastic frank-wolfe. ArXiv e-prints, arXiv:2506.04192, 2025
2025 arXiv
-
[22]
T. Sun, Q. Wang, D. Li, and B. Wang. Momentum ensures convergence of SIGNSGD under weaker assumptions. In Proceedings of the 40th International Conference on Machine Learning, pages 33077--33099, 2023
2023
-
[23]
Tang and T.-H
Z. Tang and T.-H. Chang. Fedlion: Faster adaptive federated optimization with fewer communication. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 13316--13320, 2024
2024
-
[24]
Wang, C.-H
J.-K. Wang, C.-H. Lin, and J. Abernethy. Escaping saddle points faster with stochastic momentum. In International Conference on Learning Representations, 2020
2020
-
[25]
Z. Wang, K. Ji, Y. Zhou, Y. Liang, and V. Tarokh. SpiderBoost and momentum: Faster variance reduction algorithms. In Advances in Neural Information Processing Systems 32, pages 2406--2416, 2019
2019
-
[26]
Zhang, M
L. Zhang, M. Mahdavi, and R. Jin. Linear convergence with condition number independent access of full gradients. In Advances in Neural Information Processing Systems 26, pages 980--988, 2013
2013
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.