Pith. sign in

REVIEW 3 major objections 5 minor 25 references

LionVote: Per-Layer Learning Rate Adaptation for Lion

T0 review · 3 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read At Lion's prescribed learning rate on ViT-Tiny, attention and MLP layers run 2.6–2.8× too hot and normalisation layers about 2× too hot—a 32% gap no single global rate can fix.

desk verdict Solid methods paper: derived per-layer voting for Lion plus a real (if mechanism-tied) layer-type miscalibration signal on small ViTs; gains are small, scope is narrow, but the work is careful enough to read and referee. read the letter →

arxiv 2607.09266 v1 pith:ZZTTWZ77 submitted 2026-07-10 cs.LG

Kris Atallah (New York University , New York , USA) This is my paper
classification cs.LG
keywords Lionoptimizerper-layerlearningratesvisiontransformersadaptiveoptimizersgradientdirectionstabilitymomentumhealthcompoundlevelsign-basedoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that Lion's single global learning rate systematically mis-scales different layer types. On a small vision transformer trained on CIFAR-100, per-layer compound levels that the method itself discovers show attention and MLP parameters receive an effective scale 2.6–2.8 times too high, while normalisation layers sit about 2 times too high—a 32% cross-type disparity that no uniform rate can reproduce. LionVote turns that measurement into a mechanism: each parameter tensor keeps a persistent integer level updated every few epochs by two diagnostics (gradient-direction stability and momentum health) that a validation-loss tiebreaker resolves when they conflict. The voting thresholds are derived from geometric identities, the EMA time constant, and a noise-floor estimate rather than grid search. On the same ViT/CIFAR-100 workload the method reaches 69.7% top-1 accuracy against Lion's 69.0% (statistically significant over eight seeds) and AdamW's 68.8%. The value of the adaptation is not universal: it tracks architectural heterogeneity and is task-dependent, and on uniform residual CNNs tuned SGD with cosine annealing remains strongest.

What carries the argument

The compound level: a persistent integer per parameter tensor that multiplies the base learning rate by exp(s · β1/2). It is updated every c epochs by two derived votes (gradient-direction cosine stability and momentum-to-gradient norm health) resolved by a validation-loss tiebreaker, with an asymmetric reset rule that revokes accumulated level on a single opposing vote.

What would settle it

On the same ViT-Tiny/CIFAR-100 setup, run a static per-layer-type schedule that freezes the epoch-300 multipliers from the first step (or a global rate lowered to match the average effective rate) and check whether either closes the accuracy gap to LionVote; the paper's own controls already show they do not, so a successful reverse would falsify the dynamic-adaptation claim.

Watch

Extended reading notes

Core claim

At Lion's prescribed learning rate on ViT-Tiny/CIFAR-100, the effective scale is 2.6–2.8× too high for attention and MLP parameters and roughly 2× too high for normalisation layers—a 32% cross-layer-type disparity that a single global rate cannot reproduce. LionVote's per-layer compound-level trajectories measure that miscalibration and convert it into a statistically significant accuracy gain (69.71% vs Lion 68.95%, p=0.017 over eight seeds).

Load-bearing premise

The headline 2.6–2.8× miscalibration factors rest on the compound levels that LionVote's own voting loop produces; if those levels are an artifact of the vote rules, cadence, and tiebreaker rather than a true property of Lion, the quantitative claim collapses.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LionVote, a stateful per-layer learning-rate mechanism for the Lion optimizer. Each parameter tensor holds an integer compound level that multiplies the base LR (and decoupled weight decay) by exp(s_i * β_1/2); levels are updated every c epochs by two diagnostics—epoch-mean gradient cosine alignment and momentum-to-gradient norm ratio—resolved by a global validation-loss tiebreaker when the local votes conflict or abstain. Thresholds for the two votes are derived from Sheppard’s sign-agreement identity, the EMA time-constant identity β^τ_2 = e^{-1}, and a noise-floor argument for cross-entropy; cadence and max level are structurally bounded and then ablated. On ViT-Tiny/CIFAR-100 (8 seeds) LionVote at c=8 reaches 69.71 % vs Lion 68.95 % (p=0.017) and AdamW 68.75 %; the same architecture on CIFAR-10 and both WideResNet configurations show no consistent gain, and tuned SGD remains dominant on the CNNs. Compound-level trajectories are used to claim that Lion’s prescribed rate is 2.6–2.8 imes too high for attention/MLP and ~2 imes too high for normalisation (32 % cross-type disparity). Ablations (v1-only, v2-only, no-tiebreaker, symmetric update) and controls (lower global LR, static type multipliers, WD sweep) are reported.

Significance. If the central claims hold, the work supplies three reusable pieces: (i) a derivation methodology for per-layer voting thresholds that is largely free of free parameters (Appendix A.1–A.6), (ii) a quantified, architecture-aware miscalibration statement for a sign-based optimizer that complements Zhao et al.’s qualitative observation on language models, and (iii) evidence that the value of per-layer adaptation tracks architectural heterogeneity and is task-dependent. Strengths that raise the bar relative to typical optimizer papers include 8-seed replication with Welch tests, explicit ablations of each vote and of the update rule, and control experiments that rule out simple global LR or WD fixes and static type multipliers. The portability analysis (Table 18) that separates optimizer-agnostic from Lion-specific components is also useful for follow-on work.

major comments (3)
  1. The headline numerical claim in the abstract and §5.1 (Table 3)—that Lion’s effective scale is 2.6–2.8 imes too high for attention/MLP and ~2 imes too high for normalisation—is produced by LionVote’s own compound-level trajectories (asymmetric reset, cadence-c voting, Vote-1/2 thresholds, and the validation-loss tiebreaker). Appendix B.16 shows that raw Lion gradient cosines already exhibit the qualitative attn/mlp < norm ordering, and Appendix B.15 shows that static type multipliers and a uniformly lower global LR fail to recover the accuracy gain; those controls establish that the numbers cannot be replaced by a constant schedule, but they do not establish that a different adaptive rule would settle at the same 0.36–0.50 multipliers. Because the compound multiplier also scales weight decay, the joint LR+WD interpretation inherits the same dependence. The abstract and §5.1 should theref
  2. Cadence c=8 is selected by ablation from {4,6,8} (§4.2, §5.3, Appendix A.6, B.3). The primary accuracy comparison on ViT-Tiny/CIFAR-100 is therefore partially post-hoc. The paper correctly notes that the c=8 vs Lion p-value (0.017) is marginal after Holm–Bonferroni and that the control comparisons (lower LR, static multipliers) are the stronger evidence; those controls should be elevated to the main text and the abstract should not present the 69.7 % figure as if cadence were fixed a priori.
  3. The validation-loss tiebreaker is the dominant decision path (~59 % of per-parameter votes, Appendix B.6) and is computed on the same validation set used for reporting. §5.3 flags the issue and suggests a held-out split for future work; given that the entire accuracy gain on ViT/C100 is attributed to the tiebreaker (Table 2: full system vs notie +1.07 pp, p=0.003), a held-out confirmation (or an explicit statement that the reported numbers use the training-time validation signal) is load-bearing for the claim that the mechanism improves generalisation rather than merely fitting the monitored split.
minor comments (5)
  1. Table 1 reports LionVote (c8) 69.71 % while the abstract rounds to 69.7 % and Lion to 69.0 %; keep one consistent rounding convention.
  2. Figure 2 caption says “~33 % difference in effective scale” while the text and Table 3 use 32 %; align the numbers.
  3. Appendix A.1 Remark notes that the coordinate-isotropy assumption is approximate; a short sentence in the main text (§3.3) would help readers who do not read the appendix.
  4. The symmetric-update variant reaches the study’s highest single accuracy (70.54 %) but is tested on only 2 of 4 configurations; either expand or clearly mark it as exploratory in the abstract/conclusion.
  5. Typographical consistency: “normalisation” vs “normalization”, “V ote” spacing artefacts in several places, and “LionV ote” vs “LionVote”.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline 2.6–2.8× / ~2× miscalibration factors are the inverse of LionVote’s own settled compound multipliers, so the quantification is mechanism-dependent by construction even though qualitative ordering has independent support.

  1. self definitional [Abstract; §5.1; Table 3; contribution 2]
    "Per-layer diagnostics reveal that, at the prescribed learning rate, Lion’s effective scale is 2.6–2.8× too high for attention and MLP parameters and ∼2× too high for normalisation layers on ViT-Tiny/CIFAR-100; this 32% cross-layer-type disparity cannot be reproduced by a single global rate. The measurement comes from LionVote... By epoch 300 (Table 3), Lion’s effective scale is 2.6× too high for attention parameters, 2.8× too high for MLP, and 2.0× too high for normalisation. ... The effective LR multiplier at epoch 300 is exp(s·0.9/2)"

    The reported factors are exactly 1/exp(s_i · β1/2) where the s_i are the compound levels produced by LionVote’s own voting loop (asymmetric reset, Votes 1/2, validation-loss tiebreaker, cadence c) on the same runs. Thus the numerical claim that Lion’s prescribed rate is “X× too high” is defined as the inverse of the rates the adaptive mechanism itself settled on; it is not an external measurement of Lion. Raw diagnostics recover only ordering, not these magnitudes.

full rationale

The paper’s central quantitative claim about Lion (abstract, contribution 2, §5.1, Table 3) is obtained by running LionVote itself and reading off the inverse of the compound-level multipliers exp(s·β1/2) that the voting state machine produces. This is a mild self-definitional loop for the numerical factors: the “how much too high” numbers are exactly the rates the adaptive rule chose under its own thresholds, asymmetric reset, cadence, and validation-loss tiebreaker. The paper is otherwise clean: voting thresholds are derived from geometric identities (Sheppard, EMA time-constant, noise-floor order-of-magnitude) without fitting to the accuracy numbers; raw Lion diagnostics (App. B.16) independently recover only the qualitative attn/mlp < norm ordering; and static-type / lower-global-LR controls (App. B.15) show those particular numbers cannot be replaced by a constant schedule. No self-citation chain, no uniqueness theorem imported from the authors, and no ansatz smuggled via citation. The circularity is therefore limited to the measurement of the headline scale factors, not to the existence of layer-type differentiation or to the accuracy gain itself. Score 4 reflects that the strongest scientific number remains an output of the mechanism being evaluated, while the rest of the derivation chain is self-contained.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

Most voting thresholds are pinned by standard math identities (cosine sign, Sheppard, EMA τ, L-smooth sign descent) rather than free fits; the main free choices are cadence, max level, update-rule asymmetry, and the 1% loss noise floor order-of-magnitude. Invented machinery is the compound-level state machine and the LionVote voting loop itself. Domain assumptions include coordinate isotropy for the 2/3 sign-agreement claim and that validation loss during training is an acceptable global signal.

free parameters (5)
  • voting cadence c = 8 (ablated in {4,6,8})
    Structurally bounded to roughly [4,10] then selected by ablation; c=8 is the reported best on ViT/C100 and drives the headline comparison.
  • max compound level L = 4
    Chosen as operating point balancing LR range vs cosh degradation; L=3 also called defensible.
  • LR exponent divisor d = 2
    Defended via per-level cosh cost table; any d in [1.5,3] admitted as defensible; d=2 is simplest integer.
  • tiebreaker relative threshold = 0.01
    Order-of-magnitude noise floor for n≈10^4 validation sets; not uniquely determined.
  • asymmetric vs symmetric level update = asymmetric (default)
    Design choice; symmetric can beat asymmetric on ViT/C100 but was only tested on 2/4 configs.
assumptions (5)
  • domain assumption Coordinate isotropy: per-coordinate gradient pairs behave as bivariate Gaussians with correlation equal to global cosine, so Sheppard's formula maps c=0.5 to exactly 2/3 sign agreement.
    App. A.1 Theorem A.2; paper notes gross violations make 2/3 approximate.
  • standard math EMA time-constant identity β2^τ = e^{-1} justifies Vote 2 thresholds at 1/e and e.
    Lemma A.3 / Corollary A.5; standard first-order filter identity.
  • standard math Coordinate-wise L-smoothness for sign updates yields B(α*e^ε)/B(α*)=cosh(ε), used to set divisor 2 and bound L.
    Proposition A.6 citing Bernstein et al. signSGD framework.
  • domain assumption Per-sample cross-entropy CV_ℓ is Θ(1), so relative SE at n≈10^4 is ~1%.
    App. A.3; sets tiebreaker scale.
  • ad hoc to paper Using validation loss during training (~37 decisions at c=8) as the global tiebreaker is acceptable for the reported claims.
    §3.3 and Limitations §5.3; authors flag held-out confirmation as future work.
invented entities (2)
  • compound level s_i (persistent integer per parameter tensor)
    purpose: Stateful exponential modulation of Lion's base LR and decoupled weight decay.
    Core state variable of LionVote; no independent existence outside the method.
  • LionVote two-vote + validation-loss resolution rule
    purpose: Map gradient-stability and momentum-health diagnostics into level updates.
    The paper's proposed mechanism; portability table separates Lion-specific vs agnostic pieces but entity is new.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LionVote: Per-Layer Learning Rate Adaptation for Lion." pith.science (2026). https://pith.science/paper/ZZTTWZ77

@misc{pith2026260709266,
  author       = {Pith},
  title        = {Pith review of: LionVote: Per-Layer Learning Rate Adaptation for Lion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZTTWZ77}},
  note         = {Machine review of arXiv:2607.09266}
}
read the original abstract

Per-layer diagnostics reveal that, at the prescribed learning rate, Lion's effective scale is 2.6-2.8x too high for attention and MLP parameters and ~2x too high for normalization layers on ViT-Tiny/CIFAR-100; this 32% cross-layer-type disparity cannot be reproduced by a single global rate. The measurement comes from LionVote, a per-layer learning rate mechanism in which each parameter tensor maintains a compound level, a persistent integer updated every c epochs by two diagnostics (gradient direction stability and momentum health) resolved by a validation loss tiebreaker. Voting thresholds derive from geometric identities, the EMA time constant, and a noise-floor estimate; cadence is bounded structurally and selected by ablation. On ViT-Tiny/CIFAR-100, LionVote achieves 69.7% top-1 accuracy vs. Lion's 69.0% (p < 0.02, Welch's t-test) and AdamW's 68.8%. Per-layer adaptation value depends on both architectural heterogeneity and task; on uniform CNN architectures tuned SGD with cosine annealing remains dominant, and on ViT architectures gains are task-dependent.

Figures

Figures reproduced from arXiv: 2607.09266 by the authors.

Figure 1
Figure 1. Validation accuracy vs. epoch for all four configurations (mean [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Compound level trajectories for all 152 parameters of ViT-Tiny on CIFAR-100 (cadence 8, 8-seed average). [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Effective LR+WD fingerprint for LionVote (cadence 8) on ViT-Tiny/CIFAR-100 at epoch 296 (8-seed [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Stacked area plot showing the fraction of parameters at each compound level over training (ViT-Tiny/CIFAR [PITH_FULL_IMAGE:figures/full_fig_p025_4.png]
Figure 5
Figure 5. Figure 5: Mean compound level for attention (solid/dashed) and LayerNorm (dotted) parameters across cadences [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Effective learning rate (log scale) for all configurations and optimizers. Bands for LionVote show [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: Training loss for all configurations (mean [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: Validation loss for all configurations (mean [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]
Figure 9
Figure 9. Figure 9: Generalisation gap (validation loss − training loss) for all configurations (mean ±1 std, 8 seeds). On WRN, SGD maintains the smallest gap, indicating less overfitting. On ViT-Tiny, all methods show negative gaps (validation loss < training loss), a consequence of Mixu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 2 linked inside Pith

  1. [1]

    W., Pfau, D., Schaul, T., Shillingford, B., and de Freitas, N

    Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., and de Freitas, N. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), volume 29, 2016

    2016

  2. [2]

    signSGD : Compressed optimisation for non-convex problems

    Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signSGD : Compressed optimisation for non-convex problems. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pages 560--569. PMLR, 2018

    2018

  3. [3]

    Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., and Le, Q. V. Symbolic discovery of optimization algorithms. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023

    2023

  4. [4]

    Z., and Talwalkar, A

    Cohen, J., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations (ICLR), 2021

    2021

  5. [5]

    and Mishchenko, K

    Defazio, A. and Mishchenko, K. Learning-rate-free learning by D-Adaptation . In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202, pages 7449--7479. PMLR, 2023

    2023

  6. [6]

    The road less scheduled

    Defazio, A., Yang, X., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A. The road less scheduled. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024

    2024

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021

    2021

  8. [8]

    Adaptive subgradient methods for online learning and stochastic optimization

    Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121--2159, 2011

    2011

Show all 25 references
  1. [9]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770--778, 2016

    2016

  2. [10]

    Noise-adaptive layerwise learning rates: Accelerating geometry-aware optimization for deep neural network training

    Hao, J., Gong, X., Xu, J., Wang, Z., and Liu, M. Noise-adaptive layerwise learning rates: Accelerating geometry-aware optimization for deep neural network training. arXiv preprint arXiv:2510.14009, 2025

    2025

  3. [11]

    and Ruder, S

    Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 328--339, 2018

    2018

  4. [12]

    Muon: An optimizer for hidden layers in neural networks

    Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks. Blog post, 2024

    2024

  5. [13]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015

    2015

  6. [14]

    Cautious optimizers: Improving training with one line of code

    Liang, K., Chen, L., Liu, B., and Liu, Q. Cautious optimizers: Improving training with one line of code. In International Conference on Learning Representations (ICLR), 2026

    2026

  7. [15]

    and Hutter, F

    Loshchilov, I. and Hutter, F. SGDR : Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017

    2017

  8. [16]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019

    2019

  9. [17]

    PyTorch : An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., K \"o pf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch : An i...

    2019

  10. [18]

    and Xiao, L

    Orvieto, A. and Xiao, L. An adaptive stochastic gradient method with non-negative G auss- N ewton stepsizes. arXiv preprint arXiv:2407.04358, 2024

    2024 arXiv

  11. [19]

    M., Schneider, F., and Hennig, P

    Schmidt, R. M., Schneider, F., and Hennig, P. Descending through a crowded valley --- benchmarking deep learning optimizers. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139, pages 9367--9376. PMLR, 2021

    2021

  12. [20]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017

    2017

  13. [21]

    AutoDrop : Training deep learning models with automatic learning rate drop

    Wang, J., Teng, Y., and Choromanska, A. AutoDrop : Training deep learning models with automatic learning rate drop. In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence (UAI), volume 244, pages 3603--3629. PMLR, 2024

    2024

  14. [22]

    Large batch training of convolutional networks

    You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017

    2017 arXiv

  15. [23]

    Large batch optimization for deep learning: Training BERT in 76 minutes

    You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations (ICLR), 2020

    2020

  16. [24]

    and Komodakis, N

    Zagoruyko, S. and Komodakis, N. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016

    2016

  17. [25]

    Deconstructing what makes a good optimizer for autoregressive language models

    Zhao, R., Morwani, D., Brandfonbrener, D., Vyas, N., and Kakade, S. Deconstructing what makes a good optimizer for autoregressive language models. In International Conference on Learning Representations (ICLR), 2025

    2025

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.