Pith. sign in

REVIEW 14 cited by

Symbolic Discovery of Optimization Algorithms

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2302.06675 v4 pith:RCEJQYLZ submitted 2023-02-13 cs.LG cs.AIcs.CLcs.CVcs.NE

Xiangning Chen , Chen Liang , Da Huang , Esteban Real , Kaiyuan Wang , Yao Liu , Hieu Pham , Xuanyi Dong
show 4 more authors
Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le
This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CVcs.NE
keywords lionadamtextbftrainingoptimizationprogramsearchtextit
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling Laws for Differentially Private Language Models

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Differentially private language models obey scaling laws in which compute-optimal models are roughly 10-50x smaller than non-private Chinchilla-optimal models, with large batch sizes and rapid saturation of compute.

  2. Causal Optimizer Interaction Calculus: Hidden Geometric Relaxation and Identifiable Interventions

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Under fixed innovation coupling, finite-horizon optimizers admit minimal pathwise realizations and incidence-identifiable Möbius effects, with a five-term readout transfer from hidden relaxation and a closed reduced-v...

  3. Explicit or Implicit? Encoding Physics at the Precision Frontier

    hep-ph 2026-03 conditional novelty 6.0 of 10

    On three precision classification tasks — reweighting-based unfolding, likelihood-ratio estimation, and weakly supervised anomaly detection — a Lorentz-equivariant transformer and a pretrained foundation model perform...

  4. Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation

    cs.CL 2025-12 conditional novelty 6.0 of 10

    Minimum Bayes Risk decoding with a soft F1 utility improves LLM error-span detection over standard MAP decoding on WMT24, and DPO distillation recovers the gains at greedy-search speed.

  5. How Should We Meta-Learn Reinforcement Learning Algorithms?

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A systematic comparison of black-box evolution, neural and symbolic distillation, and LLM-based proposal for meta-learning RL algorithms yields practical recommendations: warm-started LLM proposal is sample-efficient,...

  6. Simple Convergence Proof of Adam From a Sign-like Descent Perspective

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Adam's O(1/T^1/4) convergence is proven from a sign-like descent perspective, but the dimension-free claim depends on restrictive coordinate-wise assumptions.

  7. AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training

    cs.LG 2025-05 conditional novelty 6.0 of 10

    AdamS replaces AdamW's second-moment storage with a momentum-and-gradient squared denominator, matching AdamW's loss curves with half the optimizer memory.

  8. Spectral-factorized Positive-definite Curvature Learning for NN Training

    stat.ML 2025-02 conditional novelty 6.0 of 10

    The paper derives a Riemannian update rule for the spectral factors of a positive-definite preconditioner, making arbitrary matrix roots fast and numerically stable for low-precision NN training.

  9. Token Geometry

    cs.LG 2026-07 unverdicted novelty 5.5 of 10

    Ember matches Adam on embedding/LM-head tables with O(V+D) state by factoring second moments and dropping first-moment momentum, and token trajectories look nearly 1D.

  10. Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Symbolic regression over fixed-depth expressions found neural-network weight-update rules that beat tuned standard optimizers on 25 of 30 small benchmark/NN setups, but evaluation used the same data for selection and ...

  11. SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam

    cs.LG 2025-07 reject novelty 5.0 of 10

    S3, an optimizer with a p-th order momentum denominator, equal EMA coefficients, and Nesterov acceleration, is claimed to match AdamW's 100k-step perplexity at 50k steps while avoiding loss spikes.

  12. Xray-Visual Models: Scaling Vision models on Industry Scale Data

    cs.CV 2026-02 conditional novelty 4.0 of 10

    A 2-billion-parameter vision encoder trained on 15B+ image-text and billions of video-hashtag pairs reports SOTA ImageNet linear-probe, Kinetics, and retrieval numbers, but relies on proprietary data and has several v...

  13. Comparative Analysis of Lion and AdamW Optimizers for Cross-Encoder Reranking with MiniLM, GTE, and ModernBERT

    cs.IR 2025-06 reject novelty 4.0 of 10

    Lion and AdamW are compared for reranker fine-tuning, but the reported Lion gains are confounded by a 10x learning-rate difference and an inverted GPU-utilization metric.

  14. HyBDM: Multi-Scale Hybrid Experts for Time Series Forecasting with Bidirectional Dependency Modeling

    cs.LG 2026-07 conditional novelty 3.0 of 10

    HyBDM combines a Mamba-style global-pattern expert with a local window transformer and a learned router to forecast multivariate time series, reporting state-of-the-art results on six benchmarks.

Pith tools