Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Celo: Training Versatile Learned Optimizers on a Compute Diet

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Celo claims that a learned optimizer meta-trained for just 24 GPU hours can outperform tuned hand-designed and learned optimizers on 17 unseen tasks, thanks to task augmentation, a simple update rule, and a decoupled learned scheduler.

desk verdict Celo is a genuine step forward for compute-limited learned optimization, but the headline comparison needs confidence intervals and a fairer VeLO framing before I'd trust the 'outperforms' claim. read the letter →

arxiv 2501.12670 v2 pith:KBJWSTGT submitted 2025-01-22 cs.LG

Abhinav Moudgil , Boris Knyazev , Guillaume Lajoie , Eugene Belilovsky This is my paper · ORCID
classification cs.LG
keywords learnedoptimizersmeta-generalizationtaskaugmentationtwo-stagemeta-trainingIQMevaluationcompute-efficienttrainingschedulerlearningout-of-distributiontransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Learned optimizers are update rules discovered by meta-training, but the best-known ones, such as VeLO, required 4000 TPU-months to generalize. This paper claims that strong meta-generalization can be bought much more cheaply: its optimizer Celo is meta-trained for under 24 GPU hours on four tiny image-classification tasks and still transfers to 17 unseen tasks spanning MLPs, CNNs, vision transformers, language models, autoencoders, and a meta-learning problem. On that suite Celo reports a final-loss IQM of 1.20 and a speedup IQM of 1.86 relative to tuned Adam, beating both 15 hand-designed optimizers and four learned baselines. The recipe is three ingredients: task augmentation by re-parametrization, a simple per-parameter MLP update rule coupled with a global LSTM scheduler, and two-stage meta-training that learns the update rule before freezing it and learning the scheduler. If correct, this removes the main practical barrier to iterating on learned optimizers: compute.

What carries the argument

The central object is Celo, a hierarchical learned optimizer with two learned components: a per-parameter MLP update rule that maps per-parameter features to direction and magnitude, and a global LSTM scheduler that takes only task-agnostic features (normalized loss and training progress) and outputs a scalar step-size multiplier through an exponential map, $\eta_t = \alpha e^{o_t}$. The two components are meta-trained separately with truncated persistent evolution strategies (PES), a gradient estimator that avoids backpropagating through the inner loop: first the MLP alone, then the scheduler with the MLP frozen. Task augmentation re-parametrizes all inner-loop parameters by a per-iteration scalar $\tau$ sampled log-uniformly, simulating many more tasks from the four fixed meta-training tasks. The argument's engine is that decoupling step-size control from update direction prevents meta-overfitting in the low-compute regime, and the exponential scheduler lets one scalar adapt to unseen tasks and horizons.

What would settle it

Evaluate Celo, meta-trained with the paper's 24-hour budget, on a random sample or the full 83-task VeLOdrome pool at 10,000 iterations per task, and compare final-loss IQM against tuned Adam and NAdamW; if Celo's IQM advantage over the tuned baselines shrinks below noise or reverses, the claim that it outperforms tuned state-of-the-art optimizers would fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that strong meta-generalization in learned optimizers does not require massive meta-training: Celo, trained on four small image-classification tasks with 8x8 inputs for under 24 GPU hours, achieves final-loss IQM 1.20 and speedup IQM 1.86 over tuned Adam on a held-out suite of 17 tasks spanning MLPs, CNNs, ViTs, transformer and RNN language models, autoencoders, and a meta-learning task. On the same suite, the best prior learned optimizer baseline, Adafac MLP LOpt, scores 1.05 final-loss IQM and 1.30 speedup IQM. The paper argues that three ingredients carry this result: task augmentation by re-parametrization, a simple hierarchical design of a per-parameter MLP update rule plus a global LSTM scheduler, and decoupled two-stage meta-training that first learns the update rule, then freezes it and learns the scheduler.

Load-bearing premise

The load-bearing premise is that the 17-task, 2,000-iteration evaluation suite is a representative measure of out-of-distribution meta-generalization, since the tasks were picked from a larger pool to fit the compute budget and no random or diversity-based selection rule is reported.

Editorial extensions

If this is right

  • A learned optimizer with competitive meta-generalization can be meta-trained in under a day on a single GPU, making ablations and iterative design of learned optimizers affordable for research groups without large compute budgets.
  • Celo's per-parameter update costs 288 FLOPs per parameter, so its overhead relative to a forward-backward pass shrinks as batch size grows, making its runtime overhead small on large models.
  • The learned scheduler produces task- and horizon-dependent schedules (warmup, cyclic ramp-up, cosine-like decay) without manual tuning, suggesting learned step-size control transfers across unseen tasks.
  • The scheduler trained with a frozen learned update rule also improves a fixed Adam update rule (Celo-Adam) beyond tuned Adam on the 2K-step suite, indicating that the two-stage recipe isolates a reusable step-size policy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that its evaluation suite is a convenience sample of 17 tasks from a larger 83-task pool, with all comparisons made at 2,000 steps; whether Celo's margin survives randomized task selection, longer horizons, or larger models is an open empirical question.
  • Because the scheduler consumes only loss and progress features, it may be portable across optimizee architectures; a testable extension is to run Celo's scheduler on top of other hand-designed or learned update rules beyond Adam and measure transfer.
  • The paper's own AlgoPerf appendix shows full Celo is unstable on large benchmark workloads and Celo-Adam trails tuned baselines there, so scaling the update rule to production-size models remains the gap between the 24-hour result and off-the-shelf use.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Celo, a learned optimizer meta-trained on a fixed, small set of four image-classification tasks for roughly 24 GPU hours, and reports that it generalizes to 17 held-out tasks from the VeLOdrome suite. The main recipe has three components: task augmentation via random re-parametrization, a simple hierarchical architecture with a per-parameter MLP update rule plus a learned LSTM scheduler, and a two-stage training procedure where the update rule is trained first and then the scheduler is trained with the update rule frozen. The paper also adapts the IQM and optimality-gap metrics from Agarwal et al. (2021) to optimizer evaluation, defining normalized scores relative to a tuned Adam baseline. The central empirical claim is that Celo outperforms both hand-designed optimizers and previously published learned optimizers, including VeLO, on out-of-distribution tasks despite the small meta-training budget.

Significance. If the central claim holds, the paper would make an important contribution: it would show that compute-efficient meta-training can produce learned optimizers that transfer to diverse unseen tasks, substantially lowering the barrier compared to VeLO's 4000 TPU-month budget. The paper has several genuine strengths: the code is released, the meta-training protocol is controlled across all learned-optimizer baselines, the ablation study is systematic (task augmentation, scheduler form, two-stage training, tensor features), and the adoption of IQM/OG metrics is a sensible step toward robust aggregate evaluation in this area. The schedule analysis in Figure 6 is also suggestive evidence that the learned scheduler responds to task identity and horizon. However, the reported evidence does not currently support the strongest claims in the abstract, because the comparison against the released VeLO model goes in the opposite direction in Table 5, and because no confidence intervals are provided for the IQM aggregates that form the quantitative basis of the paper.

major comments (4)
  1. [Abstract and Section 6, Table 5] The abstract and introduction state that Celo outperforms VeLO, but Table 5 reports that the released pre-trained VeLO achieves a final-loss IQM of 1.41 on the same 17-task suite, whereas Celo achieves 1.20. The paper notes that the evaluation tasks are in-distribution for pre-trained VeLO, but the unqualified claim in the abstract is still contradicted by the paper's own table. The authors should either restrict the headline claim to learned optimizers meta-trained under the same compute budget, or add a more detailed comparison that makes the in-distribution/out-of-distribution distinction explicit in the abstract and conclusion.
  2. [Section 5.4 and Table 1] No confidence intervals are reported for any IQM or optimality-gap value, despite the paper explicitly adopting the IQM framework of Agarwal et al. (2021), which recommends stratified bootstrap confidence intervals. With 17 tasks and 3 seeds per task, Table 1's margins (final-loss IQM 1.20 vs. 1.05; speedup IQM 1.86 vs. 1.30) may be within sampling noise, since IQM trims the bottom 25% of the 51 normalized scores and is therefore sensitive to which runs fall in the middle 50%. The authors should provide bootstrap CIs for the headline comparisons and, ideally, increase the number of evaluation seeds.
  3. [Section 5.3 and Appendix A.6] The 17-task evaluation set is selected 'in order to fit evaluation ... within our compute budget' with no stated random or diversity-based selection rule, and all comparisons are made at 2K iterations. This makes it difficult to assess how representative the reported meta-generalization is. The paper's own Appendix A.6 shows that the full Celo is unstable on MLCommons AlgoPerf workloads and that Celo-Adam lags behind tuned baselines on those larger tasks. These statements are honest, but they substantially qualify the claim that Celo 'outperforms tuned state-of-the-art optimizers'; the paper should either add a longer-horizon evaluation on the 17-task suite or explicitly scope the claims to the 2K-iteration setting in the abstract.
  4. [Section 4, Eq. (11)] The normalized final-loss score uses the best Adam run out of a fixed number of trials as the denominator. This makes the absolute IQM values depend on how many Adam trials are available and on the selection rule for the 'best' run, which is not fully specified for all 17 tasks. The paper should state the number of Adam tuning trials used per task and whether the same best-run selection was applied consistently across all comparison optimizers, since this directly affects the comparability of the reported speedup scores.
minor comments (6)
  1. [Section 2, Eq. (2)] The notation T is used both for the unroll length in Eq. (3) and for the number of steps in the speedup definition in Eq. (12); these two uses should be distinguished to avoid confusion.
  2. [Table 1 and Table 4] The baseline name 'NNAadam LOpt' in the tables does not match the text's 'NNAdam LOpt'; the spelling should be unified.
  3. [Section 5.1] The claim that reducing VeLO's hidden size 'does not severely impact meta-generalization' is supported by Table 5 only partially: VeLO-S drops from IQM 0.96 to 0.90, which is a non-negligible change; the sentence should be softened or accompanied by a sensitivity discussion.
  4. [Appendix A.3] The meta-training setup lists a sweep over 5 learning rates for the meta-optimizer, with different optimal values for different baselines, but the paper does not report the chosen learning rate for each baseline; reporting this table would improve reproducibility.
  5. [Figure 6 and Figure 17] The schedule plots show the schedule parameter on a logarithmic axis, but the exact meaning of the y-axis values and their relationship to the effective learning rate η_t in Algorithm 1 should be stated in the caption or text.
  6. [Section 8] The limitations section is candid about the lack of large-scale evaluation, but the conclusion repeats the claim that Celo 'outperforms state-of-the-art hand-crafted and learned optimizers on unseen tasks'; the conclusion should mirror the scope of the limitations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Celo is an empirical recipe-and-benchmark paper; its transfer results are evaluated on held-out tasks, not derived from fitted quantities or self-cited theorems.

full rationale

This paper does not attempt a formal derivation; its claims are empirical. The meta-training objective (Eq. 3) minimizes mean inner-loop loss on four small image-MLP tasks, while the headline results are final-loss and speedup IQM scores computed on 17 held-out VeLOdrome tasks (Section 5.3, Table 1). The normalized score in Eq. 11 is a ratio to the best tuned Adam run; that is an evaluation convention borrowed from Agarwal et al. (2021), not a quantity fitted during meta-training, so Celo's reported transfer is not forced by construction. Task augmentation and PES are adopted with citations to Metz et al. (2022b) and Vicol et al. (2021); no load-bearing result is justified only by the authors' own prior work. The few self-citations (Moudgil et al. 2023, Knyazev et al. 2024, Thérien et al. 2024) appear only in related-work taxonomies and are explicitly described as complementary, not as evidence for Celo; per the hard rules, self-citation without load-bearing weight is not circularity. The Appendix A.6 admission that full Celo is unstable on MLCommons tasks and the non-random selection of the 17-task evaluation set are limitations on the strength of the generalization claim, but they are benchmark-selection and robustness concerns rather than circular reasoning. No equation in the paper reduces to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The recipe adds no new physical or mathematical entities. The main borrowed ingredients are task augmentation (VeLO), PES (Vicol et al.), and IQM (Agarwal et al.). The count of free parameters is modest, but the hand-picked evaluation set and the undisclosed selection rule are load-bearing.

free parameters (5)
  • task augmentation range tau = [0.001, 1000] log-uniform
    Sampled each meta-iteration; central to the meta-generalization gains (Table 2), chosen by hand.
  • scheduler scale constants alpha, lambda1, lambda2 = alpha unspecified; lambda1=lambda2=0.001
    Fixed low values in Algorithm 1 and Eq. 2 to keep meta-training stable; not tuned per task.
  • meta-training unroll and truncation lengths = max unroll 2000, truncation 50
    PES hyperparameters from prior work; no sensitivity analysis except the scaling study in A.4.
  • meta-training learning rate sweep = best of {3e-5,5e-5,1e-4,3e-4,1e-3} per optimizer
    The reported '24 GPU hours' excludes this 15-run sweep (A.3); the final Celo depends on this search.
  • evaluation horizon and seeds = 2000 iterations, 3 seeds per task
    All comparisons use this short horizon; no confidence intervals reported.
assumptions (5)
  • standard math PES gives unbiased meta-gradients in the truncated-unroll setting
    Invoked in Section 2 and used for all meta-training; cited to Vicol et al. 2021.
  • domain assumption Global task augmentation preserves the underlying optimizee function at initialization
    Section 3 rescales theta0 by 1/tau; this holds for the homogeneous MLPs used in meta-training but is assumed to transfer to other architectures.
  • ad hoc to paper The 17 selected evaluation tasks are representative of out-of-distribution generalization
    Section 5.3 selects tasks from VeLOdrome without a stated random or diversity rule; the central comparison depends on this set.
  • ad hoc to paper Released VeLO baseline data is directly comparable at the same 2K-iteration horizon
    Section 5.1 uses released baseline data from VeLO; Section 5.4 says all optimizers were optimized for 2K iterations, but no exact statement confirms the released data matches this horizon.
  • domain assumption IQM and optimality gap reliably aggregate optimizer performance across heterogeneous loss scales
    Section 4 borrows these metrics from RL (Agarwal et al. 2021); their suitability for optimizer benchmarking is assumed rather than validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Celo: Training Versatile Learned Optimizers on a Compute Diet." pith.science (2026). https://pith.science/paper/KBJWSTGT

@misc{pith2026250112670,
  author       = {Pith},
  title        = {Pith review of: Celo: Training Versatile Learned Optimizers on a Compute Diet},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KBJWSTGT}},
  note         = {Machine review of arXiv:2501.12670}
}
read the original abstract

Learned optimization has emerged as a promising alternative to hand-crafted optimizers, with the potential to discover stronger learned update rules that enable faster, hyperparameter-free training of neural networks. A critical element for practically useful learned optimizers, that can be used off-the-shelf after meta-training, is strong meta-generalization: the ability to apply the optimizers to new tasks. Recent state-of-the-art work in learned optimizers, VeLO (Metz et al., 2022), requires a large number of highly diverse meta-training tasks along with massive computational resources, 4000 TPU months, to achieve meta-generalization. This makes further improvements to such learned optimizers impractical. In this work, we identify several key elements in learned optimizer architectures and meta-training procedures that can lead to strong meta-generalization. We also propose evaluation metrics to reliably assess quantitative performance of an optimizer at scale on a set of evaluation tasks. Our proposed approach, Celo, makes a significant leap in improving the meta-generalization performance of learned optimizers and also outperforms tuned state-of-the-art optimizers on a diverse set of out-of-distribution tasks, despite being meta-trained for just 24 GPU hours.

Figures

Figures reproduced from arXiv: 2501.12670 by the authors.

Figure 1
Figure 1. Celo meta-training and architecture. where in the inner loop for t “ r0, T ´ 1]: pθt, stq “ pθ0{τ , 0q if t “ 0, (8) pθt, stq “ fϕpθt´1, ∇θt´1,Mt´1, st´1q if t ą 0; (9) ∇θt “ BL ` Xt; τθt, ϕ ˘ Bθt ; Xt „ D @t. (10) 0 500 1000 1500 2000 Iterations 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 2.6 Train loss Varying task augmentation param = 0.001 = 0.01 = 0.1 = 1.0 = 10.0 = 100.0 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Task augmentation. Learn￾ing curves for Adam with fixed (1e￾4) learning rate with different augmen￾tation parameters (τ ) for a CIFAR-10 ConvNet task. Overall, this task augmentation does not change the underlying optimizee network output at initialization, but it does affect the learning dynamics of the optimizee network in the inner loop when it is updated with adaptive or learned optimizers, thus ef￾fectively “si… view at source ↗
Figure 3
Figure 3. Two-stage training of Celo. Decoupled training of scheduler and up￾date rule. We propose to view an optimiza￾tion problem in two parts: how to control step￾size and what rule to use for parameter updates. Current state-of-the-art hand-designed optimizers such as Adam (Kingma & Ba, 2015) and Sham￾poo (Gupta et al., 2018) propose a hand-designed update rule, leaving the task of controlling or modulating global learnin… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Comparing Celo with state-of-the-art hand-crafted optimizers. Celo is our proposed learned optimizer meta-trained on a limited budget of 24 GPU hours. Optimizers are evaluated with final loss (left) and speedup (right) criteria with respect to Adam on a diverse set of …
Figure 5
Figure 5. Figure 5: Meta-generalization speedup profiles. Task augmentation improves meta-generalization performance in terms of speedup for all the learned optimizers. Celo outperforms all the learned optimizer baselines with and without task augmentation. IQMÒ optimizer w/o aug w/ aug n…
Figure 6
Figure 6. Figure 6: Visualizing schedules predicted by Celo on unseen tasks. Celo adapts its schedules based on the evaluation target length as shown for a ViT CIFAR-100 task (left) and also as per the given evaluation task by varying warmup and decay phases (right). Notably, Celo discove…
Figure 7
Figure 7. Figure 7: Training curves for selected tasks. See Appendix A.8 for detailed description of each task. task-agnostic features such as normalized loss features and training progress which remain similar between small meta-training tasks and large-scale evaluation tasks. Table 3d c…
Figure 8
Figure 8. Figure 8: Comparing Celo architecture with VeLO and RNN MLP LOpt. Our proposed approach, Celo, not only simplifies learned optimizer architecture from prior work as shown above but also vastly outperforms in meta-generalization performance (see §1). A.3 Implementation details We…
Figure 9
Figure 9. Figure 9: MLCommons AlgoPerf train curves. 0 4000 8000 12000 Iterations 0.50 0.55 0.60 0.65 0.70 0.75 Val SSIM Fast MRI 0 150000 300000 450000 Iterations 0.0 0.2 0.4 0.6 Top-1 val accuracy ImageNet ResNet-50 0 150000 300000 450000 Iterations 0.0 0.2 0.4 0.6 Top-1 val accuracy Im…
Figure 10
Figure 10. Figure 10: MLCommons AlgoPerf validation curves. We compare Celo-Adam with winner in the self￾tuning track of AlgoPerf 2024 competition Schedule-Free AdamW and prize qualification baseline NAdamW which is heavily tuned (Kasimbeg et al., 2025; Dahl et al., 2023). Although Celo-Ad…
Figure 11
Figure 11. Figure 11: All train curves. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: All test curves. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: RNN MLP LOpt augmentation training curves. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: Adafac MLP LOpt augmentation training curves. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]
Figure 15
Figure 15. Figure 15: Velo-S augmentation training curves. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]
Figure 16
Figure 16. Figure 16: Celo augmentation training curves. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]
Figure 17
Figure 17. Figure 17: Celo learned schedules for all tasks. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining

    cs.AI 2026-06 reject novelty 5.0 of 10

    A multi-agent typed-DSL search discovers RS-MR, a Muon-like matrix optimizer that beats Muon by 0.57% validation BPB on one 3.15B-token transformer pretraining run, without error bars.

Reference graph

Works this paper leans on

75 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Bellemare

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice, 2021

    work page 2021

  3. [3]

    A generalizable approach to learning optimizers

    Diogo Almeida, Clemens Winter, Jie Tang, and Wojciech Zaremba. A generalizable approach to learning optimizers. arXiv preprint arXiv:2106.00958, 2021

    arXiv 2021

  4. [4]

    Learning to learn by gradient descent by gradient descent

    Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016

    2016

  5. [5]

    Memory efficient adaptive optimization

    Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory efficient adaptive optimization. Neural Information Processing Systems, 32, 2019

    work page 2019

  6. [6]

    Scalable second order optimization for deep learning

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020

    arXiv 2002

  7. [7]

    On the distance between two neural networks and the stability of learning

    Jeremy Bernstein, Arash Vahdat, Yisong Yue, and Ming-Yu Liu. On the distance between two neural networks and the stability of learning. In Neural Information Processing Systems, 2020

    work page 2020

  8. [8]

    Optimization methods for large-scale machine learning

    L \'e on Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60 0 (2): 0 223--311, 2018

    2018

Show all 75 references
  1. [9]

    Convex optimization

    Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004

    2004

  2. [10]

    JAX : composable transformations of P ython+ N um P y programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake Vander P las, Skye Wanderman- M ilne, and Qiao Zhang. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/...

    2018

  3. [11]

    Large language models in machine translation

    Thorsten Brants, Ashok Popat, Peng Xu, Franz Josef Och, and Jeffrey Dean. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp...

    2007

  4. [12]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. 2024. URL https://openai. com/research/video-generation-models-as-world-simulators, 3, 2024

    2024

  5. [13]

    Mixtures of experts unlock parameter scaling for deep rl

    Johan Samir Obando Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep rl. In Forty-first International Conference on M...

  6. [14]

    One billion word benchmark for measuring progress in statistical language modeling

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013

    2013 arXiv

  7. [15]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.\ 1597--1607. PMLR, 2020

    2020

  8. [16]

    On empirical comparisons of optimizers for deep learning

    Dami Choi, Christopher J Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison, and George E Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint arXiv:1910.05446, 2019

    1910 arXiv

  9. [17]

    Randaugment: Practical automated data augmentation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp.\ 702--703, 2020

    2020

  10. [18]

    Benchmarking neural network training algorithms

    George E Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, et al. Benchmarking neural network training algorithms. arXiv preprint arXiv:2306.07179, 2023

    2023 arXiv

  11. [19]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  12. [20]

    Incorporating Nesterov momentum into Adam

    Timothy Dozat. Incorporating Nesterov momentum into Adam . In International Conference on Learning Representations, 2016

    2016

  13. [21]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011

    2011

  14. [22]

    Transformer-based learned optimization

    Erik G \"a rtner, Luke Metz, Mykhaylo Andriluka, C Daniel Freeman, and Cristian Sminchisescu. Transformer-based learned optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11970--11979, 2023

    2023

  15. [23]

    Can learned optimization make reinforcement learning less difficult? arXiv preprint arXiv:2407.07082, 2024

    Alexander David Goldie, Chris Lu, Matthew Thomas Jackson, Shimon Whiteson, and Jakob Nicolaus Foerster. Can learned optimization make reinforcement learning less difficult? arXiv preprint arXiv:2407.07082, 2024

    2024 arXiv

  16. [24]

    Deep learning, volume 1

    Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016

    2016

  17. [25]

    Accurate, large minibatch sgd: training imagenet in 1 hour

    P Goyal. Accurate, large minibatch sgd: training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

    2017 arXiv

  18. [26]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp.\ 1842--1850. PMLR, 2018

    2018

  19. [27]

    A closer look at learned optimization: Stability, robustness, and inductive biases

    James Harrison, Luke Metz, and Jascha Sohl-Dickstein. A closer look at learned optimization: Stability, robustness, and inductive biases. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL ht...

    2022

  20. [28]

    Highly accurate protein structure prediction with alphafold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Z \' dek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596 0 (7873): 0 583--589, 2021

    2021

  21. [29]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  22. [30]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.or...

    2015 arXiv

  23. [31]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4015--4026, 2023

    2023

  24. [32]

    Accelerating training with neuron interaction and nowcasting networks

    Boris Knyazev, Abhinav Moudgil, Guillaume Lajoie, Eugene Belilovsky, and Simon Lacoste-Julien. Accelerating training with neuron interaction and nowcasting networks. arXiv preprint arXiv:2409.04434, 2024

    2024 arXiv

  25. [33]

    Burghouts, Efstratios Gavves, Cees G

    Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J. Burghouts, Efstratios Gavves, Cees G. M. Snoek, and David W. Zhang. G raph N eural N etworks for L earning E quivariant R epresentations of N eural N etworks. In 12th International Conference on Learning Repre...

    2024

  26. [34]

    Narrowing the focus: Learned optimizers for pretrained models

    Gus Kristiansen, Mark Sandler, Andrey Zhmoginov, Nolan Miller, Anirudh Goyal, Jihwan Lee, and Max Vladymyrov. Narrowing the focus: Learned optimizers for pretrained models. arXiv preprint arXiv:2408.09310, 2024

    2024 arXiv

  27. [35]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  28. [36]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

    2012

  29. [37]

    Reinforcement learning with augmented data

    Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 0 19884--19895, 2020

    2020

  30. [38]

    The mnist database of handwritten digits

    Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 1998. URL http://yann.lecun.com/exdb/mnist

    1998

  31. [39]

    On the variance of the adaptive learning rate and beyond

    Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265, 2019

    1908 arXiv

  32. [40]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  33. [41]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  34. [42]

    Distinctive image features from scale-invariant keypoints

    David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60: 0 91--110, 2004

    2004

  35. [43]

    Learn2hop: Learned optimization on rough landscapes

    Amil Merchant, Luke Metz, Samuel S Schoenholz, and Ekin D Cubuk. Learn2hop: Learned optimization on rough landscapes. In International Conference on Machine Learning, pp.\ 7643--7653. PMLR, 2021

    2021

  36. [44]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  37. [45]

    Learning unsupervised learning rules

    Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Learning unsupervised learning rules. 2019 a

    2019

  38. [46]

    Understanding and correcting pathologies in the training of learned optimizers

    Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp.\ 4556--4565. PMLR, 2019 b

    2019

  39. [47]

    Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

    Luke Metz, Niru Maheswaranathan, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint arXiv:2009.11243, 2020 a

    2009 arXiv

  40. [48]

    Using a thousand optimization tasks to learn hyperparameter search strategies

    Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Using a thousand optimization tasks to learn hyperparameter search strategies. arXiv preprint arXiv:2002.11887, 2020 b

    2002 arXiv

  41. [49]

    Practical tradeoffs between memory, compute, and performance in learned optimizers

    Luke Metz, C Daniel Freeman, James Harrison, Niru Maheswaranathan, and Jascha Sohl-Dickstein. Practical tradeoffs between memory, compute, and performance in learned optimizers. In Conference on Lifelong Learning Agents (CoLLAs), 2022 a . URL http://github.com/google/learned_o...

    2022

  42. [50]

    Velo: Training versatile learned optimizers by scaling up

    Luke Metz, James Harrison, C Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al. Velo: Training versatile learned optimizers by scaling up. arXiv preprint arXiv:2211.09760, 2022 b

    2022 arXiv

  43. [51]

    Transformers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and Fran c ois Fleuret. Transformers are sample-efficient world models. arXiv preprint arXiv:2209.00588, 2022

    2022 arXiv

  44. [52]

    Learning to optimize with recurrent hierarchical transformers

    Abhinav Moudgil, Boris Knyazev, Guillaume Lajoie, and Eugene Belilovsky. Learning to optimize with recurrent hierarchical transformers. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023

    2023

  45. [53]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, number 2, pp.\ 4. Granada, 2011

    2011

  46. [54]

    Learning to learn with generative models of neural network checkpoints

    William Peebles, Ilija Radosavovic, Tim Brooks, Alexei Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209.12892, 2022

    2022 arXiv

  47. [55]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  48. [56]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--16. IEEE, 2020

    2020

  49. [57]

    On the convergence of adam and beyond

    Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019

    1904 arXiv

  50. [58]

    Is scaling learned optimizers worth it? evaluating the value of velo’s 4000 tpu months

    Fady Rezk, Antreas Antoniou, Henry Gouk, and Timothy Hospedales. Is scaling learned optimizers worth it? evaluating the value of velo’s 4000 tpu months. In Proceedings on, pp.\ 65--83. PMLR, 2023

    2023

  51. [59]

    Descending through a crowded valley-benchmarking deep learning optimizers

    Robin M Schmidt, Frank Schneider, and Philipp Hennig. Descending through a crowded valley-benchmarking deep learning optimizers. In International Conference on Machine Learning, pp.\ 9367--9376. PMLR, 2021

    2021

  52. [60]

    Deepobs: A deep learning optimizer benchmark suite

    Frank Schneider, Lukas Balles, and Philipp Hennig. Deepobs: A deep learning optimizer benchmark suite. arXiv preprint arXiv:1903.05499, 2019

    1903 arXiv

  53. [61]

    Bigger, better, faster: Human-level atari with human-level efficiency

    Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp.\ 30365--30380. PMLR, 2023

    2023

  54. [62]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235, 2018

    2018 arXiv

  55. [63]

    Optimizer benchmarking needs to account for hyperparameter tuning

    Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and Fran c ois Fleuret. Optimizer benchmarking needs to account for hyperparameter tuning. In International conference on machine learning, pp.\ 9036--9045. PMLR, 2020

    2020

  56. [64]

    Cyclical learning rates for training neural networks

    Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp.\ 464--472. IEEE, 2017

    2017

  57. [65]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp.\ 1139--1147. PMLR, 2013

    2013

  58. [66]

    Tieleman and G

    T. Tieleman and G. Hinton. Lecture 6.5---RmsProp: Divide the gradient by a running average of its recent magnitude . COURSERA: Neural Networks for Machine Learning, 2012

    2012

  59. [67]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  60. [68]

    Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies

    Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In International Conference on Machine Learning, pp.\ 10553--10563. PMLR, 2021

    2021

  61. [69]

    Learned optimizers that scale and generalize

    Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In International conference on machine learning, pp.\ 3751--3760. PMLR, 2017

    2017

  62. [70]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

    Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  63. [71]

    Large batch training of convolutional networks

    Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017

    2017 arXiv

  64. [72]

    Large batch optimization for deep learning: Training BERT in 76 minutes

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. arXiv preprint arXiv:1904.00962, 2019

    1904 arXiv

  65. [73]

    Adaptive methods for nonconvex optimization

    Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. Neural Information Processing Systems, 2018

    2018

  66. [74]

    Adabelief optimizer: Adapting stepsizes by the belief in observed gradients

    Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Neural Information Processing Systems, 2020

    2020

  67. [75]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.