Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Small-scale meta-training with a failure-aware resume buffer and decoupled expert supervision produces learned optimizers that, within under seven H100 GPU-hours, beat well-tuned AdamW on ImageNet-1K and GPT-2 pretraining and match Muon on

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:11 UTC pith:6U5CPNKZ

load-bearing objection ELO is a solid, practical meta-training recipe with real downstream gains; the theoretical unbiasedness claim for the resume buffer is unsupported and the evaluation has tuning-selection issues, but the empirical core deserves peer review. the 4 major comments →

arxiv 2607.06772 v4 pith:6U5CPNKZ submitted 2026-07-07 cs.LG

Efficient Long-Horizon Learning for Learned Optimization

Xiaolong Huang , Benjamin Th\'erien , James Harrison , Eugene Belilovsky This is my paper
classification cs.LG
keywords learned optimizationmeta-learninglong-horizon trainingpersistent evolution strategiesreplay bufferexpert supervisiontransfer learninglanguage model pretraining
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ELO is a recipe for training learned optimizers that recycles wasted compute: instead of always restarting inner optimization from scratch, it keeps a one-entry buffer of the most difficult stretch of the current trajectory and resumes from there with probability 0.8. To keep those long unrolls stable, it blends the optimizer's updates with a hand-designed expert and supervises the optimizer to imitate the expert's update direction and magnitude, gradually releasing the constraint. The paper's claim is that this combination yields learned optimizers that transfer from four tiny 8x8 image tasks to ImageNet-1K and GPT-2 124M/350M pretraining, with ELO-Celo2 beating AdamW everywhere and matching Muon on language modeling. A sympathetic reader would care because the meta-training cost is under seven H100 GPU-hours, which makes learned optimizers a cheap drop-in alternative rather than a research-only curiosity.

Core claim

The paper's central claim is that short-horizon meta-training failure is not a data problem but a sampling problem: random restarts keep re-training the optimizer on early optimization steps it has already mastered, starving later regions where long-horizon failures appear. ELO addresses this by maintaining a failure-aware resume buffer that stores the inner state just before the hardest loss-stagnation point, and by supervising the learned optimizer with a decoupled direction-and-magnitude imitation of a reliable expert whose weight fades out over meta-training. The paper reports that this makes meta-training converge faster and lets both element-wise (small_fc) and matrix-based (Celo2) lea

What carries the argument

The two mechanisms are the failure-aware resume buffer and decoupled progressive expert supervision. The buffer scores each inner step by a cumulative difficulty V_n = max(0, V_{n-1} - (loss_{n-1} - loss_n)), stores the state one truncation window before the worst stagnation point, and restarts unrolls from that state with probability P_B=0.8. Expert supervision interpolates the inner trajectory between the expert and the LO using weight alpha_t, and splits the imitation loss into a cosine direction term and an L1 magnitude term so that matching the expert's update direction is weighted 0.7 versus 0.3 for scale. Meta-gradients are estimated by Persistent Evolution Strategies with truncation

Load-bearing premise

The load-bearing premise is that restarting an optimization run from a saved difficult state does not bias the estimate of the gradient used to train the optimizer; if that assumption is wrong, the reported improvements could come from the sampling heuristic rather than from genuinely better optimization.

What would settle it

Meta-train ELO on a suite of small quadratic objectives where the true gradient of the meta-objective is analytically known, then compare the PES estimate obtained from buffer-resumed trajectories to the true gradient; if the estimate deviates systematically beyond Monte Carlo noise, the unbiasedness premise fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Learned optimizers can be trained on a handful of tiny vision tasks and still transfer to models hundreds of times larger, including 350M-parameter transformers.
  • Long-horizon meta-training can be made stable without abandoning task loss by anchoring early updates to an expert; the anchor can be removed entirely at the end.
  • ELO-Celo2 gives language-model pretraining validation loss on par with Muon and below AdamW at both 124M and 350M scales, so hand-designed optimizers are not an automatic ceiling.
  • The method works for two different LO families, element-wise MLP and matrix-based orthogonalizing, suggesting the recipe is architecture-agnostic.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the unbiasedness argument holds, the buffer idea transfers beyond optimizers: any meta-learning objective evaluated over long unrolled computation graphs could reuse failure checkpoints to concentrate gradient signal.
  • The expert-choice result (Adam for element-wise, Muon for matrix-based) suggests an architecture-matched expert is a free design lever; trying a structured or second-order-inspired expert for Celo2 is a natural next experiment.
  • Because the meta-training budget is so small, one can imagine meta-training an optimizer per domain or per architecture family, rather than a single universal one.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes ELO, a meta-training algorithm for learned optimizers (LOs). ELO combines a failure-aware resume buffer, which replays inner-problem states from difficult regions identified by a cumulative difficulty score, with decoupled progressive expert supervision, which imitates an expert optimizer's update direction and magnitude under an annealed weight. The paper claims that this enables efficient long-horizon meta-training of LOs, and evaluates small_fc and Celo2 meta-trained on tiny vision tasks and transferred to ImageNet (ResNet-50, ViT-B/16) and GPT-2 124M/350M on FineWeb. The headline claim is that ELO-Celo2 outperforms well-tuned AdamW across all evaluated tasks and remains competitive with Muon on language modeling, with less than 7 H100 GPU-hours of meta-training. The paper also includes ablations on buffer probability, expert choice, and the imitation objective, plus wall-clock overhead measurements.

Significance. If the empirical claims hold, the result is significant: it would demonstrate that a very cheap meta-training recipe can produce LOs that transfer to practical large-scale workloads and beat a strong hand-designed baseline. The paper is transparent about compute, provides open-source code, reports wall-clock overhead, and includes component ablations. However, the central methodological justification—unbiasedness of buffer-resumed PES—is not established, and the evaluation has selection-bias and statistical-uncertainty issues that currently prevent full confidence in the headline conclusions.

major comments (4)
  1. [Appendix A, Cor. 4.2 / Sec. 3.1] The proof that PES meta-gradients remain unbiased after resume from the failure-aware buffer is circular. Statement A.1 is stated for a full-horizon quadratic objective with a fixed, meta-parameter-independent initial state. In ELO, the restart step K is chosen from the difficulty score V_n of a previous trajectory, hence the restart state depends on current and past meta-parameters; the estimator in Algorithm 1 sums only n=K..K+N-1, omitting earlier terms, so it is not the gradient of Eq. (3). Storing theta_K alone is also not 'full PES state': the PES accumulator xi_K must be stored, but Algorithm 1 stores only s_K. Either provide a rigorous derivation for the buffer-resumed estimator (including the sampling distribution of K) or restate the claim as a heuristic and validate it empirically, e.g., by comparing ELO's gradient estimate to a full-horizon estimate on a small problem.
  2. [Sec. 4.3 / Fig. 8] The resume probability PB is selected using the validation loss of GPT-2 124M pretraining ('We then select the final PB=0.8 according to the validation loss of GPT-2 (124M, 2.5B tokens) pretraining'), and the same GPT-2 124M result is a headline downstream claim in Fig. 6. This compromises the comparison: the reported improvement over AdamW on GPT-2 124M is partially a result of model selection on that benchmark. Please report results for all PB candidates on GPT-2 124M, and treat only GPT-2 350M and ImageNet as held-out after selection, or choose PB from meta-validation only.
  3. [Sec. 4.1 / Fig. 6] All downstream comparisons appear to be single runs with no seed variance. Several key differences are small—e.g., ELO-Celo2 best validation loss 3.117 vs Muon 3.138 in Fig. 6C, and ELO-small_fc 73.92 vs AdamW 75.88 on ResNet-50 in Table 1. Without multiple seeds and error bars, 'consistently outperforms' and 'competitive with Muon' cannot be assessed. Please provide at least 3 seeds for the key AdamW/Muon comparisons in Figs. 5 and 6.
  4. [Appendix D / Fig. 9] Data augmentation strength is tuned separately per optimizer: LOs are evaluated under 'strong' augmentation and AdamW under 'medium' augmentation. This makes the comparison between ELO-Celo2 and AdamW dependent on a non-standard augmentation choice for LOs. The paper should also report results with AdamW's preferred augmentation applied to ELO-Celo2, or at least quantify the sensitivity of the headline ImageNet gains to augmentation strength.
minor comments (5)
  1. [Appendix A] Typo in Corollary proof: 'resumed frim' should be 'resumed from'; also the proof refers to 'Statement 4.1' but the statement is labelled A.1.
  2. [Eq. (4) and Algorithm 1] Notation conflict: Eq. (4) uses k as the truncation index and R as the truncation length, while Sec. 4.1 says 'truncation length of K=50' and Algorithm 1 defines R=50. Please unify the symbols.
  3. [References] References [5] and [6] are duplicates of the same Chen et al. paper ('Training stronger baselines for learning to optimize').
  4. [Appendix B] Typo: 'genralization' should be 'generalization'.
  5. [Algorithm 1] The buffer update 'B <- s_Kpush' stores only the inner state. If the PES accumulator is intended to be maintained for the unbiasedness claim, the pseudocode should explicitly store and restore it.

Circularity Check

2 steps flagged

Buffer-resumed PES unbiasedness is asserted by definition rather than proven, and PB is tuned on a benchmark that is then reported as a downstream prediction.

specific steps
  1. self definitional [Appendix A, Corollary 4.2 (and Section 3.1 claim)]
    "When meta-training with ELO, we use an identical gradient estimator ˆgPES-A-Buffer ≡ ˆgPES-A, with the only difference being that some particles may have been resumed from the particle replay buffer. ... Proof. Since Statement 4.1 has no temporal constraint on Θ, it directly applies to our case where particles are restarted from the buffer."

    The claimed unbiasedness of buffer-resumed PES gradients is made true by definition: the estimator is declared identical to ˆgPES-A because the buffer preserves the PES state, and then unbiasedness is imported from Statement A.1 of [43], which concerns a full-horizon objective with a fixed, θ-independent initial state. ELO instead restarts at K>0 chosen from the difficulty score V_n of a previous trajectory, so the start state θ_K depends on ϕ and on the buffer-selection rule; Algorithm 1 also sums only losses n=K..K+N-1, omitting n<K. Unbiasedness for this segment objective and resumed distribution is exactly what needed proof. Corollary 4.2 does not derive it; it restates the assumption that the estimator is identical and that the temporal constraint is irrelevant.

  2. fitted input called prediction [Section 4.3 (PB selection) vs Section 4.1 (GPT-2 124M result)]
    "We then select the final PB = 0.8 according to the validation loss of GPT-2 (124M, 2.5B tokens) pretraining in Appendix 8."

    PB is a free hyperparameter of ELO, and it is selected using the GPT-2-124M downstream validation loss. The same GPT-2-124M run is then reported in Section 4.1 as evidence that ELO-Celo2 outperforms AdamW, with the abstract claiming ELO-Celo2 'consistently outperforms well-tuned AdamW across all evaluated tasks.' For that specific benchmark, the comparison is selected rather than predicted. The other reported tasks (ImageNet, GPT-2 350M) were not used for this selection, so the circularity is partial rather than total.

full rationale

The paper's headline empirical claims are measured against external, hand-designed baselines and are therefore not themselves circular: ELO-Celo2's ImageNet and GPT-2-350M results stand independently, and the comparisons to AdamW/Muon are not derived from the method's assumptions. The main circularity is in the theoretical grounding of ELO. Section 3.1 and Appendix A claim that meta-gradients remain unbiased when inner problems are resumed from the failure-aware buffer. The proof of Corollary 4.2 defines the buffer-resumed estimator as identical to PES and then says Statement 4.1 'directly applies' because it has no temporal constraint on Θ. That is a restatement, not a derivation: the difficulty-selected restart distribution and the truncated segment objective are not shown to satisfy the conditions of Statement A.1. A second, smaller circularity is that the final PB value is tuned on GPT-2-124M validation loss and the same benchmark is subsequently presented among the tasks where ELO 'consistently outperforms' AdamW. This does not invalidate the independent benchmarks, but it means one of the headline comparisons is partially fit. I found no load-bearing self-citation chain: citations to Celo2 and µLO involve overlapping authors but are used as architecture/training recipes with external empirical content, not as uniqueness theorems or forbidden alternatives. Overall, the paper has genuine independent empirical content, but the central theoretical justification for the resume buffer reduces to an assumption, and one headline benchmark is used for hyperparameter selection, giving partial circularity rather than a clean non-finding.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The ledger reflects that ELO's central claim is empirical; its added degrees of freedom are hyperparameters, not a new physical entity. The strongest hidden assumption is the buffered-PES unbiasedness, which the paper asserts rather than proves; the largest fitted-value concern is PB, which is selected on the same GPT-2 124M benchmark used for the headline comparison.

free parameters (5)
  • lambda (direction loss weight) = 0.7
    Swept over {0.1,...,0.9} on meta-evaluation loss; balances direction vs magnitude imitation.
  • PB (resume probability) = 0.8
    Selected first from meta-eval and then finalized using GPT-2 124M/2.5B validation loss; controls how often unrolls restart from buffer.
  • alpha_t schedule = linear t/(T-1) over 0 to 1
    Hand-chosen annealing of expert-to-LO trajectory fusion; no automatic schedule search reported.
  • PES hyperparameters (R=50, sigma, P) = R=50; sigma and P unspecified in main text
    Truncation length fixed at 50; perturbation scale and particle count are not reported in the main paper, leaving tunable degrees of freedom.
  • Data augmentation strength per optimizer = strong for LOs, medium for AdamW
    Each method uses a different augmentation budget chosen by validation accuracy, as shown in Figure 9.
axioms (5)
  • standard math PES provides an unbiased gradient estimator of the Gaussian-smoothed meta-objective (from Vicol et al. 2021).
    Adopted without re-derivation; Eq. 4.
  • ad hoc to paper Resuming from the failure-aware buffer with preserved PES state yields unbiased meta-gradients.
    Appendix A Corollary 4.2 asserts this; the proof restates the assumption.
  • domain assumption The four tiny vision classification tasks and width-32 MLP optimizee form a sufficient meta-training distribution for transfer to ImageNet and GPT-2.
    Section 4.1; no theoretical or empirical justification beyond the observed transfer.
  • ad hoc to paper The cumulative difficulty score V_n and the Kpush=max(0,n*-R) resume point identify genuinely failure-prone regions.
    Section 3.1; heuristic based on local loss improvement.
  • domain assumption Expert updates (Adam for small_fc, Muon for Celo2) provide supervision that improves the LO rather than limiting it.
    Section 3.2/4.2 ablation; assumes imitation of expert directions transfers to better final performance than task-only training.

pith-pipeline@v1.3.0-alltime-deepseek · 15246 in / 15489 out tokens · 142516 ms · 2026-08-02T08:11:48.573423+00:00 · methodology

0 comments
read the original abstract

Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers (LOs), their meta-training remains biased toward short-unroll learning on particular tasks, resulting in redundant computation and leaving LOs often unable to compete with hand-designed optimizers. We introduce Efficient Long-hOrizon (ELO) learning, an efficient meta-training algorithm that (1) reallocates wasted meta-training compute to longer failure regimes, achieving efficient long-horizon learning, and (2) enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs. Our empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs. Across downstream language modeling (GPT-2-124M/350M on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K) tasks, ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. In particular, ELO-Celo2 consistently outperforms well-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling. \textit{Notably, all ELO baselines require less than 7 H100 GPU-hours for meta-training.}

Figures

Figures reproduced from arXiv: 2607.06772 by Benjamin Th\'erien, Eugene Belilovsky, James Harrison, Xiaolong Huang.

Figure 1
Figure 1. Figure 1: ELO improves downstream optimization and meta-training efficiency. (A) Validation loss on GPT-2 350M pretraining with FineWeb. Meta-trained only on four tiny vision-MLP tasks, ELO-Celo2 is able to generalize to realistic language modeling problem, outperforming AdamW and matching Muon. (B) Meta-evaluation loss during meta-training. ELO converges faster than other baselines over the 2K-step unroll covered b… view at source ↗
Figure 2
Figure 2. Figure 2: Ablation of ELO sampling and expert imitation. (A) ELO shifts meta-training budget toward longer inner horizons, reaching up to 17× larger inner steps than the standard baseline. (B) Complete ELO remains stable under longer-horizon sampling, whereas buffer-only training without progressive expert supervision becomes unstable. (C) Compared with direct MSE loss, the decoupled direction and magnitude objectiv… view at source ↗
Figure 3
Figure 3. Figure 3: Ablation of expert choice and resume buffer. Downstream validation loss on GPT-2 124M pretraining. (A) The best expert depends on the LO architecture: AdamW works best for the element-wise small fc, while Muon works best for the matrix-based Celo2. (B) Progressive expert supervision substantially improves the generalization of ELO-small fc to language modeling tasks, while the resume buffer further improve… view at source ↗
Figure 4
Figure 4. Figure 4: Expert supervision reduces meta￾gradient variance and improves the signal￾to-noise ratio of the ∆meta-loss between antithetic PES perturbations (See Sec. 2), leading to cleaner meta-gradient estimates. During meta-training, we maintain a failure-aware re￾sume buffer that provides difficult restart states from ongoing trajectories for future unrolls. Specifically, we identify these regions from the loss dyn… view at source ↗
Figure 5
Figure 5. Figure 5: ImageNet-1K training from scratch. Top-1 validation accuracy for ViT-B/16 (Left) and ResNet-50 (Right) trained on ImageNet-1K at 224 × 224 resolution for 50K steps with batch size 2048. ELO-trained LOs consistently improve over their corresponding LO baselines and outperform the AdamW baseline on both architectures. Insets zoom in on the final training stage. noise (ELO-small fc (w/o buffer) in [PITH_FULL… view at source ↗
Figure 6
Figure 6. Figure 6: GPT-2 124M pretraining on FineWeb. Validation loss throughout 2.5B training tokens for element-wise and matrix-based optimizers. ELO improves both LO architectures, outperforming other LO baselines and slightly exceeding their corresponding hand-designed counterparts. Language model pre-training. We evaluate ELO on realistic out-of-distribution language modeling tasks ( [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
Figure 7
Figure 7. Figure 7: Effect of direction and magnitude weights in the imitation objective. We sweep the [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Effect of the resume probability PB on long-horizon sampling and downstream performance. Top: increasing PB shifts the sampled inner steps toward later regions of the trajectory, with the maximum reached step increasing from 5,153 at PB = 0.1 to 91,221 at PB = 0.9. Bottom left: on the ImageNet-1K 32 × 32 meta-evaluation task, smaller PB generally gives better meta-validation loss, yet with one is achieved … view at source ↗
Figure 9
Figure 9. Figure 9: In our setting, learned optimizers achieve their best validation accuracy under stronger [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 9 linked inside Pith

  1. [1]

    Abdulkadirov, P

    R. Abdulkadirov, P. Lyakhov, and N. Nagornov. Survey of optimization algorithms in modern neural networks.Mathematics, 11(11):2466, 2023

    2023

  2. [2]

    Andrychowicz, M

    M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas. Learning to learn by gradient descent by gradient descent.Advances in neural information processing systems, 29, 2016

    2016

  3. [3]

    Buzzega, M

    P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara. Dark experience for general continual learning: a strong, simple baseline.Advances in neural information processing systems, 33:15920–15930, 2020

    2020

  4. [4]

    Chaudhry, A

    A. Chaudhry, A. Gordo, P. Dokania, P. Torr, and D. Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. InProceedings of the AAAI conference on artificial intelligence, volume 35, pages 6993–7001, 2021

    2021

  5. [5]

    T. Chen, W. Zhang, Z. Jingyang, S. Chang, S. Liu, L. Amini, and Z. Wang. Training stronger baselines for learning to optimize.Advances in Neural Information Processing Systems, 33:7332– 7343, 2020

    2020

  6. [6]

    T. Chen, W. Zhang, J. Zhou, S. Chang, S. Liu, L. Amini, and Z. Wang. Training stronger baselines for learning to optimize. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020

    2020

  7. [7]

    Dosovitskiy

    A. Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  8. [8]

    Florence, C

    P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. InConference on robot learning, pages 158–168. PMLR, 2022

    2022

  9. [9]

    Harrison, L

    J. Harrison, L. Metz, and J. Sohl-Dickstein. A closer look at learned optimization: Stability, robustness, and inductive biases.Advances in Neural Information Processing Systems, 35:3758– 3773, 2022

    2022

  10. [10]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778, 2016

    2016

  11. [11]

    Hester, M

    T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, J. Z. Leibo, and A. Gruslys. Deep q-learning from demonstrations. InProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), pages 3223–3230, 2018

    2018

  12. [12]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-optimal large language models, 2022

    2022

  13. [13]

    Jordan, Y

    K. Jordan, Y . Jin, V . Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. 10

    2024

  14. [14]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  15. [15]

    O. Li, J. Harrison, J. Sohl-Dickstein, V . Smith, and L. Metz. Variance-reduced gradient estimation via noise-reuse in online evolution strategies. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  16. [16]

    M. Liu, H. Zhao, Z. Yang, J. Shen, W. Zhang, L. Zhao, and T.-Y . Liu. Curriculum offline imitating learning.Advances in Neural Information Processing Systems, 34:6266–6277, 2021

    2021

  17. [17]

    Liu and J

    R. Liu and J. Zou. The effects of memory replay in reinforcement learning. In2018 56th annual allerton conference on communication, control, and computing (Allerton), pages 478–485. IEEE, 2018

    2018

  18. [18]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  19. [19]

    L. Metz, C. D. Freeman, J. Harrison, N. Maheswaranathan, and J. Sohl-Dickstein. Practical tradeoffs between memory, compute, and performance in learned optimizers. InConference on Lifelong Learning Agents (CoLLAs), 2022

    2022

  20. [20]

    L. Metz, J. Harrison, C. D. Freeman, A. Merchant, L. Beyer, J. Bradbury, N. Agrawal, B. Poole, I. Mordatch, A. Roberts, et al. Velo: Training versatile learned optimizers by scaling up.arXiv preprint arXiv:2211.09760, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    L. Metz, N. Maheswaranathan, C. D. Freeman, B. Poole, and J. Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves, 2020

    2020

  22. [22]

    L. Metz, N. Maheswaranathan, J. Nixon, D. Freeman, and J. Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. InInternational Conference on Machine Learning, pages 4556–4565. PMLR, 2019

    2019

  23. [23]

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning.nature, 518(7540):529–533, 2015

    2015

  24. [24]

    Moudgil, B

    A. Moudgil, B. Knyazev, and E. Belilovsky. Celo2: Towards learned optimization free lunch. arXiv preprint arXiv:2602.19142, 2026

    arXiv 2026

  25. [25]

    Moudgil, B

    A. Moudgil, B. Knyazev, G. Lajoie, and E. Belilovsky. Celo: Training versatile learned optimizers on a compute diet.Transactions on Machine Learning Research, 2025

    2025

  26. [26]

    A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. InIEEE International Conference on Robotics and Automation (ICRA), pages 6292–6299, 2017

    2017

  27. [27]

    T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning.Foundations and Trends® in Robotics, 7(1-2):1–179, 2018

    2018

  28. [28]

    Penedo, H

    G. Penedo, H. Kydl´ıˇcek, A. Lozhkov, M. Mitchell, C. A. Raffel, L. V on Werra, T. Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale.Advances in Neural Information Processing Systems, 37:30811–30849, 2024

    2024

  29. [29]

    D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network.Advances in neural information processing systems, 1, 1988

    1988

  30. [30]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  31. [31]

    Rajaraman, L

    N. Rajaraman, L. Yang, J. Jiao, and K. Ramchandran. Toward the fundamental limits of imitation learning.Advances in Neural Information Processing Systems, 33:2914–2924, 2020. 11

    2020

  32. [32]

    Rolnick, A

    D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne. Experience replay for continual learning.Advances in neural information processing systems, 32, 2019

    2019

  33. [33]

    S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth international conference on artifi- cial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Proceedings, 2011

    2011

  34. [34]

    Schaul, J

    T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay.arXiv preprint arXiv:1511.05952, 2015

    Pith/arXiv arXiv 2015

  35. [35]

    Schmitt, J

    S. Schmitt, J. J. Hudson, A. Zidek, S. Osindero, C. Doersch, W. M. Czarnecki, J. Z. Leibo, H. Kuttler, A. Zisserman, K. Simonyan, and S. M. A. Eslami. Kickstarting deep reinforcement learning.arXiv preprint arXiv:1803.03835, 2018

    Pith/arXiv arXiv 2018

  36. [36]

    R.-Y . Sun. Optimization for deep learning: An overview.Journal of the Operations Research Society of China, 8(2):249–294, 2020

    2020

  37. [37]

    S. Sun, Z. Cao, H. Zhu, and J. Zhao. A survey of optimization methods from a machine learning perspective.IEEE transactions on cybernetics, 50(8):3668–3681, 2019

    2019

  38. [38]

    R. S. Sutton, A. G. Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  39. [39]

    Th ´erien, C

    B. Th ´erien, C. ´Etienne Joseph, B. Knyazev, E. Oyallon, I. Rish, and E. Belilovsky. µlo: Compute-efficient meta-generalization of learned optimizers, 2024

    2024

  40. [40]

    Torabi, G

    F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation.arXiv preprint arXiv:1805.01954, 2018

    Pith/arXiv arXiv 2018

  41. [41]

    DDPG from Demonstrations

    M. Vecer´ık, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Roth¨orl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.arXiv preprint arXiv:1707.08817, 2017. This is often referred to as “DDPG from Demonstrations” (DDPGfD)

    Pith/arXiv arXiv 2017

  42. [42]

    P. Vicol. Low-variance gradient estimation in unrolled computation graphs with es-single. In International Conference on Machine Learning, pages 35084–35119. PMLR, 2023

    2023

  43. [43]

    Vicol, L

    P. Vicol, L. Metz, and J. Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In M. Meila and T. Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139 ofProceedings of Machine Learning Research, pages 10553–10563. PMLR, 2021

    2021

  44. [44]

    Wichrowska, N

    O. Wichrowska, N. Maheswaranathan, M. W. Hoffman, S. G. Colmenarejo, M. Denil, N. Freitas, and J. Sohl-Dickstein. Learned optimizers that scale and generalize. InInternational conference on machine learning, pages 3751–3760. PMLR, 2017

    2017

  45. [45]

    J. Yang, T. Chen, M. Zhu, F. He, D. Tao, Y . Liang, and Z. Wang. Learning to generalize provably in learning to optimize, 2023

    2023

  46. [46]

    Zhang and R

    S. Zhang and R. S. Sutton. A deeper look at experience replay.arXiv preprint arXiv:1712.01275, 2017. 12 A PES gradients remain unbiased when resuming from the buffer In the following sections, For the reader’s convenience, we will now restate background from [43] required to understand our proof. [ 43] derivePersistent Evolution Strategies(PES) for unroll...

    Pith/arXiv arXiv 2017