Efficient Long-Horizon Learning for Learned Optimization

Benjamin Th\'erien; Eugene Belilovsky; James Harrison; Xiaolong Huang

A cheap meta-training recipe lets tiny learned optimizers beat AdamW on real vision and language pretraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 15:56 UTC pith:6U5CPNKZ

load-bearing objection Solid, cheap meta-training recipe that actually moves LOs past AdamW on ImageNet and GPT-2; expert/PB choices are partly evaluation-aware, so treat the purest OOD claim with a grain of salt. the 4 major comments →

arxiv 2607.06772 v4 pith:6U5CPNKZ submitted 2026-07-07 cs.LG

Efficient Long-Horizon Learning for Learned Optimization

Xiaolong Huang , Benjamin Th\'erien , James Harrison , Eugene Belilovsky This is my paper

classification cs.LG

keywords learned optimizersmeta-traininglong-horizon optimizationfailure-aware resume bufferprogressive expert supervisionPESCelo2out-of-distribution generalization

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Learned optimizers are small neural nets that propose parameter updates instead of hand-designed rules such as Adam. Their usual meta-training keeps restarting short optimization runs from random starts, so most compute is spent on early steps the optimizer already handles well, while the long-horizon failures that matter later are under-trained. This paper introduces ELO: a failure-aware resume buffer that restarts unrolls near hard loss regions, paired with progressive expert supervision that fuses expert and learned trajectories and imitates expert updates with separate direction and magnitude losses. Meta-trained for under seven H100 GPU-hours on four tiny vision MLP tasks, the resulting optimizers transfer to ImageNet training of ResNet-50 and ViT-B/16 and to GPT-2 pretraining on FineWeb. ELO-Celo2 beats well-tuned AdamW on every evaluated task and stays competitive with Muon on language modeling, showing that long-horizon, generalizing learned optimizers can be obtained without large-scale meta-training.

Core claim

ELO turns short-horizon meta-training waste into long-horizon competence: a failure-aware resume buffer reallocates compute to later failure regimes, while decoupled progressive expert supervision keeps PES meta-gradients stable and improves out-of-distribution transfer. Applied to both element-wise and matrix-based learned optimizers, ELO-Celo2, meta-trained only on tiny vision MLPs for less than seven H100 hours, consistently outperforms well-tuned AdamW on ImageNet and FineWeb GPT-2 tasks and remains competitive with Muon on language modeling.

What carries the argument

ELO: a failure-aware resume buffer (difficulty score from cumulative loss non-improvement, resume probability PB) plus progressive trajectory fusion and a decoupled direction-plus-magnitude imitation loss against an architecture-matched expert, jointly used as the meta-objective under PES.

Load-bearing premise

That blending expert trajectories early and imitating them with separate direction and magnitude losses is enough to keep meta-gradients stable when most unrolls resume from hard long-horizon states, so the buffer can safely shift compute without training collapse.

What would settle it

Meta-train the same LO architectures with ELO (PB=0.8) but without progressive expert fusion or with a mismatched expert; if meta-training still converges and the resulting optimizers still beat AdamW on GPT-2-124M FineWeb and ImageNet ResNet-50/ViT-B/16, the claimed necessity of that stabilization mechanism fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid, cheap meta-training recipe that actually moves LOs past AdamW on ImageNet and GPT-2; expert/PB choices are partly evaluation-aware, so treat the purest OOD claim with a grain of salt.

read the letter

The useful takeaway is simple: ELO is a concrete, low-cost meta-training loop (failure-aware one-slot resume buffer + progressive trajectory fusion with decoupled direction/magnitude imitation) that lets small_fc and Celo2, trained only on four tiny vision MLPs for under 7 H100 hours, transfer to ResNet-50/ViT-B on ImageNet and GPT-2-124M/350M on FineWeb, with ELO-Celo2 beating well-tuned AdamW and matching Muon on language modeling.

What is new is the combination and the engineering that makes high resume probability (PB=0.8) stable. Log-uniform and curriculum sampling waste budget on early steps; pure long unrolls collapse. The buffer restarts near the onset of stagnation (via the cumulative difficulty score Vn), and the progressive expert term keeps PES gradients usable until the LO can take over. Abations are coherent: buffer alone is unstable, expert alone helps generalization but not long-horizon coverage, direction weight 0.7 beats plain MSE, and expert choice is architecture-dependent (Adam for element-wise, Muon for matrix). Code is open, PES unbiasedness under resume is argued cleanly, and the compute claim is real.

Soft spots are real but proportional. Expert and final PB selection both look at GPT-2-124M FineWeb validation, so the strongest language-modeling transfer numbers are not fully held-out from recipe design; the pure “meta-trained only on tiny vision” story is a bit softer than the abstract implies. Results are not multi-seeded, scale stops at 350M, and second-order baselines (SOAP/Shampoo) are missing. Those are limitations the authors mostly own, not hidden cracks.

This is for people who actually train or meta-train optimizers and care about wall-clock meta-cost and transfer. It is not a theory paper. I would bring it to reading group, cite the method if I am working on LO meta-training, and send it to peer review. The contribution is accept-shaped with the usual requests for multi-seed numbers and cleaner hyperparameter protocols.

Referee Report

4 major / 6 minor

Summary. The paper proposes Efficient Long-hOrizon (ELO) meta-training for learned optimizers, combining a failure-aware one-slot resume buffer that reallocates compute toward hard late-trajectory states with decoupled progressive expert supervision (trajectory fusion plus separate direction and magnitude imitation losses annealed by α_t). Meta-training is performed on four tiny vision-MLP tasks with PES; the resulting ELO-small_fc and ELO-Celo2 optimizers are evaluated on ImageNet-1K (ResNet-50, ViT-B/16) and FineWeb GPT-2-124M/350M. The central empirical claim is that ELO substantially improves long-unroll performance and OOD generalization of the base LOs, that ELO-Celo2 outperforms well-tuned AdamW on all reported tasks and remains competitive with Muon on language modeling, and that all ELO baselines require under 7 H100 GPU-hours of meta-training. Ablations support the necessity of both buffer and expert components, architecture-dependent expert choice, and direction-weighted imitation over joint MSE.

Significance. If the transfer results hold under a cleaner selection protocol, the work is a meaningful systems contribution to learned optimization: it targets a real inefficiency of short-unroll meta-training, keeps meta-training cheap, and reports competitive performance against strong hand-designed baselines (AdamW, Muon) on realistic vision and language pretraining. Strengths include open-sourced code, a clear algorithmic description (Algorithm 1), a short unbiasedness argument for PES under buffer resume (Appendix A), wall-clock meta-training costs (Table 2), and component ablations (Figs. 2–4, 7–8) that make the design choices falsifiable. The combination of failure-aware restarting with progressive expert guidance is a useful recipe even if some hyperparameter choices need tighter hold-out discipline.

major comments (4)

Sec. 4.3 and Fig. 8: the final resume probability PB=0.8 is chosen using GPT-2-124M FineWeb validation loss after a meta-eval shortlist. Fig. 8 itself shows that meta-evaluation prefers smaller PB while language-model validation prefers larger PB. Because GPT-2-124M is also a primary evaluation setting for the OOD transfer claim (Fig. 6, abstract), this selection step makes the strongest “meta-trained only on tiny vision MLPs” transfer numbers partially evaluation-aware. Please either (i) fix PB from meta-evaluation alone and re-report main LM/vision numbers, or (ii) reframe the claim and report a full sensitivity table for PB ∈ {0.1,0.5,0.8} on all main benchmarks, with the selection criterion stated before any FineWeb numbers.
Fig. 3A and Sec. 4.2: expert optimizer is selected by GPT-2-124M validation loss and is architecture-dependent (Adam for small_fc, Muon for Celo2). For ELO-Celo2, using Muon as expert while claiming competitiveness with Muon on language modeling (Fig. 1A, Fig. 6) creates a mild circularity risk: part of the gain may be progressive cloning of the evaluation baseline rather than pure task-driven discovery. Please report ELO-Celo2 with a held-out expert (e.g., Adam or SGDM only) on FineWeb, and/or quantify how much of the late-training update still matches the expert after α_t→1.
Table 1, Figs. 5–6, and the abstract claim that ELO-Celo2 “consistently outperforms” well-tuned AdamW: main results appear to be single-run point estimates with no seeds, error bars, or variance across random restarts of the optimizee/data order. Given that learning-rate, weight-decay, and augmentation are all swept per method, small absolute gaps (e.g., ResNet-50 IN-1K 76.29 vs 75.88; GPT-2-124M best val. loss 3.117 vs Muon 3.138) need multi-seed confirmation or at least two independent training runs to support “consistently.”
Sec. 4.1 / Appendix Fig. 9: LOs are evaluated under stronger data augmentation than AdamW, with the paper noting that AdamW degrades under the LO-preferred regime. This is disclosed, but the headline comparison in Table 1 and Fig. 5 is then not under a shared augmentation protocol. Please add a same-augmentation head-to-head (both under medium and under strong aug) so the optimizer effect is not confounded with augmentation choice.

minor comments (6)

Eq. (2): “NewtonShulz5” is misspelled (should be Newton–Schulz); also clarify the rms normalization convention relative to Celo2.
Algorithm 1: the demonstration assumes truncation length equals the unroll horizon, while the text uses R=50 and N sampled up to 2000; a short note that PES state is checkpointed across resumes would reduce ambiguity.
Fig. 2A y-axis “Inner Step” distribution would be clearer with a log scale or explicit max-step annotations for each method (the 17× claim is hard to read from the bars alone).
Sec. 2, Eq. (4): notation mixes K for truncation index and later K for resume start step; consider distinct symbols.
Related work: curriculum offline imitation [5,16] and prioritized/experience replay are cited, but a short explicit contrast with alternating offline imitation in CL-small_fc would help position the joint meta-loss design.
Limitations (Sec. 5) correctly note scale and architecture coverage; adding that expert/PB selection used a language-model probe would strengthen the discussion.

Circularity Check

1 steps flagged

No derivation circularity; mild evaluation-aware hyperparameter selection (PB and expert) on GPT-2-124M softens the pure OOD-transfer narrative but does not force the reported results by construction.

specific steps

fitted input called prediction [Sec. 4.3 (Buffer probability); Fig. 3A / Sec. 4.2 (Expert optimizer selection); Fig. 8]
"We first select three candidates, PB ∈ {0.1,0.5,0.8}, based on meta-evaluation loss. We then select the final PB = 0.8 according to the validation loss of GPT-2 (124M, 2.5B tokens) pretraining in Appendix 8. ... As shown in Figure 3 A, Adam gives the best validation loss when used as the expert for the element-wise small fc. In contrast, Muon works best as the expert for the matrix-based Celo2 optimizer."

PB and the expert are chosen using GPT-2-124M FineWeb validation—the same family as a primary reported evaluation—then the paper headlines OOD transfer of LOs meta-trained only on tiny vision tasks. This is mild evaluation-aware HP fitting, not a forced prediction: the LO still must actually optimize; the selection does not make ELO-Celo2 beat AdamW by construction. Softens the pure held-out transfer claim without collapsing the method into its inputs.

full rationale

This is an empirical methods paper, not a first-principles derivation. ELO is defined as a failure-aware resume buffer plus progressive decoupled expert imitation (Alg. 1, Sec. 3); meta-parameters ϕ are trained only on four tiny vision-MLP tasks; PES unbiasedness under resume is a corollary of prior PES theory (App. A), not a self-justifying uniqueness claim. Downstream ImageNet-1K (ResNet-50, ViT-B/16) and FineWeb GPT-2 results are external benchmarks against AdamW/Muon, not quantities fitted into the objective. The only mild circularity-adjacent issue is that PB and the expert optimizer are finally selected using GPT-2-124M FineWeb validation (Sec. 4.3, Fig. 3A, Fig. 8)—the same task family as a main evaluation—so the strongest language-modeling transfer numbers are not fully held-out from recipe design. That is evaluation contamination of hyperparameters, not a self-definitional reduction or a prediction forced by construction: choosing PB∈{0.1,0.5,0.8} and Adam vs Muon does not mathematically force ELO-Celo2 to beat AdamW. ImageNet results and gains over base LOs remain independent. Score 2 is proportionate; 0 would understate the soft OOD claim, 6+ would invent circularity that is not present.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard LO/PES machinery plus a small set of hand-chosen meta-training knobs and the modeling assumption that expert-guided long unrolls remain informative. No new physical entities are postulated; free parameters are the usual algorithmic hyperparameters selected by meta-eval and limited downstream search.

free parameters (6)

resume probability PB
Controls fraction of unrolls started from the failure buffer; default 0.8 chosen after meta-eval candidates and GPT-2-124M validation (Appendix Fig. 8).
direction loss weight lambda
Balances direction vs magnitude imitation; set to 0.7 after sweep on meta-evaluation loss (Appendix Fig. 7).
PES truncation length R
Fixed at 50 for meta-gradient segments; also used to place buffer push points.
progressive schedule alpha_t = t/(T-1)
Linear hand-off from expert to LO over meta-training; not derived from theory.
outer learning rate and weight decay
Searched over a small grid with AdamW meta-optimizer and cosine schedule for each baseline.
expert optimizer choice (Adam vs Muon)
Selected per LO architecture by downstream GPT-2-124M validation (Fig. 3A).

axioms (5)

standard math PES remains an unbiased estimator of the Gaussian-smoothed meta-objective when particles are resumed from a buffer that stores full PES state.
Appendix A restates Vicol et al. and argues the buffer does not introduce bias; used to justify long-horizon gradients.
domain assumption Short random-restart unrolls waste meta-training compute on already-mastered early regimes while under-training long-horizon failure regions.
Sec. 2 error-propagation argument motivating the resume buffer.
domain assumption A reliable hand-designed expert provides useful early imitation targets that stabilize long unrolls and improve task-agnostic generalization.
Sec. 3.2 progressive expert supervision; expert is architecture-dependent.
ad hoc to paper Decoupling update direction and magnitude is a better imitation objective than joint MSE for LO meta-training.
Hypothesis verified in Fig. 2C and Appendix Fig. 7; not forced by prior theory.
domain assumption Meta-training on four tiny 8x8 vision MLP tasks with width-32 optimizees is a sufficient distribution for transfer to ImageNet and GPT-2 pretraining.
Experimental setup Sec. 4; load-bearing for the OOD generalization claim.

invented entities (2)

failure-aware resume buffer (one-slot particle buffer of hard inner states) no independent evidence
purpose: Reallocate meta-training compute to long-horizon failure regimes by restarting unrolls near high cumulative difficulty scores V_n.
Core algorithmic construct of ELO; independent evidence is only the paper's own ablations and downstream gains, not external theory.
decoupled progressive expert supervision (trajectory fusion + direction/magnitude losses with schedule alpha_t) no independent evidence
purpose: Stabilize high-PB long unrolls and provide early meta-learning signal while allowing the LO to eventually exceed the expert.
Second core construct; evidence is internal ablations (Figs. 2–4).

pith-pipeline@v1.1.0-grok45 · 21332 in / 3661 out tokens · 30795 ms · 2026-07-14T15:56:31.387828+00:00 · methodology

0 comments

read the original abstract

Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers (LOs), their meta-training remains biased toward short-unroll learning on particular tasks, resulting in redundant computation and leaving LOs often unable to compete with hand-designed optimizers. We introduce Efficient Long-hOrizon (ELO) learning, an efficient meta-training algorithm that (1) reallocates wasted meta-training compute to longer failure regimes, achieving efficient long-horizon learning, and (2) enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs. Our empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs. Across downstream language modeling (GPT-2-124M/350M on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K) tasks, ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. In particular, ELO-Celo2 consistently outperforms well-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling. \textit{Notably, all ELO baselines require less than 7 H100 GPU-hours for meta-training.}