{"id":"303eaa83-b1a6-4823-bb76-b5c571bc07bc","arxiv_id":"2501.12670","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A compute-cheap learned optimizer, Celo, beats tuned hand-crafted and learned optimizers on a 17-task out-of-distribution suite by combining task augmentation, a simple hierarchical design, and two-stage decoupled meta-training.","lead":"Celo is a learned optimizer meta-trained for under 24 GPU hours on four small image-classification tasks, yet it achieves lower final losses and faster convergence than tuned Adam, Shampoo, and prior learned optimizers on 17 held-out tasks. The practical significance is that strong meta-generalization may not require the 4000-TPU-month scale of VeLO.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No confidence intervals on the headline IQM metrics: with 17 tasks and 3 seeds, the reported Celo-vs-baseline margins may be sampling noise, so the 'outperforms' claim is not yet established.","rationale":"The reader's weakest assumption is the representativeness of the 17-task suite. I agree the undisclosed selection rule is a weakness, but the more fundamental, load-bearing issue is internal validity: the paper presents its central quantitative evidence without any uncertainty quantification. Even if the 17 tasks were a perfectly random sample of VeLOdrome, the reported IQM differences need error bars to support 'outperforms'. With 3 seeds and IQM's 25% trimming, the aggregate can be driven by a handful of runs. The AlgoPerf results further show the margin does not extend to larger workloads, making it plausible that the 2K-step, small-scale evaluation favors Celo's meta-training distribution. The concrete bootstrap test uses publicly available data and directly checks whether the observed margins are statistically reliable. If the CIs exclude 0, the paper's claim is supported on its own terms, and the remaining concern is the suite's external validity (the reader's point). If the CIs include 0, the claim collapses. I therefore recommend keeping the verdict CONDITIONAL (i.e., unchanged), with the condition that the authors provide the missing confidence intervals.","tokens_in":31893,"tokens_out":13704,"duration_ms":133739,"concrete_test":"Using the released code and per-task per-seed normalized scores, compute stratified bootstrap 95% confidence intervals for the IQM differences between Celo and (a) Adafac MLP LOpt, (b) best-tuned Adam, for both final-loss and speedup criteria, following Agarwal et al. (2021) (e.g., 2000 bootstrap resamples). If the CI for any of these differences includes 0, the claim 'outperforms tuned state-of-the-art optimizers' is not statistically supported; if all CIs exclude 0, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on IQM aggregates in Table 1 (final-loss IQM 1.20, speedup IQM 1.86 vs Adafac MLP LOpt 1.05/1.30). These aggregates are computed from just 3 seeds per task over 17 tasks (51 runs), and the paper reports no confidence intervals, even though Section 4 explicitly adopts the IQM framework of Agarwal et al. (2021), which recommends stratified bootstrap CIs. IQM trims the bottom 25% of runs; with 51 runs this discards roughly 13 scores, so the reported means are sensitive to which runs fall in the middle 50%. If the observed margins are within sampling noise, the abstract's claim of outperforming tuned state-of-the-art optimizers is unsupported. The AlgoPerf appendix (A.6) shows full Celo is unstable and Celo-Adam lags tuned baselines on larger workloads, providing a concrete indication that the margin may not carry over even to in-distribution-scale tasks. The missing CIs are the load-bearing gap: without them, the reader cannot tell whether Table 1 reflects a real effect or a few favorable seeds.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Celo, a learned optimizer meta-trained on a fixed, small set of four image-classification tasks for roughly 24 GPU hours, and reports that it generalizes to 17 held-out tasks from the VeLOdrome suite. The main recipe has three components: task augmentation via random re-parametrization, a simple hierarchical architecture with a per-parameter MLP update rule plus a learned LSTM scheduler, and a two-stage training procedure where the update rule is trained first and then the scheduler is trained with the update rule frozen. The paper also adapts the IQM and optimality-gap metrics from Agarwal et al. (2021) to optimizer evaluation, defining normalized scores relative to a tuned Adam baseline. The central empirical claim is that Celo outperforms both hand-designed optimizers and previously published learned optimizers, including VeLO, on out-of-distribution tasks despite the small meta-training budget.","tokens_in":32143,"tokens_out":2851,"duration_ms":31666,"significance":"If the central claim holds, the paper would make an important contribution: it would show that compute-efficient meta-training can produce learned optimizers that transfer to diverse unseen tasks, substantially lowering the barrier compared to VeLO's 4000 TPU-month budget. The paper has several genuine strengths: the code is released, the meta-training protocol is controlled across all learned-optimizer baselines, the ablation study is systematic (task augmentation, scheduler form, two-stage training, tensor features), and the adoption of IQM/OG metrics is a sensible step toward robust aggregate evaluation in this area. The schedule analysis in Figure 6 is also suggestive evidence that the learned scheduler responds to task identity and horizon. However, the reported evidence does not currently support the strongest claims in the abstract, because the comparison against the released VeLO model goes in the opposite direction in Table 5, and because no confidence intervals are provided for the IQM aggregates that form the quantitative basis of the paper.","major_comments":[{"comment":"The abstract and introduction state that Celo outperforms VeLO, but Table 5 reports that the released pre-trained VeLO achieves a final-loss IQM of 1.41 on the same 17-task suite, whereas Celo achieves 1.20. The paper notes that the evaluation tasks are in-distribution for pre-trained VeLO, but the unqualified claim in the abstract is still contradicted by the paper's own table. The authors should either restrict the headline claim to learned optimizers meta-trained under the same compute budget, or add a more detailed comparison that makes the in-distribution/out-of-distribution distinction explicit in the abstract and conclusion.","section":"Abstract and Section 6, Table 5"},{"comment":"No confidence intervals are reported for any IQM or optimality-gap value, despite the paper explicitly adopting the IQM framework of Agarwal et al. (2021), which recommends stratified bootstrap confidence intervals. With 17 tasks and 3 seeds per task, Table 1's margins (final-loss IQM 1.20 vs. 1.05; speedup IQM 1.86 vs. 1.30) may be within sampling noise, since IQM trims the bottom 25% of the 51 normalized scores and is therefore sensitive to which runs fall in the middle 50%. The authors should provide bootstrap CIs for the headline comparisons and, ideally, increase the number of evaluation seeds.","section":"Section 5.4 and Table 1"},{"comment":"The 17-task evaluation set is selected 'in order to fit evaluation ... within our compute budget' with no stated random or diversity-based selection rule, and all comparisons are made at 2K iterations. This makes it difficult to assess how representative the reported meta-generalization is. The paper's own Appendix A.6 shows that the full Celo is unstable on MLCommons AlgoPerf workloads and that Celo-Adam lags behind tuned baselines on those larger tasks. These statements are honest, but they substantially qualify the claim that Celo 'outperforms tuned state-of-the-art optimizers'; the paper should either add a longer-horizon evaluation on the 17-task suite or explicitly scope the claims to the 2K-iteration setting in the abstract.","section":"Section 5.3 and Appendix A.6"},{"comment":"The normalized final-loss score uses the best Adam run out of a fixed number of trials as the denominator. This makes the absolute IQM values depend on how many Adam trials are available and on the selection rule for the 'best' run, which is not fully specified for all 17 tasks. The paper should state the number of Adam tuning trials used per task and whether the same best-run selection was applied consistently across all comparison optimizers, since this directly affects the comparability of the reported speedup scores.","section":"Section 4, Eq. (11)"}],"minor_comments":[{"comment":"The notation T is used both for the unroll length in Eq. (3) and for the number of steps in the speedup definition in Eq. (12); these two uses should be distinguished to avoid confusion.","section":"Section 2, Eq. (2)"},{"comment":"The baseline name 'NNAadam LOpt' in the tables does not match the text's 'NNAdam LOpt'; the spelling should be unified.","section":"Table 1 and Table 4"},{"comment":"The claim that reducing VeLO's hidden size 'does not severely impact meta-generalization' is supported by Table 5 only partially: VeLO-S drops from IQM 0.96 to 0.90, which is a non-negligible change; the sentence should be softened or accompanied by a sensitivity discussion.","section":"Section 5.1"},{"comment":"The meta-training setup lists a sweep over 5 learning rates for the meta-optimizer, with different optimal values for different baselines, but the paper does not report the chosen learning rate for each baseline; reporting this table would improve reproducibility.","section":"Appendix A.3"},{"comment":"The schedule plots show the schedule parameter on a logarithmic axis, but the exact meaning of the y-axis values and their relationship to the effective learning rate η_t in Algorithm 1 should be stated in the caption or text.","section":"Figure 6 and Figure 17"},{"comment":"The limitations section is candid about the lack of large-scale evaluation, but the conclusion repeats the claim that Celo 'outperforms state-of-the-art hand-crafted and learned optimizers on unseen tasks'; the conclusion should mirror the scope of the limitations.","section":"Section 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a well-controlled meta-training comparison and extensive ablations, but the headline claim is currently undercut by the paper's own Table 5, and the lack of confidence intervals makes the main quantitative comparison difficult to trust. I would ask the authors to reconcile the abstract with the released-VeLO comparison, add bootstrap CIs, and specify the task-selection procedure. These are fixable within the scope of the manuscript, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nCelo is a real contribution to learned optimization. The main novelty is the two-stage decoupled meta-training: first learn a per-parameter update rule, freeze it, then learn a scalar scheduler. That split makes sense, and the paper shows it beats single-stage training (IQM 1.20 vs 0.89). The scheduler without per-tensor statistics is also a nice simplification over VeLO, and the ablation in Table 3d shows it doesn't hurt. Task augmentation is borrowed, but the paper shows it helps all learned optimizers, so the recipe is well-motivated.\n\nThe empirical study is solid in design: controlled meta-training on 4 tasks, 17 held-out tasks, IQM metrics. The ablations are thorough. The appendix on AlgoPerf is honest—full Celo is unstable on those large workloads, and Celo-Adam lags tuned baselines. That tells you the recipe isn't done, but it's a fair limitation.\n\nThe soft spots are real. The headline IQM scores (Celo 1.20 vs Adafac MLP LOpt 1.05) come from 3 seeds and 17 tasks, and there are no confidence intervals. Given the paper explicitly adopts Agarwal et al.'s IQM framework, which comes with stratified bootstrap CIs, omitting them is a gap. With 51 runs, the IQM trims about 13 scores, so the aggregate is sensitive to a few runs. I'd want to see CIs before accepting the 'outperforms' claim.\n\nAlso, the abstract says Celo 'outperforms' VeLO, but Table 5 shows released VeLO gets IQM 1.41 on the same suite. It's true that suite is in-distribution for VeLO, but the abstract doesn't say that. That's a framing problem. Similarly, '24 GPU hours' is per training run; the actual hyperparameter sweep used 5 learning rates and 3 seeds, so total compute is higher. Minor, but worth tightening.\n\nTask selection: they picked 17 from 83 to fit compute budget, with no stated random or diversity rule. That's a legitimate concern for generalization claims, though they did cover a decent range of architectures.\n\nOverall, the method is plausible and the recipe is useful. The paper deserves a serious referee, but it needs revisions: add CIs, disclose task selection, and fix the VeLO comparison in the abstract. I'd suggest major revision or conditional acceptance.\n\nWould I bring it to reading group? Maybe, if the group cares about optimization. I'd cite it for the two-stage recipe and the careful ablations.","headline":"Celo is a genuine step forward for compute-limited learned optimization, but the headline comparison needs confidence intervals and a fairer VeLO framing before I'd trust the 'outperforms' claim.","tokens_in":32700,"tokens_out":2984,"would_cite":true,"duration_ms":26171,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Celo claims that a learned optimizer meta-trained for just 24 GPU hours can outperform tuned hand-designed and learned optimizers on 17 unseen tasks, thanks to task augmentation, a simple update rule, and a decoupled learned scheduler.","keywords":["learned optimizers","meta-generalization","task augmentation","two-stage meta-training","IQM evaluation","compute-efficient training","scheduler learning","out-of-distribution transfer"],"falsifier":"Evaluate Celo, meta-trained with the paper's 24-hour budget, on a random sample or the full 83-task VeLOdrome pool at 10,000 iterations per task, and compare final-loss IQM against tuned Adam and NAdamW; if Celo's IQM advantage over the tuned baselines shrinks below noise or reverses, the claim that it outperforms tuned state-of-the-art optimizers would fail.","tokens_in":31650,"feed_emoji":"⚡","tokens_out":8851,"duration_ms":77174,"temperature":0.7,"pith_summary":"Learned optimizers are update rules discovered by meta-training, but the best-known ones, such as VeLO, required 4000 TPU-months to generalize. This paper claims that strong meta-generalization can be bought much more cheaply: its optimizer Celo is meta-trained for under 24 GPU hours on four tiny image-classification tasks and still transfers to 17 unseen tasks spanning MLPs, CNNs, vision transformers, language models, autoencoders, and a meta-learning problem. On that suite Celo reports a final-loss IQM of 1.20 and a speedup IQM of 1.86 relative to tuned Adam, beating both 15 hand-designed optimizers and four learned baselines. The recipe is three ingredients: task augmentation by re-parametrization, a simple per-parameter MLP update rule coupled with a global LSTM scheduler, and two-stage meta-training that learns the update rule before freezing it and learning the scheduler. If correct, this removes the main practical barrier to iterating on learned optimizers: compute.","feed_headline":"24 GPU hours beat tuned Adam and Shampoo on unseen tasks","feed_subtitle":"Meta-trained for only 24 GPU hours on four small tasks, it beats tuned Adam, Shampoo, and VeLO on 17 unseen tasks.","key_machinery":"The central object is Celo, a hierarchical learned optimizer with two learned components: a per-parameter MLP update rule that maps per-parameter features to direction and magnitude, and a global LSTM scheduler that takes only task-agnostic features (normalized loss and training progress) and outputs a scalar step-size multiplier through an exponential map, $\\eta_t = \\alpha e^{o_t}$. The two components are meta-trained separately with truncated persistent evolution strategies (PES), a gradient estimator that avoids backpropagating through the inner loop: first the MLP alone, then the scheduler with the MLP frozen. Task augmentation re-parametrizes all inner-loop parameters by a per-iteration scalar $\\tau$ sampled log-uniformly, simulating many more tasks from the four fixed meta-training tasks. The argument's engine is that decoupling step-size control from update direction prevents meta-overfitting in the low-compute regime, and the exponential scheduler lets one scalar adapt to unseen tasks and horizons.","core_discovery":"The paper's central claim is that strong meta-generalization in learned optimizers does not require massive meta-training: Celo, trained on four small image-classification tasks with 8x8 inputs for under 24 GPU hours, achieves final-loss IQM 1.20 and speedup IQM 1.86 over tuned Adam on a held-out suite of 17 tasks spanning MLPs, CNNs, ViTs, transformer and RNN language models, autoencoders, and a meta-learning task. On the same suite, the best prior learned optimizer baseline, Adafac MLP LOpt, scores 1.05 final-loss IQM and 1.30 speedup IQM. The paper argues that three ingredients carry this result: task augmentation by re-parametrization, a simple hierarchical design of a per-parameter MLP update rule plus a global LSTM scheduler, and decoupled two-stage meta-training that first learns the update rule, then freezes it and learns the scheduler.","pith_inferences":["An implication the paper leaves implicit is that its evaluation suite is a convenience sample of 17 tasks from a larger 83-task pool, with all comparisons made at 2,000 steps; whether Celo's margin survives randomized task selection, longer horizons, or larger models is an open empirical question.","Because the scheduler consumes only loss and progress features, it may be portable across optimizee architectures; a testable extension is to run Celo's scheduler on top of other hand-designed or learned update rules beyond Adam and measure transfer.","The paper's own AlgoPerf appendix shows full Celo is unstable on large benchmark workloads and Celo-Adam trails tuned baselines there, so scaling the update rule to production-size models remains the gap between the 24-hour result and off-the-shelf use."],"forward_implications":["A learned optimizer with competitive meta-generalization can be meta-trained in under a day on a single GPU, making ablations and iterative design of learned optimizers affordable for research groups without large compute budgets.","Celo's per-parameter update costs 288 FLOPs per parameter, so its overhead relative to a forward-backward pass shrinks as batch size grows, making its runtime overhead small on large models.","The learned scheduler produces task- and horizon-dependent schedules (warmup, cyclic ramp-up, cosine-like decay) without manual tuning, suggesting learned step-size control transfers across unseen tasks.","The scheduler trained with a frozen learned update rule also improves a fixed Adam update rule (Celo-Adam) beyond tuned Adam on the 2K-step suite, indicating that the two-stage recipe isolates a reusable step-size policy."],"supporting_citations":[{"why":"Provides the VeLO baseline, the four meta-training tasks, the task-augmentation idea, and the 83-task VeLOdrome pool from which the 17 evaluation tasks are drawn.","marker":"(Metz et al., 2022b)"},{"why":"Supplies the IQM, median, and optimality-gap metrics the paper adapts to score optimizer final-loss and speedup performance robustly.","marker":"(Agarwal et al., 2021)"},{"why":"Provides the persistent evolution strategies (PES) gradient estimator used to meta-train Celo and all learned baselines.","marker":"(Vicol et al., 2021)"},{"why":"Contributes the Adafac MLP LOpt architecture that Celo's per-parameter MLP update rule builds on, and the strongest learned-optimizer baseline.","marker":"(Metz et al., 2022a)"},{"why":"Adam is the reference optimizer for the normalized score equations and the principal tuned hand-designed baseline to beat.","marker":"(Kingma & Ba, 2015)"},{"why":"NNAdam LOpt is the hybrid learned-scheduler-over-Adam baseline included in the learned-optimizer comparisons.","marker":"(Metz et al., 2020b)"}],"fun_headline_variants":["24 GPU hours: learned optimizer beats tuned Adam and Shampoo","Learned optimizer trains in a day, wins on unseen tasks","Celo: meta-training in 24 hours, beating VeLO's 4000 TPU months","Compute diet: 24 GPU hours to beat Adam and Shampoo","Celo: strong meta-generalization with 24 GPU hours of training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 17-task, 2,000-iteration evaluation suite is a representative measure of out-of-distribution meta-generalization, since the tasks were picked from a larger pool to fit the compute budget and no random or diversity-based selection rule is reported.","fun_headline_variants_meta":{"raw":{"variants":["24 GPU hours: learned optimizer beats tuned Adam and Shampoo","Learned optimizer trains in a day, wins on unseen tasks","Celo: meta-training in 24 hours, beating VeLO's 4000 TPU months","Compute diet: 24 GPU hours to beat Adam and Shampoo","Celo: strong meta-generalization with 24 GPU hours of training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1298,"prompt_tokens":968,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":231}},"tokens_in":584,"tokens_out":330,"duration_ms":3576,"temperature":1.0,"reasoning_tokens":231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:55:44.395597+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate Celo, meta-trained with the paper's 24-hour budget, on a random sample or the full 83-task VeLOdrome pool at 10,000 iterations per task, and compare final-loss IQM against tuned Adam and NAdamW; if Celo's IQM advantage over the tuned baselines shrinks below noise or reverses, the claim that it outperforms tuned state-of-the-art optimizers would fail.","supporting_citations":[{"cited_title":"Bellemare","cited_arxiv_id":null,"evidence_quote":"Supplies the IQM, median, and optimality-gap metrics the paper adapts to score optimizer final-loss and speedup performance robustly."},{"cited_title":"Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies","cited_arxiv_id":null,"evidence_quote":"Provides the persistent evolution strategies (PES) gradient estimator used to meta-train Celo and all learned baselines."}],"review_version":1}