Revisiting Learning Rate Control

Marius Lindauer; Micha Henheik; Theresa Eimer

REVIEW 4 major objections 4 minor 7 references

Revisiting Learning Rate Control

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No single method controls learning rates reliably

desk verdict A useful three-way empirical comparison of learning rate control paradigms, but the secondary claim that multi-fidelity HPO degrades on complex tasks is not supported by the evidence as configured. read the letter →

arxiv 2507.01724 v1 pith:4DORZNTN submitted 2025-07-02 cs.LG

Micha Henheik , Theresa Eimer , Marius Lindauer This is my paper

classification cs.LG

keywords learningratecontrolhyperparameter-freemulti-fidelityhyperparameteroptimizationschedulesalgorithmselectiondeepbenchmarkAutoML

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Learning rate control is usually a choice between tuning a fixed schedule, running multi-fidelity hyperparameter optimization, or using a hyperparameter-free optimizer. This paper compares all three paradigms on nine logistic-regression datasets, three computer-vision tasks, and a RoBERTa language-model pretraining run. It finds that no approach is reliable across settings: each method wins on some tasks and fails on others, and the best coverage of all tasks requires a portfolio of six different methods. The authors conclude that learning rate control should be reframed as an algorithm selection problem, and that multi-fidelity HPO becomes less effective as models and tasks grow more complex. If these results hold, AutoML researchers should shift focus toward selecting among control mechanisms and toward making hyperparameter-free methods tunable.

What carries the argument

The load-bearing instrument is the comparison protocol itself: a shared benchmark suite spanning LIBSVM logistic regression, CIFAR-10, CIFAR-100, DTD vision tasks, and RoBERTa pretraining on BookWiki, with every method scored by average difference to a per-dataset oracle and by marginal contribution to a portfolio. The methods compared are SMAC with Hyperband for multi-fidelity HPO, Cosine Annealing Warm Restarts in default and SMAC-tuned forms, and four hyperparameter-free optimizers—D-Adaptation, Prodigy, DoWG, and COCOB—run without tuning. Marginal-contribution analysis is what turns the raw accuracies into the portfolio-of-six result.

What would settle it

Run the same comparison with weight decay enabled for all optimizers and with a full-fidelity HPO baseline, such as SMAC without Hyperband or with many more trials, on BookWiki and CIFAR-100; if SMAC then matches or beats D-Adaptation and Prodigy, the claim that multi-fidelity HPO degrades on complex tasks would be directly refuted.

Watch

Extended reading notes

Core claim

The paper's central empirical claim is that the state of learning rate control is fragmented: no tested method generalizes across deep learning tasks. On LIBSVM, SMAC, the multi-fidelity HPO approach, is the only consistent method; on computer vision, Prodigy and the default learning rate lead; on RoBERTa pretraining, D-Adaptation and DoWG keep improving while high-learning-rate methods collapse. The authors show that an optimal solution of all their tasks needs six methods, and even restricting to the best method per domain leaves a portfolio of three approaches with very different performance profiles. They further observe that multi-fidelity HPO's advantage shrinks as task complexity grows, while hyperparameter-free methods are brittle but exceptionally strong when matched to the task.

Load-bearing premise

The conclusion that no method generalizes, and especially that multi-fidelity HPO loses effectiveness on complex tasks, rests on the assumption that the chosen task suite, the single HPO tool (SMAC with 50 trials and a constant learning rate), and the no-weight-decay training protocol are representative enough to support general claims about learning rate control.

Editorial extensions

If this is right

Learning rate control should be framed as algorithm selection: pick among a portfolio of methods based on the task, instead of relying on one default approach.
Multi-fidelity HPO is not a reliable default for expensive deep learning runs; on the RoBERTa task it selected a learning rate that collapsed at full budget.
Hyperparameter-free methods are strongly biased: COCOB, Prodigy, D-Adaptation, and DoWG each have a setting where they lead and others where they fail, so they are complementary rather than interchangeable.
Learning rate schedules benefit from tuning, and exposing a few key hyperparameters in hyperparameter-free methods is a promising best-of-both-worlds direction.
Evaluations of learning rate control need complex tasks: the conclusions drawn from small datasets do not transfer to larger models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A concrete next step the paper leaves implicit is meta-learning a selector that maps task features to one of the six methods, making the portfolio result directly actionable.
The opposite trajectories of COCOB, strong on small tasks and weak on language modeling, and DoWG, weak on small tasks and competitive on language modeling, hint that inductive bias tracks model scale; a controlled scaling study could test whether this ordering is monotone.
Because the vision runs use no weight decay, re-running with weight decay enabled could change the rankings; if it closed SMAC's gap, the paper's secondary HPO claim would need qualification.
The paper attributes SMAC's RoBERTa failure to Hyperband's early stopping; comparing against full-fidelity HPO with the same budget would isolate whether the problem is multi-fidelity scheduling or HPO in general.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The central negative result is credible: across logistic regression, image classification, and a RoBERTa pretraining run, no single learning-rate control method wins consistently, and the per-dataset rankings shift substantially with task complexity. That is worth knowing, and it is genuinely the first time these three paradigms—multi-fidelity HPO, fixed schedules, and hyperparameter-free methods—have been put on the same deep learning tasks. The paper is honest and well organized, and the authors ship code and raw data, which is real evidence. The per-dataset curves with 3 to 10 seeds are a solid basis for the main "no universal default" claim.

The soft spots are where the paper overreaches. The load-bearing secondary claim, that multi-fidelity HPO becomes less effective as models and tasks grow, rests on a very restricted protocol: SMAC tunes only a constant learning rate under a 50-trial budget, with search spaces that are not even log-scaled. Meanwhile, the methods it is compared against adapt the learning rate throughout training. So the RoBERTa failure in Fig. 6 likely reflects the search space and budget, not an inherent limitation of multi-fidelity HPO. The stress-test note is right about this. I'd also flag that all CV runs skip weight decay because DoWG doesn't support it, and the checklist confirms no statistical significance tests. The "portfolio of six methods" analysis is an oracle-style descriptive summary, useful for illustrating diversity but not a practical recommendation.

In short, the paper earns its keep as a benchmark-style comparison and a challenge to the idea that one paradigm will do. The main negative claim survives the soft spots; the HPO-decline claim needs re-scoping and additional experiments—e.g., tuning schedule parameters, a non-constant LR search space, or a second HPO tool—before it can be stated as a trend.

Who should read it: AutoML researchers, practitioners choosing LR strategies, and people working on algorithm selection or meta-learning. It deserves serious peer review, but with a request for revisions that narrow the HPO claim and strengthen the experimental protocol.

Referee Report

4 major / 4 minor

Summary. The paper compares three families of learning-rate control—multi-fidelity hyperparameter optimization (SMAC3/Hyperband), fixed schedules (CAWR and SMAC-tuned CAWR), and hyperparameter-free optimizers (D-Adaptation, Prodigy, DoWG, COCOB)—across LIBSVM logistic-regression datasets, CIFAR-10/100 and DTD vision tasks, and RoBERTa pretraining on a BookWiki reproduction. Using per-dataset accuracy/loss curves and oracle gaps over 3–10 seeds, the authors find that no single method is consistently best, that a portfolio of about six methods is needed to attain per-task best performance, and that multi-fidelity HPO becomes less competitive on the larger vision and language tasks. The paper concludes by arguing that learning-rate control should be framed as an algorithm-selection problem and that AutoML should engage with more complex deep learning tasks.

Significance. This is a timely and useful empirical comparison that brings together AutoML and deep-learning-native methods under one protocol. Its main strengths are the release of code and raw data, the use of multiple seeds, and the inclusion of both low-complexity and high-complexity tasks. If the central negative claim is robust, the paper provides a valuable caution to the AutoML community and a concrete motivation for algorithm selection and meta-learning for learning-rate control. However, the secondary claim that multi-fidelity HPO becomes less effective as task complexity grows is currently tied to a single HPO configuration, and the absence of statistical significance testing limits how strongly the cross-task rankings can be interpreted.

major comments (4)

[§3.4, §4, Table 8] The abstract, §3.4, and §4 state that multi-fidelity HPO becomes less effective as models and tasks grow in complexity, but this conclusion is not supported by the evidence as configured. Table 8 shows that SMAC/Hyperband is restricted to tuning a single constant learning rate with a 50-trial budget, whereas every comparator (CAWR, tuned CAWR, D-Adaptation, Prodigy, DoWG, COCOB) either schedules or adapts the learning rate during the run. The RoBERTa collapse in Fig. 6 is therefore a statement about this particular constant-LR search space and budget, not about multi-fidelity HPO as a paradigm. The claim should be re-scoped to 'SMAC with a constant-LR search space and 50 trials', or the authors should add a multi-fidelity HPO variant that can adapt or schedule the learning rate to test the complexity-trend claim.
[Checklist 2e, §3.2–3.3, Table 1] The paper's central rankings rest on gaps that are small relative to the reported standard errors, but no statistical significance tests are provided (checklist item 2e). For example, Table 1 reports differences between Prodigy (0.05%), default Adam (1.05%), and SMAC (1.96%) on the computer-vision suite, and Table 2 reports marginal contributions on CV of only 0.07–0.69 percentage points. Without paired tests or bootstrap confidence intervals on ranks, the reader cannot distinguish genuine situational failure from seed noise. Please add significance tests or confidence intervals for the main ranking and portfolio claims.
[§3.1, Tables 4–7] All computer-vision and language-modeling runs use weight decay 0 because DoWG does not support it (§3.1, Appendix A). Weight decay is a standard component of ResNet/DenseNet/ViT and RoBERTa training, so this protocol may change which methods are competitive, potentially disadvantaging fixed schedules or HPO relative to hyperparameter-free methods that already operate at very small effective learning rates. The authors should either run the comparison with a common non-zero weight decay where possible, or explicitly scope the CV and NLP conclusions to the no-weight-decay setting and discuss the likely sensitivity to this choice.
[Table 2, §3.2, §4] The 'portfolio of six methods' and the marginal contributions in Table 2 are never defined: the reader is not told how the portfolio is selected (greedy forward selection, oracle per-dataset choice, or some other procedure) or how marginal contributions are computed across datasets with different scales (accuracy versus perplexity). These numbers are used in the abstract and §4 to argue for algorithm selection, so they must be reproducible. Please specify the portfolio algorithm and, ideally, validate the resulting selector's generalization rather than only describing the same runs on which it was chosen.

minor comments (4)

[Throughout] Several typos and inconsistencies appear: 'seeeds' in the captions of Figures 4/5/6/9, 'DOWG' in §3.3, 'Tabel 4,5,7' in Appendix A, and inconsistent spacing in 'CA WR' versus 'CAWR' versus 'T. CA WR'.
[Table 2] Table 2 uses the label 'Adam' while the text refers to this method as 'the default learning rate'; please unify the naming.
[Appendix A] The total GPU hours are reported, which is helpful; please also separate the HPO tuning overhead from the method evaluation runs so that the claimed impracticality of black-box HPO can be assessed quantitatively.
[Figure 6] In Fig. 6c the y-axis is not explicitly labeled; please add a clear label such as 'scheduled/effective learning rate' and state whether the values are per-parameter or global learning rates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark comparison whose claims are read off experimental results, and its self-citations are to tools under test, not to load-bearing evidence.

full rationale

The paper's central claims — that no single learning rate control method generalizes across tasks, that a portfolio of methods is needed, and that multi-fidelity HPO tends to become less effective on more complex tasks — are direct empirical summaries of benchmark runs. No equation in the paper constructs a prediction from a fitted parameter or defines a target quantity in terms of itself. The portfolio analysis (Section 4, Table 2) is a descriptive marginal-contribution computation on the same observed accuracies, not a prediction that reduces to fitted values; it is a summary rather than a claimed derivation. The use of SMAC3 (Lindauer et al., 2022) and DACBench (Eimer et al., 2021), both authored by the same group, is not circular because these tools are the objects under comparison, not evidence used to justify the conclusions; the paper does not invoke its own prior results to establish the ranking. The restriction that SMAC searches only a constant learning rate (Section 3, Table 8) is an explicitly disclosed protocol choice and a possible limitation for the HPO-effectiveness claim, but it is not a definitional equivalence: the paper never defines 'multi-fidelity HPO' as 'constant learning rate tuning' nor derives the observed trend from that choice. Similarly, the absence of weight decay for DoWG (§3.1) is a stated experimental constraint, not a circular step. No uniqueness theorem, no imported ansatz, and no renaming of a known result appears. Concerns about representativeness, statistical significance (checklist 2e), and protocol adequacy belong under correctness risk, not circularity. The paper is self-contained as an empirical study, and its limitations are disclosed rather than hidden.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

These are the hand-set experimental controls the conclusions are sensitive to. There are no derivation parameters because the paper makes no theoretical claims; the listed free parameters and assumptions capture where the empirical conclusions could change under different settings.

free parameters (3)

SMAC optimization budget = 50 trials
The conclusion that multi-fidelity HPO is less effective on complex tasks depends on this fixed search budget; a larger budget could change rankings.
Hyperband minimum fidelity budgets = 20 (LIBSVM), 5 (CIFAR/DTD), 500 steps (RoBERTa)
Chosen by hand per task and control how much training is used to prune configurations, directly affecting HPO outcomes.
CAWR default schedule parameters = T0=10, T_mult=2, eta_min=0, eta_max=0.005
Borrowed from CIFAR-10/100 experiments in Loshchilov and Hutter 2017; not tuned per task, so CAWR results partly reflect choice of defaults rather than the schedule family.

assumptions (3)

domain assumption The evaluation protocol is a fair basis for comparing learning rate control paradigms.
All methods share optimizer defaults and no weight decay, but the no-weight-decay choice is driven by DoWG's limitations and makes CV runs non-standard; if this setting biases methods, the central ranking could change.
domain assumption The selected tasks represent deep learning settings where learning rate control matters.
Nine LIBSVM datasets, three CV tasks, and one RoBERTa setup are used to generalize about 'many deep learning settings'; task selection strongly shapes the portfolio conclusion.
domain assumption Hyperparameter-free methods are applied with original defaults and need no tuning.
The paper relies on method authors' claims that these methods have no important hyperparameters; if hidden tuning matters, the comparison would be less fair.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Revisiting Learning Rate Control." pith.science (2026). https://pith.science/paper/4DORZNTN

@misc{pith2026250701724,
  author       = {Pith},
  title        = {Pith review of: Revisiting Learning Rate Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DORZNTN}},
  note         = {Machine review of arXiv:2507.01724}
}

read the original abstract

The learning rate is one of the most important hyperparameters in deep learning, and how to control it is an active area within both AutoML and deep learning research. Approaches for learning rate control span from classic optimization to online scheduling based on gradient statistics. This paper compares paradigms to assess the current state of learning rate control. We find that methods from multi-fidelity hyperparameter optimization, fixed-hyperparameter schedules, and hyperparameter-free learning often perform very well on selected deep learning tasks but are not reliable across settings. This highlights the need for algorithm selection methods in learning rate control, which have been neglected so far by both the AutoML and deep learning communities. We also observe a trend of hyperparameter optimization approaches becoming less effective as models and tasks grow in complexity, even when combined with multi-fidelity approaches for more expensive model trainings. A focus on more relevant test tasks and new promising directions like finetunable methods and meta-learning will enable the AutoML community to significantly strengthen its impact on this crucial factor in deep learning.

Figures

Figures reproduced from arXiv: 2507.01724 by the authors.

**Figure 1.** Average differences in final training accuracy of every method to the oracle on each LIBSVM dataset. Methods are sorted according to the mean average difference across all datasets (in parentheses). In terms of overall performance, COCOB performs best among the hyperparameter-free methods, setting or matching the best training accuracy in six out of nine cases. Prodigy and the default learning rate are inconsist… view at source ↗

**Figure 3.** Training Accuracies for LIBSVM, non-hyperparameter-free methods. Figures show the mean across 10 seeds with standard error. We add the best hyperparameter-free method according to the final training accuracy. The Best HP-Free method is COCOB for every dataset except vowel and wine. For the latter it is Prodigy. best overall method in this comparison, ranking first or second on every dataset. Thus, it is the only con… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Resulting Training losses (a), Validation losses (b) and Validation Accuracies (c) for the hyperparameter-free methods (first row) and non-hyperparameter-free methods (second row) on DTD. In the second row, the best hyperparameter-free method (COCOB) is added to the pl…

**Figure 5.** Figure 5: Resulting Training losses (a), Validation losses (b) and Validation Accuracies (c) for the hyperparameter-free methods (first row) and non-hyperparameter-free methods (second row) on CIFAR-100. In the second row, the best hyperparameter-free method is added to the plot…

**Figure 6.** Figure 6: Resulting Training Perplexity (a), Validation Perplexity (b) and observed Learning Rates (c) for the all methods on BookWiki using RoBERTa. The y-axes of (a) and (b) are on a log scale and show the mean across seeeds with standard error. Adam SMAC CAWR T. CAWR D-Adapt …

**Figure 7.** Figure 7: Estimated effective learning rates of DoWG and COCOB on the RoBERTa BookWiki experiments. (a) Rank heatmap for the LIBSVM experiments according to final training loss. The average rank in denoted in parantheses. (b) Average Difference Heatmap (Training Loss) 18 [PITH…

**Figure 9.** Figure 9: Resulting Training losses (a), Validation losses (b) and Validation Accuracies (c) for the hyperparameter-free methods (first row) and non-hyperparameter-free methods (second row) on CIFAR-10. In the first row, the best hyperparameter-free method is added to the plot. …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

7 extracted references · 7 canonical work pages

[1]

For all authors. . . (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [Yes] (d) Did you read the ethics review guidelines and ensure that your paper con...

work page 2022
[2]

If you ran experiments. . . (a) Did you use the same evaluation protocol for all methods being compared (e.g., same benchmarks, data (sub)sets, available resources)? [Yes] (b) Did you specify all the necessary details of your evaluation (e.g., data splits, pre-processing, search spaces, hyperparameter tuning)? [Yes] (c) Did you repeat your experiments (e....

work page
[3]

With respect to the code used to obtain your results. . . (a) Did you include the code, data, and instructions needed to reproduce the main experimental results, including all requirements (e.g., requirements.txt with explicit versions), random seeds, an instructive README with installation, and execution commands (either in the supplemental material or a...

work page
[4]

If you used existing assets (e.g., code, data, models). . . 13 (a) Did you cite the creators of used assets? [Yes] (b) Did you discuss whether and how consent was obtained from people whose data you’re using/curating if the license requires it? [N/A] (c) Did you discuss whether the data you are using/curating contains personally identifiable information o...

work page
[5]

If you created/released new assets (e.g., code, data, models). . . (a) Did you mention the license of the new assets (e.g., as part of your code submission)? [N/A] (b) Did you include the new assets either in the supplemental material or as a url (to, e.g., GitHub or Hugging Face)? [N/A]

work page
[6]

If you used crowdsourcing or conducted research with human subjects. . . (a) Did you include the full text of instructions given to participants and screenshots, if appli- cable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (irb) approvals, if applicable? [N/A] (c) Did you include the estimated hourl...

work page
[7]

If you included theoretical results. . . (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A] 14 A Experimental Details Table 3: LIBSVM Configuration Parameter Value Architecture Logistic Regression Epochs 100 CPUs 1 ×AMD Epyc Device Batch Size 64 LR schedule ...

work page

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

For all authors. . . (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [Yes] (d) Did you read the ethics review guidelines and ensure that your paper con...

work page 2022

[2] [2]

If you ran experiments. . . (a) Did you use the same evaluation protocol for all methods being compared (e.g., same benchmarks, data (sub)sets, available resources)? [Yes] (b) Did you specify all the necessary details of your evaluation (e.g., data splits, pre-processing, search spaces, hyperparameter tuning)? [Yes] (c) Did you repeat your experiments (e....

work page

[3] [3]

With respect to the code used to obtain your results. . . (a) Did you include the code, data, and instructions needed to reproduce the main experimental results, including all requirements (e.g., requirements.txt with explicit versions), random seeds, an instructive README with installation, and execution commands (either in the supplemental material or a...

work page

[4] [4]

If you used existing assets (e.g., code, data, models). . . 13 (a) Did you cite the creators of used assets? [Yes] (b) Did you discuss whether and how consent was obtained from people whose data you’re using/curating if the license requires it? [N/A] (c) Did you discuss whether the data you are using/curating contains personally identifiable information o...

work page

[5] [5]

If you created/released new assets (e.g., code, data, models). . . (a) Did you mention the license of the new assets (e.g., as part of your code submission)? [N/A] (b) Did you include the new assets either in the supplemental material or as a url (to, e.g., GitHub or Hugging Face)? [N/A]

work page

[6] [6]

If you used crowdsourcing or conducted research with human subjects. . . (a) Did you include the full text of instructions given to participants and screenshots, if appli- cable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (irb) approvals, if applicable? [N/A] (c) Did you include the estimated hourl...

work page

[7] [7]

If you included theoretical results. . . (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A] 14 A Experimental Details Table 3: LIBSVM Configuration Parameter Value Architecture Logistic Regression Epochs 100 CPUs 1 ×AMD Epyc Device Batch Size 64 LR schedule ...

work page