Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Tune My Adam, Please!

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A surrogate pretrained on real Adam learning curves, with a rank-preserving augmentation, beats generic surrogates at both curve extrapolation and freeze-thaw hyperparameter tuning.

desk verdict A useful specialized surrogate for freeze-thaw HPO with a genuinely novel augmentation, but the paper never specifies its curve normalization, which is the difference between a real gain and a preprocessing artifact. read the letter →

arxiv 2508.19733 v2 pith:KN2YQQKW submitted 2025-08-27 cs.LG

Theodoros Athanasiadis , Steven Adriaensen , Samuel Müller , Frank Hutter This is my paper
classification cs.LG
keywords hyperparameteroptimizationfreeze-thawBayesianlearningcurveextrapolationprior-datafittednetworksAdamoptimizerdataaugmentationmeta-learningTaskSet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to show that hyperparameter tuning of Adam can be made much cheaper by replacing the generic surrogate in freeze-thaw Bayesian optimization with one that has actually seen Adam learning curves before. The authors build Adam-PFN, a small transformer pretrained on thousands of real learning curves from TaskSet, and add CDF-augment, a rank-preserving transformation that stretches and compresses curves to create more training examples. On held-out TaskSet tasks, this surrogate extrapolates partial curves more accurately than the synthetic-prior FT-PFN baseline at every context size tested. Plugged into the same freeze-thaw framework, it reaches the same normalized regret at roughly epoch 150 where FT-PFN needs roughly epoch 750, and it ranks first early on out-of-distribution tasks. The practical stake: cheap low-budget tuning of Adam can rest on an offline pretrained prior rather than a generic one.

What carries the argument

The load-bearing piece is the PFN (Prior-Data Fitted Network): a transformer trained, as in the FT-PFN baseline, to map a context of observed epoch-performance points plus a hyperparameter vector to a predictive distribution over future performance in one forward pass. The contribution is the prior: instead of sampling curves from a synthetic mixture of basis functions, Adam-PFN is trained on real Adam curves from TaskSet, augmented by CDF-augment, which applies the CDF of a Beta distribution with mode uniform in [0,1] and concentration uniform in [2,5] to each curve value. Because the CDF is monotone, the rank order of configurations is preserved while curve shape, and thereby perceived tas

What would settle it

Re-run training and evaluation with the normalized validation accuracy computed two ways: once from the full 50-epoch curve, and once from only the first 400 epochs available at inference. Train Adam-PFN on each version and compare extrapolation log-likelihood against FT-PFN. If the advantage shrinks or vanishes when normalization is context-only, the reported extrapolation gains came from full-curve leakage rather than the learned prior.

Watch

Extended reading notes

Core claim

Adam-PFN is a Prior-Data Fitted Network: a transformer that performs Bayesian learning-curve extrapolation in a single forward pass, trained on real Adam learning curves from 878 TaskSet tasks rather than on synthetic curves. The paper reports that this specialization improves extrapolation at every context size tested: at context 400, median log-likelihood rises from 3.440 (FT-PFN) to 5.326 and MSE falls from 0.00184 to 0.00054; contexts 1000 and 1600 show the same order. CDF-augment, which maps curve values through the CDF of a Beta distribution with random mode and concentration, preserves ordering while changing curve shape, and beats both no augmentation and Mixup. In the ifBO freeze-th

Load-bearing premise

The method assumes that the 'Normalized Valid. Accuracy' labels used for training and evaluation are produced by a fixed transformation that can be applied identically to partial curves, with no information from the curve's full trajectory or final value.

Editorial extensions

If this is right

  • A domain-specific surrogate can be dropped into an existing freeze-thaw BO loop without changing the acquisition function, turning saved epochs directly into cheaper hyperparameter tuning.
  • Learning-curve-only augmentation (CDF) is enough to beat both no augmentation and the two-sided Mixup augmentation; augmenting hyperparameters as well hurts performance.
  • On the TaskSet-8P benchmark the same final regret is reached with roughly one fifth of the epochs, so low-budget tuning pipelines gain the most.
  • On out-of-distribution tasks Adam-PFN is best early but FT-PFN catches up later, so a warm-start-then-switch policy is a plausible use pattern.
  • Training on randomly sampled TaskSet tasks is sufficient; carefully curated task families are not needed to learn a useful prior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gains likely transfer to other optimizers with large public curve corpora, e.g., SGD or AdamW, following the same recipe of real curves plus a monotone shape-augmentation.
  • If the normalization of validation accuracy is fixed and computable from partial curves, the 5x HPO speedup would probably also appear in settings where users provide their own budget-limit normalization; if instead it uses full-curve information, the speedup would shrink under a leakage-free reimplementation.
  • A testable extension is to combine the two priors: use Adam-PFN to select configurations early and FT-PFN to rank them later, or fine-tune FT-PFN on Adam-PFN's augmented real data.
  • CDF-augment's benefit likely comes from reweighting curve shapes toward harder and easier tasks; other monotone transformations with different curvature profiles may work as well or better.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Adam-PFN, a PFN-based surrogate for freeze-thaw Bayesian optimization specialized to Adam hyperparameters. It is pre-trained on TaskSet learning curves from 878 sampled tasks, augmented with a new CDF-augment method that applies a Beta CDF to curve values. The authors evaluate learning-curve extrapolation on 12 held-out NLP TaskSet tasks (Table 1) and HPO within ifBO (Figures 2), reporting that Adam-PFN(CDF) outperforms FT-PFN and other baselines in log-likelihood and MSE and reaches comparable normalized regret in fewer epochs. They also report early HPO gains on four PyTorch-example tasks (Section 3.4).

Significance. If the results hold, the paper makes a useful empirical contribution: it demonstrates that a domain-specific PFN prior trained on real Adam learning curves can beat a generic synthetic prior, and CDF-augment is a simple, rank-preserving augmentation. The appendices give unusually detailed training task lists and hyperparameter bounds, which help reproducibility. The main unresolved issue is the definition of the normalized accuracy used as the prediction target; until that is clarified, the quantitative gains are not fully established.

major comments (3)
  1. [Section 2.1 / Section 2.2 / Figure 1] The paper never defines 'Normalized Valid. Accuracy', the quantity plotted in Figure 1 and used as the target y in the CDF-augment equation (Eq. 1) and throughout training and evaluation. If this normalization is computed per task using all 1000 configurations or the final best validation accuracy, then it encodes future information unavailable during freeze-thaw HPO, where only partial curves are observed. The authors must state the exact transformation and, if it uses task-level statistics, show how it is computed from partial data only. This is load-bearing because the strong extrapolation and HPO gains in Table 1 and Figure 2 could be artifacts of normalization leakage.
  2. [Section 3.4 / Abstract / Figure 3] The abstract claims 'strong performance on out-of-distribution (OOD) tasks', but the paper's own text in Section 3.4 says that on average ranks FT-PFN catches up and eventually surpasses Adam-PFN (CDF) as budget increases. Figure 3(b) supports this. The OOD claim should either be qualified as early-budget performance or supported with final-budget results. As written, the OOD part of the central claim is not established.
  3. [Table 1 / Figures 2-3] The central quantitative claims are reported without uncertainty quantification. Table 1 reports medians over evaluation tasks, and Figures 2 and 3 report means over 5 seeds, but no error bars, confidence intervals, or per-task/per-seed results are provided. Since the paper claims that Adam-PFN (CDF) outperforms all baselines, the authors should report variability (e.g., standard errors or per-task values) and, where relevant, significance tests. This is needed to assess whether the reported differences are stable.
minor comments (6)
  1. [Section 2.2, after Figure 1] The sentence 'CDF-augment non-linearly transformed "task hardness" by making the task easier.' appears to be a dangling fragment; it should be integrated into the surrounding text.
  2. [Section 2.3] The phrase 'we do not sample the learning curves with and their HP configurations' contains a typo; it should read 'we do not sample the learning curves and their HP configurations'.
  3. [Section 3.3 / Figure 2] The claim that Adam-PFN(CDF) reaches the same normalized regret at epoch ~150 that FT-PFN reaches at epoch ~750 is made by visual inspection of a log-scale plot. Report the numerical values for these epochs to make the claim precise.
  4. [Appendix A, Table 2] State explicitly that hyperparameters are normalized to [0,1] using the lower/upper bounds before being fed to the model. This is mentioned in the text, but it would be clearer in the table caption.
  5. [Appendix C] The detailed training-task seed lists are helpful for reproducibility, but they would be even more useful as a machine-readable file or code release. No code or model download link is provided.
  6. [Table 1] The 'Time' column should specify the unit of measurement (seconds, milliseconds, etc.) and whether it is inference time per query or total wall-clock time.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity identified; the central claims are empirical evaluations rather than construction-equivalent derivations.

full rationale

Adam-PFN is trained on TaskSet learning curves and evaluated on held-out TaskSet tasks plus OOD PyTorch tasks; no parameter is fitted to the evaluation tasks, so the extrapolation and HPO results in Table 1 and Figure 2 are not predictions of fitted quantities. The PFN architecture (Müller et al., 2022) and the ifBO/FT-PFN framework (Rakotoarison et al., 2024) are code-reproduced prior work used as building blocks and baselines, not load-bearing self-citations that force the conclusion. CDF-augment (Eq. 1) is a fixed, hand-set transformation with no optimised constants, so it is not a fitted input renamed as a prediction. The main unaddressed risk is that the paper never defines the 'Normalized Valid. Accuracy' target (Section 2.1, Figure 1); if TaskSet's normalization uses full-curve or task-level statistics, the reported gains could be leakage artifacts. However, the manuscript provides no definition or equation exhibiting such an equivalence, so under the evidentiary standard this is a missing-support/correctness risk rather than a demonstrated circular step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about PFNs, TaskSet representativeness, and the CDF-augment, plus an unstated normalization step. There are no invented physical entities. The two CDF ranges are the only hand-set numeric inputs; they are not fitted to the evaluation.

free parameters (2)
  • CDF-augment mode range = mu ~ U[0,1]
    Augmentation distortion parameter, chosen by hand; not fitted to the evaluation data.
  • CDF-augment concentration range = kappa ~ U[2,5] for curves, U[2,3] for HP augmentation
    Design choices controlling how strongly curves are warped; chosen without formal justification.
assumptions (4)
  • domain assumption A PFN trained on learning curves performs valid approximate Bayesian inference for extrapolation in a single forward pass.
    Inherited from Prior-data Fitted Networks (Müller et al., 2022) and ifBO (Rakotoarison et al., 2024); the paper does not re-derive or validate this.
  • domain assumption Learning curves from TaskSet sampled tasks, all optimized with Adam under a fixed 8-HP search space, form a representative prior for Adam tuning on unseen tasks.
    Section 2.1 and 4; the authors themselves note the fixed search space limits scope and OOD results show the advantage fades.
  • domain assumption The Beta-CDF transformation preserves the information needed to predict final performance from early performance.
    Section 2.2; rank preservation is stated, but no proof or analysis shows that the warped curves remain predictive of HP quality.
  • ad hoc to paper Learning curve values are normalized to [0,1] by a fixed transformation identical in training and inference, without using future budget information.
    The paper never describes the normalization used to produce 'Normalized Valid. Accuracy'; this is required for the method to be well-defined and non-leaky.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tune My Adam, Please!." pith.science (2026). https://pith.science/paper/KN2YQQKW

@misc{pith2026250819733,
  author       = {Pith},
  title        = {Pith review of: Tune My Adam, Please!},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KN2YQQKW}},
  note         = {Machine review of arXiv:2508.19733}
}
read the original abstract

The Adam optimizer remains one of the most widely used optimizers in deep learning, and effectively tuning its hyperparameters is key to optimizing performance. However, tuning can be tedious and costly. Freeze-thaw Bayesian Optimization (BO) is a recent promising approach for low-budget hyperparameter tuning, but is limited by generic surrogates without prior knowledge of how hyperparameters affect learning. We propose Adam-PFN, a new surrogate model for Freeze-thaw BO of Adam's hyperparameters, pre-trained on learning curves from TaskSet, together with a new learning curve augmentation method, CDF-augment, which artificially increases the number of available training examples. Our approach improves both learning curve extrapolation and accelerates hyperparameter optimization on TaskSet evaluation tasks, with strong performance on out-of-distribution (OOD) tasks.

Figures

Figures reproduced from arXiv: 2508.19733 by the authors.

Figure 1
Figure 1. (a) A selection of learning curves from TaskSet, the color of each curve is based on the learning rate value. (b) The same set of learning curves augmented with CDF-augment. CDF-augment non-linearly transformed "task hardness" by making the task easier. 2.2 Augmentation Method: CDF-augment To generate an even more diverse training set, we introduce CDF-augment, a local learning curve augmentation method that uses th… view at source ↗
Figure 2
Figure 2. HPO results of normalized regret and average rank of Adam-PFN (CDF) against the baselines. The results are the mean across the 12 evaluation tasks for 5 random seeds. Adam-PFN (CDF) Adam-PFN (Mixup) Adam-PFN (No aug.) FT-PFN (ifBO) Hyperband ASHA Random Search 0 200 400 600 800 1000 Total epochs spent 10−5 10−4 10−3 10−2 10−1 Normalized regret PyTorch Examples (a) Normalized regret 0 200 400 600 800 1000 Total epoch… view at source ↗
Figure 3
Figure 3. HPO results of normalized regret and average rank of Adam-PFN (CDF) against the baselines on OOD tasks. The results are the mean across tasks for 5 random seeds. on a fixed search space with a pre-defined number of HPs. Tuning different sets of HPs needs to be further explored. One possible approach would be to set unused HPs to default values during training and testing. Unlike Mixup, we only consider augmentations… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (a) A selection of learning curves from TaskSet, the color of each curve is based on the learning rate value of the configuration. (b) The same set of learning curves after we apply Mixup learning curve augmentation. • GCN: This task uses a Graph Convolutional Network …
Figure 5
Figure 5. Figure 5: HPO results on real-world tasks. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: HPO results on real-world tasks. H Augmenting the HPs Mixup augments both learning curves and HP configurations, so we also extended CDF-augment to support HP augmentation. Specifically, we transformed HP configurations using the cumulative distribution function (CDF) …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. On the Nonlinearity of Learning Rate Scaling for LLM Training

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Optimal learning rate for models from 22M to 707M parameters shows nonlinear upward curvature with scale that disappears under effective learning rate and data-scale extrapolation.

Reference graph

Works this paper leans on

10 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Random Search for Hyper-Parameter Optimization

    Bergstra, J. and Y. Bengio (2012). “Random Search for Hyper-Parameter Optimization”. In: Journal of Machine Learning Research 13.10, pp. 281–305. Hochreiter, S. and J. Schmidhuber (1997). “Long Short-Term Memory”. In: Neural Comput. 9.8, pp. 1735–1780. Jamieson, K. and A. Talwalkar (2016). “Non-stochastic Best Arm Identification and Hyperparameter Optimiz...

    work page 2012

  2. [8]

    is a recently introduced FT-BO framework to which our work is closely related. CMBO is a Cost-Sensitive Multi-fidelity BO framework that introduces a new utility function, aiming to balance a user-specific tradeoff between the expected performance improvement gain during an HPO run, and the computational resources cost needed to achieve that gain. Similar...

    work page 2024

  3. [9]

    The second step involves mixing hyperparameter configurations and learning curves within the same task

    and perform learning curve augmentation as follows: 𝐿𝐶′ 𝑚 =𝜆1𝐿𝐶𝑚+( 1−𝜆1)𝐿𝐶𝑚′ for all𝑚,𝑚′∈𝑀 (3) This step produces a new learning curve for each hyperparameter configuration as a linear interpolation between the learning curves of tasks𝑚 and𝑚′. The second step involves mixing hyperparameter configurations and learning curves within the same task. Assume we...

    work page 2024

  4. [30]

    Supervising the Multi-Fidelity Race of Hyperpa- rameter Configurations

    Wistuba, M., A. Kadra, and J. Grabocka (2022). “Supervising the Multi-Fidelity Race of Hyperpa- rameter Configurations”. In: 36th Conference on Neural Information Processing Systems . Zhang, H. et al. (2018). “mixup: Beyond Empirical Risk Minimization”. In: International Conference on Learning Representations. 5 A TaskSet Hyperparameter Bounds Table 2 pro...

    work page 2022

  5. [51]

    Scaling Laws for Hyperparameter Optimization

    Proceedings of Machine Learning Research, pp. 240–248. Kadra, A. et al. (2023). “Scaling Laws for Hyperparameter Optimization”. In: 37th Conference on Neural Information Processing Systems . Kingma, D. P. and J. Ba (2015). “Adam: A Method for Stochastic Optimization”. In: 3rd International Conference on Learning Representations, ICLR 2015 . Kipf, T. N. an...

    work page 2023

  6. [1997]

    • Sequence Predictor: This task uses an LSTM to predict future values of sine wave signals

    and one using a transformer. • Sequence Predictor: This task uses an LSTM to predict future values of sine wave signals. 3All the equations in this section are from Lee et al., 2024 9 0 10 20 30 40 Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Valid. Accuracy Original Task 0.2 0.4 0.6 0.8 Learning Rate (a) 0 10 20 30 40 Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Normalized...

    work page 2024

  7. [2017]

    We train on a total of 2.0M learning curves

    with a linear warm-up schedule for the first 200 epochs. We train on a total of 2.0M learning curves. A list of the sampled tasks we used during training is presented in Appendix C. C List of TaskSet Training Tasks TaskSet provides both hand-designed tasks designed by experts, and sampled tasks, where the task parameters such as dataset, network architect...

    work page 2024

  8. [2022]

    During evaluation, we used the implementation suggested by Rakotoarison et al., 2024 and their publicly available surrogate model at https://github.com/automl/ifBO

    that performs Bayesian learning curve extrapolation, trained on synthetic datasets created by 4 weighted basis functions with additive Gaussian noise. During evaluation, we used the implementation suggested by Rakotoarison et al., 2024 and their publicly available surrogate model at https://github.com/automl/ifBO. Uniform Predictor is a simple baseline th...

    work page 2024

Show all 10 references
  1. [2023]

    is a framework that performs Bayesian Optimization with learning curve extrapolation under the assumption that learning curves follow power law functions. Using an ensemble of neural networks as a surrogate model, and Expected Improvement (EI) at the maximum budget as the acqu...

    2024

  2. [2024]

    DPL (Kadra et al.,

    framework implementation provided by Rakotoarison et al., 2024, which follows the imple- mentation details and code by Wistuba et al., 2022 from their publicly available repository at https://github.com/machinelearningnuremberg/DyHPO. DPL (Kadra et al.,

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.