{"id":"bffa09e1-1317-4e17-a1ce-2001105b7b01","arxiv_id":"2508.19733","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Adam-PFN, a prior-data fitted network trained on CDF-augmented real Adam learning curves, improves learning curve extrapolation and accelerates Adam hyperparameter tuning versus generic surrogates, with limited OOD gains.","lead":"This paper introduces Adam-PFN, a surrogate model for freeze-thaw Bayesian optimization pre-trained on real learning curves from TaskSet and augmented with a new CDF-based method. It speeds up tuning of Adam's hyperparameters on in-distribution benchmarks, with a smaller early advantage on new tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unspecified learning-curve normalization risks oracle leakage; central extrapolation gains may not hold under a fixed, partial-data normalization.","rationale":"The reader's verdict is CONDITIONAL, and their weakest assumption identifies the unspecified normalization, which I agree is the most load-bearing concern. The central quantitative claim—large improvements in extrapolation LL and HPO regret—rests entirely on the preprocessing of learning curves. If the normalization uses task-level or full-curve information, then the evaluation protocol is not a fair test of the method's ability to extrapolate from partial data, and the reported gains could be an artifact. The paper is silent on this step, so the claim is not yet verified. Other issues (missing error bars, OOD performance being overtaken by FT-PFN) are secondary, as the core in-distribution result is the strongest and most emphasized claim. The proposed test is concrete and would settle whether the concern lands: quantify the normalization and re-run the comparison under a partial-data-compatible alternative. This supports the reader's CONDITIONAL verdict, so no verdict change is needed.","tokens_in":11639,"tokens_out":4966,"duration_ms":58145,"concrete_test":"Inspect the exact preprocessing used to produce 'Normalized Valid. Accuracy' (from the TaskSet repository or by re-deriving from raw validation accuracies) and determine whether it uses any statistic that depends on the full set of 1000 configurations or on the complete training budget (e.g., the best or maximum final accuracy on that task). If yes, rerun the context-400 extrapolation experiment (Table 1) under a normalization that uses only information available at context time—for example, a fixed global scaling (e.g., accuracy/100) or a per-curve normalization using only the already-observed points—and recompute the LL and MSE for Adam-PFN (CDF) and FT-PFN. If the LL gap at context 400 (5.326 vs 3.440) substantially narrows or reverses, the reported gains rely on oracle/leaky normalization and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that Adam-PFN (CDF) substantially outperforms FT-PFN in learning-curve extrapolation (Table 1: LL 5.326 vs 3.440 at context 400) and freeze-thaw HPO (Figure 2: same regret at epoch 150 vs 750)—depends on the preprocessing of learning curves into the 'Normalized Valid. Accuracy' shown in Figure 1 and Section 2.1. Yet the paper never defines this normalization. In TaskSet, 'Normalized Valid. Accuracy' is often computed using task-level statistics such as the best final validation accuracy achieved by any configuration on that task (or a min-max scaling over all 1000 configurations). If that is the case, then every training curve and every evaluation curve encodes information that is not available during real freeze-thaw HPO, where only a partial curve from a small subset of configurations has been observed and the task-level normalization constant is unknown. The surrogate would be trained to exploit this leaked constant, and the reported gains over FT-PFN—which was trained on synthetic curves without such normalization—would be an artifact of the preprocessing rather than of the learned prior. The paper provides no statement that the normalization is fixed and computable from partial data only, so the strongest claim is unverified. This is load-bearing because without this assumption, the quantitative advantage claimed in the abstract and Section 3.2 collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adam-PFN, a PFN-based surrogate for freeze-thaw Bayesian optimization specialized to Adam hyperparameters. It is pre-trained on TaskSet learning curves from 878 sampled tasks, augmented with a new CDF-augment method that applies a Beta CDF to curve values. The authors evaluate learning-curve extrapolation on 12 held-out NLP TaskSet tasks (Table 1) and HPO within ifBO (Figures 2), reporting that Adam-PFN(CDF) outperforms FT-PFN and other baselines in log-likelihood and MSE and reaches comparable normalized regret in fewer epochs. They also report early HPO gains on four PyTorch-example tasks (Section 3.4).","tokens_in":11978,"tokens_out":5831,"duration_ms":65698,"significance":"If the results hold, the paper makes a useful empirical contribution: it demonstrates that a domain-specific PFN prior trained on real Adam learning curves can beat a generic synthetic prior, and CDF-augment is a simple, rank-preserving augmentation. The appendices give unusually detailed training task lists and hyperparameter bounds, which help reproducibility. The main unresolved issue is the definition of the normalized accuracy used as the prediction target; until that is clarified, the quantitative gains are not fully established.","major_comments":[{"comment":"The paper never defines 'Normalized Valid. Accuracy', the quantity plotted in Figure 1 and used as the target y in the CDF-augment equation (Eq. 1) and throughout training and evaluation. If this normalization is computed per task using all 1000 configurations or the final best validation accuracy, then it encodes future information unavailable during freeze-thaw HPO, where only partial curves are observed. The authors must state the exact transformation and, if it uses task-level statistics, show how it is computed from partial data only. This is load-bearing because the strong extrapolation and HPO gains in Table 1 and Figure 2 could be artifacts of normalization leakage.","section":"Section 2.1 / Section 2.2 / Figure 1"},{"comment":"The abstract claims 'strong performance on out-of-distribution (OOD) tasks', but the paper's own text in Section 3.4 says that on average ranks FT-PFN catches up and eventually surpasses Adam-PFN (CDF) as budget increases. Figure 3(b) supports this. The OOD claim should either be qualified as early-budget performance or supported with final-budget results. As written, the OOD part of the central claim is not established.","section":"Section 3.4 / Abstract / Figure 3"},{"comment":"The central quantitative claims are reported without uncertainty quantification. Table 1 reports medians over evaluation tasks, and Figures 2 and 3 report means over 5 seeds, but no error bars, confidence intervals, or per-task/per-seed results are provided. Since the paper claims that Adam-PFN (CDF) outperforms all baselines, the authors should report variability (e.g., standard errors or per-task values) and, where relevant, significance tests. This is needed to assess whether the reported differences are stable.","section":"Table 1 / Figures 2-3"}],"minor_comments":[{"comment":"The sentence 'CDF-augment non-linearly transformed \"task hardness\" by making the task easier.' appears to be a dangling fragment; it should be integrated into the surrounding text.","section":"Section 2.2, after Figure 1"},{"comment":"The phrase 'we do not sample the learning curves with and their HP configurations' contains a typo; it should read 'we do not sample the learning curves and their HP configurations'.","section":"Section 2.3"},{"comment":"The claim that Adam-PFN(CDF) reaches the same normalized regret at epoch ~150 that FT-PFN reaches at epoch ~750 is made by visual inspection of a log-scale plot. Report the numerical values for these epochs to make the claim precise.","section":"Section 3.3 / Figure 2"},{"comment":"State explicitly that hyperparameters are normalized to [0,1] using the lower/upper bounds before being fed to the model. This is mentioned in the text, but it would be clearer in the table caption.","section":"Appendix A, Table 2"},{"comment":"The detailed training-task seed lists are helpful for reproducibility, but they would be even more useful as a machine-readable file or code release. No code or model download link is provided.","section":"Appendix C"},{"comment":"The 'Time' column should specify the unit of measurement (seconds, milliseconds, etc.) and whether it is inference time per query or total wall-clock time.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"This paper is an AutoML non-archival track submission. For a journal version, the missing normalization definition is the key correctness risk. The authors should be asked to either specify the normalization and verify it is partial-data-only, or re-run the evaluation with a clearly defined fixed normalization. The OOD claim also needs to be tempered or supported with final-budget evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: Adam-PFN is a credible, well-scoped empirical contribution. The CDF-augment trick is simple and appears to beat both no augmentation and Mixup on in-distribution extrapolation. The paper is honest about its limits, especially the OOD results, where FT-PFN catches up later. If the normalization detail checks out, this is a solid drop-in improvement for ifBO.\n\nWhat is actually new: training a PFN surrogate on real TaskSet Adam curves rather than a synthetic prior, plus the Beta-CDF augmentation that preserves curve ordering. The authors also run a nice robustness check, withholding 400 HP configurations and showing that curve-only augmentation still generalizes to unseen HPs. That is real evidence the model is not just memorizing configurations.\n\nWhere it wobbles: the paper never defines \"Normalized Valid. Accuracy.\" TaskSet's standard normalization uses per-task statistics over the full set of configurations. If that is what the surrogate sees during both training and evaluation, then at HPO time the model is getting information about the final performance of all configurations on the task, which is exactly the oracle information freeze-thaw HPO is supposed to avoid. The stress-test note is right: this is unverified and load-bearing. The paper needs a clear statement of the transform and, ideally, an evaluation under a transform that uses only partial data. This is not a manufactured flaw; the omission is real and central.\n\nAlso: Table 1 reports medians without error bars or significance tests. The differences are large, so I wouldn't call the result fragile, but the reader can't tell how stable the gap is across the 12 tasks. The OOD claim is oversold in the abstract; the paper's own Figure 3 shows FT-PFN overtaking as budget grows. To the authors' credit, they say this in Section 3.4 and propose warm-starting, but the abstract should match the evidence. No code or trained model is released, which makes the normalization question harder to check.\n\nVerdict: I agree with the conditional take. The central in-distribution claim is defensible; the HPO speedup (epoch 150 vs 750) matters to practitioners, and the baselines are standard. This deserves a serious referee, not a desk reject. A revision that specifies the normalization, adds error bars, and tones down the OOD claim would make it publishable. If the normalization turns out to use full-curve statistics, the headline result may not survive; until then, treat the magnitude of the gain as unverified.","headline":"A useful specialized surrogate for freeze-thaw HPO with a genuinely novel augmentation, but the paper never specifies its curve normalization, which is the difference between a real gain and a preprocessing artifact.","tokens_in":12452,"tokens_out":2890,"would_cite":true,"duration_ms":33679,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A surrogate pretrained on real Adam learning curves, with a rank-preserving augmentation, beats generic surrogates at both curve extrapolation and freeze-thaw hyperparameter tuning.","keywords":["hyperparameter optimization","freeze-thaw Bayesian optimization","learning curve extrapolation","prior-data fitted networks","Adam optimizer","data augmentation","meta-learning","TaskSet"],"falsifier":"Re-run training and evaluation with the normalized validation accuracy computed two ways: once from the full 50-epoch curve, and once from only the first 400 epochs available at inference. Train Adam-PFN on each version and compare extrapolation log-likelihood against FT-PFN. If the advantage shrinks or vanishes when normalization is context-only, the reported extrapolation gains came from full-curve leakage rather than the learned prior.","tokens_in":11545,"feed_emoji":"⚡","tokens_out":6004,"duration_ms":58100,"temperature":0.7,"pith_summary":"The paper's aim is to show that hyperparameter tuning of Adam can be made much cheaper by replacing the generic surrogate in freeze-thaw Bayesian optimization with one that has actually seen Adam learning curves before. The authors build Adam-PFN, a small transformer pretrained on thousands of real learning curves from TaskSet, and add CDF-augment, a rank-preserving transformation that stretches and compresses curves to create more training examples. On held-out TaskSet tasks, this surrogate extrapolates partial curves more accurately than the synthetic-prior FT-PFN baseline at every context size tested. Plugged into the same freeze-thaw framework, it reaches the same normalized regret at roughly epoch 150 where FT-PFN needs roughly epoch 750, and it ranks first early on out-of-distribution tasks. The practical stake: cheap low-budget tuning of Adam can rest on an offline pretrained prior rather than a generic one.","feed_headline":"Surrogate pretrained on real curves tunes Adam five times faster","feed_subtitle":"A transformer trained on real Adam curves beats the generic surrogate at freeze-thaw hyperparameter search.","key_machinery":"The load-bearing piece is the PFN (Prior-Data Fitted Network): a transformer trained, as in the FT-PFN baseline, to map a context of observed epoch-performance points plus a hyperparameter vector to a predictive distribution over future performance in one forward pass. The contribution is the prior: instead of sampling curves from a synthetic mixture of basis functions, Adam-PFN is trained on real Adam curves from TaskSet, augmented by CDF-augment, which applies the CDF of a Beta distribution with mode uniform in [0,1] and concentration uniform in [2,5] to each curve value. Because the CDF is monotone, the rank order of configurations is preserved while curve shape, and thereby perceived tas","core_discovery":"Adam-PFN is a Prior-Data Fitted Network: a transformer that performs Bayesian learning-curve extrapolation in a single forward pass, trained on real Adam learning curves from 878 TaskSet tasks rather than on synthetic curves. The paper reports that this specialization improves extrapolation at every context size tested: at context 400, median log-likelihood rises from 3.440 (FT-PFN) to 5.326 and MSE falls from 0.00184 to 0.00054; contexts 1000 and 1600 show the same order. CDF-augment, which maps curve values through the CDF of a Beta distribution with random mode and concentration, preserves ordering while changing curve shape, and beats both no augmentation and Mixup. In the ifBO freeze-th","pith_inferences":["The gains likely transfer to other optimizers with large public curve corpora, e.g., SGD or AdamW, following the same recipe of real curves plus a monotone shape-augmentation.","If the normalization of validation accuracy is fixed and computable from partial curves, the 5x HPO speedup would probably also appear in settings where users provide their own budget-limit normalization; if instead it uses full-curve information, the speedup would shrink under a leakage-free reimplementation.","A testable extension is to combine the two priors: use Adam-PFN to select configurations early and FT-PFN to rank them later, or fine-tune FT-PFN on Adam-PFN's augmented real data.","CDF-augment's benefit likely comes from reweighting curve shapes toward harder and easier tasks; other monotone transformations with different curvature profiles may work as well or better."],"forward_implications":["A domain-specific surrogate can be dropped into an existing freeze-thaw BO loop without changing the acquisition function, turning saved epochs directly into cheaper hyperparameter tuning.","Learning-curve-only augmentation (CDF) is enough to beat both no augmentation and the two-sided Mixup augmentation; augmenting hyperparameters as well hurts performance.","On the TaskSet-8P benchmark the same final regret is reached with roughly one fifth of the epochs, so low-budget tuning pipelines gain the most.","On out-of-distribution tasks Adam-PFN is best early but FT-PFN catches up later, so a warm-start-then-switch policy is a plausible use pattern.","Training on randomly sampled TaskSet tasks is sufficient; carefully curated task families are not needed to learn a useful prior."],"supporting_citations":[{"why":"Formalizes freeze-thaw Bayesian optimization, the decision loop the new surrogate plugs into.","marker":"(Swersky et al., 2014)"},{"why":"Introduces ifBO and the FT-PFN generic synthetic-prior surrogate that Adam-PFN replaces and is compared against.","marker":"(Rakotoarison et al., 2024)"},{"why":"Provides the PFN architecture that lets a transformer perform Bayesian in-context learning-curve extrapolation.","marker":"(Müller et al., 2022)"},{"why":"TaskSet, the source of real Adam learning curves and hyperparameter configurations used for pretraining.","marker":"(Metz et al., 2020)"},{"why":"CMBO and Mixup, the nearest related method and the alternative augmentation baseline.","marker":"(Lee et al., 2024)"},{"why":"Defines Adam and the eight hyperparameters whose tuning is the target domain.","marker":"(Kingma and Ba, 2015)"},{"why":"DPL baseline and one of the sources of the evaluation task list.","marker":"(Kadra et al., 2023)"},{"why":"DyHPO baseline and another source of the same evaluation task list.","marker":"(Wistuba et al., 2022)"}],"fun_headline_variants":["Pretrained on real Adam curves, this surrogate tunes hyperparams faster","A transformer that predicts learning curves in one pass speeds up Adam tuning","CDF-augment and real-curve training make Adam hyperparameter search quicker","Adam-PFN: Bayesian extrapolation from real curves, not synthetic ones","Tune Adam faster with a surrogate pretrained on 878 TaskSet tasks"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The method assumes that the 'Normalized Valid. Accuracy' labels used for training and evaluation are produced by a fixed transformation that can be applied identically to partial curves, with no information from the curve's full trajectory or final value.","fun_headline_variants_meta":{"raw":{"variants":["Pretrained on real Adam curves, this surrogate tunes hyperparams faster","A transformer that predicts learning curves in one pass speeds up Adam tuning","CDF-augment and real-curve training make Adam hyperparameter search quicker","Adam-PFN: Bayesian extrapolation from real curves, not synthetic ones","Tune Adam faster with a surrogate pretrained on 878 TaskSet tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000992,"raw_usage":{"total_tokens":4007,"prompt_tokens":676,"completion_tokens":3331,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":420,"completion_tokens_details":{"reasoning_tokens":3234}},"tokens_in":420,"tokens_out":3331,"duration_ms":26733,"temperature":1.0,"reasoning_tokens":3234,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:30:14.108446+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run training and evaluation with the normalized validation accuracy computed two ways: once from the full 50-epoch curve, and once from only the first 400 epochs available at inference. Train Adam-PFN on each version and compare extrapolation log-likelihood against FT-PFN. If the advantage shrinks or vanishes when normalization is context-only, the reported extrapolation gains came from full-curve leakage rather than the learned prior.","supporting_citations":[{"cited_title":"DPL (Kadra et al.,","cited_arxiv_id":null,"evidence_quote":"Introduces ifBO and the FT-PFN generic synthetic-prior surrogate that Adam-PFN replaces and is compared against."},{"cited_title":"Supervising the Multi-Fidelity Race of Hyperpa- rameter Configurations","cited_arxiv_id":null,"evidence_quote":"DyHPO baseline and another source of the same evaluation task list."}],"review_version":1}