{"id":"655ac315-d61e-499f-9d19-a85ac55eab4d","arxiv_id":"2607.29626","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 30-task benchmark shows LLM agents can improve real ML experiments through sequential hyperparameter choices, but their gains are uneven and often not retained.","lead":"This paper introduces AgentHPOBench, a benchmark of 30 executable machine-learning tasks that tests whether LLM agents can improve real training runs by choosing hyperparameters step-by-step from logs and metrics. Across 12 agents and conventional HPO methods, the best agent beats baselines on most tasks but all agents struggle to preserve gains over five interventions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run reference baselines may inflate measured optimization ability; no repeated baseline noise floor is provided.","rationale":"The reader's weakest assumption focused on anchor accuracy as the normalization target. My concern is instead about the reference baseline, which is also a normalization target in Eq. (6) and the direct reference for BWR in Eq. (9). The paper itself flags the absence of a noise-based minimum meaningful difference in Sec. 12, but does not supply the repeated baseline data needed to evaluate it. This is a concrete gap that could affect the central claim of 'measurable' optimization ability, not just the absolute MAA values. However, the paper already reports three-seed repetitions for open-weight agents and conventional baselines, and the feedback ablation provides some internal control; the missing piece is specifically the baseline noise floor. I agree with the reader's overall conditional verdict: the findings are plausible and richly reported, but the benchmark should release repeated baseline runs (or at least a no-op control) to establish that the observed win rates and normalized improvements exceed what a single noisy baseline run would produce. My concern does not overturn the verdict; it adds a specific reproducibility condition. Hence verdict stays CONDITIONAL, and the reader's verdict remains appropriate.","tokens_in":71255,"tokens_out":5065,"duration_ms":58618,"concrete_test":"For each of the 30 tasks under the limited-budget protocol, execute the fixed reference baseline configuration at least 10 times with independent seeds. Compute the per-task mean and standard deviation of baseline performance. Then compute a no-op BWR: the fraction of repeated baseline runs that strictly exceed the original baseline value. Also compute a no-op MBNS by treating each repeated baseline run as the 'final result' against the original baseline. Compare these no-op metrics to the agent and conventional-HPO BWR/MBNS values in Table 2. If the no-op BWR is comparable to or above the agent BWRs (e.g., >40%), the claim of measurable optimization ability is unsupported; if it is low (e.g., <10%), the claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that agents exhibit 'measurable experimental optimization ability' rests on the reference baseline being a reliable zero point. In Eqs. (6) and (9) (Sec. 3.4), the limited-budget baseline y_{t,0} is a single executed run per task (Table 7). If that run is an outlier—unusually poor or good relative to execution stochasticity—then MBNS, BWR, and even MAA are systematically biased. The paper explicitly acknowledges this in Sec. 12: 'The current benchmark does not separately estimate a noise-based minimum meaningful difference from repeated no-op executions.' That is a missing control. For tasks with small baseline–anchor gaps (e.g., iTransformer, TimeMixer, SparseTSF), modest baseline noise can produce large normalized-score swings, and BWR counts a strict improvement over a possibly unlucky baseline as a 'win.' Without repeated baseline runs, we cannot distinguish genuine agent skill from regression to the mean or a fluke. This is more load-bearing than anchor validity: even if anchors are perfect, an unreliable baseline undermines the primary evidence for the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AgentHPOBench, a sequential hyperparameter-optimization benchmark built from 30 executable ML repositories across seven research categories. Each task fixes a reference baseline configuration, an intervention space, and a reported paper/repository anchor; an agent observes accumulated metrics and logs over five sequential interventions and proposes valid configurations, which the harness executes and audits. Agents are scored by mean bounded normalized score (MBNS), baseline win rate (BWR), and mean anchor attainment (MAA). The paper evaluates six open-weight agents, six API agents, and three conventional HPO baselines under a unified protocol, and reports additional analyses: a full-budget check, a harness ablation, an intermediate-feedback ablation, best-so-far diagnostics, and task-composition bootstrap intervals. The central claim is that current LLM agents exhibit measurable experimental optimization ability but remain limited in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.","tokens_in":71415,"tokens_out":3266,"duration_ms":40928,"significance":"If the results hold, AgentHPOBench fills a genuine gap: it targets the ability to convert repository metrics and logs into the next hyperparameter configuration, which is not captured by tabular HPO benchmarks or general research-agent benchmarks. The paper's strengths are substantial: 30 real repositories with audited per-task tables, three-seed repetitions for all locally controllable methods, 20,000-resample category-stratified bootstrap intervals, a full-budget comparison, a controlled feedback ablation, and explicit reporting of task-level raw metrics and normalized scores. The scoring definitions are transparent and no parameters are fitted to the headline claim, so there is no circularity. The main weakness is the absence of a repeated-execution noise floor for the single-run reference baselines, which is acknowledged in Sec. 12 but is load-bearing for the primary evidence.","major_comments":[{"comment":"The reference baseline y_{t,0} is a single executed run per task under the limited-budget protocol (Table 7), and MBNS, BWR, and MAA all use this value as a fixed zero point. If that single run is unusually good or poor relative to execution stochasticity, every metric is systematically biased. The paper explicitly acknowledges in Sec. 12: 'The current benchmark does not separately estimate a noise-based minimum meaningful difference from repeated no-op executions.' This is not a minor caveat: for tasks with a small baseline-anchor gap (e.g., iTransformer, TimeMixer, SparseTSF in Table 6), modest baseline noise can produce large normalized-score swings, and BWR counts a strict improvement over a possibly unlucky baseline as a win. Without repeated baseline runs, the headline claim of 'measurable experimental optimization ability' cannot be cleanly separated from regression to the mean. T","section":"Sec. 3.4, Eq. (6)-(9); Sec. 12; Table 7"},{"comment":"API agents are evaluated on a single audited run per task. The paper's justification (hosted endpoints do not expose an immutable checkpoint or reproducible seed) is reasonable, but the consequence is that the main cross-agent comparisons for the leading methods, including Claude Sonnet 4.6, rest on a single draw. The task-composition bootstrap intervals in Table 11 explicitly state that for API agents they reflect only benchmark composition, not repeated API executions. Since the open-weight agents and conventional baselines are averaged over three seeds, the API point estimates are not on the same evidential footing. At minimum, the paper should report the number of effective repetitions it would take to detect the observed differences, or clearly mark the API rankings as exploratory.","section":"Sec. 9, API-based agents; Table 11"},{"comment":"The scoring assumes that the repository anchor a_t is accurate, comparable, and attainable under the benchmark's execution budget. The paper states that anchors come from a paper, an official repository report, or a documented full-setting reproduction, but it does not audit whether those anchors are reproducible with the same evaluation code, data split, and metric definition used by the benchmark. If an anchor is stale or was obtained with a different budget or postprocessing, then the absolute values of MBNS and MAA, including the claim that agents 'remain below the reported reference performance,' are not well supported. The benchmark's shared denominator makes relative method comparisons robust to anchor miscalibration, but the absolute interpretation of the headline results depends on anchor validity. A concrete test would be to reproduce a sample of anchors under the full-budget p","section":"Sec. 7, Table 6; Sec. 3.4, Eq. (12)"}],"minor_comments":[{"comment":"The same bounded normalized score is defined twice with different notation (NS and BNS in Sec. 3.4, then again in Sec. 10 with y and g_t). Unifying the notation and pointing to a single canonical definition would reduce confusion.","section":"Eq. (7) and Eq. (12)"},{"comment":"The full-budget result for Claude Sonnet 4.6 uses a smaller set of agents than the main table. Since the full-budget baseline is the median of three observations (Sec. 7) while the limited-budget baseline is a single run, the comparison between tables should note this asymmetry explicitly.","section":"Table 2 and Table 3"},{"comment":"The best-so-far diagnostic is useful, but the text does not specify how ties between equal best metrics are broken. Given that BWR counts strict improvements, the tie-breaking rule could affect the reported BWR values.","section":"Sec. 11.3, Table 12"},{"comment":"The metric-sensitivity paragraph mentions that bounding to [-1,1] can cause score saturation, but the paper does not quantify how many tasks actually saturate at the bounds. Reporting the number of saturated tasks per method would help interpret aggregate scores.","section":"Sec. 12"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmark contribution with a clear central finding and unusually transparent reporting. The main issue is the acknowledged absence of a repeated-baseline noise floor, which weakens the causal reading of the headline claim. I believe this is fixable within the paper's scope (adding repeated no-op runs and re-computing the affected metrics), and the paper should be given a chance to address it. The API single-run issue is also important for the specific rankings but does not undermine the benchmark itself. I would not go to reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AgentHPOBench is worth engaging with. The core construction—30 executable repository tasks, each reduced to a fixed baseline, a constrained intervention space, and five sequential decisions, with agents reading logs and metrics to propose the next config—fills a real gap between MLGym-style whole-workflow benchmarks and tabular HPO suites. The evaluation is unusually transparent for this area: per-task tables, three seeds for open-weight agents and conventional baselines, bootstrap intervals over task composition, a full-budget check, a feedback ablation, and a best-so-far analysis that shows random search beating all open-weight agents under final-step scoring. That last result is the kind of honest reporting that makes me trust the headline: agents show measurable but limited capability, and the capability is fragile.\n\nThe soft spots are real but proportionate. API agents are single-run with no repeated-run error bars; the paper says so explicitly. More load-bearing, the reference baseline itself is a single execution per task in the limited-budget protocol, and the scoring equations normalize against that value. If that one run is lucky or unlucky, MBNS, BWR, and MAA all shift, and for tasks with tiny baseline–anchor gaps (iTransformer, TimeMixer, SparseTSF) the swing can be large. The paper acknowledges this in Sec. 12—'the current benchmark does not separately estimate a noise-based minimum meaningful difference from repeated no-op executions'—but acknowledges it as a missing control rather than providing it. That weakens the quantitative precision, especially for fine-grained rankings around zero. It does not, in my view, overturn the qualitative picture: the top agents' margins are large enough that a single baseline run is unlikely to explain MBNS 0.4 or BWR 77%. The anchor-validity concern the reader raised is secondary; anchors are externally sourced and clearly documented per task.\n\nThe main practical issue is that no code or machine-readable task specs are shipped yet. For a benchmark, that is the whole point. I would not desk-reject on that, but it must be a stated condition of acceptance.\n\nWho gets value: anyone evaluating LLM agents for experimental iteration, and anyone building HPO benchmarks that want a realistic, log-rich middle ground. I'd cite it and bring it to the group. Recommend peer review with the condition that artifacts are released and repeated baseline runs are added, or at least the noise floor is quantified.","headline":"A carefully built benchmark for sequential HPO from repository logs; solid results, but the single-run baselines need flagging and the artifacts aren't out yet.","tokens_in":71985,"tokens_out":2374,"would_cite":true,"duration_ms":27966,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current LLM agents, given an executable research repository, a validated baseline run, and five chances to intervene, improve the baseline on a clear majority of 30 tasks but routinely plateau or sacrifice earlier gai","keywords":["LLM agents","hyperparameter optimization","sequential decision making","executable benchmarks","experimental feedback","research repositories","HPO baselines","agent evaluation"],"falsifier":"Re-run the 30 tasks under the limited-budget protocol with a no-op agent that keeps the baseline configuration for all five interventions, repeated across seeds, and compute the distribution of the three metrics. If that execution-noise floor produces MBNS variation on the same scale as the differences between the top agents (0.148 for the best open-weight agent versus 0.018 for the BOHB-style baseline), then the headline ordering is not robust; additionally, reproduce each Table 6 anchor from the repository's own code at 10% budget and count how many are attainable, since unattainable anchors","tokens_in":71092,"feed_emoji":"🧪","tokens_out":7536,"duration_ms":78775,"temperature":0.7,"pith_summary":"AgentHPOBench asks a focused question: can an LLM agent read the logs and metrics of an executed machine-learning experiment and convert that evidence into the next hyperparameter configuration? The paper builds 30 executable tasks from real research repositories and runs 12 agents plus conventional HPO methods under a single protocol: a shared validated baseline, five sequential interventions, and roughly 10% of the original training budget. The results show that current agents do have measurable experimental-optimization ability — several improve the baseline on a clear majority of tasks, and the strongest agent wins on 76.7% of tasks at the final step — but their gains are fragile: later interventions often plateau or discard earlier improvements, and even the best agent attains only about 80% of the reported reference performance. The paper's deeper point is that 'finding a good configuration' and 'reliably refining it through feedback' are distinct capabilities: under a best-so-far criterion, conventional HPO methods that return their best trial instead of their final one outperform the open-weight agents, because agents keep experimenting past their best find.","feed_headline":"Agents tune ML runs from logs, then often lose the gains","feed_subtitle":"A 30-task benchmark shows LLM agents improve experiments from evidence, but later steps often erase earlier wins.","key_machinery":"The load-bearing mechanism is the benchmark itself: 30 executable ML repositories, each reduced to a fixed intervention space of configurable fields, a shared validated baseline, a paper- or repository-reported anchor, and a standardized per-trial trace of configuration, metric, and logs. The scoring identity carries the argument: the bounded normalized score compares the final result to the baseline and clamps that ratio to [−1, 1], while baseline win rate counts tasks where the final result strictly beats the baseline, and mean anchor attainment measures absolute nearness to the reported reference. Two further components do the analytical work: the best-so-far diagnostic, which scores the","core_discovery":"The central claim, stated on the paper's own terms, is that current LLM agents exhibit measurable experimental-optimization ability across domains but face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance. The benchmark formalizes hyperparameter optimization as a sequential decision problem over executable research repositories: the agent observes a validated baseline, then at each of five steps receives the accumulated configurations, target and auxiliary metrics, and execution logs and must propose a valid configuration; the result after the final intervention — not the best intermediate result — is wh","pith_inferences":["Inference: the anchor-relative scoring (Eq. 6) is fragile exactly where the baseline-to-anchor gap is small; an immediate extension is to compute a repeated-no-op noise floor for each task so that reported score gaps smaller than execution noise are not treated as meaningful.","Inference: because the benchmark exposes raw logs and auxiliary metrics, it is a natural test bed for a targeted hypothesis — that agents which explicitly report reading log anomalies (e.g., a diverging loss or NaN gradient) in their reasoning achieve higher normalized scores, which would directly test the 'complex log diagnosis' limitation the paper identifies.","Inference: the final-step versus best-so-far gap suggests a one-number diagnostic for future agents: the fraction of tasks on which the agent's final configuration equals or beats its best-so-far configuration (a 'gain retention rate'), which could complement baseline win rate and make the refinement failure mode more readable.","Inference: the strongest agent's edge may partly reflect conservative behavior — keeping the baseline config when unsure — rather than effective exploration; an ablation that forces at least one hyperparameter change per intervention would separate cautiousness from optimization skill."],"forward_implications":["Scoring agents by their final intervention rather than their best-so-far trial lowers their measured performance; conventional HPO methods that return the incumbent close the gap or overtake the open-weight agents, so the suite separates search quality from the ability to preserve a discovery.","Withholding intermediate metrics and logs reduces both normalized score and baseline win rate, indicating that the sequential evidence channel is an active ingredient in agent-driven optimization, not just a wrapper.","Increasing the execution budget to the full training budget raises normalized score and anchor attainment but not uniformly baseline win rate, so the three metrics must be read together rather than as interchangeable rankings.","Changing the execution harness changes all three metrics in different directions for the same agent, meaning harness choice is part of the evaluated agent system and must be held fixed for fair comparison.","Use of the final-step criterion changes which methods lead: the paper's best-so-far table shows random search with an incumbent rule ahead of every open-weight agent, which reframes the agent deficit as one of decision-making over time rather than pure search capacity."],"fun_headline_variants":["Agents tune ML from logs, then erase prior wins","LLM agents can't sustain HPO gains across steps","Sequential HPO benchmark reveals agent drift","From evidence to errors: agents fail at iterative HPO","30-task test: agents lose early HPO progress"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the paper- or repository-reported anchors used as normalization targets are accurate, comparable across the 30 tasks, and actually achievable under the 10%-budget protocol; if any anchor is stale, overly optimistic, or unreachable at that budget, the normalized scores and anchor-attainment percentages systematically misstate how close agents are to reference performance.","fun_headline_variants_meta":{"raw":{"variants":["Agents tune ML from logs, then erase prior wins","LLM agents can't sustain HPO gains across steps","Sequential HPO benchmark reveals agent drift","From evidence to errors: agents fail at iterative HPO","30-task test: agents lose early HPO progress"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1210,"prompt_tokens":660,"completion_tokens":550,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":404,"completion_tokens_details":{"reasoning_tokens":483}},"tokens_in":404,"tokens_out":550,"duration_ms":6157,"temperature":1.0,"reasoning_tokens":483,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:11:17.437731+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 30 tasks under the limited-budget protocol with a no-op agent that keeps the baseline configuration for all five interventions, repeated across seeds, and compute the distribution of the three metrics. If that execution-noise floor produces MBNS variation on the same scale as the differences between the top agents (0.148 for the best open-weight agent versus 0.018 for the BOHB-style baseline), then the headline ordering is not robust; additionally, reproduce each Table 6 anchor from the repository's own code at 10% budget and count how many are attainable, since unattainable anchors","supporting_citations":[],"review_version":1}