{"id":"de2240e8-cb96-4e52-9147-01b38bab63ce","arxiv_id":"2506.10764","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OPT-BENCH is a 30-task benchmark showing that LLM agents generally improve optimization results when they are given their own past solutions and feedback.","lead":"A new benchmark, OPT-BENCH, tests whether large language models can get better at optimization problems by learning from their own previous attempts, using 20 Kaggle machine learning tasks and 10 classic NP-hard problems. The authors also introduce an agent framework, OPT-Agent, and report that giving the model its own history improves results on most models and tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline evidence for 'significantly enhances' rests on an Improvement Rate that mixes lower- and higher-is-better metrics; as defined in Section 3.2, IR>1 is not a valid sign of improvement, so the central quantitative claim needs correction before the abstract's claim is supported.","rationale":"The abstract's central claim requires both a clean history-vs-no-history comparison and summary statistics that mean what they claim. The reader's weakest assumption targets the cleanliness of the baseline; that is a legitimate reproducibility concern, but the paper explicitly asserts the sole difference is historical information, so I do not treat it as the most decisive issue. The Improvement Rate formula in Section 3.2 is a more concrete and verifiable flaw: it averages ratios for lower-is-better and higher-is-better metrics without inverting one class, so IR>1 cannot be read as improvement. Table 3's NP win counts are also mixed (e.g., gpt-4o 2/8 at 20 steps), which reinforces the need for a significance test before claiming enhancement 'across both ML and NP tasks.' These are correctable flaws in an otherwise useful benchmark: the task suite, open-source artifacts, and qualitative traces are real contributions. Because the quantitative pillar can be repaired by recomputation and the reader already issued a CONDITIONAL verdict, I see no reason to change that verdict; the conditions should include fixing the IR definition/reporting and adding a significance check.","tokens_in":16189,"tokens_out":10048,"duration_ms":110954,"concrete_test":"Using the released OPT-BENCH code and raw per-task metrics, recompute every IR(w,w.o) cell in Table 2 with direction-corrected ratios: for lower-is-better metrics use β_i/α_i, and for higher-is-better metrics use α_i/β_i, then average over the 20 tasks. If the corrected values are not >1 for the large majority of model×step cells, the quantitative evidence for history-driven enhancement fails; report the corrected table and, as a secondary check, a paired sign test on the NP Win Counts (e.g., gpt-4o's 2/8 at 20 steps) to see whether the cross-domain claim differs from chance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines IR(α,β) = (1/n) Σ α_i/β_i, with α_i the metric under the history condition and β_i the baseline. OPT-BENCH-ML's 20 tasks (Table 7) mix 12 lower-is-better metrics (MSE, RMSE, MAE, SMAPE, RMSLE, log loss) with 8 higher-is-better metrics (accuracy, AUC, F1, quadratic kappa). The formula never accounts for metric direction: for a lower-is-better metric, an actual improvement yields α_i/β_i < 1, while for a higher-is-better metric it yields α_i/β_i > 1. Averaging these incompatible quantities makes IR(w,w.o)>1 uninterpretable as 'improvement'; a large mean can be driven by the eight higher-is-better tasks while lower-is-better tasks regress. Section 3.3 explicitly reads IR(w,w.o)>1 as 'underscoring the value of contextual information,' and the abstract's claim that history 'significantly enhances' optimization rests in part on this evidence. The paper either inverted lower-is-better ratios without saying so, or the reported IRs and the conclusions drawn from them are invalid as written. With no error bars or significance tests, IR is one of only two quantitative supports for the headline claim, alongside Win Count.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OPT-BENCH, a benchmark containing 20 Kaggle machine-learning tasks and 10 classical NP problems, together with OPT-Agent, an LLM agent framework that iteratively drafts, improves, and debugs solutions using historical feedback. The authors evaluate nine LLMs from six families and report that including historical context improves optimization performance, as measured by win counts and an Improvement Rate, and also study the effects of iteration count, temperature, and draft versus refine settings. The benchmark, code, and evaluation tools are open-sourced.","tokens_in":16475,"tokens_out":4896,"duration_ms":53340,"significance":"If the empirical claims are validated, OPT-BENCH fills a clear gap: existing LLM benchmarks largely test single-pass responses, whereas iterative optimization with feedback is an important and under-measured capability. The benchmark's assets—real-world ML tasks, human-verified initial solutions, automated NP validation scripts, and an open-source framework—are valuable resources for the community. The paper's main weakness is that its central quantitative claim rests on a metric that is defined and interpreted inconsistently, and on win counts without statistical support. The benchmark itself is sound as a contribution; the empirical analysis needs correction before the headline claim is acceptable.","major_comments":[{"comment":"The Improvement Rate as defined, IR(α,β) = (1/20) Σ α_i/β_i, with α_i the metric under the improved setting and β_i the baseline metric, is not a valid measure of improvement when the 20 ML tasks mix lower-is-better metrics (12 tasks per Table 7, e.g., MSE, RMSE, RMSLE, log loss) with higher-is-better metrics (8 tasks, e.g., accuracy, AUC, F1, kappa). For a lower-is-better metric, an actual improvement gives α_i/β_i < 1, while for a higher-is-better metric it gives a ratio > 1; averaging these incompatible quantities makes IR > 1 uninterpretable. Table 2 reports IR(w,w.o) > 1 for every model and step count, which is only possible if lower-is-better ratios were inverted without documentation. Section 3.3 then reads IR > 1 as \"underscoring the value of contextual information,\" and the abstract's claim that history \"significantly enhances\" optimization relies on this measure. Please re-analyze the data using direction-aware ratios, report per-task signed improvements, and correct the conclusions accordingly.","section":"Section 3.2, Table 2"},{"comment":"The win-count comparisons are based on a single run per condition, with no repeated seeds, confidence intervals, or significance tests. With only 20 ML tasks and 10 NP problems, win counts such as 12/8, 11/9, or 10/10 are within the range of random variation, so the abstract's word \"significantly\" is not supported by the evidence. Please report the number of seeds or runs, and provide a statistical test (e.g., a binomial test against the null of equal performance, or a paired test on per-task metric changes) to substantiate the claim.","section":"Section 3.2, Tables 2-3"},{"comment":"The baseline \"without historical information\" is described as differing only by the absence of historical information, but the prompt template in Figure 4 contains separate fields for \"History Information,\" \"Previous Solution,\" \"Previous (buggy) Implementation,\" and \"Previous (buggy) Output.\" If the baseline omits all of these, the comparison does not isolate \"history\" from \"having a previous solution,\" and the improvement might simply reflect the ability to start from an existing draft rather than learning from feedback. Please specify exactly which prompt fields are kept and which are removed in the baseline, and describe what action the baseline agent performs at each optimization step.","section":"Section 3.2 and Figure 4"},{"comment":"The paper's own Limitations section acknowledges that \"averaging performance metrics may introduce scale inconsistencies\" for OPT-BENCH-ML. The Improvement Rate in Section 3.2 is exactly such an average of ratios across heterogeneous metrics (MSE, accuracy, etc.). Since the central claim depends on this rate, the acknowledgment is not sufficient: the analysis should be corrected, for example, by reporting per-task signed improvements or normalized changes, and the corrected metric should be used to re-evaluate all claims in Section 3.3 and the abstract.","section":"Section 5, Limitations"}],"minor_comments":[{"comment":"There is a typo: \"which is a key sapect for future improvement\" should read \"a key aspect.\"","section":"Section 3.3"},{"comment":"The NP example is labeled \"Hamiltonian-cycle\" but the task description mentions finding a subgraph and maximizing the number of visited vertices, while the validation script checks for a cycle. Please clarify whether the task is to find a Hamiltonian cycle or to find the longest cycle/subgraph in a graph, and align the description with the actual validation logic.","section":"Figure 2"},{"comment":"The table refers to \"NPHardBench [10]\" but the reference list and Section 4.1 use \"NPHardEval\"; the names should be consistent.","section":"Table 1"},{"comment":"The text says each NP problem contains \"five distinct instances,\" but it is not specified how win counts and ranks aggregate across those instances. Please state whether win counts are computed per instance, per problem, or per problem set, and report the aggregation rule.","section":"Section 2.1, Tables 7-8"},{"comment":"The trace diagrams are dense and hard to read at the current resolution. Please ensure the published figures are legible, or provide a zoomed view of representative paths.","section":"Figures 5 and 6"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and framework are useful contributions, and the data appear to be genuinely collected rather than synthesized. The main issue is that the headline quantitative measure, IR, is internally inconsistent as defined and interpreted, and the win-count evidence lacks statistical support. Both are fixable within the scope of a revision, so I would not recommend rejection; however, the abstract and Section 3.3 should not be accepted as they currently stand."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, OPT-BENCH is a genuinely useful resource: 20 Kaggle ML tasks and 10 NP problems with validation scripts, initial solutions, and a clean history-on/history-off agent comparison across 9 LLMs. Second, the paper's quantitative evidence for 'significantly enhances' is not trustworthy as written, because the Improvement Rate in Section 3.2 averages ratios of metrics without accounting for metric direction. For lower-is-better tasks (MSE, RMSE, etc.) improvement gives a ratio below 1; for higher-is-better (accuracy, AUC) it gives a ratio above 1. Mixing them makes IR>1 uninterpretable. The paper never inverts the lower-is-better ratios, so the IR columns in Tables 2, 4, and 6 don't support the abstract's claim. The stress-test note is right.\n\nThe win count metric is the better evidence: it compares per-task which condition won, and it mostly favors history for ML and NP. But there are no error bars, no repeated seeds, and no significance tests. 'Consistently enhances' and 'significantly' are overclaims. Also, the reported IR(w,init) trends over steps may be partly an artifact of the same formula problem, since those also mix directions.\n\nThe baseline control seems okay: the paper states the only difference is inclusion of historical information, and the prompt templates show the History Information field is what gets added. I wouldn't call that underspecified.\n\nCredit where due: the dataset curation is careful, the validation scripts for NP tasks are concrete, the scope (30 tasks, 9 models, multiple step counts and temperatures) is solid, and everything is open-sourced. The limitation section even admits that averaging heterogeneous metrics may cause scale inconsistencies, but then the paper uses exactly such an average without fixing it.\n\nWho's this for? Researchers building agents for iterative code/ML optimization; folks benchmarking LLM self-improvement. The benchmark itself could be a valuable addition, but the paper needs a corrected IR definition (or remove it), plus error bars or at least cautious language. I'd send it to review—the resource deserves referee attention—but expect a major revision before the headline claim is acceptable.","headline":"A useful new agent benchmark whose headline 'history helps' claim is undermined by a flawed Improvement Rate formula that mixes lower- and higher-is-better metrics.","tokens_in":17029,"tokens_out":2190,"would_cite":false,"duration_ms":24304,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Incorporating historical context improves LLM optimization across 20 ML tasks and 10 NP problems, according to a new benchmark and agent framework.","keywords":["LLM agents","iterative optimization","historical feedback","benchmarking","machine learning tasks","NP-hard problems","solution refinement","agent evaluation"],"falsifier":"Run both conditions on the same 30 tasks, but in the history condition replace the agent's own previous solutions and metrics with an equal-length history drawn from a different task of the same type. If the advantage over the no-history baseline persists, the gain is not specific to learning from the current problem's feedback and the central claim would be falsified; if the advantage disappears, the paper's interpretation is supported.","tokens_in":16013,"feed_emoji":"🔁","tokens_out":10163,"duration_ms":106688,"temperature":0.7,"pith_summary":"OPT-BENCH asks whether an LLM can get better at a problem by seeing what it tried before and what score that attempt earned, instead of answering once from scratch. The paper introduces a 30-task benchmark, 20 machine-learning tasks drawn from Kaggle competitions and 10 classical NP-hard problems, and an agent loop, OPT-Agent, that drafts a solution, validates it, then improves or debugs it using the history of previous solutions, metrics, and error messages. Across nine LLMs from six model families and 5, 10, and 20 optimization steps, the paper reports that supplying this historical context raises win counts and improvement rates relative to a no-history baseline for most models, on both task families. The reason to care is that real optimization is iterative: a benchmark that measures feedback-driven refinement tests a different capability than single-pass question answering.","feed_headline":"Nine LLMs improve on 30 tasks when they get past feedback","feed_subtitle":"A 30-task benchmark shows that iterative refinement with history beats single-shot generation for most models.","key_machinery":"The load-bearing mechanism is OPT-Agent's draft-improve-debug loop. At each step the LLM receives the problem description, submission format, and, when history is enabled, the previous solution, its metric, and any validation error; it then either drafts a fresh solution, refines the last valid one, or repairs the last buggy one. A rule-based or metric-based validator turns every attempt into a concrete score or a validity verdict, so the loop always has a checkable quality signal to feed back into the next prompt. The headline comparison metric is Win Count, the number of tasks where the history-augmented run beats the no-history run under otherwise identical settings, supplemented by Improvement Rate, Buggy Rate, and Rank.","core_discovery":"The central claim is that historical context is a measurable and general driver of LLM optimization quality. Using identical task definitions, datasets, validators, and evaluation metrics, OPT-Agent with history outperforms OPT-Agent without history on a majority of the 20 ML tasks and 10 NP problems for most of the nine LLMs tested; the paper's improvement-rate numbers are above 1 in nearly all ML configurations, and win counts favor the history condition on NP problems more often than not. The paper also reports that longer optimization horizons generally improve final metrics, that moderate temperatures tend to balance exploration and solution validity, and that open-source models produce more invalid solutions and trail proprietary models on NP tasks. On the ML side, the benefit of history appears as better hyperparameter choices and feature engineering; on the NP side, the benefit is real but weaker, because models often discard the previous solution and start over instead of repairing the specific failing constraint. The intended conclusion is that evaluating LLMs on iterative, feedback-driven optimization is feasible and that this capability is partly independent of one-shot reasoning strength.","pith_inferences":["The paper's central result does not isolate why history helps; a natural follow-up is to give the no-history baseline a fixed amount of extra context that is task-irrelevant, so any remaining gain in the history condition is specifically about the current problem's feedback.","The NP half of the benchmark could be used to test whether models genuinely reuse valid substructures from earlier attempts or merely regenerate answers, by logging how often the final solution contains segments of previous valid solutions.","Because the paper names growing prompt size as a limitation, a testable extension is that compressing history, summaries of what changed and by how much rather than full previous code, preserves most of the benefit at much lower token cost.","A skeptical reading is that the ML win counts may partly reflect the no-history baseline losing access to the previous code it needs for debugging; isolating that would require a baseline that receives the previous error message and metric but not the previous solution."],"forward_implications":["Benchmarks for LLM problem-solving should include iterative loops with feedback, since a single generation no longer measures the capability these results target.","Optimization budgets matter: the paper reports that improvement over the initial solution rises with the number of steps, from 5 to 20, for most models.","Temperature should be treated as a per-model hyperparameter for optimization agents, with low-to-moderate values usually but not always best.","Draft-based optimization should be evaluated separately from refinement, since it yields higher invalid-solution rates yet often higher improvement rates when solutions are valid.","Future LLM development can use the 30 validated tasks and public scripts as a reproducible yardstick for optimization ability."],"supporting_citations":[{"why":"The preceding agent loop that this framework extends; it also generates the initial ML solutions that the benchmark refines.","marker":"[18]"},{"why":"The static, single-shot complexity-class benchmark whose lack of iterative feedback motivates the new benchmark.","marker":"[10]"},{"why":"A comparison baseline in the table showing a prior ML-agent benchmark without long-horizon optimization.","marker":"[5]"},{"why":"A comparison baseline for ML agent experimentation whose success-rate framing OPT-BENCH extends with win counts and improvement rates.","marker":"[17]"},{"why":"The verbal-reinforcement baseline that the history condition is designed to test under controlled, reproducible tasks.","marker":"[30]"},{"why":"The iterative self-refinement approach that supplies the general prior result the benchmark evaluates across models and temperatures.","marker":"[23]"}],"fun_headline_variants":["Past feedback boosts LLM optimization on 30-task benchmark","LLM agents improve optimization with history in 30 tasks","OPT-BENCH: history-aware LLMs outperform on search problems","Iterative refinement with history wins on 30 optimization tasks","LLMs with history beat single-shot on 30 optimization tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison only demonstrates that history helps if the no-history baseline is otherwise identical in task information, prompt formatting, and starting conditions; the paper states that the only difference is the absence of historical information but does not fully specify how the baseline prompt is constructed, leaving open the possibility that the gain comes from extra context rather than from learning.","fun_headline_variants_meta":{"raw":{"variants":["Past feedback boosts LLM optimization on 30-task benchmark","LLM agents improve optimization with history in 30 tasks","OPT-BENCH: history-aware LLMs outperform on search problems","Iterative refinement with history wins on 30 optimization tasks","LLMs with history beat single-shot on 30 optimization tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3477,"prompt_tokens":998,"completion_tokens":2479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":2395}},"tokens_in":614,"tokens_out":2479,"duration_ms":18600,"temperature":1.0,"reasoning_tokens":2395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:18:06.921102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run both conditions on the same 30 tasks, but in the history condition replace the agent's own previous solutions and metrics with an equal-length history drawn from a different task of the same type. If the advantage over the no-history baseline persists, the gain is not specific to learning from the current problem's feedback and the central claim would be falsified; if the advantage disappears, the paper's interpretation is supported.","supporting_citations":[],"review_version":1}