{"id":"84624ca8-17fb-45b0-a229-810d94d9f14b","arxiv_id":"2502.00964","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ML-Dev-Bench introduces 30 ML workflow tasks and finds agent success drops sharply as tasks become more open-ended, with OpenHands-Sonnet best at 50%.","lead":"ML-Dev-Bench, a 30-task benchmark for applied machine learning workflows, ranks AI coding agents on dataset handling, training, debugging, and API integration. In a single-run evaluation, OpenHands with Claude Sonnet scored highest at 50%, but the small differences and lack of repeated runs make the ranking only weakly supported.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparative ranking and the 'open-ended tasks fail' claim rest on single-run binary outcomes per task; a single flip among the 30 binary outcomes would reorder OH-Sonnet and ReAct-Sonnet.","rationale":"The reader's weakest_assumption—single-run evaluation without variance—is exactly the load-bearing concern. The strongest claim about declining success with open-endedness is anchored in 0/6 Model Performance results and small category denominators; the comparative ranking is anchored in a one-task margin. The paper is honest about this limitation (Section 7 future work), and the benchmark artifact itself is a useful contribution, so I do not recommend rejection. But the empirical claims need either repeated trials, deterministic seeds, or a clear statement that the results are single-run observations without comparative significance. I agree with the reader's conditional verdict and would not change it. The concrete test I propose is deliberately modest: re-run the decisive configurations a few times to see whether the one-task margin and the 0/6 model-performance outcomes persist. If they do, the conditional acceptance is justified; if not, the verdict should move toward rejection of the comparative claims.","tokens_in":6599,"tokens_out":1319,"duration_ms":12118,"concrete_test":"Re-run the five agent configurations on at least the 14 tasks that are jointly successful for OH-Sonnet and ReAct-Sonnet, plus the Model Performance category, for 5 independent trials each (approximately 250 runs, feasible with the public Calipers framework). If ReAct-Sonnet's mean success rate meets or exceeds OH-Sonnet's, or if any agent completes a Model Performance task in any trial, the headline ranking and 'no agent succeeded in open-ended tasks' claim would need to be weakened to a single-run observation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—that success declines with open-endedness and that OH-Sonnet outperforms ReAct-Sonnet—is supported only by single-run binary success/failure per agent-task pair (Table 2: OH-Sonnet 15/30 vs ReAct-Sonnet 14/30). The paper itself lists 'studying variance in success metrics across multiple runs' as future work in Section 7, confirming that variance was not measured. For a 15/30 vs 14/30 comparison, a single task outcome is the entire margin. More importantly, the headline pattern across categories is also fragile: within each category the denominators are tiny (e.g., 3/3 vs 3/3 vs 2/3 vs 1/3 vs 0/3 in Dataset Handling; 0/6 for Model Performance in every column), so category-level rates are dominated by one or two tasks. Because the binary success criterion is the only outcome metric and the agent runs are stochastic, the reported ordering and the monotonic-decline narrative are not statistically distinguishable from noise without repeated trials or at least a documented deterministic seeding. This is not an internal inconsistency but a correctness-risk issue: the empirical basis is too thin to support the strength of the comparative conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ML-Dev-Bench, a benchmark of 30 tasks intended to evaluate AI agents on applied machine-learning development workflows, spanning dataset handling, model training, debugging, model implementation, API integration, and performance improvement. The authors also present Calipers, their evaluation framework, and report results for five agent-model configurations: ReAct-Sonnet, OpenHands-Sonnet, OpenHands-Gemini, AIDE-4o, and ReAct-4o. The central findings are that overall success rates are modest (best 50%), that the two best agents are OpenHands-Sonnet and ReAct-Sonnet, and that performance degrades as tasks become more open-ended, with no agent solving any model-performance task.","tokens_in":6791,"tokens_out":2764,"duration_ms":31910,"significance":"If the benchmark and its results are taken at face value, the paper addresses a genuine gap: most coding benchmarks focus on isolated code edits, whereas ML-Dev-Bench targets multi-stage workflows involving data, training, existing codebases, and external tools. The open-source release of the benchmark and the Calipers framework, together with the availability of agent traces, are concrete contributions that can support community reuse and extension. The qualitative observation that all evaluated agents fail on open-ended model-performance tasks is a useful, falsifiable finding. However, the comparative agent ranking and the category-level decline narrative are currently supported only by single-run binary outcomes on small per-category task counts, so the evidential weight of the specific numbers is considerably weaker than the prose suggests. The paper is best viewed as a benchmark-and-initial-evaluation report whose methodological limitations should be addressed before the comparative claims are treated as established.","major_comments":[{"comment":"The headline comparison between OpenHands-Sonnet (15/30) and ReAct-Sonnet (14/30) rests on a margin of exactly one task, and each agent-task outcome is a single stochastic run. No repeated runs, seeds, or uncertainty estimates are reported, and Section 7 explicitly lists 'studying variance in success metrics across multiple runs' as future work. As presented, the data do not support the statement that OpenHands-Sonnet outperforms ReAct-Sonnet; the observed difference is within the noise expected from one-run-per-task evaluations. The authors should either add repeated-run results with variance or confidence intervals, or explicitly relabel the ranking as a preliminary observation rather than a measured result.","section":"§6 and Table 2"},{"comment":"The claim that success rates 'decrease as tasks become more open-ended and complex' is based on category-level aggregates with very small denominators: Dataset Handling has 3 tasks, API Integration has 1, Model Training has 6, Debugging has 7, Model Implementation has 7, and Model Performance has 6. A one-task change can alter a category rate by 10 to 33 percentage points, so the monotonic-decline narrative is not statistically distinguishable from noise at this scale. Moreover, the paper does not operationally define 'open-endedness' or provide any independent rating of task openness; the categories themselves are assumed to form an openness ordering. I recommend adding per-task difficulty annotations or at least a sensitivity analysis showing how the qualitative trend changes when individual tasks are excluded.","section":"§6, Table 2, and Table 1"},{"comment":"There is an internal inconsistency in the reported model-performance failures: Section 6.2 states that OpenHands-Sonnet 'failed to complete any of the model performance tasks (0/5)', while Table 2 reports 0/6 for Model Performance for every agent, and Table 3 lists six distinct performance tasks. This discrepancy is load-bearing because the model-performance category is central to the paper's main qualitative claim. The authors should correct the count and verify that all per-task entries in Table 3 are consistent with the aggregate numerators and denominators in Table 2.","section":"§6.2 vs Table 2"},{"comment":"The evaluation relies entirely on binary success/failure determined by task-specific validation logic, but the paper gives only a high-level list of validation types (running tests, checking artifacts, evaluating checkpoints, querying wandb) without describing the actual validators, their thresholds, or any manual verification of a sample of failures and successes. Since the central results are measurements derived from these validators, a bug or overly strict/lenient validator in even a few tasks could change the reported ranking. The authors should document the validation code or include a detailed per-task validation specification in the appendix, and ideally report any human checks performed on the generated artifacts.","section":"§4.1 and §3.1"}],"minor_comments":[{"comment":"There are several typographical and grammatical errors that should be corrected, including 'While existing benchmarks... don't capture' in Section 2, 'performing agents performing agents' in Section 6, 'doesnt' in multiple places (Sections 5 and 6), and inconsistent hyphenation of 'Openhands' versus 'OpenHands'.","section":"Abstract and §1, §2"},{"comment":"Table 3 is very hard to read: only failure marks (×) are shown for most cells, while the caption refers to 'Success Rate' but no explicit success symbol appears in most columns; the reader must infer success from the row subtotals. Please use explicit check/cross symbols in every cell and align the table with the reported numerators and denominators.","section":"Table 3"},{"comment":"The token-cost comparison in Table 4 covers only ReAct-Sonnet and OpenHands-Sonnet, while Section 6 discusses cost trends for other agents. The authors should state clearly why costs for the other configurations are omitted, and in the text distinguish 'total tokens' from 'cost' since the two columns do not correlate monotonically in several rows.","section":"§5.2 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv-style benchmark report rather than a full journal article. The benchmark itself appears genuinely useful and the open-source release is commendable, but the empirical evaluation as currently designed (single-run binary outcomes, small per-category counts, one inconsistent count) is not yet at the standard required for a published comparative claim. The authors' own future-work section acknowledges the variance issue. I would encourage the editor to treat this as a benchmark-plus-initial-results contribution that needs methodological hardening—preferably repeated runs or explicit uncertainty quantification, and a corrected, fully consistent set of tables—before acceptance. The lack of statistical testing is not by itself disqualifying, but the one-task margin between the top two agents makes the current ranking claims unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ML-Dev-Bench is a worthwhile addition to the ML-agent benchmarking space, but do not trust the head-to-head ranking. The paper gives you 30 realistic ML development tasks, open-sourced with a validation framework, and it evaluates five agent-model configurations. The design is honest: tasks span dataset handling, training, debugging, model implementation, API integration, and performance improvement, and validation checks go beyond code tests to include artifacts, checkpoints, and wandb logs. The cost and token tables are a nice extra. I appreciate that the authors published the benchmark, list failure modes, and admit variance analysis is still to be done.\n\nThe soft spots are real. Every agent-task pair is a single binary run. OH-Sonnet's 15/30 beats ReAct-Sonnet's 14/30 by exactly one task, so the headline ranking is a coin flip away from reversing. Category-level numbers are even thinner: Model Performance is 0/6 across the board, but each cell is one attempt, and Debugging 57% is 4/7. The paper's claim of a \"consistent pattern\" of decline with open-endedness is directionally plausible, and the fact that all five configurations score zero on every performance task is suggestive, but without repeated runs or seeded determinism, you cannot distinguish the trend from noise. There is also a small internal inconsistency: Section 6.2 says OH-Sonnet failed 0/5 Model Performance tasks, while Table 2 shows 0/6. And the HumanEval citation points to the wrong reference. Both are easy fixes.\n\nThe benchmark itself is the real contribution, not the comparative numbers. The tasks look practical and the validation logic seems reasonable. I would treat this as a resource to build on rather than a settled evaluation. For peer review, I would send it to a workshop or a journal with a revision requirement: add repeated trials, report variance or at least seed control, fix the count and citation issues, and soften the ranking claim. Nobody should desk-reject this; it is sincere and useful. Just make the empirical claims match the evidence.\n\nRecommendation: engage with it, but as a benchmark contribution, not as evidence about which agent is best. Take it to reading group if you discuss benchmarking methodology.","headline":"A useful, honest ML-dev benchmark whose head-to-head ranking is fragile because every agent-task pair is a single binary run.","tokens_in":7290,"tokens_out":2462,"would_cite":true,"duration_ms":24050,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Applied machine-learning agents pass well-specified subtasks and fail open-ended model-improvement work, the paper's 30-task benchmark shows.","keywords":["AI agents","machine learning development","agent benchmarks","open-ended tasks","model fine-tuning","debugging","iterative improvement","empirical evaluation"],"falsifier":"Run the 30-task suite several times per agent configuration and recompute per-category success rates; if any model-performance task is ever completed by an agent in this family, or if the best-versus-second gap flips across runs, the claim that open-endedness predicts failure and the headline ranking would need revision.","tokens_in":6380,"feed_emoji":"🧪","tokens_out":5218,"duration_ms":53726,"temperature":0.7,"pith_summary":"This paper introduces ML-Dev-Bench, a 30-task benchmark for applied machine-learning development work, spanning dataset handling, model training, debugging, model architecture changes, API integration, and baseline improvement. The authors ran three agent architectures through the suite and measured binary pass/fail per task. The central result is a gradient: agents succeed on well-specified tasks such as dataset setup and simple debugging, succeed sometimes on structured training and implementation work, and fail completely on open-ended 'improve this model's performance' tasks. The best configuration passed 15 of 30 tasks, so the practical conclusion is that today's agents can assist with bounded ML subtasks but cannot yet autonomously drive iterative model improvement.","feed_headline":"No AI agent solved an open-ended ML tuning task","feed_subtitle":"In a 30-task benchmark, the best agent passed 50 percent; all six baseline-improvement tasks failed.","key_machinery":"The benchmark's organizing device is an openness gradient: 30 tasks binned into six categories ordered by how much specification the agent receives, from download-and-preprocess datasets to 'improve this baseline'. Paired with binary validation—tests pass, required artifacts exist, model checkpoints reach a threshold, or logged metrics are queryable—this gradient lets a single success-rate number per category expose where agents stop succeeding. The mechanism carrying the argument is the comparison of those category-wise success rates across agent configurations.","core_discovery":"The paper's central claim is that agent competence on ML-Dev-Bench tracks how open-ended a task is rather than any single skill. In categories with explicit instructions and immediate validation—downloading and preprocessing datasets, logging with an experiment tracker, fixing a pointed-out bug—the strongest agents were at or near ceiling. In model implementation tasks, where agents had to modify existing architectures with partial specifications, success collapsed. In the six model-performance tasks, which ask agents to improve a baseline through iteration and hypothesis testing, every agent configuration failed. The paper takes this as evidence that current agent scaffolds are reliable for structured ML subtasks but have not yet crossed a threshold into autonomous experimentation.","pith_inferences":["The binary successful/failed metric may hide near misses; a partial-credit scoring could show whether agents approach performance solutions or stop early due to excessive verification seeking.","The single-run protocol makes the 50%-versus-47% ranking unstable; repeated runs are needed before treating one agent as meaningfully stronger than another.","The zero on performance tasks may reflect scaffold design as much as agent limits: none of the tested scaffolds used explicit experiment planning or result-driven backtracking, and an agent with those mechanisms would be the natural next test.","The category ordering suggests a curriculum: start with dataset and logging tasks, then structured training and debugging, before attempting open-ended tuning."],"forward_implications":["Current agents can be trusted to automate dataset engineering, standard fine-tuning, and well-specified debugging in production pipelines.","Tool builders should treat open-ended performance work as the frontier: none of the tested configurations could raise a baseline model's score.","The gap between the best configuration and the others suggests that model choice and scaffold persistence matter, but even the best scaffold stalls at half the suite.","The openness gradient can function as a diagnostic: an agent that solves performance tasks while keeping structured-task success would mark a clear capability jump."],"supporting_citations":[{"why":"Supplies the reasoning-and-acting loop used as one baseline agent.","marker":"[8]"},{"why":"Supplies the generalist coding agent evaluated with two underlying models.","marker":"[7]"},{"why":"Defines the data-science style evaluation this benchmark contrasts with and provides the data-science agent used here.","marker":"[2]"},{"why":"Gives the real-world repository-level issue benchmark whose format this work extends toward ML workflows.","marker":"[5]"},{"why":"Provides a prior machine-learning experimentation benchmark in the same problem space.","marker":"[4]"},{"why":"Provides a repository-level ML task benchmark whose scope this work narrows to end-to-end workflow tasks.","marker":"[6]"},{"why":"Supplies the image dataset used in the noisy-label tasks.","marker":"[3]"}],"fun_headline_variants":["AI agents fail all open-ended ML tuning tasks","Best agent scores 50% on ML tasks, all tuning attempts fail","ML-Dev-Bench: agent performance tracks task openness","Six tuning tasks, six failures: AI can't improve ML models","Agents ace structured ML work but flop on open-ended tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Each agent received exactly one attempt at each task, and the two best agents differ by only one successful task, so a re-run could change the ranking and the apparent drop-off.","fun_headline_variants_meta":{"raw":{"variants":["AI agents fail all open-ended ML tuning tasks","Best agent scores 50% on ML tasks, all tuning attempts fail","ML-Dev-Bench: agent performance tracks task openness","Six tuning tasks, six failures: AI can't improve ML models","Agents ace structured ML work but flop on open-ended tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1376,"prompt_tokens":815,"completion_tokens":561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":431,"completion_tokens_details":{"reasoning_tokens":475}},"tokens_in":431,"tokens_out":561,"duration_ms":6077,"temperature":1.0,"reasoning_tokens":475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:03:56.355896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 30-task suite several times per agent configuration and recompute per-category success rates; if any model-performance task is ever completed by an agent in this family, or if the best-versus-second gap flips across runs, the claim that open-endedness predicts failure and the headline ranking would need revision.","supporting_citations":[{"cited_title":"Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H","cited_arxiv_id":null,"evidence_quote":"Supplies the reasoning-and-acting loop used as one baseline agent."},{"cited_title":"Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the generalist coding agent evaluated with two underlying models."},{"cited_title":"Mlagentbench: Evaluating language agents on machine learning experimentation, 2024","cited_arxiv_id":null,"evidence_quote":"Gives the real-world repository-level issue benchmark whose format this work extends toward ML workflows."},{"cited_title":"Imagenette","cited_arxiv_id":null,"evidence_quote":"Provides a prior machine-learning experimentation benchmark in the same problem space."},{"cited_title":"Mle-bench: Evaluating machine learning agents on machine learning engineering, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the image dataset used in the noisy-label tasks."}],"review_version":1}