{"id":"5a072f19-ecfc-43ea-a9a0-f9f956514a79","arxiv_id":"2507.08038","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AblationBench is a new benchmark for testing AI planning of ablation experiments, and it shows that current language models recover only a minority of the human reference ablations.","lead":"This paper releases AblationBench, a benchmark for testing whether language models can plan ablation experiments from scientific papers, from both an author's and a reviewer's perspective. Current frontier models recover only a minority of the human-created reference ablations, suggesting the task is far from solved.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark scores 'ablation planning' entirely as recovery of the authors' chosen ablations; a model proposing valid but non-reference ablations scores zero, so the reported 45%-versus-human gap may not be a general planning gap.","rationale":"The abstract's headline—'the best-performing LM system identifies only 45% of the original ablations... below human-level'—is the paper's central empirical claim. The reader's weakest-assumption analysis correctly identifies the load-bearing premise: the benchmark's GT is defined as the ablations the original authors performed (AuthorAblation) or that reviewers/LM-extraction flagged (ReviewerAblation). The paper itself acknowledges in Section 3 and in the Limitations that models may generate meaningful ablations outside the GT, and that evaluating such novel suggestions is left to future work. If a meaningful fraction of model proposals are valid but non-reference, the reported gap is a gap in reference recovery, not in general ablation-planning ability. I agree with the reader that this is the key assumption and that it is acknowledged rather than an oversight.\n\nI also note secondary issues that do not change the verdict: (a) the human baseline is small (10 included responses after excluding 5 of 15) and humans self-selected papers in their expertise with at least one hour of effort, making the 'below human-level' claim statistically thin; (b) Table 4 reports F1@k 0.35 for the best planner while the Conclusion states 0.30, and the Abstract's 45% is macro-averaged recall across two tasks with different k and GT sizes—numeric presentation should be cleaned; (c) ReviewerAblation GT is produced by a GPT-4o filter calibrated only on a development set, adding machine-generated noise to the 'ground truth.' These are real but secondary.\n\nIn favor of the paper: the data and code are released, the judges are validated against human annotations with kappa around 0.6, and the paper honestly states the GT-overlap limitation. The central result that models recover fewer than half of reference ablations is plausibly robust to the numerical fixes. Therefore, the reader's CONDITIONAL verdict is appropriate; my analysis does not change it. The proposed expert-quality-rating study would directly test whether the GT-overlap metric understates model capability, and would also quantify how conservative the human baseline is on the same grounds.","tokens_in":49728,"tokens_out":6558,"duration_ms":66707,"concrete_test":"Take a random sample of 20 AuthorAblation test papers. For each, collect the five proposals from the best LM planner (GPT-5.4 LM-PLANNER, Table 7) and the human proposals from Appendix B.3. Have three ML-expert annotators, blind to the ground truth, rate every proposal on relevance, component isolation, feasibility, and expected insight, using the paper's own annotation criteria (Figure 6) but without treating GT membership as a quality signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that frontier LMs are far below human-level ablation planning rests on a metric that equates planning quality with exact recovery of the reference ablations. In AuthorAblation, the ground truth is the set of ablations the original authors chose to run and explicitly labeled as ablations (Figure 6); in ReviewerAblation, the ground truth is the suggestions extracted by an LM pipeline from reviews (Section 5.1). Section 3 and the Limitations state that models may propose meaningful, feasible ablations not in the GT and that no GT exists for such suggestions, yet every non-matching proposal is scored as a miss. Consequently, a system that proposes a correct but non-reference ablation receives no credit, and the reported F1@k of 0.35 (Table 4) and 45% recall reflect reference recovery, not planning quality per se. The human baseline does not resolve this: humans are measured on the same reference-recovery metric, and the comparison rests on only 10 included responses after excluding 5 of 15 (Appendix B.3), with participants selecting papers in their expertise and spending at least one hour. Thus, \"below human-level performance\" is a claim about a narrow matching task under a small, filtered sample. The benchmark is still a valid measure of reference-ablation recovery, but the abstract's unqualified phrasing overstates what the numbers establish.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AblationBench introduces a two-task benchmark for evaluating how well LM-based systems plan ablation studies from raw papers: AuthorAblation (83 papers, ground truth from the original papers) and ReviewerAblation (350 ICLR submissions, ground truth from official reviews). The paper develops LM-based judges with ensemble voting, validates them against a small human-annotated evaluation set, and evaluates LM and SWE-agent planners across several frontier LMs. The central empirical claim is that the best system achieves only 0.35 F1@k (45% recall) overall and remains below human performance on an AuthorAblation subset. The paper also reports an inverse performance trend between the author and reviewer tasks and concludes that chain-of-thought prompting outperforms an agent-based approach.","tokens_in":50012,"tokens_out":7524,"duration_ms":82456,"significance":"If the benchmark results hold, AblationBench is a useful resource for the rapidly growing area of AI co-scientist evaluation: it ships data and code, provides two complementary task formulations, and is one of the few attempts to evaluate ablation planning from raw paper text rather than curated summaries. The judge validation against human annotations (63 AuthorEval and 60 ReviewerEval plans) is a genuine strength, as is the explicit comparison of LM and agent scaffolds. However, the headline claims are currently stronger than the quantitative support in the paper: the stated inverse trend is not visible in the aggregate tables, the conclusion's best F1 number contradicts Table 4, and the human-baseline description is internally inconsistent. The central metric measures reference recovery rather than unimpeded planning quality, and the paper's own limitations section acknowledges this; the abstract and introduction should therefore qualify the 'below human-level' claim accordingly.","major_comments":[{"comment":"The abstract states that the paper observes an inverse performance trend between the author and reviewer tasks, and Section 7.4 claims that 'models that perform well on one task tend to perform poorly on the other.' The aggregate tables do not support this claim. In Table 7, GPT-5.4 has the highest AuthorAblation F1@5 (0.39 for LM-PLANNER, 0.40 for AGENT-PLANNER), and in Table 8 it also has the highest ReviewerAblation F1@2 (0.31 and 0.30). The cross-model correlation between the two task scores is near zero for the LM-PLANNER rows (Pearson r approximately 0.1), not negative. The qualitative Qwen-vs-Llama contrast in Appendix B is anecdotal and cannot carry a general claim of an inverse trend. The quantitative data need to be analyzed with a correlation or the claim should be substantially softened.","section":"Section 7.4 and Abstract; Tables 7 and 8"},{"comment":"The conclusion reports that 'the best model achieves an F1 score of 0.30 on AblationBench,' but Table 4 reports the best F1@k as 0.35 for both LM-PLANNER and AGENT-PLANNER with GPT-5.4. Since Section 7.2 defines the AblationBench score as the average of the two tasks, the value in Table 4 is the correct reported result, and the conclusion should be reconciled with it. The discrepancy is small but load-bearing because the headline number of the paper is the benchmark's best F1 score.","section":"Section 8 vs. Table 4"},{"comment":"The human baseline is described inconsistently. Section 7.5 says 'We recruit 10 participants' and reports humans achieving F1@5 of 0.66 compared with 0.43 for GPT-5.4 on the same subset. Appendix B.3 states that 15 participants were recruited, 5 responses were excluded, and the remaining 10 were compared against the 'best-performing model on this task, Qwen 3.5.' These two accounts disagree about the recruited sample size and about which model was compared. The paper must state the correct participant count, the exclusion procedure, and the exact model used in the comparison, and should provide the per-paper subset results in a table.","section":"Section 7.5 vs. Appendix B.3"},{"comment":"The benchmark scores every generated ablation against the reference set of ablations chosen by the original authors or reviewers. Section 3 explicitly acknowledges that models may propose new ablation ideas for which no ground truth exists, and the Limitations section repeats this concern. Consequently, a system that proposes a valid, feasible ablation not present in the reference set receives zero credit, and the reported F1@k and recall values measure reference recovery rather than ablation-planning quality per se. The human baseline is measured on the same recovery metric, so this concern does not invalidate the comparison, but the abstract's unqualified statement that the best system is 'below human-level performance' overstates what the metric establishes. The paper should consistently phrase the headline claim as 'below human-level performance at recovering the reference ablations' or provide additional evidence that non-reference proposals are not systematically better than the scored ones.","section":"Section 3 and Limitations"}],"minor_comments":[{"comment":"The human-baseline results are currently reported only in prose; including a small table with the subset size, human F1, model F1, precision, and recall would make the comparison easier to verify.","section":"Section 7.5"},{"comment":"The abstract's '45% of the original ablations' corresponds to recall@k in Table 4; the paper should state this explicitly so that readers do not confuse it with the F1@k value of 0.35.","section":"Abstract and Table 4"},{"comment":"The model name 'Qwen3.5-122B-A10B' appears with and without a hyphen in different tables and in the text; the naming should be unified.","section":"Tables 3-8 and Appendix B"},{"comment":"The text says the pipeline 'results in 5,960 papers in ReviewerAblation' and then states that 50 development and 300 test papers are sampled; since the main text says the benchmark contains 350 instances, clarify that 5,960 is the full filtered pool and 350 is the benchmark subset.","section":"Appendix A.2"},{"comment":"The sentence 'The best model achieve an F1 score' contains a subject-verb agreement error and should read 'achieves.'","section":"Section 8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has the flavor of a workshop contribution, but the benchmark resource itself is timely and the judge validation is a real strength. The main reasons for revision are internal: the inverse-trend claim is contradicted by the paper's own tables, the conclusion's headline F1 does not match Table 4, and the human-baseline description in Section 7.5 disagrees with Appendix B.3. I would like to see these fixed before the paper is considered further, and I would also encourage the authors to frame the human-level comparison strictly as reference recovery rather than ablation-planning ability in general."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a real, reusable artifact: an 83-paper author-side benchmark and a 350-paper reviewer-side benchmark, with LM judges validated against human annotations and data/code released. Second, the central claim — best LM finds 45% of reference ablations, below humans — is only about recovering the authors' or reviewers' chosen ablations, not about planning quality in general. A model that proposes a valid, feasible ablation not in the GT gets zero credit. The authors say this explicitly in Section 3 and in the Limitations, so it's an acknowledged design choice, but the abstract's phrasing 'below human-level performance' will read as a broader capability gap.\n\nThe work extends AbGen in a real way: raw paper input instead of manually processed snippets, a second reviewer task, and automatic judges instead of human rating. The judges get a proper validation set (63 + 60 human-annotated plans) and the best majority-vote judge reaches ~0.79 F1 and kappa ~0.62, which is good enough for rough screening. The inverse author/reviewer trend (more grounded models do better as authors, worse as reviewers) is a plausible and interesting observation, even if the mechanism is qualitative.\n\nThe soft spots are real but not fatal. The human baseline is thin: 10 responses after excluding 5 of 15, with participants choosing papers in their area and spending at least an hour. That's fine for a sanity check, but the 'below human-level' claim carries a lot of weight for such a small, filtered sample. There are also outright inconsistencies in the text: the conclusion says best F1 is 0.30 while Table 4 says 0.35; Section 7.5 compares humans to GPT-5.4 while Appendix B.3 says Qwen 3.5; and the participant count shifts from 10 to 15. These need fixing before this can be cited cleanly.\n\nBottom line: the benchmark is worth having and the honest limitations section earns credit. It deserves a serious referee and, after a revision that resolves the numeric mismatches and recalibrates the abstract's phrasing, I'd trust it as a baseline for ablation-planning agents.","headline":"AblationBench is a genuinely useful, released benchmark for ablation planning, but its headline 'below-human' gap is a statement about reference recovery, not general planning quality, and the write-up has several numeric inconsistencies.","tokens_in":50546,"tokens_out":5679,"would_cite":true,"duration_ms":50302,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frontier language models asked to plan ablation experiments for research papers recover at most 45% of the ablations that human authors or reviewers chose, and PhD-level humans still beat the best model by a wide margin.","keywords":["ablation studies","benchmark","language model agents","LM-as-judge","scientific reasoning","author assistance","reviewer assistance","chain-of-thought prompting"],"falsifier":"Take the non-matching top-k proposals a top planner generates on the benchmark's test papers and have independent machine-learning researchers rate each for relevance, feasibility, and scientific value; if those proposals are rated on par with the gold ablations, then the reported 45% recall and 0.35 F1 reflect scoring against a single reference set rather than a true capability ceiling.","tokens_in":49514,"feed_emoji":"🧪","tokens_out":13418,"duration_ms":114152,"temperature":0.7,"pith_summary":"This paper sets out to measure whether language-model agents can plan ablation experiments, the studies researchers run to establish which components of a proposed method actually matter. To do this it builds AblationBench, a two-part benchmark: AuthorAblation, where a planner sees 83 papers truncated at their method sections and must propose the ablations the authors later ran (230 gold ablations in total), and ReviewerAblation, where a planner reads 350 ICLR submissions and must propose the missing ablations that official reviewers requested. Plans are scored by an ensemble of LM judges against these human ground-truth labels. The paper's main finding is that the best frontier system identifies only about 45% of the original ablations on average, while PhD-level humans score F1 around 0.65 against the best model's roughly 0.42 on the same papers, so ablation planning marks a real, unsolved capability for AI research agents.","feed_headline":"Language models recover 45% of human-designed ablations at best","feed_subtitle":"A new benchmark scores AI planners against the ablations real authors and reviewers chose — and humans still win.","key_machinery":"The central object is the benchmark suite itself, whose gold labels are human practice rather than synthetic answers. In AuthorAblation, each of 83 papers is truncated to its method section and paired with the 230 ablations the original authors ran, manually extracted as structured JSON entries (ablated component, action, replacement, metrics); in ReviewerAblation, each of 350 ICLR submissions is paired with the ablation requests found in its official reviews, and ablation order in the source papers serves as a proxy for importance. The load-bearing mechanism is the LM-based judge, a three-model majority-vote ensemble with randomized side assignment and shuffling that counters intra-model, contextual, and positional bias, deciding whether a generated plan matches a gold ablation under conservative criteria; the judge's agreement with human annotations (kappa approximately 0.62) is itself validated on a dedicated JudgeEval set. Matching requires the same ablated component, an action the reference ablation explicitly allows, and at least one shared replacement when one is specified.","core_discovery":"On the paper's own terms, the discovery is that automated ablation planning from raw paper text is now measurable and currently far from solved. A chain-of-thought-prompted GPT-5.4 reaches an overall F1 around 0.35 (0.39 on AuthorAblation at k=5 proposals, 0.31 on ReviewerAblation at k=2), and no tested model beats a single CoT prompt, including SWE-agent-style agents that may inspect files and iterate. The human gap is large: on a ten-paper subset, machine-learning PhD students recover the gold ablations at F1 approximately 0.65 with precision 0.62, versus approximately 0.42 for the best model. The authors read the pattern through model grounding: models that adhere closely to the paper text decompose methods well for the author task but propose fewer reviewer-style missing ablations, while less grounded models show the opposite trade-off, and standardized CV papers yield higher F1 (0.45) than more varied NLP papers (0.30).","pith_inferences":["The ground truth inherits the contingency of what authors chose to run and reviewers happened to request; a planner that proposes a scientifically better ablation than anything in the reference set receives a false negative, so the true capability gap may be smaller than the 45% figure suggests.","A direct test follows: take a planner's non-matching proposals on the test papers and have independent machine-learning researchers rate them for relevance, feasibility, and scientific value; if those rate on par with the gold ablations, matching-only scoring understates model capability.","Individual judges agree with humans at kappa around 0.45-0.59, so the matching subtask itself is not solved; improving judge reliability through calibrated confidence or pairwise comparisons would sharpen every downstream planner score.","The ReviewerAblation pipeline is fully automatic and could be pointed at other conferences and years, which would let the community track whether future models close the gap across settings or improve only on this benchmark's reference set."],"forward_implications":["No current frontier model is close to human-level ablation planning: the best system recovers under half of the ablations an author or reviewer would name, so claims about end-to-end AI research agents should be read with that ceiling in mind.","A single chain-of-thought prompt with the full paper in context outperforms an agent that can browse, edit, and iterate, so for this skill reasoning over complete context beats tool use, and agent scaffolds must be redesigned rather than scaled.","Removal-type ablations are substantially easier for models than modification-type ablations that require proposing feasible replacements, which marks the replacement problem as the higher-value target for future work.","Because human experts recover the gold ablations at high F1, the ground-truth target is a fair standard, and the roughly 0.2 F1 gap to the best model defines concrete headroom for improvement.","The inverse author/reviewer trend means a single model is unlikely to excel at both perspectives, and performance on one planning task does not indicate capability on the other."],"supporting_citations":[{"why":"AbGen, the prior ablation-planning benchmark that AblationBench extends by moving from processed inputs to raw paper content and by adding the reviewer perspective.","marker":"Zhao et al., 2025"},{"why":"PaperBench, which contributes source papers to AuthorAblation and supplies the judge scaffolding that the LM-based judges are modeled on.","marker":"Starace et al., 2025"},{"why":"SWE-agent, the ReAct-style scaffold used to build the agent planners and agent judges across both tasks.","marker":"Yang et al., 2024b"},{"why":"Source of the roughly 22,800 ICLR submission forums and 89,100 reviews from which ReviewerAblation is filtered down to its 350 instances.","marker":"OpenReview, 2025"},{"why":"CSR-Bench, a source of candidate papers with ablations and open-source code used to construct AuthorAblation.","marker":"Xiao et al., 2025"},{"why":"SUPER, an additional source of research-repository papers used to construct the AuthorAblation paper pool.","marker":"Bogin et al., 2024"},{"why":"Grounds the majority-vote ensemble of judge models used to mitigate intra-model bias in automatic evaluation.","marker":"Verga et al., 2024"}],"fun_headline_variants":["AI ablation planning benchmark: best hit 45% human","New benchmark shows AI misses most human ablations","AblationBench: chain-of-thought beats agents, humans lead","AI plans only 45% of ablations humans would choose","Benchmark exposes gap in AI's ablation planning skill"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Overlap with what the original authors happened to run and what reviewers happened to request is treated as the definition of good ablation planning, so a system that proposes correct, valuable ablations that appear in neither reference set is scored as failing.","fun_headline_variants_meta":{"raw":{"variants":["AI ablation planning benchmark: best hit 45% human","New benchmark shows AI misses most human ablations","AblationBench: chain-of-thought beats agents, humans lead","AI plans only 45% of ablations humans would choose","Benchmark exposes gap in AI's ablation planning skill"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1468,"prompt_tokens":992,"completion_tokens":476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":608,"tokens_out":476,"duration_ms":6097,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:54:12.902694+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the non-matching top-k proposals a top planner generates on the benchmark's test papers and have independent machine-learning researchers rate each for relevance, feasibility, and scientific value; if those proposals are rated on par with the gold ablations, then the reported 45% recall and 0.35 F1 reflect scoring against a single reference set rather than a true capability ceiling.","supporting_citations":[],"review_version":1}