{"id":"6a0705ac-4bdc-4730-b51b-368490541aca","arxiv_id":"2507.13300","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A benchmark of 1,500 expert-annotated ablation study designs from 807 NLP papers shows frontier LLMs underperform human experts and that LLM-as-a-judge evaluations correlate weakly with human judgments.","lead":"AbGen is a new benchmark that tests whether large language models can design ablation studies for NLP research papers, and it finds that even the best models lag behind human experts. It also finds that current automated LLM judges do not reliably agree with human assessments when grading these designs, which matters for anyone hoping to use LLMs to speed up scientific review or experimental planning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The model ranking and meta-evaluation rest on a single-annotator, 100-example gold standard with no reported annotator balance across models; the headline gap and reliability claims are statistically fragile.","rationale":"The paper is a substantial contribution: the AbGen construction pipeline is described carefully, the dataset and code are released, and the reference ablation score of 4.77 on the 100-example set provides a matched human-quality anchor that supports the qualitative gap between LLMs and expert-level designs. The inter-annotator Kappa of 0.71-0.78 is acceptable but far from perfect. The main risk is that the two headline numbers, the 4.80 human-expert score and the low machine-human correlations, are produced from a gold standard with a single annotator per output and no demonstrated balance of annotators across models. This is a statistical fragility rather than a fundamental flaw, so the claims remain plausible but are not yet pinned down. My concern partially overlaps with the reader's weakest assumption (small sample and inter-annotator agreement), but it is more specific about annotator-assignment imbalance and the unmatched human-expert baseline, and it extends to the missing human-human reliability ceiling in the meta-evaluation. The reader's conditional verdict is appropriate, and I recommend no change to it; the key condition is the release of per-example scores and annotator IDs so that the ranking and the reliability ceiling can be re-analyzed independently.","tokens_in":17944,"tokens_out":13404,"duration_ms":143998,"concrete_test":"Request the per-example human scores and annotator IDs, then fit a mixed-effects model with annotator random intercepts, bootstrap the 18 model marginal means, and re-score human experts on the same 100 testmini examples used for LLM evaluation. If the top-model ordering changes, or if the R1-0528-to-human-expert gap falls below the reported 0.69 on the matched set, the headline performance claims need qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claims are anchored by the human-evaluation results in Table 2. For every model, 100 testmini outputs are each scored by a single expert evaluator, but the paper never reports how the 18 models' outputs were distributed across the four evaluators. Systematic differences in annotator severity can move a model's mean by more than the observed 0.11-point gap between DeepSeek-R1-0528 (4.11) and o4-mini (4.00), so the model ordering is not robust unless assignment is balanced or annotator effects are modeled. The human-expert baseline of 4.80 is computed on only 20 examples (10 papers) solved by two annotators, not on the same 100 examples used to score LLMs, so the headline \"4.80 vs 4.11\" is not a same-items comparison; the only matched human-quality reference on the LLM set is the original-paper ablation at 4.77. The same single-annotator scores become the gold standard in AbGen-Eval, and the paper reports Cohen's Kappa on 40 outputs but never a human-human Pearson or ICC correlation on the same scoring task. Without this reliability ceiling, the low machine-human correlations in Table 5 (e.g., 0.164-0.329 for GPT-4.1-mini) cannot be interpreted as definitive evidence that automated evaluation is unreliable, since they may partly reflect noise in the single-annotator gold standard. All of these issues are fixable by releasing per-example scores and annotator IDs, which the limitations section says are planned.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AbGen, a benchmark of 1,500 expert-annotated examples from 807 NLP papers, in which an LLM is given a research context (background, methodology, main experiments) and the name of a module or process, and must produce a detailed ablation study design. The authors evaluate 18 frontier LLMs using expert human ratings on importance, faithfulness, and soundness, and compare these with an LLM-as-judge automated evaluator. They also construct AbGen-Eval, a meta-evaluation benchmark built from the same human ratings, and report that current automated evaluators have low instance-level correlation with human judgments. Two user studies examine whether researcher feedback improves LLM-generated ablation designs and whether the approach transfers to biomedical and computer-network domains.","tokens_in":18124,"tokens_out":5239,"duration_ms":56362,"significance":"If the results hold, AbGen is a useful first benchmark for a genuinely scientific task—ablation study design—and the public data and code will let other groups build on it. The paper also ships a meta-evaluation benchmark that directly targets the reliability of LLM-as-judge scoring, which is a timely contribution. The benchmark construction is reasonably transparent: source selection, annotation, validation, and revision (273 of 1,500 examples revised) are described, and a 100-example quality check reports over 95% satisfaction. The main weakness is statistical: the human gold standard that supports both the model ranking and the meta-evaluation consists of 100 testmini examples per model, each scored by a single annotator, with no reported annotator balance across models and no human-human Pearson/ICC reliability ceiling. These issues are fixable and do not invalidate the benchmark, but they currently weaken the central quantitative claims.","major_comments":[{"comment":"The model ranking is built on 100 testmini examples per model, with each output scored by a single expert evaluator, yet no information is given about how the four evaluators were assigned across the 18 models. Because annotator severity differences can move a model's mean by more than the observed 0.11-point gap between DeepSeek-R1-0528 (4.11) and o4-mini (4.00), the ordering is not robust unless assignment is balanced or annotator effects are modeled. Please report per-example scores with annotator IDs, test for balance across models, and provide severity-adjusted estimates (e.g., a mixed-effects model) or an equivalent analysis.","section":"Section 4.2, Table 2"},{"comment":"The human-expert baseline of 4.80 is computed on only 20 examples from 10 papers, solved by Annotators 1 and 4, and is not the same 100-item set used for LLM scoring. The headline comparison 'human expert 4.80 vs DeepSeek 4.11' is therefore not a same-items comparison; the only matched reference available on the 100-example LLM set is the original-paper ablation at 4.77. Please either collect expert scores on the same 100 examples used for LLM evaluation or clearly present the expert baseline as a separate, non-matched estimate rather than implying a direct gap.","section":"Section 4.1, 'Measuring Performance of Real Paper and Expert'"},{"comment":"Single-annotator human scores are used as the gold standard in AbGen-Eval, and the only inter-annotator statistic reported is Cohen's Kappa on 40 outputs. Without a human-human Pearson or ICC on the same scoring task, the low machine-human correlations (e.g., 0.164 to 0.329 for GPT-4.1-mini) cannot be definitively interpreted as evidence that automated evaluation is unreliable, because part of the disagreement may reflect noise in the single-annotator gold standard. Please report human-human reliability on a larger overlap, use weighted Kappa/ICC for ordinal 1-5 scores, and consider using aggregated multiple ratings for the gold standard before drawing conclusions about automated evaluator reliability.","section":"Sections 3.2 and 5.1, Tables 2 and 5"},{"comment":"The automated evaluation scores are reported on the test subset (1,000 examples) while human scores are reported on 100 examples from testmini. The apparent automated-versus-human discrepancy in this table is therefore not a same-item comparison; the same-item evidence appears only later via AbGen-Eval. Please compute automated scores on the same 100 human-evaluated examples for Table 2, or explicitly label the comparison as cross-subset and move the reliability argument entirely to the matched AbGen-Eval setting.","section":"Table 2 caption and Section 4.2"}],"minor_comments":[{"comment":"The caption says 'Human evaluation over 200 samples' while Section 2.5 says the final quality check was on 100 examples; please reconcile this discrepancy.","section":"Appendix A.1, Table 6"},{"comment":"Equation (1) writes the task as arg max over A of P(A|C,M) but does not define the probability distribution; clarify that this is the model's generation distribution or replace the notation.","section":"Equation (1)"},{"comment":"The caption says 'A summary of GPT-4o's failure cases' while Section 4.2 says the error analysis is based on top-performing LLMs more generally; clarify exactly which models contributed the 100 failure cases.","section":"Table 3"},{"comment":"The sentence '18 LLM outputs × 100 human assessments = 1,800 examples' is ambiguous because each output receives three human scores; please say '1,800 model-output assessments' or clarify that each assessment is one (output, score) pair.","section":"Section 5.1"},{"comment":"There are minor typos, including 'requried' in Section 2.4 and 'ambiguious' in Section 3.1; a careful proofread would fix these.","section":"Sections 2.4 and 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful and timely contribution, and the core benchmark resource is well documented. The requested revisions are statistical and presentational: they require releasing per-example human scores and annotator IDs, expanding the inter-annotator reliability analysis, and making the Table 2 comparison same-item. These are all within the manuscript's scope and should be feasible, so I do not see a need to reject on technical grounds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AbGen is a genuinely new benchmark, and the core negative result—frontier LLMs are noticeably worse than human experts at designing ablations, and LLM-as-judge is not reliable for scoring this task—is interesting. The dataset construction is the strongest part: 1,500 expert-annotated examples from 807 recent NLP papers, with a validation pass that revised 273 examples and a 100-example quality check above 95% satisfaction. The authors also release data and code, and the appendix gives enough detail to audit the pipeline. The reference-annotation protocol that strips ablation-related content from the research context is a thoughtful design choice, and the meta-evaluation AbGen-Eval is a useful resource for people working on automated scientific evaluation. The citation pattern is fine, with appropriate links to prior benchmark and evaluation work.\n\nThe soft spots are statistical, mostly about the human-evaluation anchor. The 18-model ranking rests on 100 testmini examples, a single annotator per output, and no reported balance of annotators across models. Systematic severity differences between the four evaluators could easily move a model's mean by more than the 0.11 gap between DeepSeek-R1-0528 and o4-mini. The human-expert ceiling of 4.80 comes from a different set—20 examples from 10 papers solved by two annotators—so the headline '4.11 vs 4.80' is not a same-items comparison; the only matched reference on the LLM set is the original-paper ablation at 4.77. The meta-evaluation inherits the same issue: the gold standard is one expert's score per output, and the paper reports Cohen's Kappa on 40 outputs but no human-human Pearson or ICC on the actual scoring task. Without that reliability ceiling, the low instance-level correlations in Table 5 are hard to assign confidently; they likely reflect both machine failure and gold-standard noise. There is also a minor comparison problem in Table 2: automated scores are on the test subset while human scores are on testmini, so the side-by-side discrepancy is confounded by different items.\n\nNone of this invalidates the paper. The testmini/test split means the benchmark can be re-checked, and the authors say they will release model outputs, which should allow per-example human scores if they follow through. If an independent panel rescored a larger sample with balanced assignment, we would know how much of the ranking is real. I would send this to peer review and recommend revision that addresses annotator balance and publishes the human scores. The core contribution is solid; it just needs to be pinned down more carefully.","headline":"AbGen is a genuinely new benchmark with a careful construction pipeline, but its headline human-vs-model comparison rests on a statistically fragile 100-example gold standard.","tokens_in":18793,"tokens_out":3247,"would_cite":true,"duration_ms":34953,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AbGen, the first benchmark for LLM ablation-study design, shows frontier models trail human experts and that automated judges are not yet reliable for this task.","keywords":["ablation study design","LLM benchmark","scientific experimental design","meta-evaluation","LLM-as-judge","natural language processing","human evaluation"],"falsifier":"Take a fresh sample of 200 examples from the held-out test set, have all four expert annotators score every model output, and recompute the model ranking and the instance-level Pearson correlation of GPT-4.1-mini; if the best-model lead over o4-mini becomes statistically insignificant, or if any LLM judge's instance-level correlation exceeds about 0.5, the paper's two central claims — that LLMs trail experts and that automated evaluation is unreliable — would need revision.","tokens_in":17641,"feed_emoji":"🧪","tokens_out":9371,"duration_ms":83740,"temperature":0.7,"pith_summary":"The paper introduces AbGen, the first benchmark built to test whether large language models can design sound ablation studies for scientific research. Drawing on 1,500 expert-annotated examples from 807 NLP papers, it gives a model a research context and the name of a module, then asks it to propose an experiment that isolates that module's contribution. On a 100-example sample scored by experts, the best model, DeepSeek-R1-0528, averaged 4.11 on a 1-5 scale while human experts averaged 4.80, a gap the authors read as evidence that LLMs cannot yet plan controlled experiments on their own. The paper then builds AbGen-Eval, a meta-evaluation set of 1,800 human-scored outputs, and reports that current LLM-as-judge systems correlate only weakly with human ratings at the instance level, with Pearson correlations as low as 0.164. A careful reader takes away that AbGen offers a first standard yardstick for this scientific skill, and that automated evaluation of it is not yet trustworthy.","feed_headline":"Best LLM scores 4.11 vs 4.80 for human experts on new benchmark","feed_subtitle":"This 1,500-example benchmark from 807 NLP papers shows automated judges still can't match human review.","key_machinery":"The load-bearing machinery is the benchmark construction pipeline and the meta-evaluation protocol built on top of it. AbGen is built by filtering arXiv NLP papers, restructuring each paper into a three-part research context with all ablation content removed, annotating each reference ablation study as objective, experiment process, and result discussion, and validating the annotations with a separate annotator. Model outputs are scored by experts on three 1-5 Likert dimensions — importance, faithfulness, and soundness — with evaluators scoring the design without seeing the reference first and then adjusting their scores. AbGen-Eval then treats the 1,800 human-scored outputs as the gold standard and measures each automated evaluator by system-level Kendall correlation and instance-level Pearson correlation; the low instance-level correlations are the specific quantity that exposes current LLM-as-judge systems as unreliable.","core_discovery":"AbGen establishes a measurable standard for one specific scientific skill: designing an ablation study. Given a research context divided into background, methodology, and main experiments, plus the name of a module, an LLM must produce a research objective and a replicable experimental procedure that would isolate that module's role. The paper's central finding is two-fold. First, frontier LLMs underperform human experts by a wide margin: the strongest system, DeepSeek-R1-0528, receives an average human score of 4.11 versus 4.80 for human experts, with the largest deficits in faithfulness and soundness rather than in picking an important question. Second, automated evaluation cannot yet stand in for human judgment: on AbGen-Eval, every LLM-based evaluator tested shows low instance-level Pearson correlation with human scores, so automatic ratings would misorder the systems on individual examples.","pith_inferences":["Editorial inference: the weak instance-level correlations suggest that holistic judges miss the hard part — whether a control condition would actually isolate the module — so future evaluators may need to check experimental logic step by step rather than assign an overall score.","Editorial inference: a natural harder test is to score designs by whether the proposed ablation would change the paper's known main result; this would replace subjective importance ratings with an objective, falsifiable criterion.","Editorial inference: stripping all ablation content from the research context may make AbGen harder than real-world design, where a researcher already knows which ablations were actually run; the benchmark measures generative design from a clean slate, not recognition of a missing experiment.","Editorial inference: with only 100 examples judged and annotators only partially overlapping, the ranking of closely spaced models (4.11 versus 4.00) is a point estimate; re-running all four annotators on the full testmini set could shift the ordering."],"forward_implications":["If AbGen is a fair yardstick, current frontier LLMs are not ready to replace a researcher in planning even a single controlled experiment: every evaluated model averages below human experts on all three criteria.","The benchmark's testmini/test split and publicly released model outputs let future systems be compared on the same 500 and 1,000 examples without rerunning expert evaluation.","An interactive use case is viable: expert feedback raised GPT-4o's faithfulness score from 2.1 to 4.2 and Llama-3.1-70B's from 1.8 to 3.9 in the user study, suggesting LLMs are useful as drafts that a researcher corrects.","Small user studies on biomedical and computer-network papers show similar performance to the NLP setting, indicating the benchmark pipeline generalizes even though current model capability does not.","Automated evaluation of ablation designs needs new methods before it can rank models on this task; the reported instance-level correlations rule out today's LLM-as-judge systems for that purpose."],"supporting_citations":[{"why":"supplies the document-parsing tool that converts LaTeX source files into JSON for the paper collection pipeline.","marker":"Lo et al. (2020)"},{"why":"prior work on peer-review generation that relies on human evaluation and shapes AbGen's evaluation design.","marker":"D'Arcy et al. (2024)"},{"why":"prior benchmark for scientific direction generation that establishes the pattern of human-evaluation-based assessment AbGen extends.","marker":"Wang et al. (2024a)"},{"why":"established the meta-evaluation methodology that treats human judgments as gold standard and uses correlation metrics.","marker":"Fabbri et al. (2021)"},{"why":"prior adversarial meta-evaluation of factuality that motivates checking whether automated metrics agree with humans.","marker":"Chen et al. (2021)"},{"why":"prior work benchmarking generation and evaluation capabilities that supports the meta-evaluation design.","marker":"Liu et al. (2024)"},{"why":"provides evidence that LLMs can comprehend HTML tables, justifying the conversion of tables in the research context.","marker":"Sui et al. (2024)"},{"why":"cited for the decision not to evaluate fluency because leading LLMs already produce fluent text.","marker":"Zeng et al. (2024)"}],"fun_headline_variants":["LLMs lag humans in ablation design: 4.11 vs 4.80","New benchmark shows LLMs can't design ablations like experts","Automated judges fail on AbGen: humans still needed","Ablation design benchmark: humans beat top LLMs by 0.69","First benchmark for LLM ablation design exposes big gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that expert human ratings collected on a 100-example sample from testmini, with partially overlapping annotators, define a stable gold standard that can rank 18 models and ground the meta-evaluation; if that sample is not representative, both the model gap and the evaluator-unreliability finding weaken.","fun_headline_variants_meta":{"raw":{"variants":["LLMs lag humans in ablation design: 4.11 vs 4.80","New benchmark shows LLMs can't design ablations like experts","Automated judges fail on AbGen: humans still needed","Ablation design benchmark: humans beat top LLMs by 0.69","First benchmark for LLM ablation design exposes big gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000462,"raw_usage":{"total_tokens":2301,"prompt_tokens":925,"completion_tokens":1376,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1284}},"tokens_in":541,"tokens_out":1376,"duration_ms":9030,"temperature":1.0,"reasoning_tokens":1284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:24:35.140186+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of 200 examples from the held-out test set, have all four expert annotators score every model output, and recompute the model ranking and the instance-level Pearson correlation of GPT-4.1-mini; if the best-model lead over o4-mini becomes statistically insignificant, or if any LLM judge's instance-level correlation exceeds about 0.5, the paper's two central claims — that LLMs trail experts and that automated evaluation is unreliable — would need revision.","supporting_citations":[],"review_version":1}