{"id":"55141545-86e0-482a-9490-bc19758971f1","arxiv_id":"2510.24891","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GPT-5 scores 62% on Idea2Plan, a new benchmark that grades AI-generated research plans against rubrics built from 200 post-cutoff ICML 2025 papers — the strongest result, with substantial headroom.","lead":"This paper introduces Idea2Plan, a benchmark that tests how well AI models turn a one-paragraph research idea into a concrete scientific research plan, using 200 recent ICML papers the models could not have memorized. The strongest model, GPT-5, scores about 62%, well below the estimated 82% ceiling, showing clear room for improvement in AI research planning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rubric questions may be too specific to the reference paper, so the benchmark could reward reconstructing the original plan rather than general planning ability; human validation did not test fairness to valid alternatives.","rationale":"The reader's weakest assumption identified the general risk of auto-generated rubrics and limited human validation. I agree that this is the weakest point, but I sharpen it to a specific mechanism: rubric questions may be overfit to the reference paper's choices, and the LLM judge's strict 'explicitly satisfies' rule would then penalize equally valid alternative plans. This is more than a stylistic preference of the judge; it changes what is being measured. The paper's own rubric generation guidelines (Tables 8–10) try to prevent this, and the 8-paper human study rates rubrics as 'acceptable,' but the study does not test the fairness dimension against alternative plans. Because the central ranking claim depends on the rubrics being method-agnostic, this concern is load-bearing. The proposed test—comparing expert-written plans' acceptance and comparing rankings under human vs. LLM grading—would directly settle whether the bias is real. Given the paper's transparency and the plausibility of the benchmark, the existing CONDITIONAL verdict is appropriate; my concern does not move it to REJECT but reinforces the need for expanded validation before the benchmark is used for capability tracking.","tokens_in":32221,"tokens_out":3572,"duration_ms":44452,"concrete_test":"Randomly select 25–30 test-set papers. For each, recruit 2–3 AI/ML researchers to write a research plan from the extracted idea only (no access to the paper or rubric). Grade these expert plans with the existing o4-mini rubric and judge; then have independent human graders evaluate the same plans against a rubric they write from the idea alone. Measure (a) the fraction of 'No' judgments on expert plans that human experts consider unreasonable, and (b) whether the relative model ranking (GPT-5 vs. o4-mini vs. others) computed by o4-mini judge matches the ranking computed by human grader on a larger set of at least 100 generated plans across models. If expert plans are frequently rejected due to wording/method specificity, or if the ranking shifts under human grading, the benchmark is not a valid measure of planning ability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Idea2Plan Bench measures LLM research-planning ability and that GPT-5 leads on it. This requires the auto-generated rubrics to reward any valid plan for the idea, not just plans that mirror the reference paper's specific methodological choices. The rubric generation prompt (§3.2, Table 8) instructs generalization, and section-by-section guidance (Table 9) says to avoid requiring exact datasets/models unless necessary. However, the human validation (§3.4, Table 2) only asked experts whether rubric questions 'cover the essential parts' of a plan; it did not test whether questions unfairly penalize alternative but sound plans that make different design choices. The LLM judge prompt (Table 13) compounds this: it instructs a strict interpretation, answering 'Yes' only if the plan 'explicitly satisfies' the question as written. If a rubric question is phrased around the reference paper's specific method (e.g., 'Does the plan describe a supervised fine-tuning approach?' when the paper used SFT), a plan using RLHF or prompt-based alignment could be marked 'No' even though it is equally valid. With 192 of 200 test-set rubrics never human-checked for this kind of bias, the benchmark may systematically favor models that are better at guessing the original paper's design choices from the abstract-level idea. GPT-5's higher scores could then reflect stronger inference of the reference paper's conventions rather than superior open-ended planning. This is a load-bearing concern because it directly threatens the core claim that the ranking measures research-planning ability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes the Idea2Plan task — turning a research idea into a structured research plan — and introduces Idea2Plan Bench, built from 200 ICML 2025 Spotlight/Oral papers whose arXiv versions postdate the tested LLMs' training cutoffs. Ideas and reference plans are extracted from paper abstracts/full texts using o4-mini, rubrics are generated from the reference plans with o4-mini, and final grading is performed by an o4-mini judge. The authors validate the extraction and rubric quality with 8 expert-rated papers and validate the judge on 10 plans against human annotations (o4-mini F1=0.91). They evaluate GPT-5, GPT-5-mini, o4-mini, GPT-4.1, DeepSeek-V3/R1, and Phi-4 under naïve, 0-shot, 1-shot, and ReAct-agent settings with three trials per configuration, plus length-controlled and curated-literature probes. The headline results are that GPT-5 and GPT-5-mini score highest (naïve average Planning Score 62.0% and 56.7%; ReAct 61.9% and 61.1%), while a full-paper upper bound gives 81.9%. The paper also reports that ReAct does not beat simpler prompting and that SFT on idea–plan pairs degrades performance.","tokens_in":32438,"tokens_out":3495,"duration_ms":42480,"significance":"If the evaluation instrument is valid, this is a useful contribution: it is one of the first dedicated benchmarks for research planning, with deliberate contamination controls (post-cutoff papers, blocklisted search in ReAct), a transparent rubric-based scoring protocol, a small judge-validation set, and length-robustness checks. The result that frontier models leave large headroom (≈20 points below the upper bound) is a falsifiable, potentially actionable finding. However, the benchmark's validity rests on whether auto-generated rubrics reward any sound plan for an idea rather than plans that reconstruct the reference paper's specific methodological choices. The human validation currently does not test this fairness property, and the provided worked example (Jailbreak-Tax) shows rubric questions that appear to require the specific attack taxonomy of the reference paper. Because the ranking is the central claim, this concern is load-bearing rather than cosmetic.","major_comments":[{"comment":"The rubric generation prompt (Table 8) instructs the model to 'not ask about anything that is unique or special to this particular research plan' and to judge intent rather than terminology. The worked Jailbreak-Tax rubric in Tables 26–29, however, contains questions that violate this instruction. For example, the Methods section independently requires a prompt-based jailbreak attack, a fine-tuning-based jailbreaking approach, a demonstration-based (many-shot) attack, optimization-based adversarial attacks, genetic/evolutionary strategies, translation-based attacks, and iterative prompt rewriting — seven attack families. The research idea only says 'apply representative jailbreak strategies'; a sound plan could reasonably choose three or four of these families. As written, the rubric mechanically penalizes any plan that does not mirror the reference paper's full attack suite. This conver","section":"§3.2, Tables 8 and 26–29"},{"comment":"The human validation is too weak to establish rubric fairness. In Table 2, experts were asked only whether the extracted rubric questions 'cover the essential parts' of a plan for the idea; they were not asked whether a competent alternative plan that makes different methodological choices would be unfairly penalized. The JudgeEval set (Table 3) is only 10 plans, with a single macro-F1 of 0.91 for o4-mini; no confidence intervals or per-paper breakdown are given. With 200 test rubrics generated by the same model that later serves as judge, a systematic bias in rubric wording (e.g., requiring the reference paper's specific methods) would not be detected by either validation stage. The authors should either (a) run a targeted human study in which expert annotators judge whether each rubric question would be satisfied by a valid alternative plan for the same idea, or (b) release rubrics and","section":"§3.4 and §3.6"},{"comment":"The judge prompt compounds the rubric-specificity problem. It instructs 'only answer Yes if the plan explicitly satisfies the rubric question as written' and 'No if the answer is vague or only implied.' Combined with questions that enumerate the reference paper's attack families (e.g., 'Does the plan include a genetic algorithm or evolutionary strategy?'), this strictness converts any omitted-but-optional design element into a 'No.' The JudgeEval F1 of 0.91 against human annotations does not address this, because the human annotations were produced for the same rubric questions and thus inherit the same specificity bias. The paper should report a sensitivity analysis in which the judge is re-run on rubrics that are explicitly pruned of reference-specific items, or in which alternative human-written reference plans are graded.","section":"§3.5, Table 13"},{"comment":"The paper reports means over three runs but provides no measures of uncertainty or significance testing for the main ranking. The claim that 'GPT-5 and GPT-5-mini substantially outperform other models' relies on differences of a few points (e.g., 62.0 vs 45.7 for o4-mini in naïve), but there are 200 papers and three trials; paired bootstrap or a Wilcoxon test per model pair would be inexpensive and would strengthen the claim. As written, the lack of error bars makes it difficult to assess whether, for example, GPT-5-mini's ReAct score (61.1) is truly distinguishable from GPT-5's ReAct score (61.9). This is secondary to the rubric-fairness issue but should be addressed during revision.","section":"§4.2, Figure 2 and Table 4"}],"minor_comments":[{"comment":"The abstract says 'GPT-5 achieves the strongest performance' while §1 and §4.2 say 'GPT-5 and GPT-5-mini' lead. Please harmonize the wording. Also, there are missing spaces in the typeset text (e.g., 'Idea2Plantask', 'ReActagent', 'AI SCIENTISTproposes'), likely from LaTeX macro expansion.","section":"Abstract and Introduction"},{"comment":"The statement 'Code will be released upon institutional approval' is not a commitment. Since reproducibility of the benchmark depends on public release of the 200 rubrics and ideas, please clarify whether the dataset artifacts (ideas, plans, rubrics, judge outputs) will be released, and under what license.","section":"§1, Footnote 1"},{"comment":"The expert evaluation uses only 8 papers, but the appendix lists 8 selected papers. The domain coverage is reasonable, but the small N means the reported subsection means (e.g., 4.12±0.83 for Introduction plans) have wide confidence intervals; this should be acknowledged in the text.","section":"§3.4, Table 2"},{"comment":"The SFT experiment is described as a 'potential training strategy,' but the negative result is interesting. The explanation that fine-tuning on extracted plans causes hallucination is plausible but not directly measured (no hallucination metric). Please either add a quantitative hallucination check or soften the causal claim.","section":"§4.5 and Appendix J"},{"comment":"Some in-text citations in the appendix are not in the reference list (e.g., 'Perdomo, J. C. (2024)' appears in Table 22 but is not listed in the bibliography). Also, several references cited in the Introduction (e.g., 'Baek et al., 2025') are listed, but a few arXiv identifiers in Table 23 have inconsistent formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea — a post-cutoff, rubric-based benchmark for research planning — is sound and worth publishing if the evaluation instrument is made robust to alternative plans. My main concern is that the worked Jailbreak-Tax rubric in the appendix looks exactly like the kind of reference-paper-specific rubric that the generation prompt tries to avoid; if this is typical, the benchmark measures plan reconstruction, not planning. The human validation does not test this. I would encourage the editor to ask for a targeted fairness study or for evidence that valid alternative plans score well, rather than accepting the current validation at face value. The paper’s contribution would also be stronger with confidence intervals on the model ranking."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your time if you care about evaluating AI research agents. It formalizes the Idea2Plan task, builds a 200-paper benchmark from post-cutoff ICML 2025 papers, and adds a small judge-reliability set. The contamination control is genuine: arXiv dates before the training cutoff are filtered, the ReAct search is blocklisted against the target paper and its citations, and the upper-bound experiment is a sensible sanity check. The reporting is also transparent, including negative results that the ReAct agent doesn't beat simple prompting and SFT degrades performance.\n\nThe main thing to probe is the evaluation instrument. The rubrics, the reference plans, and the final grading are all produced by o4-mini; human validation covers only 8 plans and 10 judge evaluations. The rubric generation prompt tries to force generalization (\"cite X or similar work on Y\", \"do not require exact method match unless it is the only viable option\"), and the judge prompt says examples in parentheses are for reference only. So the stress-test concern about rubric specificity is real but partially mitigated. Still, the jailbreak-tax example shows the problem: the Methods section includes a question specifically about supervised fine-tuning, while an equally reasonable RLHF-based plan would get a No on all three alignment questions. The human study asked whether questions cover essential parts, not whether they unfairly penalize alternative plans. With 192 of 200 rubrics never human-checked for this, the ranking could partly reflect how well models guess the original paper's design choices from the abstract-level idea. The lack of significance tests on the model gaps makes that uncertainty worse.\n\nTwo smaller issues. The curated-literature experiment in Section 4.4 feeds the target paper's own introduction and references into the selector (Table 23), so it leaks the literature answer key; those gains are not evidence that curated retrieval helps. And the abstract mentions Nature Mental Health papers that never appear in the body.\n\nWho is this for? People building or evaluating research agents, and anyone designing rubric-based LLM benchmarks. The benchmark contribution is plausible and the limitations section is honest about LLM-judge bias. It deserves a serious referee, but the authors should expand human validation of the auto-generated rubrics, release the artifacts, add uncertainty quantification, and fix the leak and the abstract discrepancy before publication.","headline":"Solid benchmark for research planning with real contamination control; main risk is that the auto-generated rubric/judge pipeline may reward reconstructing the reference paper rather than general planning skill.","tokens_in":33114,"tokens_out":2259,"would_cite":true,"duration_ms":23873,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Idea2Plan Bench, a benchmark built from papers published after LLM training cutoffs, to measure how well language models turn a research idea into an executable plan, and reports that GPT-5 leads all tested models whil","keywords":["research planning","LLM evaluation","benchmark","rubric-based evaluation","LLM-as-a-judge","data contamination","autonomous research agents","research ideas"],"falsifier":"Take a random sample of, say, fifty generated plans from GPT-5 and o4-mini, have human experts grade them against the same auto-generated rubrics, and compare the resulting scores and ranking to the LLM-judge scores. If human graders do not reproduce the GPT-5 advantage, or if the absolute scores shift by more than a few points, the benchmark's validity as a measure of planning ability is falsified.","tokens_in":31965,"feed_emoji":"📋","tokens_out":2477,"duration_ms":25747,"temperature":0.7,"pith_summary":"The authors set out to measure a capability that has been largely unexamined: how well large language models can bridge the gap from a raw research idea to a concrete, testable research plan. To do this, they built a benchmark from 200 ICML 2025 papers released after the models' training cutoff, so the models could not have memorized the papers. Each benchmark item pairs a research idea with a rubric of yes/no questions derived from the published plan, and a language-model judge grades the generated plans. On this measure, GPT-5 and GPT-5-mini outperform all other tested models, but even the best plans cover only about 62 percent of rubric elements, well short of an estimated 81.9 percent upper bound set by giving a model the full paper. The core contribution is a reproducible instrument for tracking research-planning ability, a capability that matters for building autonomous research agents.","feed_headline":"GPT-5 leads new benchmark for turning ideas into plans","feed_subtitle":"Built on 200 post-cutoff papers, the test leaves roughly 20 points of headroom for better research planners.","key_machinery":"The central object is the auto-generated grading rubric. For each research idea, the authors extract a reference plan from the corresponding paper, then prompt a model to generate a structured rubric of binary yes/no questions that any valid plan for that idea should satisfy. A separate language-model judge, validated against a small set of expert annotations, grades each plan section by section; the section accuracies are macro-averaged into the Planning Score.","core_discovery":"On the paper's own terms, the central claim is that research planning can be isolated and rigorously evaluated, and that current frontier models are already competent but far from saturated at this task. Using a rubric-based scoring protocol, GPT-5 achieves an average Planning Score of 62.0 percent under a simple naïve prompt, and 61.9 percent under a ReAct agentic setup; GPT-5-mini follows at 56.7 and 61.1 percent respectively. The authors find that a ReAct agent with arXiv search and reading tools does not beat simpler prompting, and that fine-tuning on paper-derived idea–plan pairs degrades performance and increases hallucination. They also report that an upper-bound setup, in which o4-mi","pith_inferences":["If the ranking holds beyond this benchmark, GPT-5-class models are a reasonable starting point for building research-planning assistants, but the absolute scores caution against treating current agents as reliable autonomous planners.","Because the rubrics and the judge are both generated and run by the same model family, the absolute Planning Scores may be inflated or deflated in ways that a human-only benchmark would correct; the relative ranking is the more robust claim.","A natural next experiment is to train a judge on rubric judgments from a different model family to test whether the ranking flips, effectively probing whether the scores measure planning ability or evaluator preference.","The benchmark could be extended to other fields by sampling papers from other venues with post-cutoff release dates, but rubric generation would likely need domain-specific adaptation."],"forward_implications":["If the benchmark is accepted, research planning becomes a measurable, trackable capability, giving the field a standardized yardstick for autonomous research agents.","The reported headroom—from roughly 62 percent to an 81.9 percent upper bound—implies that substantially better planners should be achievable without new scientific knowledge.","The finding that retrieval tools did not help the ReAct agent suggests that naive retrieval can introduce noise and that knowledge conflicts are a bottleneck for agentic planning.","The failure of straightforward fine-tuning on paper-derived idea–plan pairs points away from imitation-only training and toward other training signals, such as rubric feedback or execution results.","The consistently weak performance on literature-review sections across all models identifies a specific capability gap to target next."],"fun_headline_variants":["GPT-5 tops new benchmark but research planning still lags","New benchmark grades AI on turning ideas into research plans","Research planning: AI scores just 62% on new test","Benchmark reveals AI's planning gap: room to improve","Idea2Plan: GPT-5 best, but AI planners have far to go"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire evaluation instrument is auto-generated by one AI model: the ideas come from an AI extraction, the rubrics from an AI prompt, and the grading from an AI judge, with only a small sample of human checks; if that model's notion of a valid plan does not generalize to human standards, the reported scores and ranking could reflect evaluator bias rather than planning ability.","fun_headline_variants_meta":{"raw":{"variants":["GPT-5 tops new benchmark but research planning still lags","New benchmark grades AI on turning ideas into research plans","Research planning: AI scores just 62% on new test","Benchmark reveals AI's planning gap: room to improve","Idea2Plan: GPT-5 best, but AI planners have far to go"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000709,"raw_usage":{"total_tokens":3026,"prompt_tokens":740,"completion_tokens":2286,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":2198}},"tokens_in":484,"tokens_out":2286,"duration_ms":18049,"temperature":1.0,"reasoning_tokens":2198,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T07:37:51.510515+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, fifty generated plans from GPT-5 and o4-mini, have human experts grade them against the same auto-generated rubrics, and compare the resulting scores and ranking to the LLM-judge scores. If human graders do not reproduce the GPT-5 advantage, or if the absolute scores shift by more than a few points, the benchmark's validity as a measure of planning ability is falsified.","supporting_citations":[],"review_version":1}