{"id":"f2a2bfb5-dab0-4db2-acdb-f552f6db93d1","arxiv_id":"2504.14191","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper builds a post-cutoff benchmark dataset and a multi-metric evaluation framework for scoring LLM-generated research ideas against real paper motivations and experiment plans.","lead":"AI Idea Bench 2025 is a dataset of 3,495 recent AI papers paired with their inspiration papers, plus a six-metric evaluation framework that scores research ideas generated by large language models. It aims to give the AI-for-science field a standardized, leakage-controlled way to compare idea-generation systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Leakage control is incomplete: RAG baselines query Semantic Scholar with target-derived topics, so the target paper itself can enter generation prompts; Deepseek V3's cutoff is also unaudited.","rationale":"The reader's weakest assumption was judge memorization by Deepseek V3. That is a real risk, but a more direct and arguably more load-bearing failure is that the generation-stage RAG pipelines can retrieve the target paper itself into the prompt, bypassing the carefully chosen knowledge cutoff of GPT-4o. This is not a dispute with consensus or an external critique; it is an internal inconsistency between the stated leakage-control design and the described operation of the baselines. The paper deserves credit for assembling a large dataset, releasing code, and designing a multi-metric framework, but the empirical comparison is the centerpiece, and its validity depends on the target paper being unseen by both the generator context and the judge. Neither condition is verified in the manuscript. The proposed test would settle the retrieval question directly. If retrieval leakage is confirmed, the reported rankings would need to be recomputed with target papers blocked. If not confirmed, the judge-cutoff audit would still be necessary. Since the reader already made the verdict conditional on leakage auditing, I do not change the verdict, but I would strengthen the stated condition: block target papers from retrieval and audit Deepseek V3's exposure before relying on any reported score.","tokens_in":35026,"tokens_out":7221,"duration_ms":67661,"concrete_test":"Instrument a re-run of the AI-Scientist and SCIPIP pipelines on a random subset of 100 topics, logging every Semantic Scholar query and result before any LLM re-ranking. Count how many retrieved lists contain the target paper (by title or arXiv ID). Then re-run with target paper IDs/titles hard-blocked from retrieval and compare IMCQ/I2I scores. If the target appears in retrieval and scores drop when blocked, retrieval leakage is confirmed; if not, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's leakage-control claim rests on selecting target papers after GPT-4o's Oct 3, 2023 cutoff (footnote 2, Section 3.1), but this only controls the generator's parametric memory, not the full evaluation pipeline. Section 4.3 states that AI-Scientist and SCIPIP use Semantic Scholar during generation, and 'AI-Scientist enriches the generation prompt by embedding selected supplementary literature.' The retrieval queries are built from topics and inspiration papers derived from the target paper (Section 3.1; prompts in Appendices I, J, K). Because the target paper is published and indexed, Semantic Scholar can return the target paper itself as 'supplementary literature.' No blocking or filtering of the target paper's ID/title is reported in Section 4.1. If this happens, the ground-truth content appears in the generator's prompt, trivially inflating I2I and IMCQ (Section 3.2.1) for exactly the baselines that use retrieval-augmented prompts. Separately, the judge Deepseek V3 has no audited training cutoff, so it may have memorized the same target papers; this would inflate all judge-based metrics, including Equations 1-4 and the idea competition in Section 3.2.2. The 'robust evaluation methodology' therefore depends on the unverified assumption that no target paper is observed at any stage of the pipeline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AI Idea Bench 2025, a dataset of 3,495 AI papers published after October 3, 2023, together with their inspiration papers, and a six-metric evaluation framework for LLM-generated research ideas. The framework measures alignment with target papers via multiple-choice questions (IMCQ), idea-to-idea matching (I2I), and idea-to-topic matching (I2T), and further evaluates ideas through pairwise competition, novelty assessment against historical/contemporary literature, and a citation-based feasibility score. Four baselines are compared: AI-Researcher, AI-Scientist, SCIPIP, and VIRSCI. The central claims are that the benchmark provides a leakage-controlled, ground-truth-based evaluation methodology and that the reported comparisons identify meaningful differences among current idea-generation systems.","tokens_in":35319,"tokens_out":5627,"duration_ms":48233,"significance":"If the methodology is valid, AI Idea Bench 2025 would be a useful community resource: the dataset is large, the inspiration-target pairing is valuable for grounded evaluation, the authors provide detailed prompts and case studies, and the multi-faceted evaluation goes beyond simple novelty scoring. The paper also makes a serious attempt to control generator-side knowledge leakage by selecting papers after a fixed cutoff. However, the benchmark's validity hinges on two unverified assumptions: that the judge model has not memorized the target papers, and that retrieval-augmented baselines never receive the target paper in their prompts. The absence of any uncertainty quantification further weakens the comparative claims. These issues are load-bearing for the paper's stated contribution as a 'robust evaluation methodology,' but they are addressable in a revision.","major_comments":[{"comment":"The evaluation judge DeepSeek-V3 is used both to construct the ground-truth summaries (Appendix G) and to score all target-paper metrics in Eqs. (1)–(4), yet its training cutoff is never audited. The dataset intentionally contains papers published after October 3, 2023 (footnote 2, §3.1) to avoid leakage into the generator, but the same protection is not established for the judge. If DeepSeek-V3 has memorized any of the 3,495 target papers or their arXiv preprints, IMCQ and I2I scores are inflated for all baselines and the ranking in Table 1 is unreliable. The authors should either use a judge with a verified cutoff, run a control test (e.g., ask the judge to identify whether a generated idea is the target paper), or report an audit of DeepSeek-V3's exposure to the target corpus.","section":"§3.2.1 (Eqs. 1–4), Appendix G"},{"comment":"Leakage control is incomplete for retrieval-augmented baselines. Section 4.3 states that AI-Scientist 'enriches the generation prompt by embedding selected supplementary literature' retrieved through Semantic Scholar, and SCIPIP 'reconstructs an entirely new research background by synthesizing information from both the input and the retrieved literature.' The retrieval queries are built from topics and inspiration papers derived from the target paper (§3.1), so the target paper itself can be returned as 'supplementary literature.' The paper does not report any filtering of the target paper's ID or title from retrieval results. If this occurs, the ground-truth content enters the generator's prompt, trivially inflating I2I and IMCQ for exactly the baselines that use retrieval. The authors should block the target paper in retrieval and report the retrieval logs or a leakage audit.","section":"§4.1, §4.3"},{"comment":"The comparative claims are made without any measure of uncertainty. Section 4.1 states that each baseline generates two ideas per inspiration set, but the reported means in Tables 1 and 2 have no error bars, standard deviations, confidence intervals, or significance tests. Many reported differences are very small (e.g., I2T motivation 4.974 vs. 4.983 vs. 5.0; FA total score 24.873 vs. 24.917), so the conclusion in §4.2.2 that 'AI-Researcher significantly outperforms the other baselines' is unsupported. A permutation test or paired bootstrap over target papers should accompany the headline rankings.","section":"§4.1, Tables 1–2"},{"comment":"The feasibility metric contains a hand-set parameter and an algebraic error. In Eq. (8), lambda=50 is arbitrary and the calibration statement is inaccurate: 'when the number of citations reaches 100, the influence is increased to 0.9' should be 1−exp(−100/50)=0.865. In Eq. (9), the second sum starts at yl−2 and the first sum ends at yl−2, so the term for year yl−2 is counted twice when yp≤yl−2. Since FA and FPS in Table 2 are derived from this formula, the metric needs to be redefined and re-run before the feasibility ranking can be accepted.","section":"§3.2.2, Eqs. (8)–(9)"}],"minor_comments":[{"comment":"The caption contains typos: 'genearation pipline' should be 'generation pipeline'.","section":"Figure 1 caption"},{"comment":"'reuslt' should be 'result'; also the definition of S_M as a binary variable is clear but the notation could be unified with the rest of the paper.","section":"Eq. (2)"},{"comment":"Case 2 labels the target paper 'MDAgents' but cites reference [28], which is 'Buffer of Thoughts'; the citation and reference list entry are mismatched.","section":"Appendix D"},{"comment":"The competition scoring awards 3 points for a win and 0 for a loss, but no tie rule is defined; pairwise LLM comparisons in the described prompt may produce ties.","section":"§3.2.2, Eq. (5)"},{"comment":"The notation is inconsistent: the text defines M^IT_D, while Eq. (4) uses F^D_IT; unify the notation for the similarity function.","section":"§3.2.1, Eq. (4)"},{"comment":"No data availability statement, versioning information, or license is provided beyond a project URL; for a benchmark paper these details are important for community adoption and reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is a benchmark, so data release, evaluation code, and leakage auditing are essential. The judge-cutoff issue and the retrieval-leakage issue are the main risks; if the authors can provide a leakage audit, add uncertainty quantification, and correct the feasibility formula, the benchmark could be a useful community resource. The current manuscript is not ready for acceptance because the headline comparisons rest on unverified assumptions about the evaluation pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution—a 3,495-paper post-cutoff dataset with paired inspiration papers, plus a multi-metric evaluation—but the 'robust evaluation methodology' claim is not yet earned. The stress-test note is right, and it lands harder than the reader's report suggests.\n\nWhat's new: the dataset construction is genuinely useful. Papers from top venues after October 2023, with the ten most-cited candidates shortlisted by two researchers down to five inspiration papers, make a workable input-output set. The human step is expensive and, as far as I can tell, unaudited—no inter-annotator stats, no release of the selections—so the paired inputs could carry hidden biases. Still, the scale is new. The six metrics (MCQ, I2I, I2T, competition, novelty, feasibility) are a reasonable attempt to move beyond single-score evaluation, and the case studies make the behavior concrete.\n\nThe soft spots are real. First, the judge, Deepseek V3, writes the ground-truth summaries (Section 3.1, prompt G) and then scores alignment with those summaries (Equations 1–4). No study shows that the judge's own summary doesn't become the reference. Second, leakage control is incomplete. The paper only controls the generator's cutoff (GPT-4o-2024-11-20, Oct 3 2023). But the retrieval-augmented baselines query Semantic Scholar during generation, and the query terms come from the target paper. AI-Scientist explicitly embeds retrieved 'supplementary literature' into the prompt; SCIPIP synthesizes a new background from input and retrieved literature. Nothing filters the target paper itself from those API results. If the target appears in the prompt, I2I and MCQ scores are inflated for exactly the systems that lost on other axes. The paper's own conclusion says it 'eliminat[es] the risk of knowledge leakage'—that is too strong. Third, all scores are point estimates; no error bars, no significance tests, and lambda=50 in Eq. 8 is a hand-set knob with no sensitivity analysis.\n\nWhat holds up: the central ranking—AI-Scientist ahead on alignment, AI-Researcher ahead on feasibility—is plausible and consistent with the workflows described. The dataset alone is worth refereeing. This paper should go to peer review, with the expectation of major revision: audit the judge's cutoff, filter target IDs from retrieval, add uncertainty, and release the inspiration-paper selections.\n\nIf I were working on idea generation, I would cite it; I'd otherwise wait for the revision.","headline":"An honest, potentially useful benchmark that is not yet leakage-proof: the judge writes the ground truth and the RAG baselines can retrieve the target itself.","tokens_in":35846,"tokens_out":3531,"would_cite":false,"duration_ms":30629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AI Idea Bench 2025 scores LLM-generated research ideas against 3,495 real papers, using ground-truth alignment plus reference-based novelty and feasibility metrics.","keywords":["AI research idea generation","benchmark","knowledge leakage","LLM evaluation","scientific discovery","ground truth","feasibility assessment","novelty assessment"],"falsifier":"Train or obtain a judge LLM whose training corpus is verifiably free of the 3,495 target papers (for example, a local model fine-tuned only on pre-2023 data), re-run the six evaluation protocols, and compare scores; if IMCQ and I2I scores drop markedly or the ranking of AI-Scientist versus AI-Researcher flips, the original numbers were contaminated by judge memorization.","tokens_in":34858,"feed_emoji":"🧪","tokens_out":5113,"duration_ms":43222,"temperature":0.7,"pith_summary":"AI Idea Bench 2025 is a benchmark for measuring whether large language models can generate research ideas that match what real AI researchers later published. The paper's central claim is that existing idea-generation evaluations are undermined by data leakage, missing ground truth, and prompt-limited feasibility checks, and that a dataset of 3,495 recent papers paired with their inspiring works fixes all three. On this benchmark, AI-Scientist produces ideas that align most closely with the target papers, while AI-Researcher wins the reference-based competition and feasibility scores, and both outperform the SCIPIP baseline. If the benchmark holds up, it gives the field a reusable, leakage-controlled yardstick for comparing idea-generation systems and for automating scientific discovery.","feed_headline":"New benchmark checks AI ideas against real papers","feed_subtitle":"3,495 post-cutoff papers give grounded scores for novelty, feasibility, and match to ground truth.","key_machinery":"The central object is the paired dataset: 3,495 target papers published after the GPT-4o knowledge cutoff of October 3, 2023, each accompanied by five curated inspiration papers, extracted motivations, experimental designs, and anonymized topics. The argument runs through the evaluation pipeline built on those pairs: IMCQ (multiple-choice with one ground-truth answer and three distractors), I2I and I2T matching scored by the judge model, pairwise idea competition, a novelty formula using historical and contemporary paper distances and citation counts, and a feasibility formula that averages time-decayed influence of reference methods. The judge model is Deepseek V3, and the same model performs the matching, selection, and competition judgments.","core_discovery":"The paper establishes that idea-generation quality can be measured quantitatively in two complementary ways. First, by feeding a model the inspiration papers behind a real AI paper and asking whether the model's generated idea matches the target paper's own motivation and experimental plan, scored by multiple-choice selection, idea-to-idea similarity, and idea-to-topic similarity. Second, by evaluating generated ideas against external reference material, through pairwise competition among baselines, a citation-weighted novelty score, and a stepwise feasibility score grounded in the cited influence of reference methods. The reported results show AI-Scientist with the highest alignment scores (idea-to-idea motivation 3.591/5 and multiple-choice motivation 0.611), AI-Researcher leading the competition ranking and overall feasibility, and AI-Scientist leading novelty. The paper argues these results validate the benchmark's ability to discriminate meaningful differences among idea generators.","pith_inferences":["Editorial inference: A natural extension would be to hold out the judge model's own training data — replacing Deepseek V3 with a model whose cutoff is verified to precede the target papers would confirm the scores are not contamination artifacts.","Editorial inference: The benchmark's MCQ success criterion counts only whether the ground-truth option is selected at least once across the generated idea cluster, so future versions could measure partial overlap or rank position of the correct match.","Editorial inference: The same paired target-and-inspiration construction could be transferred to other disciplines, such as chemistry or neuroscience, where conference papers and citation graphs are equally available."],"forward_implications":["Anyone running an idea-generation system can drop in this benchmark and obtain three target-aligned scores and four reference-based scores without hiring human experts.","Because the targets are all published after the generator's knowledge cutoff, high alignment scores become evidence that the model is doing more than recalling a paper it memorized.","The benchmark converts feasibility from a vibe judgment into a citation-grounded number, making it possible to compare experimental plans across methods.","The reported ranking gives a concrete baseline for future work: AI-Scientist-style relevance re-ranking helps alignment, while AI-Researcher-style iterative self-reflection helps feasibility and competition standing.","The six metric scores can be combined as a leaderboard that tracks progress in automating scientific discovery."],"supporting_citations":[{"why":"Provides the Deepseek V3 model used as the judge for all matching, selection, and competition evaluations in Equations 1-4.","marker":"[4]"},{"why":"Supplies the AI-Scientist baseline whose alignment results are compared in the main experiments.","marker":"[15]"},{"why":"Supplies the AI-Researcher baseline that leads the reference-based competition and feasibility assessments.","marker":"[21]"},{"why":"Supplies the VIRSCI baseline and the historical-difference novelty measure that Formula 7 adapts.","marker":"[19]"},{"why":"Supplies the SCIPIP baseline used for motivation-only comparison.","marker":"[18]"},{"why":"Defines the earlier IdeaBench ground-truth evaluation limited to titles and abstracts, which this paper extends to full motivations and experiment plans.","marker":"[14]"},{"why":"Supplies the pre-processing approach for inspiration papers that the dataset construction follows.","marker":"[20]"}],"fun_headline_variants":["AI idea quality now measurable against real papers","Benchmark grades AI ideas on novelty and feasibility","3,495 papers give AI ideas a grounded reality check","AI idea generation scored by alignment, not just vibe","New test separates strong AI ideas from weak ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark rests on whether the judge model used for scoring has not already memorized the 3,495 target papers; if it has, every alignment and competition score is inflated and the ranking is unreliable.","fun_headline_variants_meta":{"raw":{"variants":["AI idea quality now measurable against real papers","Benchmark grades AI ideas on novelty and feasibility","3,495 papers give AI ideas a grounded reality check","AI idea generation scored by alignment, not just vibe","New test separates strong AI ideas from weak ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000133,"raw_usage":{"total_tokens":1104,"prompt_tokens":883,"completion_tokens":221,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":148}},"tokens_in":499,"tokens_out":221,"duration_ms":2673,"temperature":1.0,"reasoning_tokens":148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:54:10.435567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train or obtain a judge LLM whose training corpus is verifiably free of the 3,495 target papers (for example, a local model fine-tuned only on pre-2023 data), re-run the six evaluation protocols, and compare scores; if IMCQ and I2I scores drop markedly or the ranking of AI-Scientist versus AI-Researcher flips, the original numbers were contaminated by judge memorization.","supporting_citations":[],"review_version":1}