Pith. sign in

REVIEW 4 major objections 6 minor 9 cited by

AI Idea Bench 2025: AI Research Idea Generation Benchmark

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AI Idea Bench 2025 scores LLM-generated research ideas against 3,495 real papers, using ground-truth alignment plus reference-based novelty and feasibility metrics.

desk verdict An honest, potentially useful benchmark that is not yet leakage-proof: the judge writes the ground truth and the RAG baselines can retrieve the target itself. read the letter →

arxiv 2504.14191 v3 pith:QEI4MOFF submitted 2025-04-19 cs.AI cs.CL

Yansheng Qiu , Haoquan Zhang , Zhaopan Xu , Ming Li , Diping Song , Zheng Wang , Kaipeng Zhang This is my paper · ORCID
classification cs.AIcs.CL
keywords AIresearchideagenerationbenchmarkknowledgeleakageLLMevaluationscientificdiscoverygroundtruthfeasibilityassessmentnovelty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AI Idea Bench 2025 is a benchmark for measuring whether large language models can generate research ideas that match what real AI researchers later published. The paper's central claim is that existing idea-generation evaluations are undermined by data leakage, missing ground truth, and prompt-limited feasibility checks, and that a dataset of 3,495 recent papers paired with their inspiring works fixes all three. On this benchmark, AI-Scientist produces ideas that align most closely with the target papers, while AI-Researcher wins the reference-based competition and feasibility scores, and both outperform the SCIPIP baseline. If the benchmark holds up, it gives the field a reusable, leakage-controlled yardstick for comparing idea-generation systems and for automating scientific discovery.

What carries the argument

The central object is the paired dataset: 3,495 target papers published after the GPT-4o knowledge cutoff of October 3, 2023, each accompanied by five curated inspiration papers, extracted motivations, experimental designs, and anonymized topics. The argument runs through the evaluation pipeline built on those pairs: IMCQ (multiple-choice with one ground-truth answer and three distractors), I2I and I2T matching scored by the judge model, pairwise idea competition, a novelty formula using historical and contemporary paper distances and citation counts, and a feasibility formula that averages time-decayed influence of reference methods. The judge model is Deepseek V3, and the same model performs the matching, selection, and competition judgments.

What would settle it

Train or obtain a judge LLM whose training corpus is verifiably free of the 3,495 target papers (for example, a local model fine-tuned only on pre-2023 data), re-run the six evaluation protocols, and compare scores; if IMCQ and I2I scores drop markedly or the ranking of AI-Scientist versus AI-Researcher flips, the original numbers were contaminated by judge memorization.

Watch

Extended reading notes

Core claim

The paper establishes that idea-generation quality can be measured quantitatively in two complementary ways. First, by feeding a model the inspiration papers behind a real AI paper and asking whether the model's generated idea matches the target paper's own motivation and experimental plan, scored by multiple-choice selection, idea-to-idea similarity, and idea-to-topic similarity. Second, by evaluating generated ideas against external reference material, through pairwise competition among baselines, a citation-weighted novelty score, and a stepwise feasibility score grounded in the cited influence of reference methods. The reported results show AI-Scientist with the highest alignment scores (idea-to-idea motivation 3.591/5 and multiple-choice motivation 0.611), AI-Researcher leading the competition ranking and overall feasibility, and AI-Scientist leading novelty. The paper argues these results validate the benchmark's ability to discriminate meaningful differences among idea generators.

Load-bearing premise

The benchmark rests on whether the judge model used for scoring has not already memorized the 3,495 target papers; if it has, every alignment and competition score is inflated and the ranking is unreliable.

Editorial extensions

If this is right

  • Anyone running an idea-generation system can drop in this benchmark and obtain three target-aligned scores and four reference-based scores without hiring human experts.
  • Because the targets are all published after the generator's knowledge cutoff, high alignment scores become evidence that the model is doing more than recalling a paper it memorized.
  • The benchmark converts feasibility from a vibe judgment into a citation-grounded number, making it possible to compare experimental plans across methods.
  • The reported ranking gives a concrete baseline for future work: AI-Scientist-style relevance re-ranking helps alignment, while AI-Researcher-style iterative self-reflection helps feasibility and competition standing.
  • The six metric scores can be combined as a leaderboard that tracks progress in automating scientific discovery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: A natural extension would be to hold out the judge model's own training data — replacing Deepseek V3 with a model whose cutoff is verified to precede the target papers would confirm the scores are not contamination artifacts.
  • Editorial inference: The benchmark's MCQ success criterion counts only whether the ground-truth option is selected at least once across the generated idea cluster, so future versions could measure partial overlap or rank position of the correct match.
  • Editorial inference: The same paired target-and-inspiration construction could be transferred to other disciplines, such as chemistry or neuroscience, where conference papers and citation graphs are equally available.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces AI Idea Bench 2025, a dataset of 3,495 AI papers published after October 3, 2023, together with their inspiration papers, and a six-metric evaluation framework for LLM-generated research ideas. The framework measures alignment with target papers via multiple-choice questions (IMCQ), idea-to-idea matching (I2I), and idea-to-topic matching (I2T), and further evaluates ideas through pairwise competition, novelty assessment against historical/contemporary literature, and a citation-based feasibility score. Four baselines are compared: AI-Researcher, AI-Scientist, SCIPIP, and VIRSCI. The central claims are that the benchmark provides a leakage-controlled, ground-truth-based evaluation methodology and that the reported comparisons identify meaningful differences among current idea-generation systems.

Significance. If the methodology is valid, AI Idea Bench 2025 would be a useful community resource: the dataset is large, the inspiration-target pairing is valuable for grounded evaluation, the authors provide detailed prompts and case studies, and the multi-faceted evaluation goes beyond simple novelty scoring. The paper also makes a serious attempt to control generator-side knowledge leakage by selecting papers after a fixed cutoff. However, the benchmark's validity hinges on two unverified assumptions: that the judge model has not memorized the target papers, and that retrieval-augmented baselines never receive the target paper in their prompts. The absence of any uncertainty quantification further weakens the comparative claims. These issues are load-bearing for the paper's stated contribution as a 'robust evaluation methodology,' but they are addressable in a revision.

major comments (4)
  1. [§3.2.1 (Eqs. 1–4), Appendix G] The evaluation judge DeepSeek-V3 is used both to construct the ground-truth summaries (Appendix G) and to score all target-paper metrics in Eqs. (1)–(4), yet its training cutoff is never audited. The dataset intentionally contains papers published after October 3, 2023 (footnote 2, §3.1) to avoid leakage into the generator, but the same protection is not established for the judge. If DeepSeek-V3 has memorized any of the 3,495 target papers or their arXiv preprints, IMCQ and I2I scores are inflated for all baselines and the ranking in Table 1 is unreliable. The authors should either use a judge with a verified cutoff, run a control test (e.g., ask the judge to identify whether a generated idea is the target paper), or report an audit of DeepSeek-V3's exposure to the target corpus.
  2. [§4.1, §4.3] Leakage control is incomplete for retrieval-augmented baselines. Section 4.3 states that AI-Scientist 'enriches the generation prompt by embedding selected supplementary literature' retrieved through Semantic Scholar, and SCIPIP 'reconstructs an entirely new research background by synthesizing information from both the input and the retrieved literature.' The retrieval queries are built from topics and inspiration papers derived from the target paper (§3.1), so the target paper itself can be returned as 'supplementary literature.' The paper does not report any filtering of the target paper's ID or title from retrieval results. If this occurs, the ground-truth content enters the generator's prompt, trivially inflating I2I and IMCQ for exactly the baselines that use retrieval. The authors should block the target paper in retrieval and report the retrieval logs or a leakage audit.
  3. [§4.1, Tables 1–2] The comparative claims are made without any measure of uncertainty. Section 4.1 states that each baseline generates two ideas per inspiration set, but the reported means in Tables 1 and 2 have no error bars, standard deviations, confidence intervals, or significance tests. Many reported differences are very small (e.g., I2T motivation 4.974 vs. 4.983 vs. 5.0; FA total score 24.873 vs. 24.917), so the conclusion in §4.2.2 that 'AI-Researcher significantly outperforms the other baselines' is unsupported. A permutation test or paired bootstrap over target papers should accompany the headline rankings.
  4. [§3.2.2, Eqs. (8)–(9)] The feasibility metric contains a hand-set parameter and an algebraic error. In Eq. (8), lambda=50 is arbitrary and the calibration statement is inaccurate: 'when the number of citations reaches 100, the influence is increased to 0.9' should be 1−exp(−100/50)=0.865. In Eq. (9), the second sum starts at yl−2 and the first sum ends at yl−2, so the term for year yl−2 is counted twice when yp≤yl−2. Since FA and FPS in Table 2 are derived from this formula, the metric needs to be redefined and re-run before the feasibility ranking can be accepted.
minor comments (6)
  1. [Figure 1 caption] The caption contains typos: 'genearation pipline' should be 'generation pipeline'.
  2. [Eq. (2)] 'reuslt' should be 'result'; also the definition of S_M as a binary variable is clear but the notation could be unified with the rest of the paper.
  3. [Appendix D] Case 2 labels the target paper 'MDAgents' but cites reference [28], which is 'Buffer of Thoughts'; the citation and reference list entry are mismatched.
  4. [§3.2.2, Eq. (5)] The competition scoring awards 3 points for a win and 0 for a loss, but no tie rule is defined; pairwise LLM comparisons in the described prompt may produce ties.
  5. [§3.2.1, Eq. (4)] The notation is inconsistent: the text defines M^IT_D, while Eq. (4) uses F^D_IT; unify the notation for the similarity function.
  6. [General] No data availability statement, versioning information, or license is provided beyond a project URL; for a benchmark paper these details are important for community adoption and reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; benchmark scores are externally anchored, with leakage risks being contamination rather than circularity.

full rationale

The paper does not derive a scientific prediction from fitted parameters; it constructs an external benchmark. The target papers are external documents, and the inspiration-paper inputs are collected from the target papers' reference lists. The metrics in Equations 1-10 are computed against the target paper text, Semantic Scholar citation data, reference-based LLM judgments, and pairwise comparisons among baseline outputs, none of which is defined in terms of another reported score. No parameter is fitted to the quantity it later predicts: the lambda in Equation 8 is explicitly hand-set, and the novelty formula in Equation 7 is adopted from prior work rather than derived from this paper's own data. The use of Deepseek V3 both for extracting target summaries and for judging similarity is a validity and possible contamination risk, as is the possibility that retrieval-augmented baselines retrieve the target paper itself through Semantic Scholar, but these are leakage/robustness concerns rather than circular reductions: the central comparison between baseline methods is not forced by construction, and the target-paper ground truth remains external to the generated ideas. There is no self-citation chain carrying the argument, no fitted input rebranded as a prediction, and no result that is equivalent to its inputs by definition.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity depends on four unverified domain assumptions about knowledge cutoffs, LLM judge reliability, inspiration-paper selection, and citation counts as feasibility proxies.

free parameters (1)
  • impact factor lambda in influence normalization (Eq. 8) = 50
    Hand-set to calibrate the curve so 35 citations give 0.5 influence and 100 give 0.9; this choice directly shapes feasibility scores.
assumptions (4)
  • domain assumption Target papers published after October 3, 2023 are unknown to GPT-4o-2024-11-20.
    The entire leakage-control design rests on this cutoff date, stated in Section 1 footnote 2 and Section 3.1, without a verifiable source.
  • domain assumption Deepseek V3's judgments (similarity, MCQ choices, competition) are valid and bias-free proxies for research idea quality.
    All six evaluation metrics depend on Deepseek V3, yet the paper provides no calibration against human experts or other judges.
  • domain assumption The five 'inspiration papers' selected per target paper by Deepseek V3 plus two researchers accurately capture the true intellectual antecedents.
    Section 3.1 describes the selection but gives no criteria, audit trail, or reproducibility details.
  • domain assumption Citation counts from Semantic Scholar reflect the feasibility and influence of an experimental method.
    Feasibility assessment (Eq. 8-10) equates high citations of related methods with high feasibility of the proposed idea.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI Idea Bench 2025: AI Research Idea Generation Benchmark." pith.science (2026). https://pith.science/paper/QEI4MOFF

@misc{pith2026250414191,
  author       = {Pith},
  title        = {Pith review of: AI Idea Bench 2025: AI Research Idea Generation Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QEI4MOFF}},
  note         = {Machine review of arXiv:2504.14191}
}
read the original abstract

Large-scale Language Models (LLMs) have revolutionized human-AI interaction and achieved significant success in the generation of novel ideas. However, current assessments of idea generation overlook crucial factors such as knowledge leakage in LLMs, the absence of open-ended benchmarks with grounded truth, and the limited scope of feasibility analysis constrained by prompt design. These limitations hinder the potential of uncovering groundbreaking research ideas. In this paper, we present AI Idea Bench 2025, a framework designed to quantitatively evaluate and compare the ideas generated by LLMs within the domain of AI research from diverse perspectives. The framework comprises a comprehensive dataset of 3,495 AI papers and their associated inspired works, along with a robust evaluation methodology. This evaluation system gauges idea quality in two dimensions: alignment with the ground-truth content of the original papers and judgment based on general reference material. AI Idea Bench 2025's benchmarking system stands to be an invaluable resource for assessing and comparing idea-generation techniques, thereby facilitating the automation of scientific discovery.

Figures

Figures reproduced from arXiv: 2504.14191 by the authors.

Figure 1
Figure 1. Comparison with current idea genearation pipline. (a) Current idea-generation methods retrieve relevant literature based on topics and use it as a corpus for idea generation, which leads to a lack of reference for idea evaluation. (b) Our The AI Idea Bench 2025 first identifies the target paper, then determines the corpus for idea generation by extracting its content, and uses this as the ground truth when evaluatin… view at source ↗
Figure 2
Figure 2. Overall pipeline of AI Idea Bench 2025. First, we decompose and summarize the motivation, experimental steps, topic, and the inspiration papers from the target paper. Then, we extract the motivation and experimental steps from the inspiration papers, and generate a cluster of ideas in combination with the topic of the target paper. Finally, we compare the idea-generation methods in six evaluations: idea multiple-cho… view at source ↗
Figure 3
Figure 3. A case of idea generation on motivation. In the visual annotations, text highlighted with a green background denotes areas of overlap between the generated ideas and those of the target paper. The red background indicates elements within the generated ideas that are thematically aligned with current research based on the given topic. While SCIPIP, AI-Scientist, and AI-Researcher all employ Semantic Scholar’s API for… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Tree-of-Ideas: Automated Research Ideation via Cross-Trajectory Reasoning over Scholarly Evolution

    cs.AI 2026-08 conditional novelty 7.0 of 10

    A citation-graph system that traces how research gaps evolve along branching literature trajectories can generate AI research ideas rated near human-paper quality on novelty and groundedness.

  2. ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Conference accept/reject outcomes yield 15 operational ideation patterns that, as an LLM skill suite, improve automated-judged research-proposal quality over no-skill and generic-skill baselines.

  3. LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    LigBench evaluates research ideas through formalization, pairwise comparison against 11k+ papers with debiased OpenReview scores, and Elo-based score aggregation, validated against human experts and paper acceptance outcomes.

  4. AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

    cs.CL 2026-02 reject novelty 6.0 of 10

    AD-Bench evaluates LLM agents on real advertising analytics tasks using replayed expert tool-call trajectories, and finds even top models drop sharply on hard multi-step queries.

  5. Creativity in LLM-based Multi-Agent Systems: A Survey

    cs.HC 2025-05 conditional novelty 6.0 of 10

    A taxonomy-driven survey organizes the emerging field of creativity in LLM-based multi-agent systems across workflows, techniques, personas, datasets, and evaluation metrics.

  6. SciDER: Scientific Data-centric End-to-end Researcher

    cs.AI 2026-03 unverdicted novelty 5.0 of 10

    SciDER is a data-centric multi-agent system that automates ideation, raw-data analysis, experiment coding, and critique, with reported leading results on six scientific-agent benchmarks.

  7. SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents

    cs.AI 2025-05 reject novelty 5.0 of 10

    SafeScientist adds prompt, discussion, tool-use, and output-review safety checks to an AI scientist, with a new domain benchmark, but its reported evaluation is internally inconsistent.

  8. InternAgent: When Agent Becomes the Scientist -- Building Closed-Loop System from Hypothesis to Verification

    cs.AI 2025-05 conditional novelty 5.0 of 10

    A closed-loop LLM-agent framework that auto-generates research ideas and code, reported to improve baseline performance on all 12 tasks it was tested on.

  9. AI for Auto-Research: Roadmap & User Guide

    cs.AI 2026-05 unverdicted novelty 4.0 of 10

    The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.

Reference graph

Works this paper leans on

75 extracted references · 42 canonical work pages · cited by 9 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  3. [3]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    arXiv 2024

  4. [4]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  5. [5]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023

    arXiv 2023

  6. [6]

    Leandojo: Theorem proving with retrieval-augmented language models

    Kaiyu Yang, Aidan M Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. Leandojo: Theorem proving with retrieval-augmented language models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track ,

    work page

  7. [7]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  8. [8]

    The impact of large language models on scientific discovery: a preliminary study using gpt-4

    Microsoft Research AI4Science and Microsoft Azure Quantum. The impact of large language models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361, 2023

    arXiv 2023

Show all 75 references
  1. [9]

    Large language models for automated open-domain scientific hypotheses discovery

    Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. Large language models for automated open-domain scientific hypotheses discovery. arXiv preprint arXiv:2309.02726, 2023

    2023 arXiv

  2. [10]

    Scimon: Scientific inspiration machines optimized for novelty

    Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259, 2023

    2023 arXiv

  3. [11]

    Hypothesis generation with large language models

    Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. Hypothesis generation with large language models. arXiv preprint arXiv:2404.04326, 2024

    2024 arXiv

  4. [12]

    Researchagent: Iterative research idea generation over scientific literature with large language models

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738, 2024

    2024 arXiv

  5. [13]

    Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement

    Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, et al. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. arXiv preprint arXi...

    arXiv

  6. [14]

    Ideabench: Benchmarking large language models for research idea generation

    Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Eric Xie, Stefan Bekiranov, and Aidong Zhang. Ideabench: Benchmarking large language models for research idea generation. arXiv preprint arXiv:2411.02429, 2024

    2024 arXiv

  7. [15]

    The ai scientist: Towards fully automated open-ended scientific discovery

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024

    2024 arXiv

  8. [16]

    Mlr-copilot: Autonomous machine learning research based on large language models agents

    Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. Mlr-copilot: Autonomous machine learning research based on large language models agents. arXiv preprint arXiv:2408.14033, 2024

    2024

  9. [17]

    Nova: An iterative planning and search approach to enhance novelty and diversity of llm generated ideas

    Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, and Zhenzhong Lan. Nova: An iterative planning and search approach to enhance novelty and diversity of llm generated ideas. arXiv preprint arXiv:2410.14255, 2024

    arXiv

  10. [18]

    Scipip: An llm-based scientific paper idea proposer

    Wenxiao Wang, Lihui Gu, Liye Zhang, Yunxiang Luo, Yi Dai, Chen Shen, Liang Xie, Binbin Lin, Xiaofei He, and Jieping Ye. Scipip: An llm-based scientific paper idea proposer. arXiv preprint arXiv:2410.23166, 2024. 10

    2024 arXiv

  11. [19]

    Two heads are better than one: A multi-agent system has the potential to improve scientific idea generation

    Haoyang Su, Renqi Chen, Shixiang Tang, Xinzhe Zheng, Jingzhe Li, Zhenfei Yin, Wanli Ouyang, and Nanqing Dong. Two heads are better than one: A multi-agent system has the potential to improve scientific idea generation. arXiv preprint arXiv:2410.09403, 2024

    arXiv

  12. [20]

    Chain of ideas: Revolutionizing research via novel idea development with llm agents

    Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, et al. Chain of ideas: Revolutionizing research via novel idea development with llm agents. arXiv preprint arXiv:2410.13185, 2024

    arXiv

  13. [21]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024

    2024 arXiv

  14. [22]

    Openess: Event- based semantic scene understanding with open vocabularies

    Lingdong Kong, Youquan Liu, Lai Xing Ng, Benoit R Cottereau, and Wei Tsang Ooi. Openess: Event- based semantic scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15686–15698, 2024

    2024

  15. [23]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  16. [24]

    Ev-segnet: Semantic segmentation for event-based cameras

    Inigo Alonso and Ana C Murillo. Ev-segnet: Semantic segmentation for event-based cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 0–0, 2019

    2019

  17. [25]

    Event-based vision: A survey

    Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(1):15...

    2020

  18. [26]

    Ess: Learning event-based semantic segmentation from still images

    Zhaoning Sun, Nico Messikommer, Daniel Gehrig, and Davide Scaramuzza. Ess: Learning event-based semantic segmentation from still images. In European Conference on Computer Vision, pages 341–357. Springer, 2022

    2022

  19. [27]

    Slic superpixels compared to state-of-the-art superpixel methods

    Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11):2274–2282, 2012

    2012

  20. [28]

    Buffer of thoughts: Thought- augmented reasoning with large language models, 2024

    L Yang, Z Yu, T Zhang, S Cao, M Xu, W Zhang, JE Gonzalez, and B Cui. Buffer of thoughts: Thought- augmented reasoning with large language models, 2024. URL https://arxiv. org/abs/2406.04271

    2024 arXiv

  21. [29]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  22. [30]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023

    2023

  23. [31]

    Expertprompting: Instructing large language models to be distinguished experts

    Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models to be distinguished experts. arXiv preprint arXiv:2305.14688, 2023

    2023 arXiv

  24. [32]

    Pal: Program-aided language models

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning , pages 10764–10799. PMLR, 2023

    2023

  25. [33]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  26. [34]

    Large language models are few-shot clinical information extractors

    Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. Large language models are few-shot clinical information extractors. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998–2022, 2022

    2022

  27. [35]

    Nlice: Synthetic medical record generation for effective primary healthcare differential diagnosis

    Zaid Al-Ars, Obinna Agba, Zhuoran Guo, Christiaan Boerkamp, Ziyaad Jaber, and Tareq Jaber. Nlice: Synthetic medical record generation for effective primary healthcare differential diagnosis. In 2023 IEEE 23rd International Conference on Bioinformatics and Bioengineering (BIBE)...

    2023

  28. [36]

    Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images

    Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric Chang, Tackeun Kim, et al. Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images. Advances in Neural Information Processing S...

    2023

  29. [37]

    Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians

    Michael L Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W Bates. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians. JAMA network open, 2 (3):e190096–e190096, 2019

    2019

  30. [38]

    The use of analytic hierarchy process for measuring the complexity of medical diagnosis

    Ofir Ben-Assuli, Nanda Kumar, Ofer Arazy, and Itamar Shabtai. The use of analytic hierarchy process for measuring the complexity of medical diagnosis. Health Informatics Journal, 26(1):218–232, 2020

    2020

  31. [39]

    Koala: Key frame-conditioned long video-llm

    Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13581–13591, 2024

    2024

  32. [40]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  33. [41]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems , 36:46212–46244, 2023

    2023

  34. [42]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision , pages...

    2019

  35. [43]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  36. [44]

    Fine-grained cross-view geo- localization using a correlation-aware homography estimator

    Xiaolong Wang, Runsen Xu, Zhuofan Cui, Zeyu Wan, and Yu Zhang. Fine-grained cross-view geo- localization using a correlation-aware homography estimator. Advances in Neural Information Processing Systems, 36:5301–5319, 2023

    2023

  37. [45]

    R-tuning: Instructing large language models to say ‘i don’t know’

    Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say ‘i don’t know’. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computati...

    2024

  38. [46]

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, 2023

    2023

  39. [47]

    On faithfulness and factuality in abstractive summarization

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661, 2020

    2005 arXiv

  40. [48]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  41. [49]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321–1330. PMLR, 2017

    2017

  42. [50]

    Step 1: Gather Datasets

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 12 Contents 1 Int...

    2023 arXiv

  43. [52]

    Diagnostic Accuracy

    "Diagnostic Accuracy": "Percentage of correct diagnoses.",

  44. [53]

    Temporal Consistency Score

    "Temporal Consistency Score": "Measures how often outputs conform to temporal constraints in the input data.",

  45. [54]

    Explainability Assessment

    "Explainability Assessment": "Qualitative evaluation of the intermedi- ate steps." * "Step 3: Construct Temporal Reasoning Prompts": · "Examples of Prompts for Temporal Decomposition": "Input: ’The patient developed a headache on Tuesday, vomiting on Thursday, and sleep distur...

    2018

  46. [55]

    No Similarity: Entirely distinct problems, contexts, and methodologies

  47. [56]

    Low Similarity: Minimal overlap in one aspect (e.g., tangential problem mention but divergent focus/theory)

  48. [57]

    technology)

    Moderate Similarity: Shared problem domain but differing approaches/theories (e.g., both address climate change mitigation but focus on policy vs. technology)

  49. [58]

    High Similarity: Aligned problem focus and theory with minor differences in scope or application (e.g., both study AI bias in healthcare, but one targets diagnostics and the other patient data)

  50. [59]

    motivation_similarity

    Complete Similarity: Identical problems, theories, and contextual applications. Output Format: Return a JSON object for each pair with: • A numeric ‘rating‘ (1–5) • A concise ‘explanation‘ highlighting specific overlaps and key distinctions (1–2 sentences). Example Output: { "...

  51. [60]

    No similarity: Entirely different problems/theories/structures

  52. [61]

    Low similarity: Minor overlap in one criterion (e.g., same problem but divergent theories)

  53. [62]

    Moderate similarity: Align in 1–2 criteria with clear differences (e.g., shared theory but distinct structures)

  54. [63]

    High similarity: Align in 2–3 criteria with minor discrepancies (e.g., same problem and structure but different theories)

  55. [64]

    experiment_plan_similarity

    Complete similarity: Identical in all criteria. Output Requirements: • Return JSON format with ‘rating‘ (1–5) and ‘explanation‘. • The explanation must explicitly reference structure, theory, and problem focus. Example Output: { "experiment_plan_similarity": { "rating": 3, "ex...

  56. [65]

    The motivation does not address the topic, and the Experimental plan does not effectively test the research question posed

    (Very Poor Alignment): The motivation and/or Experimental plan are completely un- related to the topic. The motivation does not address the topic, and the Experimental plan does not effectively test the research question posed

  57. [66]

    The motivation addresses the topic but in a vague or unclear manner, and the Experimental plan partially, but not effectively, tests the research question

    (Poor Alignment): The motivation and/or Experimental plan are weakly connected to the topic. The motivation addresses the topic but in a vague or unclear manner, and the Experimental plan partially, but not effectively, tests the research question

  58. [67]

    The motivation explains the topic, but with some gaps in clarity, and the Experimental plan mostly addresses the research question, though some methods may not be optimal

    (Moderate Alignment): The motivation and Experimental plan are somewhat aligned with the topic. The motivation explains the topic, but with some gaps in clarity, and the Experimental plan mostly addresses the research question, though some methods may not be optimal

  59. [68]

    The motivation explains the research problem well, and the Experimental plan is suitable and addresses the research question effectively

    (Good Alignment): The motivation and Experimental plan are clearly aligned with the topic. The motivation explains the research problem well, and the Experimental plan is suitable and addresses the research question effectively

  60. [69]

    motivation

    (Excellent Alignment): The motivation and Experimental plan are perfectly aligned with the topic. The motivation provides a strong, clear rationale for the study, and the Experimental plan is highly suitable and directly addresses the research question in a methodologically so...

  61. [70]

    Novelty: Are the problems or approaches new? Is this a novel combination of familiar techniques? Is it clear how this work differs from previous contributions? Is related work adequately referenced?

  62. [71]

    Significance: Are the ideas important? Are other people (practitioners or researchers) likely to use these ideas or build on them? Does the idea address a difficult problem in a better way than previous research? Does it provide a unique theoretical or pragmatic approach?

  63. [72]

    Quality: Is there a clear rationale for each step of the experimental design? Are the baseline and evaluation metrics chosen appropriately? Has the design taken into account the potential advantages and limitations of the methods used? Can this experimental design effectively ...

  64. [73]

    Feasibility: Can the idea be realized with existing technology or methods? Are there any technical difficulties or bottlenecks? Is the idea clear and logical? Is there any obvious error or unreasonable part in the idea, and can the experiment be designed normally according to ...

  65. [74]

    Clarity: Is the motivation and experiment clearly written? Does it provide enough information for the expert reader to understand the experiment? Is it well organized? Does it adequately inform the reader?

  66. [75]

    Novelty_choice

    Relevance to Topic: Does the idea align with the provided topic? Does it address the core themes or issues raised by the topic? How closely is the idea tied to the overall focus of the competition? Note: Avoid any position biases and ensure that the order in which the response...

  67. [2023]

    URL https://openreview.net/forum?id=g7OX2sOJtn

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.