Pith. sign in

REVIEW 5 major objections 7 minor 12 cited by

ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dedicated deep research systems already yield expert-recognizable insight on unsolved AI questions, say the authors, and a 65-question rubric benchmark puts the top two systems 20–30% ahead, with citation coverage largely decoupled from…

desk verdict ResearcherBench is a real, open-sourced benchmark with a plausible ranking, but its headline gap may be more about checklist coverage and verbosity than about research insight. read the letter →

arxiv 2507.16280 v1 pith:LH4RUVI3 submitted 2025-07-22 cs.AI

Tianze Xu , Pengrui Lu , Lyumanshan Ye , Xiangkun Hu , Pengfei Liu This is my paper
classification cs.AI
keywords deepresearchsystemsfrontierAIquestionsexpertrubricevaluationcitationfaithfulnessgroundednessopenconsultingbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ResearcherBench asks whether current deep research systems can help on genuinely unsolved AI research questions, not just retrieve and summarize known material. The paper compiles 65 frontier questions from lab discussions, researcher interviews, and scientific forums, then scores system responses with expert-designed rubrics for insight coverage and with automated checks of citation faithfulness and grounding. On this benchmark, the two leading commercial deep research systems outperform all others by 20–30% in rubric coverage, and every system is strongest on open-ended consulting questions. The authors take this as evidence that such systems are becoming genuine research partners rather than report generators, while the persistent high-faithfulness, low-groundedness pattern shows how much of their output remains unsourced. If the benchmark measures what it claims, it provides a standardized way to track progress toward AI-assisted research and recursive self-improvement.

What carries the argument

The load-bearing mechanism is the weighted rubric: each frontier question is decomposed into ten to fifteen expert-validated binary insight items, each carrying a weight from 1 to 3, and a judge model independently decides whether the response covers each item, producing a weighted coverage score. The factual arm extracts claims from each report, links them to any citing URL, and verifies whether the source text supports the claim, yielding faithfulness and groundedness scores. Together the two arms separate insight quality from citation reliability, which is what lets the benchmark argue that a system can be high-quality in insight while still poorly grounded.

What would settle it

Have a panel of human experts score every response on all 65 questions with the same rubrics used by the automated judge, then compare the system ordering; if human scores shrink or reverse the 20–30% gap between the top two systems and the rest, the automated judge rather than the systems explains the result.

Watch

Extended reading notes

Core claim

The core claim is that state-of-the-art deep research systems can already produce valuable, expert-recognizable insights on frontier AI questions, and that this capability is measurable. On ResearcherBench, the two leading systems achieve weighted rubric coverage scores of about 0.70, a 20–30% margin over the next tier, with particular strength on open consulting questions. The authors further report that all systems show high faithfulness, meaning citations, when given, usually support their claims, but low groundedness, meaning large fractions of claims carry no citation at all, with the top rubric scorer at only 0.34 groundedness. The paper concludes that frontier research assistance is dominated by synthesis and reasoning rather than source coverage, and that citation coverage is not a reliable proxy for research quality.

Load-bearing premise

The whole ranking rests on the assumption that the automated judge's yes-or-no coverage verdicts reflect real conceptual coverage, even though the judge was checked against human experts on only 10 responses and the rubric seeds were partly produced by another AI system.

Editorial extensions

If this is right

  • The leading deep research systems can already serve as idea-generation partners on open-ended research questions, where they outperform their own performance on technical details and literature review questions.
  • Frontier AI research evaluation should shift from retrieval- and report-focused benchmarks toward insight-centered rubrics that human experts design.
  • Low groundedness in high-scoring responses means that even the best current research assistants should be used with caution: their insights may be unsourced even when their citations are accurate.
  • The benchmark can support longitudinal tracking of deep research systems as new model versions appear, measuring how quickly frontier research assistance improves.
  • Dedicated research agents add measurable value over ordinary LLMs with web search, although a strong reasoner paired with search tools can close part of the gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The rubric-construction pipeline starts with an AI system producing candidate insights from source material, so the scores may partly measure alignment with AI-generated expectations rather than purely human standards; a version with fully human-authored rubrics would test this directly.
  • Because all 65 questions come from AI research, the observed open-consulting advantage may not transfer to physics, biology, or other sciences; an extension to those domains would show whether the pattern is general.
  • The 20–30% gap between the top two systems and the rest could shrink quickly as vendor systems improve, so the benchmark's long-term value depends on continuously adding fresh frontier questions.
  • There is a potential contamination risk: if frontier questions resembling these appear in training data, high coverage scores could reflect memorization rather than live reasoning, and a held-out question set created after each model cutoff would expose this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces ResearcherBench, a benchmark of 65 frontier AI research questions sourced from laboratory discussions, interviews, and scientific forums, categorized into technical details, literature review, and open consulting. Systems under test include commercial deep research systems (OpenAI Deep Research, Gemini Deep Research, Grok, Perplexity) and LLMs with web search. The proposed dual evaluation framework uses expert-designed, weighted rubrics scored by an LLM judge (o3-mini) to produce a coverage score, and a factual assessment pipeline based on GPT-4.1 that extracts claims, checks cited claims against source URLs, and computes faithfulness and groundedness scores. The main empirical claim is that OpenAI Deep Research and Gemini Deep Research significantly outperform other systems on rubric assessment, with 20-30% advantages, and that low groundedness does not correlate with research quality. The authors open-source the benchmark.

Significance. If the evaluation is valid, ResearcherBench would be a useful step beyond retrieval-and-summary benchmarks for deep research agents, because it targets open-ended questions without definitive answers and uses expert-designed rubrics rather than generic LLM-generated criteria. The paper has concrete strengths: an authentically sourced question set, human involvement in rubric construction and question filtering, a documented meta-evaluation of judge models, a dual rubric/factual framework, and a public repository. However, the central ranking rests on a coverage metric whose construct validity is not established, and the LLM judge is validated on only 10 responses. The finding that frontier research assistance may already be meaningful, and the 'high faithfulness, low groundedness' pattern, are interesting but currently supported mainly by a single checklist-style metric. The contribution is potentially significant, but the evidence base needs strengthening before the leaderboard claims can be accepted.

major comments (5)
  1. [Section 5.4, Table 3] The meta-evaluation that justifies using o3-mini as the rubric judge is conducted on only 10 responses. With n=10, the reported weighted F1 of 0.80 has a wide confidence interval, and the sample is too small to support stable estimates across 65 questions and three question types. Since every coverage score in Table 2 is produced by this judge, the entire system ranking depends on this validation. Please expand the human-annotated validation set substantially, report per-question-type agreement, and provide confidence intervals or a bootstrap analysis.
  2. [Section 4.1.2, Eq. (1); Appendix F.2] The coverage score measures only whether a response 'clearly includes or adequately expresses the main content' of each rubric item, as a binary judgment. It does not measure depth, novelty, correctness, or reasoning quality, and there is no control for response length or verbosity. A longer, enumerative report can mechanically satisfy more checklist items, so the reported 0.7032 versus 0.44-0.48 gap may reflect report style rather than research insight as claimed in the abstract. Please provide evidence that coverage scores correlate with expert holistic judgments of insight quality, and include length-controlled analyses or a non-binary human-rated validation.
  3. [Section 5.2.1, Table 2] The text states that OpenAI and Gemini 'significantly outperform' other systems, but no statistical significance tests, confidence intervals, or effect sizes are reported. With 65 questions and only one run per system on many questions, the observed differences could be within sampling noise. Please report per-question score distributions and run paired bootstrap or permutation tests across the 65 questions for each pairwise comparison.
  4. [Section 4.2, Eqs. (2)-(3); Appendix F.1] Groundedness as defined in Eq. (3) is the fraction of all extracted claims that carry a citation URL; it is a citation-coverage measure, not a semantic grounding measure. Claims without directly attached citations are counted as ungrounded even if they are supported by context or by other cited passages, and claims classified as 'unknown' are excluded from metric calculations in Appendix F.1. The 'high faithfulness, low groundedness' pattern may therefore be an artifact of the claim-extraction and citation-mapping rules. The factual assessment pipeline itself is not meta-evaluated: there is no human validation of GPT-4.1 claim extraction or claim-verification decisions. Please validate the factual pipeline on a human-annotated sample and report the rates of 'unknown' and uncited claims per system.
  5. [Appendix D, Appendix E] The evaluated systems were tested in different time windows (e.g., OpenAI Deep Research March 24-April 29; Gemini-2.5-Pro Deep Research April 15-21; Grok DeeperSearch April 18-19). Commercial systems can change between releases and even within a deployment window, so the cross-system comparison may be confounded by temporal drift. Please state the exact system versions used, run at least a subset of systems in a common short window, or explicitly analyze the sensitivity of the ranking to collection dates.
minor comments (7)
  1. [Section 5.2] The phrase 'significantly outperform' in Section 5.2.1 is not supported by any statistical test and should be replaced with descriptive wording until tests are provided.
  2. [Section 5.4, Table 3] The table lists 'o3' without defining which model variant and endpoint is used; please clarify how it differs from 'o3-mini' and why it was not selected.
  3. [Figure 3] Figure 3 appears to contain garbled character sequences rather than a readable plot; the figure needs to be regenerated and the axis labels verified.
  4. [Appendix F.1] The exclusion of 'unknown' claims from the factual metrics is described only in the appendix; it should also be stated in Section 4.2, with the per-system rates of unknown claims reported.
  5. [References] References [21] and [22] are duplicates of the same Search-R1 paper, and reference [13] lacks a venue or arXiv identifier; please clean up the bibliography.
  6. [Section 4.1.1] The rubric construction pipeline starts from Claude-3.7-Sonnet extractions; the manuscript should more explicitly quantify how much of the final rubric content originates from the model versus human experts, and report inter-annotator agreement on rubric weights.
  7. [Abstract and Section 1] There is a typo in the Introduction ('faithfulness score and and groundedness score') and the abstract's claim that the benchmark is 'the first' should be softened given the related benchmarks cited in Section 2.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ranking is an empirical measurement whose rubric and judge are externally validated, not a derivation from its own outputs.

full rationale

The paper's central claim is a comparative evaluation result, not a predicted quantity derived from fitted parameters. The construction chain is: (1) collect authentic questions; (2) use Claude-3.7-Sonnet to extract candidate insights from source materials; (3) have human experts review and transform those insights into weighted rubrics (Section 4.1.1); (4) have an LLM judge o3-mini decide binary coverage of each rubric item (Eq. 1), with the judge selected by agreement against human expert judgments on a separate 10-response sample (Section 5.4). No step fits a parameter to the evaluated DARS systems or defines the metric in terms of the target ranking. The rubric items are derived from question context, not from the responses being scored, so a high coverage score is not guaranteed by construction. The factual metrics (Eqs. 2-3) are similarly computed from independent claim extraction and source verification, with no input re-appearing as output. The only overlapping-author citation (Zheng et al., 2025, used to define DARS in Sections 1 and 2.1) is contextual and not load-bearing for any benchmark result. Concerns about construct validity (whether binary checklist coverage captures "insight quality"), the small 10-sample meta-evaluation, and possible verbosity confounds are evaluation-validity risks, not circular reasoning; honest non-finding is therefore appropriate.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The benchmark's central claims rest on human-crafted rubrics, an LLM judge, and a citation-verification pipeline, none of which are independently established beyond the small validation set. No new physical or mathematical entities are introduced.

free parameters (3)
  • Rubric item weights = 1-3 per item
    Assigned by human annotators without reported inter-annotator reliability; directly change coverage scores in Eq. 1.
  • Question selection threshold = Average score >= 4.0 on 1-5 scale
    Hand-chosen cutoff for filtering questions; affects dataset composition (Section 3.3).
  • Judge model choices = o3-mini for rubric; GPT-4.1 for factual
    Selected based on meta-evaluation on 10 responses; may not generalize (Section 5.4).
assumptions (3)
  • domain assumption Human-expert rubric items are a valid measure of 'insight quality' for frontier research questions.
    The paper rests its main evaluation on expert-designed rubrics but does not provide external validation that these rubrics capture what experts value in frontier answers (Section 4.1).
  • domain assumption LLM judge agreement with humans on 10 responses extrapolates to the full 65-question benchmark.
    Meta-evaluation sample is tiny and not stratified by question type (Section 5.4).
  • domain assumption Citation-support verification via URL text extraction is a faithful measure of factual grounding.
    Claims are extracted and verified by LLMs; 'unknown' exclusions may bias scores (Section 4.2, Appendix F.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry." pith.science (2026). https://pith.science/paper/LH4RUVI3

@misc{pith2026250716280,
  author       = {Pith},
  title        = {Pith review of: ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LH4RUVI3}},
  note         = {Machine review of arXiv:2507.16280}
}
read the original abstract

The emergence of deep research systems presents significant capabilities in problem-solving, extending from basic queries to sophisticated research tasks. However, existing benchmarks primarily evaluate these systems as agents for web retrieval and report generation, overlooking their potential to discover novel insights on the frontiers of scientific research. To address this gap, we introduce ResearcherBench, the first benchmark focused on evaluating the capabilities of these advanced, agentic systems - which we refer to as Deep AI Research Systems (DARS) - on frontier AI scientific questions. We compiled a dataset of 65 research questions expertly selected from real-world scientific scenarios such as laboratory discussions and interviews, spanning 35 different AI subjects and categorized into three types: technical details, literature review, and open consulting. Our dual evaluation framework combines rubric assessment, which uses expert-designed criteria to evaluate insight quality, with factual assessment, which measures citation accuracy (faithfulness) and coverage (groundedness). We evaluated several leading commercial DARS and baseline systems. Results show that OpenAI Deep Research and Gemini Deep Research significantly outperform other systems, with particular strength in open-ended consulting questions. Such capabilities represent a meaningful step toward AI self-improvement, aligning with the vision of ASI for AI. We open-source ResearcherBench to provide a standardized platform for promoting the development of next-generation AI research assistants, hoping to foster a new perspective in AI research evaluation for a novel pattern of scientific collaboration: https://github.com/GAIR-NLP/ResearcherBench.

Figures

Figures reproduced from arXiv: 2507.16280 by the authors.

Figure 1
Figure 1. ResearcherBench Framework Overview. The framework consists of three main components from top to bottom: (1) Dataset collection from authentic research scenarios leading to expert-generated rubrics, (2) Rubric assessment to evaluate coverage against rubrics, and (3) Factual assessment to measure faithfulness and groundedness scores. 1 * Equal contribution. 2 † Corresponding author. 1 arXiv:2507.16280v1 [cs.AI] 22 Jul… view at source ↗
Figure 2
Figure 2. AI Benchmark Topic Distribution with Representative Examples. Left Side: Pie chart showing the distribution of AI subjects in the benchmark. Right Side: Concrete question examples from major subjects. 3.1 Data Collection Strategy Our benchmark construction follows a systematic approach designed to capture authentic frontier research questions from real-world scientific scenarios. We identified three primary contexts… view at source ↗
Figure 3
Figure 3. Performance Analysis by Question Type (Rubric Assessment Coverage). Performance comparison [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

    cs.AI 2026-05 unverdicted novelty 8.0 of 10

    A new benchmark with cognitive traps shows frontier deep research agents achieve only 13-16% acceptance on expert consulting tasks under combined verifier and rubric criteria.

  2. Can AI Agents Synthesize Scientific Conclusions?

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    A new benchmark and clean-room harness show frontier AI agents reach only 0.337 factual F1 when synthesizing conclusions from scientific evidence.

  3. Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    New benchmark evaluates three frontier deep research agents on 42 SME prompts with verifiers and rubrics, reporting low acceptance rates of 9.5-21.4% and agent-specific failure modes.

  4. SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    LLMs predict outcomes of real scientific experiments at 14-26% accuracy, comparable to human experts, but lack calibration on prediction reliability while humans demonstrate strong calibration.

  5. ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    ScientistOne introduces Chain-of-Evidence and an audit system that achieves zero hallucinated references, perfect score verification, and top method-code alignment while matching or beating human experts on five front...

  6. ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents

    cs.CR 2026-05 conditional novelty 6.0 of 10

    Clarification-seeking in LLM agents amplifies prompt injection attack success from ~2% to over 30% across ten frontier models in a new 728-scenario benchmark.

  7. FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

    cs.AI 2026-02 conditional novelty 6.0 of 10

    In FIRE-Bench's rediscovery test — question-only prompt, methods withheld — the best agent averaged 46.7 F1 and no agent reached 50, with failures concentrated in research planning and conclusion formation.

  8. AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

    cs.AI 2026-01 conditional novelty 6.0 of 10

    AgencyBench is a new benchmark with 138 tasks in 32 scenarios that measures autonomous agent performance on extended real-world problems using simulated feedback and sandboxed assessment.

  9. SafeSearch: Automated Red-Teaming of LLM-Based Search Agents

    cs.AI 2025-09 conditional novelty 6.0 of 10

    An automated red-teaming framework and 300-case benchmark show that a single unreliable website can induce unsafe responses in LLM search agents, with attack success rates up to 90.5%.

  10. Characterizing Deep Research: A Benchmark and Formal Definition

    cs.CL 2025-08 conditional novelty 6.0 of 10

    Deep research is characterized by high search and reasoning intensity; the new LiveDRBench measures claim-level precision and recall, where the best current model scores 0.55 F1.

  11. Personalized Deep Research: A User-Centric Framework, Dataset, and Hybrid Evaluation for Knowledge Discovery

    cs.IR 2026-05 conditional novelty 5.0 of 10

    PDR is a user-context-aware framework for LLM research agents that improves report relevance over static baselines, supported by a new dataset and hybrid evaluation.

  12. AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery

    cs.AI 2026-05 unverdicted novelty 4.0 of 10

    A survey organizing AI-powered research automation into five workflow stages, defining AutoResearch and Vibe Research, and proposing five evaluation dimensions while noting domain-conditioned limits on autonomy.

Reference graph

Works this paper leans on

76 extracted references · 60 canonical work pages · cited by 11 Pith papers

  1. [1]

    Precise technical specifications and parameters

    work page

  2. [2]

    Detailed algorithmic descriptions and mathematical formulations

    work page

  3. [3]

    Implementation considerations and computational requirements

    work page

  4. [4]

    Performance metrics and efficiency analyses

    work page

  5. [5]

    Technical limitations and edge cases

    work page

  6. [6]

    Optimization techniques and fine-tuning procedures

    work page

  7. [7]

    Code examples or pseudocode where applicable 15 C.1 Key Insight Extraction SII-GAIR

    work page

  8. [8]

    System architecture and component interactions

    work page

Show all 76 references
  1. [9]

    Technical dependencies and environmental requirements

  2. [10]

    Debugging approaches and common technical pitfalls Literature Review Questions - Key Insight Guidelines For literature review questions, the analysis focuses on extracting structured key insights with emphasis on:

  3. [12]

    Ronald A. Fisher. 1936. The use of multiple measurements in taxonomic problems.Annals of Eugenics, 7(2):179–188

    1936

  4. [13]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey

    2024

  5. [14]

    Google. 2025. Gemini deep research - your personal research assistant. Accessed: April 14, 2025

    2025

  6. [15]

    Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jim´enez Guti´errez, Yiheng Shu, et al. 2025. Mind2web 2: Evaluating agentic search with agent-as-a- judge.arXiv preprint arXiv:2506.21506

    2025 arXiv

  7. [16]

    Mourad Gridach, Jay Nanavati, Khaldoun Zine El Abidine, Lenon Mendes, and Christina Mack. 2025. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions

    2025

  8. [17]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge

    2025

  9. [18]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948

    2025 arXiv

  10. [19]

    Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, Kun Shao, and Jun Wang. 2025. Deep research agents: A systematic examination and roadmap

    2025

  11. [20]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276

    2024 arXiv

  12. [21]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025a. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516

    arXiv

  13. [22]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025b. Search-r1: Training llms to reason and leverage search engines with reinforcement learning

  14. [23]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551

    2017 arXiv

  15. [24]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174

    1977

  16. [25]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhat- tacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025a. From generation to judgment: Opportunities and challenges of llm-as-a-judge

  17. [26]

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025b. Search-o1: Agentic search-enhanced large reasoning models.arXiv preprint arXiv:2501.05366

    arXiv

  18. [27]

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery

    2024

  19. [28]

    Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. 2025. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?arXiv preprint arXiv:2502.12115

    2025 arXiv

  20. [29]

    OpenAI. 2024. Introducing chatgpt search. Accessed: 2024-10-31

    2024

  21. [30]

    OpenAI. 2025. Deep research system card. Accessed: April 14, 2025

    2025

  22. [31]

    OpenAI. 2025. Gpt-4o search preview. OpenAI Platform Documentation

    2025

  23. [32]

    OpenAI. 2025. Introducing deep research. Accessed: April 14, 2025

    2025

  24. [33]

    Perplexity AI. 2025. Introducing perplexity deep research. Accessed: April 14, 2025

    2025

  25. [34]

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. 2025. Humanity’s last exam.arXiv preprint arXiv:2501.14249

    2025 arXiv

  26. [35]

    David M. W. Powers. 2011. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation.Journal of Machine Learning Technologies, 2(1):37–63. 11 A. Limitations and Future Works SII-GAIR

    2011

  27. [36]

    Google Research. 2025. Accelerating scientific breakthroughs with an ai co-scientist. https://research. google/blog/accelerating-scientific-breakthroughs-with-an-ai-co-scientist/

    2025

  28. [37]

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning.arXiv preprint arXiv:2503.05592

    2025 arXiv

  29. [38]

    Stanford HAI. 2025. The 2025 ai index report. Accessed: April 14, 2025

    2025

  30. [39]

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. 2025. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848

    2025 arXiv

  31. [40]

    Brian Uzzi, Satyam Mukherjee, Michael Stringer, and Ben Jones. 2013. Atypical combinations and scientific impact.Science, 342(6157):468–472

    2013

  32. [41]

    Xiaomei Wang and Xiaoyu Chen. 2024. Towards human-ai mutual learning: A new research paradigm

    2024

  33. [42]

    Ziting Wang, Haitao Yuan, Wei Dong, Gao Cong, and Feifei Li. 2024. Corag: A cost-constrained retrieval optimization system for retrieval-augmented generation.arXiv preprint arXiv:2411.00744

    2024

  34. [43]

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516

    2025 arXiv

  35. [44]

    Jackson Woodrow, Nour Nassour, John Y Kwon, Soheil Ashkani-Esfahani, and Mitchel Harris. 2025. From algorithms to academia: An endeavor to benchmark ai-generated scientific papers against human standards. Archives of Bone and Joint Surgery, 13(4):212

    2025

  36. [45]

    xAI. 2025. Grok 3 beta — the age of reasoning agents. Accessed: April 14, 2025

    2025

  37. [46]

    Renjun Xu and Jingwen Peng. 2025. A comprehensive survey of deep research: Systems, methodologies, and applications

    2025

  38. [47]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600

    2018 arXiv

  39. [48]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren’s song in the ai ocean: a survey on hallucination in large language models.arXiv preprint arXiv:2309.01219

    2023 arXiv

  40. [49]

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments.arXiv preprint arXiv:2504.03160

    2025 arXiv

  41. [50]

    Yuxiang Zheng, Shichao Sun, Lin Qiu, Dongyu Ru, Cheng Jiayang, Xuefeng Li, Jifan Lin, Binjie Wang, Yun Luo, Renjie Pan, et al. 2024. Openresearcher: Unleashing ai for accelerated scientific research.arXiv preprint arXiv:2408.06941. A Limitations and Future Works A.1 Limitation...

    2024 arXiv

  42. [51]

    Comprehensive overview of the technological landscape

  43. [52]

    Historical development and evolution of relevant technologies

  44. [53]

    Current state-of-the-art approaches and methodologies

  45. [54]

    Comparative analysis of different technical solutions

  46. [55]

    Key research papers, influential publications and bibliographic references

  47. [56]

    Emerging trends and future research directions

  48. [57]

    Major contributors and research groups in the field

  49. [58]

    Theoretical foundations and fundamental principles

  50. [59]

    Cross-disciplinary connections and applications

  51. [60]

    Benchmark datasets and evaluation frameworks commonly used in the field Open Consulting Questions - Key Insight Guidelines For open consulting questions, the analysis focuses on extracting structured key insights with emphasis on:

  52. [61]

    Provision of new insights beyond common knowledge or existing literature

  53. [62]

    In-depth analysis of the question from multiple perspectives

  54. [63]

    Critical thinking and identification of key challenges and core problems

  55. [64]

    Novel hypotheses, conceptual frameworks, or alternative viewpoints

  56. [65]

    Strategic discussions on potential research directions or practical solutions

  57. [66]

    Integration of cross-disciplinary knowledge to enrich the analysis

  58. [67]

    Reflection on the broader implications, including societal, ethical, and industrial impacts

  59. [68]

    Exploration of future trends and transformative opportunities

  60. [69]

    Expert judgment supported by logical reasoning and evidence

  61. [70]

    unknown” category in addition to “yes

    Creative and thought-provoking ideas that inspire further discussion C.1.1 Key Insight Extraction Prompt We employ the following prompt to extract key insights from the contextual source material of questions, and generate auxiliary materials as reference. Key Insight Extracti...

  62. [71]

    A claim should be a complete, standalone factual statement

  63. [72]

    Maintain the original wording where possible, but remove unnecessary context

  64. [73]

    Extract all factual claims regardless of whether they have citation support

  65. [74]

    Only consider to map citation markers (numbers, author names, etc.) to their corresponding URLs in the references section when it directly follow the claim statement

  66. [75]

    Exclude opinions, speculations, or methodological descriptions

  67. [76]

    Extract the context passage containing each claim for verification purposes

  68. [77]

    id": 1,

    If multiple claims are associated with the same citation, extract them as separate entries ## Citation URL Mapping • If URLs appear directly after claims, use those URLs directly • Citation markers (e.g. follows a number or [number]) must directly follow the claim to be consid...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.