Pith. sign in

REVIEW 3 major objections 4 minor 48 references

The paper claims that current large language models, despite mastering textbook biology, cannot reliably derive genuinely new biological findings, and offers a monthly-updated, contamination-controlled benchmark—DBench-Bio—as evidence and a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

DBench-Bio builds a dynamic biology benchmark from post-release abstracts, but LLM-generated gold answers and unverified per-model temporal separation undermine its claim to measure knowledge discovery.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A clean dynamic benchmark pipeline, but the task asks models to reproduce withheld experimental findings, so low scores don't demonstrate a lack of discovery ability. the 3 major comments →

arxiv 2603.03322 v2 pith:CQEELBWJ submitted 2026-02-10 cs.CL cs.AI

Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery

Chaoqun Yang , Xinyu Lin , Shulin Li , Wenjie Wang , Ruihan Guo , Fuli Feng , Tat-Seng Chua This is my paper
classification cs.CL cs.AI
keywords knowledge discoveryLLM evaluationdynamic benchmarkdata contaminationtemporal separationbiological knowledgescientific reasoningbenchmark automation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DBench-Bio is the paper's attempt to settle whether large language models can derive genuinely new scientific knowledge, rather than retrieve facts they have already seen. It builds a monthly-updated biology benchmark where every question comes from a paper published after the evaluated models' training cutoffs, so the answer cannot have been memorized. The paper reports that all current models score low on these questions, even though the same models do well on textbook biology, and interprets this as evidence that knowledge discovery is not a skill that emerges from scaling up pre-training data alone. A sympathetic reader would take the central claim to be: current LLMs cannot yet turn known biology into new biological findings, and a dynamic, automated benchmark is needed to measure when they can.

Core claim

DBench-Bio shows that when evaluation data are guaranteed to post-date a model's training, current large language models cannot answer questions whose answers are the actual findings of newly published biological papers. On a standard exam-style biology test the same models score near-perfectly, but on DBench-Bio their scores are low, and failure analysis reveals four recurring patterns: proposing mechanisms that are biologically plausible but wrong, substituting generic textbook mechanisms, refusing to answer, and confidently reasoning to hallucinated conclusions. The authors take this as evidence that knowledge discovery is a distinct capability that does not emerge from scaling pre-traini

What carries the argument

The load-bearing object is DBench-Bio itself, a three-stage pipeline: it selects abstracts from authoritative biology journals published after a model-release cutoff (temporal separation); an LLM converts each abstract into one hypothesis-style question plus a bulleted answer that captures the core finding; and a second LLM filters pairs on relevance, clarity, and centrality, keeping only pairs that pass strict thresholds. A judge model then scores candidate answers against the gold answer. The mechanism that makes the benchmark work is the monthly refresh: because new abstracts are ingested continuously, no static answer set can be memorized.

Load-bearing premise

The verdict assumes both that the correct answers are inferable from pre-cutoff knowledge by reasoning alone and that no evaluated model has seen the source abstracts; if either fails, low scores no longer indicate a discovery deficit.

What would settle it

Take a single DBench-Bio item whose answer is a specific mechanism—say, how a drug inhibits a cancer pathway—and query a model whose training provably excludes that paper. If the model can state the exact mechanism, the claim that the task is impossible is falsified. A second, cheaper test: run the identical pipeline on papers published before the model cutoffs and compare scores; low scores there would mean DBench-Bio measures difficulty, not novelty.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If DBench-Bio is accepted, the field gains a reusable, contamination-controlled way to measure whether any future model can actually discover new biology, rather than memorize old answers.
  • The low baseline scores imply that improving discovery ability will require new training objectives, inference-time mechanisms, or architectures—not just larger datasets.
  • The finding that tool-using and agentic models improve scores but do not close the gap suggests that retrieval and decomposition alone are insufficient; the missing component is the ability to compose known facts into a specific, novel mechanism.
  • The benchmark's pipeline is domain-agnostic, so the same method can produce monthly discovery benchmarks for other sciences, allowing cross-field comparison of discovery ability.
  • Because the benchmark is monthly, it can track progress over time and serve as a target for research on knowledge discovery.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference: the benchmark operationalizes 'discovery' as reproducing one particular published outcome, so a model that proposes a plausible, novel, but different mechanism receives a low score. True discovery might be better measured by whether a hypothesis is experimentally confirmable, not whether it matches the eventual paper.
  • My inference: a decisive control would run the same pipeline on older papers (whose answers are already known) with models whose cutoffs pre-date those papers. If scores are equally low there, DBench-Bio is measuring task difficulty, not the novelty gap; if scores are high, it is measuring genuine temporal separation.
  • My inference: the contamination guarantee is only as strong as the commitment to keep the source abstracts out of public training corpora. Once DBench-Bio is released, nothing technical prevents future models from being trained on the same papers, so the benchmark's edge will depend on continuous withholding and refresh.
  • My inference: adapting the pipeline to fields with more computational derivation (e.g., mathematical biology) would separate two failure modes—inability to reason about novel combinations versus inability to guess empirical results. This would tell us where to focus improvement efforts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes DBench-Bio, a dynamic benchmark pipeline for evaluating LLMs' ability to discover new biological knowledge. It collects JCR Q1 journal abstracts published after a claimed temporal cutoff, uses DeepSeek-V3.2-thinking to generate QA pairs from those abstracts, applies an LLM-based filter for relevance/clarity/centrality, and then evaluates candidate models with an LLM-as-a-judge protocol. The authors instantiate the pipeline with December 2025 and January 2026 snapshots across 12 biomedical sub-domains and report that current state-of-the-art models score uniformly low. They conclude that LLMs cannot derive genuinely new knowledge and that this capability cannot be acquired simply by scaling pre-training data.

Significance. If the benchmark were valid, it would provide a reusable living resource for contamination-limited evaluation and a strong negative result about LLM discovery capabilities. The automated pipeline, released code, and monthly-update design are valuable engineering contributions, and the use of an Alt-test to validate the LLM judge is a good practice. However, the central claim is not supported as stated. The gold answers are specific empirical findings from withheld papers, temporal separation is not actually implemented per model, and the gold answers themselves are unverified LLM outputs. As a result, the reported low scores are better interpreted as evidence that models cannot guess or retrieve a specific unseen experimental result, not that they lack the ability to derive new knowledge. The benchmark may be useful as a living test of 'new-finding recall' or 'abstract-grounded QA,' but it does not measure knowledge discovery in a way that supports the paper's conclusions.

major comments (3)
  1. [§3.2, §4.5, §4.1.2] The benchmark's gold answers are specific contingent empirical findings from withheld papers, making the task impossible without access to those papers. The QA extraction prompt (§A.3) instructs converting the abstract's 'core conclusion' into an answer, so the correct answer is essentially a paraphrase of the withheld result. The examples in §4.5 (BRU–P4HA2–H3K9lac/H3K18lac–HBO1–TTK; SYB-1–CYP450–PTGS2) are not entailed by any pre-existing biological knowledge. Tool access is restricted to literature before Nov 30, 2025 (§4.1.2), so even agentic models cannot retrieve the finding. Low scores are therefore the expected outcome for any honest model without the withheld paper, and the conclusion that 'knowledge discovery is a capability that cannot be sufficiently acquired simply by scaling up pre-training data' (§1, §5) does not follow. The task measures recall or guessing of a specific e
  2. [§3.1, Appendix A.2] Temporal separation is claimed per model but implemented as shared monthly snapshots. The release dates in Appendix A.2 show mid-month releases (e.g., GPT-5.2 on Dec 11, 2025; GLM-4.7 on Dec 23, 2025; Qwen3-Max thinking on Jan 26, 2026). The December snapshot includes papers from the entire month, including days before the GPT-5.2 and GLM-4.7 release dates; the January snapshot includes Jan 1–25 papers for Qwen3-Max thinking, which precede its release. No per-model abstract filtering is described. Thus the benchmark is not contamination-free for a substantial subset of evaluated models, undermining the central motivation of the work.
  3. [§3.2, §3.3, §3.4, Table 1] The Alt-test validates only the LLM judge's agreement with human experts on relevance/clarity/centrality and on scoring candidate answers; it does not validate the correctness of the gold answers. The gold answers are generated by DeepSeek-V3.2-thinking from abstracts, with no human verification sample reported. Consequently, the evaluation scores reflect agreement with an LLM-generated paraphrase of the abstract, and the use of the same model family as both answer generator and judge introduces circularity. The paper should report a human-verified subset of gold answers and, ideally, use independent judge models to support the reliability claim.
minor comments (4)
  1. [Figures 2–5] The bar/line figures report aggregate scores without error bars or confidence intervals. Because the benchmark is supposed to support comparative claims across models and sub-domains, reporting variance (e.g., bootstrap CIs) would help assess the stability of rankings.
  2. [§3.1, Table 1] No dataset statistics are provided: the total number of QA pairs, per-sub-domain counts, and the number of abstracts retained after filtering are not reported. This makes it difficult to judge coverage and potential domain imbalance.
  3. [Appendix A.2] The list of release dates does not include training-cutoff dates, which are what actually matter for contamination. Release date is only a proxy and can be misleading for models with delayed public API deployment or continuous training.
  4. [References and format] The ACM template shows 'Conference ’XX, June 03–05,2018' and '©2026' inconsistently; the venue and date placeholders should be corrected. Reference [23] is a URL-only citation without a formal access date.

Circularity Check

2 steps flagged

Gold answers are LLM paraphrases of the withheld abstracts, so low scores reflect task impossibility, not absence of discovery ability.

specific steps
  1. self definitional [§3.2 QA Extraction; §4.5 Case Study; §5.2 Conclusion]
    "We design specific prompts to instruct the model: the Question must represent a precise scientific inquiry, while the corresponding Answer must encapsulate the core conclusion of the scientific finding derived directly from the abstract. ... we provided a dynamic, contamination-free evaluation benchmark that reliably assesses complex scientific reasoning over new knowledge."

    The gold answer is, by construction, an LLM's restatement of the target abstract (§3.2). The benchmark then measures whether any model can reproduce that exact restatement without access to the paper. The required answers are contingent experimental findings—e.g., 'BRU directly binds to and promotes the degradation of P4HA2' and 'SYB-1 modulates arachidonic acid metabolism via CYP450 and PTGS2' (§4.5)—which are not entailed by prior knowledge. With tool retrieval restricted to pre-collection literature, no model can discover them. Hence low scores follow by construction from the benchmark's definition of an answer; they show only that models cannot access the withheld finding, not that knowledge discovery 'cannot be sufficiently acquired simply by scaling up pre-training data' (§1/§5.2). T

  2. other [§3.4 Evaluation Protocol]
    "Following the previous work [7, 19], we employ an 'LLM-as-a-Judge' framework to evaluate the performance of target models. The primary objective is to assess the consistency between the answers generated by the models and the ground-truth answers in our benchmark. A judge LLM is provided with the gold standard answer and the model's generated answer."

    The 'gold standard answer' is itself one LLM's paraphrase of the abstract (produced in §3.2), and the judge is another LLM. The evaluation therefore scores agreement with an LLM-authored summary, not with independently verified discovery. The Alt-test on 100 items gives partial human anchoring, so this is not a fully closed loop; nevertheless, the paper's broad conclusion that all SOTA LLMs fail at knowledge discovery is inferred from scores whose reference answers were generated by the same type of system under test.

full rationale

DBench-Bio's construction is transparent, but the load-bearing inference is circular in a specific way. In §3.2, gold answers are produced by DeepSeek-V3.2-thinking from the abstracts that are later withheld from evaluated models; in §3.4, an LLM judge scores candidate answers against these gold paraphrases. Hence the headline result—that SOTA models 'perform poorly' on discovering new knowledge—reduces to the fact that models cannot reproduce LLM restatements of post-cutoff, contingent experimental findings that are not derivable from any prior knowledge. The case-study examples (§4.5) confirm that the gold answers name specific undisclosed mechanisms (BRU→P4HA2→glycolysis→lactylation→TTK; SYB-1→CYP450/PTGS2) that no reasoner could infer. The Alt-test on 100 items provides partial external validation of the annotation, which prevents a fully closed loop, and there is no self-citation chain. But because the conclusion about fundamental inability to discover is inferred directly from scores that are forced by task construction, the circularity is substantial rather than cosmetic.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No physical entities are invented. The benchmark's load-bearing assumptions are the LLM-generated gold answers, the per-model temporal separation, and the derivability of post-cutoff findings.

free parameters (4)
  • QA filter thresholds = Relevance >= 4, Clarity >= 5, Centrality >= 5
    Hand-chosen cutoffs in §3.3; no sensitivity analysis, and they determine which QA pairs enter the benchmark.
  • Tool retrieval cutoff = Before Nov 30, 2025 for Dec 2025 snapshot
    Hand-chosen in §4.1.2; not tied to individual model release dates, so it does not implement the stated per-model temporal separation.
  • Temporal snapshot windows = Dec 2025 and Jan 2026
    Two monthly windows chosen as initial implementation (§3.1); the 'monthly-updated' claim is extrapolation.
  • QA extraction model = DeepSeek-V3.2-thinking
    Choice of extractor (§3.2) determines all gold answers; no comparison of extractors.
axioms (5)
  • domain assumption JCR Q1 'Biology & Biochemistry' abstracts are rigorous, authoritative, and representative of new biological knowledge.
    Invoked in §3.1 to justify data source; not validated within the paper.
  • domain assumption A single QA pair generated from an abstract captures the core discovery at the right level of detail.
    Used in §3.2; no evidence that one question per abstract is sufficient or that LLM-generated phrasing preserves scientific content.
  • domain assumption Findings published after a model's release date are absent from its training data.
    Basis of the contamination-free claim (§1, §3.1); treated as a guarantee though model training cutoffs are not verified and shared monthly snapshots are used for models with different release dates.
  • ad hoc to paper The specific empirical conclusion of a post-cutoff paper is derivable from general biological knowledge.
    Needed for the benchmark to measure knowledge discovery rather than recall of withheld information; contradicted by the case studies in §4.5 where correct answers contain experiment-specific details.
  • ad hoc to paper LLM-as-a-judge scores are a valid measure of answer quality for the entire dataset.
    Supported only by Alt-test on 100 QA pairs (§3.4); no human validation of all gold answers or all judge scores.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery." pith.science (2026). https://pith.science/paper/CQEELBWJ

@misc{pith2026260303322,
  author       = {Pith},
  title        = {Pith review of: Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQEELBWJ}},
  note         = {Machine review of arXiv:2603.03322}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advancements in Large Language Model (LLM) agents have demonstrated remarkable potential in automatic knowledge discovery. However, rigorously evaluating an AI's capacity for knowledge discovery remains a critical challenge. Existing benchmarks predominantly rely on static datasets, leading to inevitable data contamination where models have likely seen the evaluation knowledge during training. Furthermore, the rapid release cycles of modern LLMs render static benchmarks quickly outdated, failing to assess the ability to discover truly new knowledge. To address these limitations, we propose DBench-Bio, a dynamic and fully automated benchmark designed to evaluate AI's biological knowledge discovery ability. DBench-Bio employs a three-stage pipeline: (1) data acquisition of rigorous, authoritative paper abstracts; (2) QA extraction utilizing LLMs to synthesize scientific hypothesis questions and corresponding discovery answers; and (3) QA filter to ensure quality based on relevance, clarity, and centrality. We instantiate this pipeline to construct a monthly-updated benchmark covering 12 biomedical sub-domains. Extensive evaluations of SOTA models reveal current limitations in discovering new knowledge. Our work provides the first dynamic, automatic framework for assessing the new knowledge discovery capabilities of AI systems, establishing a living, evolving resource for AI research community to catalyze the development of knowledge discovery.

Figures

Figures reproduced from arXiv: 2603.03322 by Chaoqun Yang, Fuli Feng, Ruihan Guo, Shulin Li, Tat-Seng Chua, Wenjie Wang, Xinyu Lin.

Figure 1
Figure 1. Figure 1: The overall pipeline of DBench-Bio, which consists of three stages. (1) Data Acquisition: We source abstracts from JCR [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall results on DBench-Bio [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Results for agent-based methods on DBench-Bio. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results across different domains on DBench-Bio. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results for base models on MMLU-Pro (Biology) [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 8 linked inside Pith

  1. [1]

    Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. 2024. A Survey on RAG with LLMs.Procedia Computer Science246 (2024), 3781–3790

    2024

  2. [2]

    Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annota- tors with LLMs. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vienna, Austria, 16051–1608...

    doi:10.18653/v1/2025.acl-long.782 2025

  3. [3]

    Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, et al. 2025. Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 10091–10109

    2025

  4. [4]

    Tingting Chen, Srinivas Anumasa, Beibei Lin, Vedant Shah, Anirudh Goyal, and Dianbo Liu. 2025. Auto-Bench: An Automated Benchmark for Scientific Discovery in LLMs.arXiv preprint arXiv:2502.15224(2025)

    Pith/arXiv arXiv 2025

  5. [5]

    Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik

  6. [6]

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. 2025. Towards an AI Co-scientist.arXiv preprint arXiv:2502.18864(2025)

    Pith/arXiv arXiv 2025

  7. [7]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A Survey on LLM-as-a-judge.The Innovation(2024)

    2024

  8. [8]

    Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, et al . 2024. Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38. 18099–18107

    2024

  9. [9]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Lan- guage Understanding. InInternational Conference on Learning Representations. https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  10. [10]

    Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, Yusuf Roohani, Ryan Li, Lin Qiu, Gavin Li, Junze Zhang, et al. 2025. Biomni: A General- purpose Biomedical AI Agent.biorxiv(2025)

    2025

  11. [11]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al . 2023. C-Eval: A Multi-level Multi-discipline Chinese Evaluation Suite for Foundation Models. Advances in Neural Information Processing Systems36 (2023), 62991–63010

    2023

  12. [12]

    Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. 2024. OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI. Advances in Neural Information Processing Systems37 (2024), 19209–19253

    2024

  13. [13]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. InThe Thirteenth International Conference on Learning Repre- sentations

    2025

  14. [14]

    Hangyi Jia, Yuxi Qian, Hanwen Tong, Xinhui Wu, Lin Chen, and Feng Wei. 2025. Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, And Metric Optimization.arXiv preprint arXiv:2509.09321(2025)

    Pith/arXiv arXiv 2025

  15. [15]

    Ruofan Jin, Yucheng Guo, Yuanhao Qu, Ming Yang, Chun Shang, Qirong Yang, Linlin Chao, Yi Zhou, Ruilai Xu, Ziyao Xu, et al. 2025. BioLab: End-to-End Au- tonomous Life Sciences Research with Multi-agents System Integrating Biological Foundation Models.BioRxiv(2025), 2025–09

    2025

  16. [16]

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021. Highly Accurate Protein Structure Prediction with Al- phaFold.Nature596, 7873 (2021), 583–589

    2021

  17. [17]

    Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tetlock. 2025. ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. InThe Thirteenth International Conference on Learning Representations

    2025

  18. [18]

    Eldar Kurtic, Amir Moeini, and Dan Alistarh. 2024. Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models. InProceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing. 17020–17027

    2024

  19. [19]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. LLMs-as-judges: A Comprehensive Survey on LLM-Based Evaluation Methods.arXiv preprint arXiv:2412.05579(2024)

    Pith/arXiv arXiv 2024

  20. [20]

    Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. 2025. ResearchBench: Bench- marking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition. arXiv preprint arXiv:2503.21248(2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. InACL (Findings)

    2024

  22. [22]

    Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. GSM-Symbolic: Understanding the Limi- tations of Mathematical Reasoning in Large Language Models. InThe Thirteenth International Conference on Learning Representations

    2024

  23. [23]

    2025.Introducing GPT-5.2

    OpenAI. 2025.Introducing GPT-5.2. https://openai.com/index/introducing-gpt- 5-2/

    2025

  24. [24]

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al . 2025. Humanity’s Last Exam.arXiv preprint arXiv:2501.14249(2025)

    Pith/arXiv arXiv 2025

  25. [25]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. InFirst Conference on Language Modeling

    2024

  26. [26]

    Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K. Reddy. 2025. LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models. InForty-second International Conference on Machine Learning. https://openreview.net/forum? id=SyQPiZJVWY

    2025

  27. [27]

    Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. 2024. SciEval: A Multi-level Large Language Model Evaluation Benchmark for Scientific Research. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19053–19061

    2024

  28. [28]

    Yuwei Wan, Yixuan Liu, Aswathy Ajith, Clara Grazian, Bram Hoex, Wenjie Zhang, Chunyu Kit, Tong Xie, and Ian Foster. 2024. SciQAG: A Framework for Auto- Generated Science Question Answering Dataset with Fine-Grained Evaluation. arXiv preprint arXiv:2405.09939(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A Stickier Bench- mark for General-Purpose Language Understanding Systems.Advances in neural information processing systems32 (2019)

    2019

  30. [30]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. MMLU-Pro: A More Robust and Challenging Multi-task Language Understand- ing Benchmark.Advances in Neural Information Processing Systems37 (2024), 95266–95290

    2024

  31. [31]

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Sid- dhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, et al

  32. [32]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. ReAct: Synergizing Reasoning and Acting in Language Models. InThe eleventh international conference on learning representations

    2022

  33. [33]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9556–9567

    2024

  34. [34]

    Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, Yali Liao, Yixiao Tian, Jin- peng Wang, Zaiyuan Wang, Yang Yang, Lingyue Yin, et al. 2025. FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction.arXiv preprint arXiv:2508.11987(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Yaping Zhang, Qixuan Zhang, Xingquan Zhang, Zhiyuan Chen, Wenwen Zhuang, Yupu Liang, Lu Xiang, Yang Zhao, Jiajun Zhang, Yu Zhou, et al. 2025. HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery.arXiv preprint arXiv:2512.22899(2025). A Appendix A.1 List of 12 Sub-domains

    arXiv 2025

  36. [38]

    Biochemistry & molecular biology

  37. [39]

    Biotechnology & applied microbiology

  38. [40]

    Cell & tissue engineering

  39. [41]

    Chemistry, medicinal

  40. [42]

    Mathematical & computational biology

  41. [43]

    Pathology Conference ’XX, June 03–05,2018, Woodstock, NY Yang et al

    2018

  42. [44]

    Pharmacology & pharmacy

  43. [45]

    •GPT-5-Mini: Released on August 7, 2025

    Physiology A.2 Model Details The details of the model we use are as follows: •GPT-5: Released on August 7, 2025. •GPT-5-Mini: Released on August 7, 2025. •GPT-5.2: Released on December 11, 2025. • Gemini-3-Pro: Released on November 18, 2025. Does not support disabling the thinking mode. •Gemini-3-Flash: Released on December 18, 2025. •DeepSeek-V3.2: Relea...

    2025

  44. [46]

    The question corresponds to the main scientific question of the abstract, and the answer corresponds to the scientific discovery result

  45. [47]

    Does X regulate Y phenomenon ?

    **Focus on Scientific Discovery**: Examples: - **Causal Inference**: e.g., "Does X regulate Y phenomenon ?" or "What are the implications of Z on disease progression?" - **Mechanistic Explanation**: e.g., "How does X regulate Y phenomenon?" or "What is the mechanism behind Z effect?"

  46. [48]

    based on the text

    **Clarity and Natural Phrasing**: - Questions and answers must be self-contained, written in natural English. Avoid phrases like "based on the text" or "in this study". - Answers should be bullet-style, clear and concise. - Focus on high-level concepts, and avoid fine-grained details like specific parameters or numerical values. # Input paper abstract # O...

    2018

  47. [2024]

    Empowering Biomedical Discovery with AI Agents.Cell187, 22 (2024), 6125–6151

    2024

  48. [2025]

    In The Thirteenth International Conference on Learning Representations

    LiveBench: A Challenging, Contamination-Limited LLM Benchmark. In The Thirteenth International Conference on Learning Representations

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.