Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding

T0 review · 5 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that LLM gene-to-function reasoning is systematically uneven: reliability drops predictably for low-attention genes, sparse evidence, multi-answer queries, and shifts in reference context, and it introduces a 540K-question

desk verdict A genuinely useful automated gene-centric benchmark, but the core hallucination metric depends on treating missing database annotations as 'no right answer' — an assumption that needs validation before the reliability-gap claim can stand. read the letter →

arxiv 2601.12805 v4 pith:EIKF5R2E submitted 2026-01-19 q-bio.GN cs.AIcs.CL

Xiaohan Huang , Meng Xiao , Chuan Qin , Qingqing Long , Jinmiao Chen , Yuanchun Zhou , Hengshu Zhu This is my paper · ORCID
classification q-bio.GNcs.AIcs.CL
keywords largelanguagemodelsbenchmarkgenefunctionhallucinationknowledgecompletenessresearchattentionontologyfunctionalgenomics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds a 540K-question benchmark covering 190K human genes to test whether large language models can reliably move from gene-level knowledge to functional understanding. The benchmark evaluates models along four behavioral axes: sensitivity to research attention, tendency to hallucinate when no grounded answer exists, completeness on multi-answer questions, and use of provided literature context. Across 27 models, the authors find a consistent reliability gap: performance degrades predictably for low-attention genes, sparse evidence, multi-answer queries, and shifts in reference context. The paper argues this makes gene-level failure modes systematic and measurable, and that current biomedical-specialised models do not consistently outperform general-purpose ones. If right, the benchmark gives a scalable, automatic way to characterise a core skill for AI-assisted cell-atlas interpretation.

What carries the argument

The benchmark's core mechanism is a gene-centric knowledge integration: each of 190K human genes is linked to structured attributes (name, synonyms, chromosome location, gene type, related proteins, expression patterns, GO terms, functional summary) and a set of PubMed references. Question cardinality determines format—single-answer, multi-answer, generative, or 'no right answer' for missing attributes. Four evaluation perspectives operationalise the paper's behavioral hypotheses: research attention is proxied by the LOC gene-symbol prefix (low PubMed counts), hallucination is measured by selection of any substantive option when the gold label is 'no right answer', completeness by macro-aver

What would settle it

Take a random sample of genes whose attributes are missing in the benchmark, manually curate the true answers from the literature, and recompute hallucination rates; if a substantial fraction of those genes are actually annotated in the literature, the reported hallucination numbers are not measuring what they claim.

Watch

Extended reading notes

Core claim

SciHorizon-GENE is a gene-centric evaluation built from NCBI Gene, Gene Ontology, and PubMed. From curated attributes it auto-generates single-answer, multi-answer, and open-ended questions, and labels attributes with no recorded value as having 'no right answer' to test hallucination. Evaluating 27 open and closed models, the paper reports substantial heterogeneity and a systematic reliability gap: accuracy falls for LOC-prefixed (low-attention) genes, models often select a substantive option when none is correct, recall lags precision on multi-answer tasks, and supplying PubMed abstracts does not consistently improve GO prediction or functional summaries and can increase hallucination. The

Load-bearing premise

The load-bearing premise is that a missing value in NCBI/GO means there is no correct answer, so 'no right answer' is a true ground truth rather than an artifact of incomplete annotation.

Editorial extensions

If this is right

  • If the reliability gap is real, model selection for gene interpretation should be guided by gene-attention profiles rather than overall biomedical QA scores.
  • Biomedical pretraining is not sufficient: specialised models may need structured gene-level supervision to close the gap on low-attention genes and multi-answer tasks.
  • Supplying abstracts as context should not be assumed to help; systems that ground outputs in controlled vocabularies need explicit mechanisms to map narrative text to GO terms.
  • The benchmark enables automatic and reproducible evaluation of gene-centric reliability without manual scoring, allowing broad model comparisons.
  • The predictability of failures suggests targeted data augmentation or prompting interventions could improve handling of under-studied genes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The hallucination metric treats a missing database value as biological absence; a manual audit of a sample of missing-attribute genes could reveal how much of the reported 'hallucination' is actually incomplete annotation.
  • Editorial inference: The LOC-prefix proxy may confound research attention with gene class (mostly uncharacterized loci); matching genes on PubMed reference counts would isolate whether attention or gene type drives the performance gap.
  • Editorial inference: The consistent recall lag on multi-answer tasks suggests a calibration or decoding issue; exploring retrieval-augmented generation or confidence-thresholded sampling for these tasks could be a testable extension.
  • Editorial inference: Because the gap is systematic, accuracy on low-attention genes could serve as a probe for pretraining corpus exposure; a simple extension is to plot accuracy against continuous PubMed counts rather than binary LOC status.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces SciHorizon-Gene, a large-scale benchmark for evaluating LLMs' gene-to-function reasoning. It constructs a gene-centric knowledge base from NCBI Gene, Gene Ontology, and PubMed, generating over 540K questions across three biological scenarios (nomenclature, genomic knowledge, and literature-influenced tasks). The benchmark defines four evaluation perspectives: research attention sensitivity, hallucination tendency, knowledge completeness, and literature influence. The authors evaluate 27 general-purpose and biomedical LLMs and report a consistent reliability gap: model performance declines for low-attention genes, sparse-evidence settings, multi-answer queries, and shifts in reference context. The paper claims the benchmark provides a principled, scalable, and automatically computable foundation for characterizing gene-level understanding in LLMs.

Significance. If the construct-validity issues are resolved, this benchmark would fill a real gap: existing biomedical LLM benchmarks mostly operate at document or clinical-questions level, not at the gene-centric granularity needed for cell-atlas interpretation and functional annotation workflows. The paper's strengths include its large scale (190K genes, 540K questions), automatic metrics inspired by CAFA, cyclic permutation to mitigate positional bias, and the breadth of models evaluated. The reported finding that biomedical-specialized models do not consistently outperform general-purpose LLMs is practically relevant. However, the benchmark's validity as a measure of hallucination and research-attention sensitivity is currently undermined by the absence-of-annotation assumption, the LOC-prefix proxy, missing human/random baselines, and lack of public data. These issues must be addressed before the central claims can be accepted.

major comments (5)
  1. [Section 3.4, Section 5.2] The hallucination metric treats missing attribute values (|a_ij|=0) as the ground truth 'no right answer' and scores any substantive output as hallucination. This equates database absence with biological absence. The paper provides no snapshot currency analysis, no manual audit of missing-value cases, and no human/random baseline. If an LLM supplies a true fact not present in the NCBI/GO snapshot, it is falsely penalized, which would invert the metric's meaning. The 'sparse-evidence reliability gap' claimed in Section 5.2 therefore rests on an unvalidated assumption. Please add an independent validation: cross-check a sample of missing attributes against current authoritative resources, and include human/random baselines for these questions.
  2. [Section 3.4, Figure 3, Table 5] Using the LOC gene-symbol prefix as the proxy for low research attention is confounded. LOC-designated genes are often uncharacterized loci and differ systematically from non-LOC genes in gene type, annotation completeness, and name meaningfulness. Table 5 shows large low-attention vs high-attention gaps (e.g., GPT-5 gene full name: 36.19 vs 99.17), but this could reflect lexical cues or annotation status rather than 'research attention.' The authors acknowledge the lexical-cue explanation in Section 5.1 but do not control for it. A matched design (e.g., non-LOC genes with similarly few PubMed references, or a continuous attention variable) is needed to separate attention from confounds.
  3. [Section 4.2, Appendix B] The 'representative subset' of 2,710 questions is asserted but not demonstrated. The paper states only that a subset was sampled to cover all scenarios and perspectives; no stratification details, randomization procedure, or per-task sample sizes are given. Table 3 reports total benchmark counts but not the counts for the sampled subset. Because all model rankings and headline claims in Tables 2, 5, and 6 are computed on this subset, the lack of a precise sampling protocol prevents reproduction and verification of the reported results. Please provide the exact sampling scheme, seed, and a composition table for the 2,710-question subset.
  4. [Appendix G, Section 1] The paper claims 'fully reproducible evaluation without manual scoring' (Section 1) and 'completely automated metrics,' but Appendix G states that the benchmark data are encrypted and 'will be released' later on the SciHorizon platform. As of submission, the core benchmark and evaluation data are not available, so no external reproducibility check is possible. Please release the benchmark data, evaluation scripts, and model outputs in a persistent archive with clear access instructions, at least upon publication.
  5. [Section 1, Figure 1, Section 3.4] There is a circularity concern: the four evaluation perspectives are motivated by the same preliminary model behaviors later reported as findings (Figure 1 and Section 3.4, then Section 5). The benchmark is thus partly shaped by the phenomena it claims to discover. This does not invalidate the benchmark as a diagnostic tool, but the paper should clarify whether the question sets and metrics were frozen before the preliminary observations, and should state this temporal ordering explicitly. An independent confirmation on held-out questions or models would strengthen the generalizability claim.
minor comments (4)
  1. [Table 3, Table 5] Typos: 'Answer Completness' should be 'Answer Completeness' in Table 3, and 'Hallucination Resistence' should be 'Hallucination Resistance' in Table 5.
  2. [Appendix D, Eq. (4)] The expression-pattern score uses alpha and beta with equal weights (0.5) but no sensitivity analysis or justification is provided. Since alpha and beta are free parameters, please report whether the main results are robust to alternative choices (e.g., alpha=0.3/0.7).
  3. [Table 6] Many entries in Table 6 are em-dashes (e.g., Gemma 3-27B, PMC-LLaMA, MedAlpaca). Please clarify whether these indicate unparseable outputs, tasks not run, or formatting failures, and specify how missing values are handled in the overall score calculations.
  4. [Appendix D, Eq. (6)] The functional summary score is an unweighted arithmetic mean of ROUGE-L, BERTScore, and fluency. ROUGE-L and BERTScore are both lexical-semantic overlap measures and may be highly correlated; consider reporting them separately or using a weighted/combined metric with rationale.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's measurements are not fitted to or derived from the outputs they report; the main caveats are construct-validity and reproducibility concerns, not circular steps.

full rationale

The paper's construction chain is external databases (NCBI Gene, GO, PubMed) to a unified gene-centric knowledge base, automated question generation, four predefined metrics, and model evaluation. No parameter is fitted to the model outputs, no gold score is used to derive the benchmark design, and no reported 'prediction' is equivalent to an input by construction. The four evaluation perspectives were motivated by pilot observations (Fig. 1, Sec. 1), so the Sec. 6 conclusion that LLMs show these failure modes is partly a confirmation of the design hypotheses; however, this is benchmark motivation, not a derivation. The missing-attribute-as-'no right answer' rule (Sec. 3.2, Sec. 3.4) is a gold-label convention; its validity depends on database absence equaling biological absence, but that is an external-validity threat, not a circular reduction. Similarly, the LOC-prefix proxy for low research attention (Sec. 3.4) is an operationalization that may be confounded, but the comparison itself is not forced by the proxy. No load-bearing self-citation chain exists; self-citations such as [9], [12], [46], [55], and [60] are contextual or background references. Limitations such as the encrypted, not-yet-released data (Appendix G) and the asserted representativeness of the 2,710-question subsample (Sec. 4.2) affect reproducibility and generalizability, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the construct validity of automatically generated gene questions: database completeness, absence-as-unknown, LOC-as-low-attention, and subset representativeness. These are domain assumptions, not established facts.

free parameters (3)
  • alpha (expression category weight) = 0.5
    Set by hand in Eq. 4 to balance exact category match and tissue F1; not fitted to data but influences the expression-pattern scores.
  • beta (tissue F1 weight) = 0.5
    Set by hand in Eq. 4, together with alpha; affects the expression-pattern scores.
  • functional summary component weights = 1/3 each
    Set by hand in Eq. 6 to average ROUGE-L, BERTScore, and fluency; affects the functional summary scores.
assumptions (5)
  • domain assumption NCBI Gene, GO, and PubMed records are authoritative and sufficiently complete for ground-truth gene knowledge.
    Used in Section 3.1 to build the gene-centric knowledge base; if records are incomplete or incorrect, gold answers are wrong.
  • domain assumption Absence of a recorded value means there is no valid answer.
    Section 3.4 defines hallucination as selecting any substantive option when no value exists; database absence may reflect incomplete curation, not biological absence.
  • domain assumption LOC-prefixed gene symbols are a valid proxy for low research attention.
    Section 3.4 compares LOC and non-LOC genes; LOC genes also differ in naming conventions and biological characterization, so the proxy may be confounded.
  • domain assumption The sampled 2,710-question subset is representative of the full 540K benchmark and all evaluation perspectives.
    Section 4.2 samples a subset for evaluation but does not describe a stratified sampling procedure.
  • domain assumption ROUGE-L, BERTScore, and CAFA-style GO scoring capture functional understanding.
    Section 4.3; lexical/semantic overlap may not equal biological correctness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding." pith.science (2026). https://pith.science/paper/EIKF5R2E

@misc{pith2026260112805,
  author       = {Pith},
  title        = {Pith review of: SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EIKF5R2E}},
  note         = {Machine review of arXiv:2601.12805}
}
read the original abstract

Large language models (LLMs) have shown growing promise in biomedical research, particularly for knowledge-driven interpretation tasks. However, their ability to reliably reason from gene-level knowledge to functional understanding, a core requirement for knowledge-enhanced cell atlas interpretation, remains largely underexplored. To address this gap, we introduce SciHorizon-GENE, a large-scale gene-centric benchmark constructed from authoritative biological databases. The benchmark integrates curated knowledge for over 190K human genes and comprises more than 540K questions covering diverse gene-to-function reasoning scenarios relevant to cell type annotation, functional interpretation, and mechanism-oriented analysis. Motivated by behavioral patterns observed in preliminary examinations, SciHorizon-GENE evaluates LLMs along four biologically critical perspectives: research attention sensitivity, hallucination tendency, answer completeness, and literature influence, explicitly targeting failure modes that limit the safe adoption of LLMs in biological interpretation pipelines. We systematically evaluate a wide range of state-of-the-art general-purpose and biomedical LLMs, revealing substantial heterogeneity in gene-level reasoning capabilities and persistent challenges in generating faithful, complete, and literature-grounded functional interpretations. Our benchmark establishes a systematic foundation for analyzing LLM behavior at the gene scale and offers insights for model selection and development, with direct relevance to knowledge-enhanced biological interpretation.

Figures

Figures reproduced from arXiv: 2601.12805 by the authors.

Figure 1
Figure 1. Observations of LLM behavior on gene-related tasks, motivating the need for our gene-centric benchmark. (a) Model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The benchmark integrates curated biological databases and verified literature sources to construct gene nodes. These [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. PubMed reference count distribution for human [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Model performance on three tasks for high- and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Completeness evaluation of LLMs. All questions [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Comparison of Gene Ontology answering perfor [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Comparison of functional summary answering per [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 10
Figure 10. Figure 10: Each example corresponds to a specific evaluation perspective. A and B indicate variants within the same genomic [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Prompt templates for all question types, including the unified system prompt and task-specific instruction prompts. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BioHarness: Substrate-Aware Evidence Assembly for Biomedical Question Answering across Literature, Knowledge Bases, and Biological Atlases

    q-bio.QM 2026-06 unverdicted novelty 6.0 of 10

    BioHarness improves pooled biomedical QA score from 65.9 to 71.0 on 19,302 items by using staged, substrate-aware evidence assembly that escalates only when needed.

Reference graph

Works this paper leans on

73 extracted references · 15 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Ajay Agrawal, John McHale, and Alexander Oettl. 2024. Artificial intelligence and scientific discovery: A model of prioritized search.Research Policy53, 5 (2024), 104989

    2024

  3. [3]

    Mistral AI. 2024. Ministral 8B. https://mistral.ai/news/ministraux

    2024

  4. [4]

    Mistral AI. 2024. Mistral Large. https://mistral.ai/news/mistral-large

    2024

  5. [5]

    Mistral AI. 2025. Mistral Medium 3.1. https://mistral.ai/news/mistral-medium-3- 1/

    2025

  6. [6]

    Mistral AI. 2025. Mistral Small 3.1. https://mistral.ai/news/mistral-small-3-1

    2025

  7. [7]

    Anthropic. 2024. Claude 3.5 Model Family. https://www.anthropic.com. Accessed: 2025-02-01

    2024

  8. [8]

    Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. 2023. Accurate medium-range global weather forecasting with 3D neural networks. Nature619, 7970 (2023), 533–538

    2023

Show all 73 references
  1. [9]

    Xunxin Cai, Chengrui Wang, Qingqing Long, Yuanchun Zhou, and Meng Xiao

  2. [10]

    Zhiyuan Cao, Vipina K Keloth, Qianqian Xie, Lingfei Qian, Yuntian Liu, Yan Wang, Rui Shi, Weipeng Zhou, Gui Yang, Jeffrey Zhang, et al. 2025. The development landscape of large language models for biomedical applications.Annual Review of Biomedical Data Science8 (2025)

    2025

  3. [11]

    Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B Singer, Xuguang Ai, Po-Ting Lai, Zhizheng Wang, et al. 2025. Bench- marking large language models for biomedical natural language processing applications and recommendations.Nature communication...

    2025

  4. [12]

    Zhijian Chen, Chuan Hu, Min Wu, Qingqing Long, Xuezhi Wang, Yuanchun Zhou, and Meng Xiao. 2024. GeneSum: Large Language Model-based Gene Summary Extraction. In2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 1438–1443

    2024

  5. [13]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic...

    2025 arXiv

  6. [14]

    ContactDoctor. 2024. ContactDoctor-Bio-Medical: A High-Performance Biomed- ical Language Model. https://huggingface.co/ContactDoctor/Bio-Medical-Llama- 3-8B

    2024

  7. [15]

    ContactDoctor. 2025. Bio-Medical-CoT: Advanced Reasoning for Healthcare Applications. https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B- CoT-012025

    2025

  8. [16]

    John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S Rosen, Gerbrand Ceder, Kristin A Persson, and Anubhav Jain. 2024. Structured information extraction from scientific text with large language models.Nature communications15, 1 (2024), 1418

    2024

  9. [17]

    Gene Ontology Consortium. [n. d.]. Gene Ontology Resource. http:// geneontology.org/. Accessed: 2025-07-29

    2025

  10. [18]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  11. [19]

    Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023. MedAlpaca–an open-source collection of medical conversational AI models and training data.arXiv preprint arXiv:2304.08247(2023)

    2023 arXiv

  12. [20]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Lan- guage Understanding. InInternational Conference on Learning Representations. https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  13. [21]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    2024 arXiv

  14. [22]

    Maria Jackson, Leah Marks, Gerhard HW May, and Joanna B Wilson. 2018. The genetic basis of disease.Essays in biochemistry62, 5 (2018), 643–723

    2018

  15. [23]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thoma...

    2023 arXiv

  16. [24]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts.arXiv preprint arXiv:2401.04088(2024)

    2024 arXiv

  17. [25]

    Yuxiang Jiang, Tal Ronnen Oron, Wyatt T Clark, Asma R Bankapur, Daniel D’Andrea, Rosalba Lepore, Christopher S Funk, Indika Kahanda, Karin M Ver- spoor, Asa Ben-Hur, et al. 2016. An expanded evaluation of protein function prediction methods shows an improvement in accuracy.Gen...

    2016

  18. [26]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences11, 14 (2021), 6421

    2021

  19. [27]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu

  20. [28]

    Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. BioASQ-QA: A manually curated corpus for Biomedical Question Answering.Scientific Data10, 1 (2023), 170

    2023

  21. [29]

    Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-antoine Gourraud, Mick- aël Rouvier, and Richard Dufour. 2024. BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains. In62th Annual Meeting of the Association for Computational Linguisti...

    2024

  22. [30]

    Tuuli Lappalainen, Yang I Li, Sohini Ramachandran, and Alexander Gusev. 2024. Genetic and molecular architecture of complex traits.Cell187, 5 (2024), 1059– 1075

    2024

  23. [31]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out. 74–81

    2004

  24. [32]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, (2024)

    2024 arXiv

  25. [33]

    Chengwu Liu, Ye Yuan, Yichun Yin, Yan Xu, Xin Xu, Zaoyu Chen, Yasheng Wang, Lifeng Shang, Qun Liu, and Ming Zhang. 2025. Safe: Enhancing Mathematical Rea- soning in Large Language Models via Retrospective Step-aware Formal Verifica- tion. InProceedings of the 63rd Annual Meeti...

    2025

  26. [34]

    Mohammad Lotfollahi, Yuhan Hao, Fabian J Theis, and Rahul Satija. 2024. The future of rapid and automated single-cell data analysis using reference mapping. Cell187, 10 (2024), 2343–2358

    2024

  27. [35]

    Jiarui Lu, Xiaoyin Chen, Stephen Zhewen Lu, Chence Shi, Hongyu Guo, Yoshua Bengio, and Jian Tang. 2025. Structure Language Models for Protein Conformation Generation. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=OzUNDnpQyd

    2025

  28. [36]

    Minghai Lu, Benjamin Delaware, and Tianyi Zhang. 2024. Proof automation with large language models. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1509–1520

    2024

  29. [37]

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. 2025. Large language model agent: A survey on methodology, applications and challenges.arXiv preprint arXiv:2503.21460(2025)

    2025 arXiv

  30. [38]

    Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining.Briefings in bioinformatics23, 6 (2022), bbac409

    2022

  31. [39]

    Pingchuan Ma, Tsun-Hsuan Wang, Minghao Guo, Zhiqing Sun, Joshua B Tenen- baum, Daniela Rus, Chuang Gan, and Wojciech Matusik. 2024. LLM and sim- ulation as bilevel optimizers: a new paradigm to advance physical scientific discovery. InProceedings of the 41st International Conf...

    2024

  32. [40]

    1999.Foundations of statistical natural language processing

    Christopher Manning and Hinrich Schutze. 1999.Foundations of statistical natural language processing. MIT press

    1999

  33. [41]

    National Center for Biotechnology Information. [n. d.]. NCBI Gene. https: //www.ncbi.nlm.nih.gov/gene. Accessed: 2025-07-29

    2025

  34. [42]

    National Library of Medicine. [n. d.]. PubMed. https://pubmed.ncbi.nlm.nih.gov/. Accessed: 2025-07-29

    2025

  35. [43]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback.Advances in neural information processing systems35 (...

    2022

  36. [44]

    Liu Pai, Wenyang Gao, Wenjie Dong, Lin Ai, Ziwei Gong, Songfang Huang, Li Zongsheng, Ehsan Hoque, Julia Hirschberg, and Yue Zhang. 2024. A survey on open information extraction from rule-based model to large language model. Findings of the association for computational linguis...

    2024

  37. [45]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. InConference on health, inference, and learning. PMLR, 248– 260. Conference acronym ’XX, June 03–05, 2018, Woo...

    2022

  38. [46]

    Chuan Qin, Xin Chen, Chengrui Wang, Pengmin Wu, Xi Chen, Yihang Cheng, Jingyi Zhao, Meng Xiao, Xiangchao Dong, Qingqing Long, et al. 2025. Scihorizon: Benchmarking ai-for-science readiness from scientific data to large language mod- els. InProceedings of the 31st ACM SIGKDD Co...

    2025

  39. [47]

    Chandan K Reddy and Parshin Shojaee. 2025. Towards scientific discovery with generative ai: Progress, opportunities, and challenges. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 28601–28609

    2025

  40. [48]

    Dmitry Scherbakov, Nina Hubig, Vinita Jansari, Alexander Bakumenko, and Leslie A Lenert. 2025. The emergence of large language models as tools in literature reviews: a large language model-assisted systematic review.Journal of the American Medical Informatics Association32, 6 ...

    2025

  41. [49]

    Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. 2025. Medgemma technical report.arXiv preprint arXiv:2507.05201(2025)

    2025 arXiv

  42. [50]

    Xinyi Shang, Xu Liao, Zhicheng Ji, and Wenpin Hou. 2025. Benchmarking large language models for genomic knowledge with GeneTuring.Briefings in Bioinformatics26, 5 (2025), bbaf492

    2025

  43. [51]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al

  44. [52]

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al . 2025. Toward expert-level medical question answering with large language models. Nature Medicine(2025), 1–8

    2025

  45. [53]

    Gemma Team, Aishwarya Kamath, Johan Ferret, and etc. 2025. Gemma 3 Techni- cal Report. arXiv:2503.19786 [cs.CL] https://arxiv.org/abs/2503.19786

    2025 arXiv

  46. [54]

    Chong Wang, Mengyao Li, Junjun He, Zhongruo Wang, Erfan Darzi, Zan Chen, Jin Ye, Tianbin Li, Yanzhou Su, Jing Ke, et al. 2025. A survey for large language models in biomedicine.Artificial Intelligence in Medicine(2025), 103268

    2025

  47. [55]

    Chengrui Wang, Qingqing Long, Meng Xiao, Xunxin Cai, Chengjun Wu, Zhen Meng, Xuezhi Wang, and Yuanchun Zhou. 2024. Biorag: A rag-llm framework for biological question reasoning.arXiv preprint arXiv:2408.01107(2024)

    2024 arXiv

  48. [56]

    Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al . 2023. Scientific discovery in the age of artificial intelligence.Nature620, 7972 (2023), 47–60

    2023

  49. [57]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. 2024. Large language models are not fair evaluators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2024

  50. [58]

    Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. 2024. PMC-LLaMA: toward building open-source language models for medicine.Journal of the American Medical Informatics Association31, 9 (2024), 1833–1843

    2024

  51. [59]

    Shican Wu, Xiao Ma, Dehui Luo, Lulu Li, Xiangcheng Shi, Xin Chang, Xiaoyun Lin, Ran Luo, Chunlei Pei, Changying Du, et al. 2025. Automated literature research and review-generation method based on large language models.National Science Review12, 6 (2025), nwaf169

    2025

  52. [60]

    Meng Xiao, Xunxin Cai, Qingqing Long, Chengrui Wang, Yuanchun Zhou, and Hengshu Zhu. 2025. Knowledge-Driven Agentic Scientific Corpus Dis- tillation Framework for Biomedical Large Language Models Training. (2025). arXiv:2504.19565 [cs.CL] https://arxiv.org/abs/2504.19565

    2025

  53. [61]

    Fengli Xu, Qianyue Hao, Chenyang Shao, Zefang Zong, Yu Li, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Jie Feng, Chen Gao, and Yong Li. 2025. Toward Large Reason...

    2025

  54. [62]

    An Yang, Baosong Yang, Beichen Zhang, and etc. 2024. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115(2024)

    2024 arXiv

  55. [63]

    Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. Sciinstruct: a self-reflective in- struction annotated dataset for training scientific language models.Advances in Neural Information Processing Systems37 (...

    2024

  56. [64]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675(2019)

    2019 arXiv

  57. [65]

    Yanbo Zhang, Sumeer A Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, et al

  58. [66]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882(2023)

    2023 arXiv

  59. [67]

    Juexiao Zhou, Haoyang Li, Siyuan Chen, Zhangtianyi Chen, Zhongyi Han, and Xin Gao. 2025. Large language models in biomedicine and healthcare.npj Artificial Intelligence1, 1 (2025), 44

    2025

  60. [68]

    Xuechao Zou, Kai Li, Junliang Xing, Yu Zhang, Shiying Wang, Lei Jin, and Pin Tao

  61. [69]

    Exploring the role of large language models in the scientific method: from hypothesis to discovery.npj Artificial Intelligence1, 1 (2025), 14

    2025

  62. [2019]

    PubMedQA: A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vince...

    2019 doi

  63. [2023]

    Large language models encode clinical knowledge.Nature620, 7972 (2023), 172–180

    2023

  64. [2024]

    DiffCR: A Fast Conditional Diffusion Framework for Cloud Removal From Optical Satellite Images.IEEE Transactions on Geoscience and Remote Sensing62 (2024), 1–14. doi:10.1109/TGRS.2024.3365806 SciHorizon-Gene: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to ...

    2024

  65. [2025]

    Knowledge hierarchy guided biological-medical dataset distillation for domain llm training.arXiv preprint arXiv:2501.15108(2025)

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.