Pith. sign in

REVIEW 4 major objections 4 minor 40 references

BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces a 26,933-protocol corpus and a 556,171-instance benchmark showing that leading LLMs understand biological protocol text but systematically fail to reason about its procedures, quantities, and safety.

desk verdict A genuinely useful benchmark for biological protocol reasoning, but the ProAgent in-benchmark gains are not yet credible because the RAG index may contain the very protocols the test instances were built from. read the letter →

arxiv 2505.07889 v4 pith:EJL353UF submitted 2025-05-11 cs.CL

Yuyang Liu , Liuzhenghao Lv , Xiancheng Zhang , Jingya Wang Li Yuan , Yonghong Tian This is my paper · ORCID
classification cs.CL
keywords biologicalprotocolsproceduralreasoninglargelanguagemodelsbenchmarkdatasetretrieval-augmentedgenerationprotocollabautomationscientificAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to build the first large-scale resource for testing whether language models can handle biological experimental protocols as procedures, not just as text. It assembles BioProCorpus, 26,933 human-written protocols, and programmatically derives 556,171 instances covering question answering, step ordering, error correction, protocol generation, and explicit reasoning. On this benchmark, ten leading LLMs score well on basic comprehension and poorly wherever exact numbers, causal ordering, safety, or complete step sequences matter. The paper further claims that grounding an LLM in the corpus through retrieval (ProAgent) recovers much of this gap, raising accuracy on protocol question answering from about 70% to 85% and step recall in generated protocols from 42.8% to 62.24%. If these results hold, the corpus and benchmark give the field a standard way to measure and train procedural competence in scientific AI.

What carries the argument

The load-bearing object is BioProCorpus, a corpus of 26,933 full-text protocols collected from six public sources, cleaned by deduplication and formatting removal, and enriched with a hierarchical top-step/child-step structure plus metadata such as problem, method, innovation, and application. From this corpus the pipeline programmatically builds five tasks: Protocol Question Answering (PQA), which fills blanks for reagents, parameters, and operations; Step Ordering (ORD), which reconstructs shuffled top-level or child steps; Error Correction (ERR), which judges targeted perturbations as true or false; Protocol Generation (GEN), which reconstructs a full protocol as a step list; and Protocol Reasoning (REA), which repeats the ERR and GEN formats with explicit structured chain-of-thought. The evaluation is carried by keyword precision, recall, and F1 computed with KeyBERT using $k=64$ keywords, together with step recall and step precision computed by embedding each step and thresholding cosine similarity at $\delta = 0.7$.

What would settle it

One decisive check would be to list the BioProCorpus protocol IDs from which the held-out PQA, ORD, GEN, and REA test instances were built, then inspect the ProAgent retrieval index to see whether those exact protocols are present; if any are present, rerun the reported experiments with them removed and compare PQA accuracy, step recall, and the LAB-Bench score. If the gains vanish, the benchmark's central practical claim fails; if they persist, the corpus genuinely improves procedural reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that procedural reasoning in biology is a distinct, measurable capability that current large language models largely lack, and that a purpose-built corpus of human-authored protocols provides both a diagnostic tool and a remedy. The authors assert that all scientific facts, procedural steps, numerical values, and ground-truth answers in BioProBench are extracted programmatically and directly from the source protocols, with the LLM confined to a constrained role such as generating distractors or applying minimal perturbations. The benchmark adds domain-specific metrics beyond lexical overlap: keyword-based content metrics and embedding-based structural metrics, including step recall and step precision with a cosine-similarity threshold of $\delta = 0.7$. Evaluated across ten models, the results show high basic comprehension but systematic failures in step ordering, quantitative precision, error correction, and complete protocol generation; best step-ordering exact match is 51.80%, and best generation step recall is below 43%. The paper then shows that a retrieval-augmented agent grounded in BioProCorpus substantially raises these numbers, and that the same retrieval grounding transfers to an out-of-distribution benchmark.

Load-bearing premise

The load-bearing premise is that the retrieval index behind ProAgent does not contain the same source protocols that were used to build the held-out test questions, because the paper never states that these protocols were excluded; if they are in the index, the reported gains could be direct answer retrieval rather than procedural reasoning.

Editorial extensions

If this is right

  • ProAgent's observed gains—85.08% PQA accuracy, 81.9% ERR F1-score, 62.24% step recall in GEN, and 64% accuracy on LAB-Bench—imply that a high-fidelity protocol corpus, used as retrieval context, can directly address known LLM failure modes in scientific procedures.
  • The consistent failure pattern across ten models (best step-ordering exact match 51.80%, best generation step recall below 43%) implies that procedural reasoning is a bottleneck independent of model scale or source, so progress on this axis likely requires dedicated training resources, not just larger models.
  • The novel metrics—keyword-based content metrics and embedding-based step recall and step precision—give protocol evaluation a way to penalize omitted or fabricated steps, which lexical metrics such as BLEU fail to detect.
  • The validated LLM-as-a-judge consistency (94.21% agreement with human experts on 200 sampled instances) supports using semantic judges to evaluate reasoning chains in protocol tasks, at least when the judge's role is restricted to matching stated reasons to ground truth.
  • The transfer gain on LAB-Bench suggests that protocol knowledge captured from one set of manuals generalizes to protocol questions drawn from other sources, which is exactly what an autonomous agent in a new laboratory would need.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper never states that the source protocols for the held-out test instances were excluded from ProAgent's retrieval index, the headline gains (PQA accuracy +15 points, step recall from 42.8% to 62.24%) could in part reflect direct retrieval of the very texts the questions were built from; a rerun with those protocols removed from the index would settle whether the corpus improves proc
  • The metrics depend on specific components: the $\delta = 0.7$ cosine threshold for step matching and the KeyBERT keyword extractor, so model rankings on GEN and REA could shift if those components are replaced; this is a testable sensitivity analysis the paper does not report.
  • If the released 380,697 training instances are used to fine-tune future protocol models, those models would then be benchmarked on instances derived from the same corpus, creating a contamination risk unless the held-out set is frozen and the corpus version is tracked.
  • The authors' stated plan to add multimodal protocols implies a practical consequence: real laboratory work is visual and physical, so a text-only benchmark likely understates the gap between current models and the requirements of autonomous experimentation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. BioProBench is a corpus and multi-task benchmark for biological protocol reasoning. The authors collect human-written protocols from six public sources, structure them into a 'BioProCorpus,' and programmatically construct five task families (PQA, ORD, ERR, GEN, REA) totaling over 550,000 instances. The benchmark is evaluated with 10 LLMs, and the authors report that current models perform well on basic comprehension but struggle with quantitative precision, step ordering, error correction, and long-form protocol generation. To demonstrate the corpus's practical value, the paper introduces ProAgent, a retrieval-augmented agent grounded in BioProCorpus, and reports large in-benchmark gains (e.g., PQA accuracy 70.27% to 85.08%, GEN Step Recall 42.8% to 62.24%) as well as a smaller out-of-distribution gain on a LAB-Bench subset (60% to 64%).

Significance. If the benchmark construction is sound, this is a potentially valuable resource: it is large, spans many biology subfields, includes expert validation of the test set, uses deterministic programmatic construction for the ordering and perturbation tasks, and proposes content- and structure-aware metrics that are plausibly more informative than lexical overlap for procedural text. The explicit statement that ground-truth content is derived from human-authored protocols is a strength. However, the central empirical claim that BioProCorpus drives the large ProAgent gains is not yet supported, because the paper never states whether the source protocols behind the held-out test instances were excluded from the retrieval index. The stress-test concern lands: without that exclusion, the in-benchmark improvements may be due to answer leakage rather than improved procedural reasoning. The benchmark itself may still be a useful diagnostic, but the practical-utility claim requires a corrected experiment or explicit disclosure.

major comments (4)
  1. [5.2, Figure 7] The evaluation never states whether the ProAgent knowledge database ('Knowledge Database from BioProBench Corpus', Figure 4) excluded the source protocols from which the held-out test instances were constructed. Because every test instance is derived from a source protocol in BioProCorpus, retrieval from the full corpus can surface the exact protocol text, including the numerical answer for PQA, the correct step order for ORD, and the reference steps for GEN. The reported gains (PQA 70.27 to 85.08; GEN SR 42.8 to 62.24; ERR F1 59.27 to 81.9) are therefore not interpretable as evidence of improved procedural reasoning until this contamination route is closed. The paper should either state that test-protocol documents were removed from the index and describe how, or re-run the ProAgent experiments with such removal. The perturbation analysis in Appendix I addresses model memorization, not RAG index leakage.
  2. [Abstract and Section 2.1, Figure 1] The paper contains unresolved numerical inconsistencies: the abstract reports 22,413 protocols and 523,784 instances; Section 2.1 and the corpus table report 26,933 protocols; the Introduction and Conclusion say 'nearly 27,000' and '27,000'; and the task table sums to 556,171 instances. The repository URLs also differ between the abstract (github.com/YuyangSunshine/bioprobench and huggingface.co/BioProBench) and the body (github.com/YuyangSunshine/bioprotocolbench and huggingface.co/datasets/BioProBench/BioProBench). Please reconcile these numbers and links, since they are the first things a reader or user checks.
  3. [2.2 and Impact Statement] The paper's core principle is that 'all scientific facts, procedural steps, numerical values, and ground-truth answers are extracted programmatically and directly from the human-authored source protocols,' yet the Impact Statement acknowledges 'reliance on LLMs for task structuring, which may introduce subtle model-specific artifacts.' The reader cannot tell which parts of the released instances are LLM-generated: distractors and perturbations are described as LLM-assisted, but the enrichment fields (problem, method, innovation, application) and any chain-of-thought ground truths in REA are not specified. Please provide a per-task table of LLM involvement and confirm that no ground-truth answer is proposed or edited by an LLM, or revise the strong claim accordingly.
  4. [3, Eqs. (1)-(2)] The step metrics SR and SP depend on the cosine threshold delta = 0.7 and the keyword metrics on k = 64; these are free choices with no sensitivity results in the main text. The manuscript cites Appendix F.2 for justification, but the main text should at least summarize the threshold-sweep results, because the ProAgent GEN comparison (SR 42.8 to 62.24) is expressed in these units. Without such evidence, the claim that these metrics are robust is unverified.
minor comments (4)
  1. [6.2] 'remian limited' should be 'remain limited'.
  2. [Abstract and 5.2] The statement that ProAgent 'substantially advances the state-of-the-art' is not supported by comparisons with any RAG or agent baselines other than the underlying LLM; please either add such baselines or temper the claim.
  3. [5] The description of the held-out test set as 'approximately 1,000 instances per task' should be reconciled with the per-subtask counts in Figure 1(b), where REA-GEN and REA-ORD appear with very different counts; please clarify which subsets form the held-out test set.
  4. [Figure 1] The 'Data Sources and Raw Data Counts' table appears twice in the same figure; the duplicate should be removed.

Circularity Check

1 steps flagged · score 5.0 of 10

ProAgent's in-benchmark gains may stem from retrieving the very source protocols from which test answers were extracted; no exclusion from the RAG index is stated.

  1. fitted input called prediction [Section 2.2 (Dataset and Benchmark Construction), Section 4/Figure 4 (ProAgent), Section 5.2 (ProAgent Performance Analysis)]
    "To demonstrate the practical utility of the BioProBench corpus and to establish a strong baseline for future research, we developed ProAgent. The objective of ProAgent is not to introduce a novel agent architecture, but rather to serve as a standardized validation of our central hypothesis: that grounding LLMs in a high-fidelity, procedural knowledge corpus can directly and substantially address the critical weaknesses identified by our benchmark. ... [Figure 4:] Knowledge Database from BioProBench Corpus ... In PQA, ProAgent increases accuracy by 15 points to 85.08%. ..."

    Every BioProBench ground truth is programmatically extracted from the human-authored source protocols in BioProCorpus (Section 2.2: 'all scientific facts, procedural steps, numerical values, and ground-truth answers are extracted programmatically and directly from the human-authored source protocols'). ProAgent retrieves from a 'Knowledge Database from BioProBench Corpus', and the paper never states that the source protocols underlying the held-out test instances were excluded from that index. For PQA, the correct value appears verbatim in the source protocol; for GEN, the reference step list is the source protocol's own steps. A retriever that can surface the source document therefore answers the test item by lookup, not by procedural reasoning.

full rationale

The benchmark construction itself is not circular: instances are programmatically derived from source protocols, expert-validated, and partitioned into train and held-out test sets, and the per-task metrics are defined externally. The circularity is confined to the ProAgent validation claim. Because the test instances are defined from BioProCorpus and ProAgent's knowledge database is built from BioProCorpus, the paper must show that test-protocol sources were excluded from the retrieval index before in-benchmark gains can be attributed to corpus knowledge. The paper does not provide that exclusion statement, so the large PQA/ERR/GEN improvements may be dominated by exact-protocol retrieval rather than improved procedural reasoning. The external LAB-Bench result (60% to 64%, +4 points) provides some non-circular evidence of transfer, which is why the overall score is moderate rather than severe. There are no load-bearing self-citations or imported uniqueness theorems; the circularity, if any, is in the evaluation protocol, not in a mathematical derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on assumptions about source protocol quality, the unbiasedness of LLM-assisted task construction, expert validation sufficiency, and the validity of embedding-based semantic matching. No new physical or formal entities are introduced.

free parameters (2)
  • similarity threshold δ for Step Recall/Precision = 0.7
    Used in Eq. 1-2 to decide whether a generated step matches a reference step; chosen through sensitivity analysis on the benchmark's own data (Section 3, Appendix F.2).
  • keyword count k for KeyBERT extraction = 64
    Top-k keywords extracted from reference and generated texts for keyword metrics (Section 3). Choice appears arbitrary and affects all keyword-based scores.
assumptions (4)
  • domain assumption Protocols from the six sources are authoritative, accurate, and representative of real biological experiments.
    The corpus quality underpins every derived task; no independent verification of the source protocols is provided.
  • domain assumption LLM-generated distractors and perturbations, when programmatically constrained and expert-validated, do not introduce systematic bias into the tasks.
    PQA and ERR rely on LLM generation for negatives; the Impact Statement admits model-specific artifacts may remain.
  • domain assumption The human expert review of a 10% sample and the full test set ensures task correctness.
    Expert review is the ground truth; its reliability is asserted but not measured with inter-annotator agreement.
  • domain assumption Semantic embeddings (all-mpnet-base-v2) and cosine similarity capture procedural step equivalence at threshold 0.7.
    The structural metrics' validity depends on this model and threshold; validation is only against LLM-as-judge and a 200-instance human sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science." pith.science (2026). https://pith.science/paper/EJL353UF

@misc{pith2026250507889,
  author       = {Pith},
  title        = {Pith review of: BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJL353UF}},
  note         = {Machine review of arXiv:2505.07889}
}
read the original abstract

The realization of autonomous scientific experimentation is currently limited by LLMs' struggle to grasp the strict procedural logic and accuracy required by biological protocols. To address this fundamental challenge, we present \textbf{BioProBench}, a comprehensive resource for procedural reasoning in biology. BioProBench is grounded in \textbf{BioProCorpus}, a foundational collection of 22,413 human-written protocols. From this corpus, we systematically constructed a dataset of 523,784 task instances, offering both a large-scale training resource and a rigorous benchmark with novel metrics. Evaluating 10 mainstream LLMs, we find that while general comprehension is high, performance drops significantly on tasks demanding deep reasoning, quantitative precision, and safety awareness. To demonstrate the value of BioProCorpus in mitigating these issues, we developed \textbf{ProAgent}, grounded in our corpus, ProAgent substantially advances the state-of-the-art. https://github.com/YuyangSunshine/bioprobench and https://huggingface.co/BioProBench.

Figures

Figures reproduced from arXiv: 2505.07889 by the authors.

Figure 3
Figure 3. Representative samples for each task in the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. The architecture of ProAgent. To demonstrate the practical utility of the Bio￾ProBench corpus and to establish a strong baseline for future research, we developed ProAgent. The objective of ProAgent is not to introduce a novel agent architecture, but rather to serve as a standardized validation of our central hypothesis: that grounding LLMs in a high-fidelity, procedural knowledge cor￾pus can directly and substantia… view at source ↗
Figure 5
Figure 5. Performance Comparison on (a) the PQA Task, measured by Accuracy (Acc) and Brier [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Comprehensive Performance Comparison on GEN Task under Direct and Zero-Shot Chain ProAgent [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comprehensive Performance Comparison on ProAgent and Gemini-2.5. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 20 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Palm 2 technical report.arXiv preprint arXiv:2305.10403, 2023

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report.arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  3. [3]

    Claude 3.7 sonnet system card, Feb 2025

    Anthropic. Claude 3.7 sonnet system card, Feb 2025. Hybrid reasoning model;https://www.anthropic. com/news/claude-3-7-sonnet

    work page 2025

  4. [4]

    Cardbiomedbench: A benchmark for evaluating large language model performance in biomedical research.bioRxiv, pages 2025–01, 2025

    Owen Bianchi, Maya Willey, Chelsea X Alvarado, Benjamin Danek, Marzieh Khani, Nicole Kuznetsov, Anant Dadu, Syed Shah, Mathew J Koretsky, Mary B Makarious, et al. Cardbiomedbench: A benchmark for evaluating large language model performance in biomedical research.bioRxiv, pages 2025–01, 2025

    work page 2025

  5. [5]

    Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    2023

  6. [6]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 2020

    work page 2020

  7. [7]

    Probio: A protocol-guided multimodal dataset for molecular biology lab.Advances in Neural Information Processing Systems, 36:41543–41571, 2023

    Jieming Cui, Ziren Gong, Baoxiong Jia, Siyuan Huang, Zilong Zheng, Jianzhu Ma, and Yixin Zhu. Probio: A protocol-guided multimodal dataset for molecular biology lab.Advances in Neural Information Processing Systems, 36:41543–41571, 2023

    work page 2023

  8. [8]

    Introducing gemini 2.0: Our new ai model for the agentic era, Dec 2024

    Google AI Blog. Introducing gemini 2.0: Our new ai model for the agentic era, Dec 2024. Google DeepMind blog; https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/

    work page 2024

Show all 40 references
  1. [9]

    Introducing gemini 2.5 pro experimental (model id gemini-2.5-pro-exp-03-25)

    Google AI Blog. Introducing gemini 2.5 pro experimental (model id gemini-2.5-pro-exp-03-25). Google DeepMind Blog, April 2025. Also documented in Google Cloud Vertex AI model card: https: //cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro

    2025

  2. [10]

    Keybert: Minimal keyword extraction with bert., 2020

    Maarten Grootendorst. Keybert: Minimal keyword extraction with bert., 2020

    2020

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    Biolp-bench: Measuring understanding of biological lab protocols by large language models

    Igor Ivanov. Biolp-bench: Measuring understanding of biological lab protocols by large language models. bioRxiv, pages 2024–08, 2024

    2024

  5. [13]

    A comprehensive evaluation of large language models on benchmark biomedical text processing tasks.Computers in biology and medicine, 171:108189, 2024

    Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Xiangji Huang. A comprehensive evaluation of large language models on benchmark biomedical text processing tasks.Computers in biology and medicine, 171:108189, 2024

    2024

  6. [14]

    Pubmedqa: A dataset for biomedical research question answering.EMNLP 2019, 2019

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering.EMNLP 2019, 2019

    2019

  7. [15]

    Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022. 10

    2022

  8. [16]

    Lab-bench: Measuring capabilities of language models for biology research.arXiv preprint arXiv:2407.10362, 2024

    Jon M Laurent, Joseph D Janizek, Michael Ruzo, Michaela M Hinks, Michael J Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. Lab-bench: Measuring capabilities of language models for biology research.arXiv preprint arXiv:2407.10362, 2024

    2024 arXiv

  9. [17]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020

    2020

  10. [18]

    Can large language models reason about medical questions?Patterns, 5(3), 2024

    Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions?Patterns, 5(3), 2024

    2024

  11. [19]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  12. [20]

    Biogpt: generative pre-trained transformer for biomedical text generation and mining.Briefings in Bioinformatics, 23(6):bbac409, 2022

    Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining.Briefings in Bioinformatics, 23(6):bbac409, 2022

    2022

  13. [21]

    Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine.arXiv preprint arXiv:2308.09442, 2023

    Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine.arXiv preprint arXiv:2308.09442, 2023

    2023 arXiv

  14. [22]

    Bixbench: a comprehensive benchmark for llm-based agents in computational biology.arXiv preprint arXiv:2503.00096, 2025

    Ludovico Mitchener, Jon M Laurent, Benjamin Tenmann, Siddharth Narayanan, Geemi P Wellawatte, Andrew White, Lorenzo Sani, and Samuel G Rodriques. Bixbench: a comprehensive benchmark for llm-based agents in computational biology.arXiv preprint arXiv:2503.00096, 2025

    2025

  15. [23]

    Laboratory automation and high-throughput biology, 2024

    Tal Murthy and Jamien Lim. Laboratory automation and high-throughput biology, 2024

    2024

  16. [24]

    Can generalist foundation models outcompete special- purpose tuning? case study in medicine.arXiv preprint arXiv:2311.16452, 2023

    Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete special- purpose tuning? case study in medicine.arXiv preprint arXiv:2311.16452, 2023

    2023 arXiv

  17. [25]

    Openai announces gpt-4 turbo, 2023

    OpenAI. Openai announces gpt-4 turbo, 2023. API Documentation; https://platform.openai.com/ docs/models/gpt-4-and-gpt-4-turbo

    2023

  18. [26]

    Introducing openai o3 and o4-mini

    OpenAI. Introducing openai o3 and o4-mini. OpenAI Blog, April 2025

    2025

  19. [27]

    Bioplanner: automatic evaluation of llms on protocol planning in biology

    Odhran O’Donoghue, Aleksandar Shtedritski, John Ginger, Ralph Abboud, Ali Ghareeb, and Samuel Rodriques. Bioplanner: automatic evaluation of llms on protocol planning in biology. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 26...

    2023

  20. [28]

    Qwen2.5 -72b-instruct, 2024

    Qwen Team. Qwen2.5 -72b-instruct, 2024. Model card; https://huggingface.co/Qwen/Qwen2. 5-72B-Instruct

    2024

  21. [29]

    Qwq-32b, 2025

    Qwen Team. Qwq-32b, 2025. Model card;https://huggingface.co/Qwen/QwQ-32B

    2025

  22. [30]

    Towards scientific intelligence: A survey of llm-based scientific agents.arXiv preprint arXiv:2503.24047, 2025

    Shuo Ren, Pu Jian, Zhenjiang Ren, Chunlin Leng, Can Xie, and Jiajun Zhang. Towards scientific intelligence: A survey of llm-based scientific agents.arXiv preprint arXiv:2503.24047, 2025

    2025

  23. [31]

    Toward expert-level medical question answering with large language models.Nature Medicine, pages 1–8, 2025

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models.Nature Medicine, pages 1–8, 2025

    2025

  24. [32]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  25. [33]

    Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    1930

  26. [34]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  27. [35]

    An overview of the bioasq large-scale biomedical semantic indexing and question answering competition

    George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. An overview of the bioasq large-scale biomedical semantic indexing and...

    2015

  28. [36]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  29. [37]

    Advancing multimodal medical capabilities of gemini

    Lin Yang, Shawn Xu, Andrew Sellergren, Timo Kohlberger, Yuchen Zhou, Ira Ktena, Atilla Kiraly, Faruk Ahmed, Farhad Hormozdiari, Tiam Jaroensri, et al. Advancing multimodal medical capabilities of gemini. arXiv preprint arXiv:2405.03162, 2024

    2024 arXiv

  30. [38]

    Biobart: Pretraining and evaluation of a biomedical generative language model.arXiv preprint arXiv:2204.03905, 2022

    Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. Biobart: Pretraining and evaluation of a biomedical generative language model.arXiv preprint arXiv:2204.03905, 2022

    2022 arXiv

  31. [39]

    Benchmarking large language models on safety risks in scientific laboratories.Nature Machine Intelligence, 2026

    Yujun Zhou, Jingdong Yang, Yue Huang, Kehan Guo, Zoe Emory, Bikram Ghosh, Amita Bedar, Sujay Shekar, Zhenwen Liang, Pin-Yu Chen, et al. Benchmarking large language models on safety risks in scientific laboratories.Nature Machine Intelligence, 2026

    2026

  32. [40]

    Medxpertqa: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025. 12

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.