Pith. sign in

REVIEW 3 major objections 6 minor 70 references

Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new benchmark of 1,116 expert questions from real laboratory practice shows that the strongest multimodal AI still answers fewer than half correctly, and the paper argues the bottleneck is managing visual evidence rather than accessing…

desk verdict A solid, carefully built benchmark with a genuinely new diagnostic, but the headline 48.7% needs a human-expert baseline before the interpretation lands. read the letter →

arxiv 2608.06931 v1 pith:LT5IFTO7 submitted 2026-08-07 cs.AI

Taolin Han , Yuchen Zhang , Jinghang Wang , Yun Wu , Wai Yuet Chiu , Zhaohai Li , Yifei Zhang , Jinxin Wang
show 17 more authors
Yuhao Zhou Chen Zhao Jiajia Li Jiaxin Li Qile Jin Kewei Sun Shuang Wu Weiqi Zhai Renquan Lv Junchao Li Ruodan Chen Qingteng Chen Zhibo Yang Hu Wei Lin Qu Shuai Bai Bing Zhao
This is my paper · ORCID
classification cs.AI
keywords ScienceEdgeEvaluationmultimodallargelanguagemodelsscientificbenchmarkingevidence-boundedreasoningexperimentaldatainterpretationvisual-agenttool-augmentedhallucinationdiagnosis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Science Edge Evaluation (SEE), a benchmark of 1,116 questions written by active researchers and grounded in peer-reviewed literature and real laboratory practice across chemistry, biology, and materials science. It sets out to test whether current multimodal language models can make justified, evidence-bounded inferences from experimental data rather than recalling scientific facts. On all 19 models evaluated, the best score is 48.7% and no model exceeds 50%; adding web search and a code interpreter raises the best score to 52.7% but leaves large errors intact. A sympathetic reader should care because the claim is diagnostic: the bottleneck is not knowledge access but the ability to manage multimodal evidence, which is exactly the capability AI would need to contribute new experimental insights rather than only explain established ones.

What carries the argument

The central machinery is SEE itself: a multimodal benchmark whose questions combine text with the visual evidence of real laboratory work, including spectra, microscopy, gel images, diffraction patterns, and thermal curves, and whose scoring requires exact matches to expert answers. It carries the argument by measuring whether a model stays within the evidence, and by adding two controlled perturbations: a text-only ablation that removes the image to test whether a model notices missing evidence, and a tool-augmented visual-agent protocol that adds web search and code execution to test whether more information fixes the gap. The benchmark also records discipline labels and task types so that failures can be attributed to perception, inference, or evidence-management stages.

What would settle it

Recruit a panel of expert chemists, biologists, and materials scientists, give them the same 1,116 SEE questions with the same images and strict binary scoring, and compare their accuracy with the models' 48.7%; expert accuracy at or below that level would indicate the benchmark itself is the problem, while expert accuracy near 80% or higher would confirm a genuine multimodal reasoning gap.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that no current multimodal large language model reliably performs evidence-bounded scientific reasoning. SEE's headline result is that the strongest of 19 models reaches 48.7% accuracy, with a general-purpose average of 34.9% compared with 22.2% for science-specialized models, so domain specialization does not close the gap. Removing the image from questions makes every model worse, by 12.2 points on average, yet models explicitly acknowledge the missing image in only 4.6% of 20,088 text-only instances, indicating that they prefer to answer from linguistic priors rather than recognize insufficient evidence. In the tool-augmented visual-agent setting, web search and a code interpreter raise the best accuracy to 52.7%, but trajectory analysis attributes 145 new errors to tool use, including evidence-integration and action-selection failures. The paper concludes that the central limitation is managerial: models can gather more information, but they cannot yet keep tool-derived information within the boundaries of the original experimental evidence.

Load-bearing premise

The load-bearing premise is that the expert gold answers are correct and every one of the 1,116 questions is genuinely answerable from the supplied text and images, yet the paper reports no human-expert accuracy or inter-annotator agreement, so ambiguous or unsupported gold answers would make the low model scores reflect benchmark flaws rather than a reasoning deficit.

Editorial extensions

If this is right

  • No current multimodal model can be trusted to draw conclusions from laboratory data without human oversight.
  • Science-specialized training and retrieval augmentation are not sufficient; general multimodal flexibility and evidence handling matter more.
  • Tool access helps only if the model can select actions, judge tool outputs, integrate them with the original observation, and stop when evidence is sufficient.
  • Scientific agent evaluation must score trajectory quality, including justification, uncertainty recognition, and revision, not just final-answer accuracy.
  • The low missing-image awareness rate implies a concrete risk in autonomous lab workflows: models will confidently answer from priors when data are absent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report a human-expert baseline; running the same 1,116 questions through trained scientists under identical binary scoring would calibrate whether 48.7% is a model deficit or a benchmark difficulty.
  • A direct testable extension is to allow models to abstain: if evidence-boundedness is the bottleneck, a properly calibrated model should keep high accuracy while refusing questions whose evidence is insufficient, whereas current models would need to learn when to stop.
  • Because the benchmark shows more information does not reliably improve reasoning, a follow-up could deliberately inject conflicting or irrelevant retrieved evidence and measure whether models hold the original experimental evidence as the authority.
  • The authors point toward agentic discovery; extending SEE from fixed questions to open-ended tasks where a model proposes a conclusion from a new dataset and experts judge whether each step is justified would test the transition from recognition to derivation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript introduces Science Edge Evaluation (SEE), a multimodal benchmark of 1,116 expert-curated questions in chemistry, biology, and materials science, grounded in peer-reviewed literature and real experimental practices. The authors evaluate 19 MLLMs under a strict binary scoring protocol and report that the best model reaches 48.7% accuracy, that general-purpose models outperform science-specialized ones on average, that models rarely acknowledge missing visual evidence in a text-only ablation (explicit acknowledgment rate 4.6%), and that a tool-augmented visual-agent setting improves the best accuracy to 52.7% while also introducing new error modes. The paper concludes that current MLLMs cannot reliably make justified, evidence-bounded inferences from experimental results, and that the main barrier is the management of multimodal evidence rather than knowledge access.

Significance. If the benchmark is valid, SEE would be a valuable complement to existing multimodal scientific benchmarks by focusing on experimental evidence, interdisciplinary reasoning, and evidence-bounded inference rather than exam-style knowledge recall. The evaluation pipeline has several strengths: strict binary scoring with exact-match multiple-choice and tolerance-based numerical answers; an LLM-judge sensitivity analysis reporting Cohen's kappa >= 0.99 and explicitly checking the primary judge against self-favoring bias; public-subset reproducibility tables showing the withheld 67 questions do not drive the results; and a text-only ablation in which every instance remains in the denominator. The tool-augmented visual-agent evaluation with human-in-the-loop trajectory attribution also goes beyond aggregate accuracy by diagnosing action selection, observation interpretation, evidence integration, and termination failures. However, the headline 'reliability' claim rests on the assumption that the gold answers are unambiguously correct and answerable from the provided evidence, which the paper has not yet established quantitatively.

major comments (3)
  1. [Section 3.2 / Figure 1 / Table 5] The central claim that the best model reaches only 48.7% and therefore that MLLMs 'cannot reliably make justified and evidence-bounded inferences' is uncalibrated without a human-expert baseline and inter-annotator agreement on the gold answers. Section 3.2 and Figure 1 state that questions pass 'information sufficiency' and 'answer uniqueness' checks by expert review, but the paper reports no human expert accuracy on SEE, no Cohen's or Fleiss' kappa for the gold-answer construction, and no count of questions rejected or revised during the expert QA stages. Table 5 demonstrates that the 67 withheld questions do not drive the public-subset results, but it does not validate the gold answers themselves. If human experts also score near 50% on the same evidence, or if experts disagree with the gold answers on a nontrivial fraction of items, the observed clustering of model scores could reflect benchmark ambiguity rather than a model reasoning deficit. A human baseline, and preferably per-question ambiguity or difficulty annotations, is needed to support the 'reliability' interpretation.
  2. [Section 3.1 / Supplementary B.1 / Figure 1] The dataset description (Section 3.1) says the associated visual files 'may include both visual evidence presented with the question and images used for expert verification' and that 'reported image counts are not the exact number of figures displayed in the question itself.' Supplementary B.1 states that models are given 'the image inputs retained in the evaluation payload,' but the paper never specifies which images were actually shown to the evaluated models. If the evaluation payload includes the expert-verification images, then the information-sufficiency check (Figure 1) and the text-only ablation (Section 4.4) are not well defined; if it excludes them, the released dataset metadata is misleading. This ambiguity directly affects the multimodal evidence claim, because the models' inputs and the benchmark's evidence-bounding rationale depend on exactly which images constitute the question. Please clarify the relationship between released entries and evaluation payloads, and audit the released files so that each question's required visual evidence is unambiguous and reproducible.
  3. [Section 4.2 / Figure 4] The claim that 'general-purpose models outperform science-specialized models on average' is confounded by model scale and recency: the general-purpose set includes frontier proprietary models (e.g., GPT-5.6-Sol, Gemini 3.1 Pro) while the science-specialized set includes much smaller open models (e.g., S1-VL-32B-RL) and only one model near the top tier (Intern-S2 Preview-397B). The observed gap may be an artifact of model selection rather than evidence that domain specialization does not help. The paper's further conclusion that 'simply specializing a model through post-training or augmenting it with RAG does not necessarily lead to improved performance' goes beyond what an uncontrolled comparison of these particular models can support. Please add a matched or at least scale- and recency-constrained comparison, or explicitly reframe the claim as a property of the tested model set rather than a general property of domain specialization.
minor comments (6)
  1. [Section 4.1 / Table 5] With 1,116 questions, the 95% confidence interval around the 48.7% point estimate is roughly +/- 2.9 percentage points, so the statement 'no model exceeds 50%' would be more precise if accompanied by confidence intervals or a note about sampling uncertainty.
  2. [Table 8 / Supplementary B.9] Several sub-disciplines have very small denominators (e.g., Physiology has 19 questions, Immunology 32, Genetics 36), so the per-label accuracy differences may not be meaningful; please report uncertainty intervals or aggregate these tail labels.
  3. [Section 4.4 / Supplementary B.7] The 'missing-image acknowledgment' metric depends on how the text-only prompt is phrased and how acknowledgments are classified; please provide the exact text-only prompt and a few concrete examples of responses counted as acknowledgments versus generic refusals.
  4. [Supplementary B.4] The manual inspection of the judge disagreement cases is described qualitatively but no sample size or human-verification agreement rate is given; a small human-verified set of judge decisions would further strengthen the LLM-judge validation.
  5. [Section 4.5 / Table 4] The trajectory attribution procedure reports 411 improvements and 145 regressions attributed to tool use, but no inter-annotator agreement is provided for the human-in-the-loop categorization; please report agreement or a detailed resolution protocol for the expert-validated labels.
  6. [Supplementary B.5] The tool-augmented visual-agent evaluation uses each model's own official web-search and code-interpreter implementations, so the tool environments are not standardized across models; please state whether this was the case and discuss how it affects cross-model comparisons of tool gains.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark accuracy figures are external measurements against expert-constructed gold answers, and the only self-referential element (LLM-as-judge overlap) is explicitly controlled by an independent judge check.

full rationale

The paper's central claim is an empirical benchmark result: 19 MLLMs score between 15.9% and 48.7% on SEE, with tool use raising the best score to 52.7%. No equation is fitted to any subset of these scores and then used to predict a closely related quantity. The gold answers are expert-curated from peer-reviewed literature and experimental practice; model outputs are compared against these external ground truths. The only self-referential element is that Gemini 3.1 Pro serves as the LLM judge while also being one of the evaluated models. The paper directly addresses this with an independent GPT-5.5 judge, reporting Cohen's kappa >= 0.99 and a maximum accuracy difference of 0.46 percentage points, with manual inspection indicating the primary judge is stricter rather than lenient toward itself. This is an independent robustness check, not a circular derivation. The image-necessity filtering, where questions answerable from text alone are removed or revised, is a benchmark construction choice; it does not make the low model scores equivalent to the construction. The absence of a human-expert baseline is a validity limitation, but it is not an instance of a claim reducing to its own input. Therefore no circular step is identifiable and the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark conclusions depend on four domain assumptions: gold-answer validity, LLM judge reliability, no source-paper leakage through web search, and the text-only ablation being a clean diagnostic of evidence awareness. None are malicious, but each is load-bearing for the paper's interpretation of the accuracy numbers.

assumptions (4)
  • domain assumption Each SEE gold answer is scientifically correct and the question is answerable from the provided text and images.
    The benchmark's core measurement equates model accuracy with scientific reasoning. If gold answers are wrong or ambiguous, the 48.7% result does not measure what the paper claims. No human baseline or inter-annotator agreement is reported. Invoked in Section 3.2 and Figure 1.
  • domain assumption The strict LLM-as-a-judge protocol with Gemini 3.1 Pro correctly maps model responses to ground truth.
    All accuracy numbers depend on automated judging. The paper provides a sensitivity analysis with GPT-5.5 (kappa at or above 0.99) that partially supports this, but the check covers a subset of five models and about 11,000 judgments, not all runs. Invoked in Section 3.3 and Supplementary B.4.
  • domain assumption Web search in the tool-augmented setting does not retrieve the source paper or the exact answer to the question.
    Questions carry DOI-like sources in their metadata, and models can issue web searches. The paper assumes search provides background knowledge rather than the answer, but it does not test this. Invoked in Section 4.5 and Supplementary B.5.
  • domain assumption The text-only ablation isolates the effect of missing visual evidence, so the accuracy drop and low acknowledgment rate measure evidence-boundary awareness.
    Removing images while keeping text is a reasonable diagnostic, but generic refusals and non-answers are not counted as evidence awareness, which could underestimate the true rate of boundary recognition. Invoked in Section 4.4 and Supplementary B.7.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery." pith.science (2026). https://pith.science/paper/LT5IFTO7

@misc{pith2026260806931,
  author       = {Pith},
  title        = {Pith review of: Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LT5IFTO7}},
  note         = {Machine review of arXiv:2608.06931}
}
read the original abstract

Large language models (LLMs) are increasingly involved in scientific discovery, yet it remains unclear whether they can support complex real laboratory science. Here we introduce Science Edge Evaluation (SEE), a multimodal benchmark of expert-curated questions grounded in peer-reviewed literature and experimental practice in chemistry, biology, and materials science. Evaluation of 19 multimodal large language models (MLLMs) shows that even the best-performing model reaches only 48.7% accuracy. Moreover, general-purpose models outperform science-specialized models on average. In the visual-agent evaluation, the use of tools increases the best accuracy to 52.7%. Tool use can expand the information available to models, but more information does not necessarily lead to reliable scientific reasoning. The key challenge is whether models can manage tool-derived information within the boundaries of the original experimental evidence. Together, these findings reveal that current MLLMs still cannot reliably make justified and evidence-bounded inferences from experimental results, which is an essential capability in real scientific discovery. Bridging this gap requires MLLMs to transition from explaining established scientific concepts to deriving novel and evidence-based insights from experimental data.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 31 canonical work pages

  1. [1]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. “Measuring Massive Multitask Language Understanding”. In:International Conference on Learning Representations (ICLR). 2021. arXiv:2009.03300 [cs.CY].url: https://openreview.net/forum?id=d7KBjmI3GmQ

    arXiv 2021

  2. [2]

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Un- derstanding Benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, et al. “MMLU-Pro: A More Robust and Challenging Multi-Task Language Un- derstanding Benchmark”. In:Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. 2024. arXiv:2406.01574 [cs.CL].url: https://procee dings.neurips.cc/paper...

    arXiv 2024

  3. [3]

    Bowman.GPQA: A Graduate-Level Google- Proof Q&A Benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman.GPQA: A Graduate-Level Google- Proof Q&A Benchmark. 2023. arXiv:2311.12022 [cs.AI].url: https://arxiv.org/abs /2311.12022

    arXiv 2023

  4. [4]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin.CMMLU: Measuring Massive Multitask Language Understanding in Chinese. 2023. arXiv:2306.09212 [cs.CL].url:https://arxiv.org/abs/2306.09212

    arXiv 2023

  5. [5]

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, et al.Humanity’s Last Exam. 2025. arXiv: 2501.14249 [cs.LG].url:https://arxiv.org/abs/2501.14249

    arXiv 2025

  6. [6]

    Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. “Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering”. In:Advances in Neural Information Processing Systems (NeurIPS). 2022. arXiv:2209.09513 [cs.CL].url: https://proceed ings.neurips.cc/paper_file...

    arXiv 2022

  7. [7]

    MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI”. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, pp. 9556–9567.doi: 10.110...

    arXiv 2024

  8. [8]

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig.MMMU- Pro: A More Robust Multi-Discipline Multimodal Understanding Benchmark. 2024. arXiv: 2409.02813 [cs.CL].url:https://arxiv.org/abs/2409.02813

    arXiv 2024

Show all 70 references
  1. [9]

    MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. “MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities”. In:International Conference on Machine Learning (ICML). Vol. 235. 2024, 18 pp. 57730–57754. arX...

    2024 arXiv

  2. [10]

    MathVista: Evaluating Mathe- matical Reasoning of Foundation Models in Visual Contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. “MathVista: Evaluating Mathe- matical Reasoning of Foundation Models in Visual Contexts”. In:International Conference on Learning Represe...

    2024 arXiv

  3. [11]

    Jonathan Roberts, Kai Han, Neil Houlsby, and Samuel Albanie.SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation. 2024. arXiv:2405.08807 [cs.CV].url:https://arxiv.org/abs/2405.08807

    2024 arXiv

  4. [12]

    OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. “OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scient...

    2024

  5. [13]

    MMSciBench: Bench- marking Language Models on Chinese Multimodal Scientific Problems

    Xinwu Ye, Chengfan Li, Siming Chen, Wei Wei, and Robert Tang. “MMSciBench: Bench- marking Language Models on Chinese Multimodal Scientific Problems”. In:Findings of the Association for Computational Linguistics: ACL 2025. Vienna, Austria: Association for Com- putational Lingui...

    2025 doi

  6. [14]

    MME- SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models

    Jiacheng Ruan, Dan Jiang, Xian Gao, Ting Liu, Yuzhuo Fu, and Yangyang Kang. “MME- SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models”. In:Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). Vol. 40

  7. [15]

    8760–8768.doi: 10.1609/aaai.v40i11.37829

    2026, pp. 8760–8768.doi: 10.1609/aaai.v40i11.37829. arXiv: 2508.13938 [cs.CL]. url:https://ojs.aaai.org/index.php/AAAI/article/view/37829

    2026 arXiv

  8. [16]

    Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams

    Yiming Cui, Xin Yao, Yuxuan Qin, Xin Li, Shijin Wang, and Guoping Hu. “Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams”. In:Communications Chemistry8 (2025). USNCO-V benchmark, p. 402.doi:10.1038/s42004-025-01782-x

    2025 doi

  9. [17]

    A Framework for Evaluating the Chemical Knowledge and Reasoning Abilities of Large Language Models against the Expertise of Chemists

    Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martíño Ríos-García, Benedict Emoekabu, Aswanth Krishnan, Kevin Maik Jablonka, et al. “A Framework for Evaluating the Chemical Knowledge and Reasoning Abilities of Large Language Models against the Expertise of Chemists”. In:Na...

    2025 doi

  10. [18]

    SciBench: Evaluating College- Level Scientific Problem-Solving Abilities of Large Language Models

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. “SciBench: Evaluating College- Level Scientific Problem-Solving Abilities of Large Language Models”. In:International Conference on Machine...

    2024

  11. [19]

    SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific 19 Research

    Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. “SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific 19 Research”. In:Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). 2024. url:https://...

    2024

  12. [20]

    Laurent, Joseph D

    Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Ham- merling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques.LAB-Bench: Measuring Capabilities of Language Models for Biology Research

  13. [21]

    Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Refer- ence Algorithm

    Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. “Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Refer- ence Algorithm”. In:npj Computational Materials6.1 (2020), p. 138.doi: 10.1038/s41524 -020-00406-3

    2020 doi

  14. [22]

    Probing the Limi- tations of Multimodal Language Models for Chemistry and Materials Research

    Nawaf Alampara, Mara Schilling-Wilhelmi, Martíño Ríos-García, et al. “Probing the Limi- tations of Multimodal Language Models for Chemistry and Materials Research”. In:Nature Computational Science(2025). MaCBench.doi:10.1038/s43588-025-00836-3

    2025 doi

  15. [23]

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic.Galactica: A Large Language Model for Science. 2022. arXiv:2211.09085 [cs.CL].url: https://arxiv.org/abs/2211 .09085

    2022 arXiv

  16. [24]

    Zhiyuan Huang, Baichuan Yang, Zikun He, Yanhong Wu, Hongyu Fang, Zhenhe Liu, Dong- sheng Lin, and Bing Su.ChemVTS-Bench: Evaluating Visual–Textual–Symbolic Reasoning of Multimodal Large Language Models in Chemistry. 2025. arXiv:2511.17909 [cs.AI] . url:https://arxiv.org/abs/2511.17909

    2025

  17. [25]

    Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, and Jonathan Frankle.BioMedLM: A 2.7B Parameter Language Model Trained on Biomedical Text. 2024. arXiv:2403.18421 [cs.CL].url: https://arxiv.org /abs/2403.18421

    2024 arXiv

  18. [26]

    BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

    Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. “BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining”. In:Briefings in Bioinformatics23.6 (2022), bbac409.doi:10.1093/bib/bbac409

    2022 doi

  19. [27]

    Toward Expert-Level Medical Question Answering with Large Language Models

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Amin Mohamed, Le Hou, Kevin Clark, Stephen R. Pfohl, et al. “Toward Expert-Level Medical Question Answering with Large Language Models”. In:Nature Medicine31 (2025). Med-PaLM, pp. 943–950.doi: 10.1038/s41591-0...

    2025 doi

  20. [28]

    A Study of Generative Large Language Model for Medical Research and Healthcare

    Cheng Peng, Xi Yang, Aokun Chen, Kaleb E. Smith, Nima PourNejatian, Anthony B. Costa, Cheryl Martin, Mona G. Flores, Ying Zhang, et al. “A Study of Generative Large Language Model for Medical Research and Healthcare”. In:npj Digital Medicine6 (2023). GatorTronGPT, p. 210.doi: ...

    2023 doi

  21. [29]

    Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, et al.ChemLLM: A Chemical Large Language Model. 2024. arXiv: 2402.06852 [cs.CL].url:https://arxiv.org/abs/2402.06852

    2024 arXiv

  22. [30]

    Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, et al.MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. 2023. arXiv:2311.16079 [cs.CL].url: https://arxiv.org/abs/2311.16079. 20

    2023 arXiv

  23. [31]

    Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models

    Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. “Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models”. In:International Conference on Learning Representations (ICLR). 2024. arXiv:2...

    2024 arXiv

  24. [32]

    Developing ChemDFM as a Large Language Foundation Model for Chemistry

    Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, et al. “Developing ChemDFM as a Large Language Foundation Model for Chemistry”. In: Cell Reports Physical Science6.4 (2025), p. 102523.doi:10.1016/j.xcrp.2025.102523. url: https://www.cell.com/ce...

    2025

  25. [33]

    HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science

    Yu Song, Santiago Miret, Huan Zhang, and Bang Liu. “HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science”. In:Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore: Association for Computational Linguistics, 202...

    2023 arXiv

  26. [34]

    Baker, Ziqi Chen, Xia Ning, and Huan Sun.LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset

    Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, and Huan Sun.LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset. 2024. arXiv:2402.09391 [cs.CL].url: https://arxiv.org /abs/2402.09391

    2024 arXiv

  27. [35]

    LLaVA-Med: Training a Large Language- and-Vision Assistant for Biomedicine in One Day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. “LLaVA-Med: Training a Large Language- and-Vision Assistant for Biomedicine in One Day”. In:Advances in Neural Information Processing Systems. Vol. ...

    2023

  28. [36]

    A Multimodal Large Language Model for Materials Science

    Yingheng Tang, Wenbin Xu, Jie Cao, Weilu Gao, Steve Farrell, Benjamin Erichson, Michael W. Mahoney, Andy Nonaka, and Zhi Yao. “A Multimodal Large Language Model for Materials Science”. In:Nature Machine Intelligence(2026). MatterChat.doi: 10.1038/s42 256-026-01214-y.url:https:...

    2026 doi

  29. [37]

    BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine

    Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine. 2023. arXiv:2308.09442 [cs.CV].url:https://arxiv.org/abs/2308.09442

    2023 arXiv

  30. [38]

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, et al.Towards Generalist Biomedical AI. 2023. arXiv:2307.14334 [cs.AI].url:https://arxiv.org/abs/2307.14334

    2023 arXiv

  31. [39]

    Official model page

    InternLM.Intern-S2 Preview. Official model page. 2026.url: https://huggingface.co/i nternlm/Intern-S2-Preview

    2026

  32. [40]

    S1-VL: Scientific Multimodal Reasoning Model with Thinking-with-Images

    Qingxiao Li, Lifeng Xu, QingLi Wang, Yudong Bai, Mingwei Ou, Shu Hu, and Nan Xu. S1-VL: Scientific Multimodal Reasoning Model with Thinking-with-Images. We evaluate the S1-VL-32B-RL checkpoint. 2026. arXiv:2604.21409 [cs.CV].url: https://arxiv.org/a bs/2604.21409. 21

    2026 arXiv

  33. [41]

    Shanghai AI Laboratory.Intern-S1 Pro. 2026. arXiv:2603.25040 [cs.CL].url: https: //arxiv.org/abs/2603.25040

    2026

  34. [42]

    Official model page

    InternLM.Intern-S2 Preview-397B. Official model page. 2026.url: https://huggingface .co/internlm/Intern-S2-Preview-397B

    2026

  35. [43]

    Autonomous Chemical Research with Large Language Models

    Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. “Autonomous Chemical Research with Large Language Models”. In:Nature624.7992 (2023), pp. 570–578.doi: 10.1038/s41586-023-06792-0

    2023 doi

  36. [44]

    Augmenting Large Language Models with Chemistry Tools

    Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. “Augmenting Large Language Models with Chemistry Tools”. In:Nature Machine Intelligence6.5 (2024). ChemCrow, pp. 525–535.doi:10.1038/s42256-024-00832-8

    2024 doi

  37. [45]

    Agent Laboratory: Using LLM Agents as Research Assistants

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. “Agent Laboratory: Using LLM Agents as Research Assistants”. In:Findings of the Association for Computational Linguistics: EMNLP 2025. Suzhou, C...

    2025 arXiv

  38. [46]

    The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. 2024. arXiv: 2408.06292 [cs.AI].url:https://arxiv.org/abs/2408.06292

    2024 arXiv

  39. [47]

    Holistic Evaluation of Language Models

    PercyLiang,RishiBommasani,TonyLee,DimitrisTsipras,DilaraSoylu,MichihiroYasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. H...

    2023 arXiv

  40. [48]

    Accelerating Sci- entific Discovery with Co-Scientist

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Petar Sirkovic, Artiom Myaskovsky, Grzegorz Glowaty, Felix Weissenberger, Alessio Orlandi, et al. “Accelerating Sci- entific Discovery with Co-Scientist”. In:Nature(2026).doi: 10.1038/s41586-026-10644-y. url:https://www....

    2026 doi

  41. [49]

    Official announcement

    OpenAI.GPT-5.6-Sol. Official announcement. 2026.url: https://openai.com/index/gp t-5-6/

    2026

  42. [50]

    Official model page

    Google DeepMind.Gemini 3.1 Pro. Official model page. 2026.url:https://deepmind.go ogle/models/gemini/pro/

    2026

  43. [51]

    Official announcement

    Qwen Team, Alibaba Cloud.Qwen3.8-Max. Official announcement. 2026.url:https://qw en.ai/blog?id=qwen3.8

    2026

  44. [52]

    Official announcement

    Anthropic.Claude Opus 5. Official announcement. 2026.url: https://www.anthropic.co m/news/claude-opus-5

    2026

  45. [53]

    Technical report

    Moonshot AI.Kimi K3: Open Frontier Intelligence. Technical report. 2026.url:https: //arxiv.org/abs/2607.24653

    2026

  46. [54]

    Official announcement

    OpenAI.GPT-5.5 (xhigh). Official announcement. 2026.url: https://openai.com/zh-Ha ns-CN/index/introducing-gpt-5-5/. 22

    2026

  47. [55]

    Official announcement

    Anthropic.Claude Opus 4.8 (Max). Official announcement. 2026.url:https://www.anthr opic.com/news/claude-opus-4-8

    2026

  48. [56]

    Official model page

    Google DeepMind.Gemini 3.5 Flash. Official model page. 2026.url:https://deepmind.g oogle/models/gemini/flash/

    2026

  49. [57]

    Official an- nouncement

    Qwen Team, Alibaba Cloud.Qwen3.7-Plus: Multimodal Agent Intelligence. Official an- nouncement. 2026.url: https://www.alibabacloud.com/blog/qwen3-7-plus-multimo dal-agent-intelligence_603206

    2026

  50. [58]

    Official product page

    ByteDance Seed.Seed2.1 Pro. Official product page. 2026.url:https://seed.bytedance .com/en/seed2_1

    2026

  51. [59]

    Official blog post

    MiniMax.MiniMax-M3: Frontier Coding, 1M Context, Native Multimodality. Official blog post. 2026.url:https://www.minimax.io/blog/minimax-m3

    2026

  52. [60]

    Official product page

    ByteDance Seed.Seed2.0 Pro. Official product page. 2026.url:https://seed.bytedance .com/zh/seed2

    2026

  53. [61]

    Official documentation

    Zhipu AI (Z.ai).GLM-5V-Turbo. Official documentation. 2026.url:https://docs.z.ai /guides/vlm/glm-5v-turbo

    2026

  54. [62]

    Official blog post

    Moonshot AI.Kimi K2.6: Advancing Open-Source Coding. Official blog post. 2026.url: https://www.kimi.com/blog/kimi-k2-6

    2026

  55. [63]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”. In:Advances in Neural Information Proce...

    2023 arXiv

  56. [64]

    Official model page

    LLM-Core Team, Xiaomi.MiMo-V2.5. Official model page. 2026.url:https://mimo.xia omi.com/mimo-v2-5

    2026

  57. [66]

    General-Purpose Large Language Models Outperform Specialized Clinical AI Tools on Medical Benchmarks

    Krithik Vishwanath, Anton Alyakin, Mrigayu Ghosh, Ali Hage, et al. “General-Purpose Large Language Models Outperform Specialized Clinical AI Tools on Medical Benchmarks”. In:Nature Medicine(2026).doi: 10.1038/s41591-026-04431-5.url: https://www.natu re.com/articles/s41591-026-...

    2026 doi

  58. [67]

    If the student’s answer is a full response containing an analysis process, first identify the final answer from it, and then compare it with the reference answer

  59. [68]

    5" and "5.0

    Judge whether the core meaning is consistent. For choice questions, 25 the letters must be identical. For fill-in-the-blank or short-answer questions, the numerical value or key phrase must be identical. Format differences such as "5" and "5.0" are allowed

  60. [69]

    5-10", "[0.8, 1.2]

    If the reference answer is a numerical range or contains an error tolerance, such as "5-10", "[0.8, 1.2]", "greater than 100", "5 +/- 0.5", "about 0.05", or "<3.2", the student’s answer is correct as long as the numerical value falls within the range. Range boundaries are also...

  61. [70]

    correct": true or false,

    If the student refuses to answer or no answer can be identified, mark it as incorrect. Strictly output the following JSON format, and do not include Markdown formatting: { "correct": true or false, "reason": "judgment reason within 50 characters" } For models that support mult...

  62. [2024]

    arXiv:2407.10362 [cs.AI].url:https://arxiv.org/abs/2407.10362

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.