Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MISS-QA, the first benchmark for schematic-diagram reading in scientific papers, finds all 18 tested multimodal models trail human experts.

desk verdict A useful first benchmark for schematic-diagram QA with a plausible but methodologically under-supported human gap; accept with revisions requiring judge validation and inter-annotator agreement. read the letter →

arxiv 2507.10787 v1 pith:I4N2ZMKB submitted 2025-07-14 cs.CL cs.CV

Yilun Zhao , Chengye Wang , Chuhan Li , Arman Cohan This is my paper
classification cs.CLcs.CV
keywords MISS-QAschematicdiagramunderstandingmultimodalquestionansweringscientificliteratureinformation-seekingQAunanswerablequestionsLLM-as-a-judgefoundationmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MISS-QA, the first benchmark designed to test whether multimodal foundation models can interpret schematic diagrams in scientific papers. Each of the 1,500 expert-annotated questions points to a highlighted module in an overview figure and requires the model to find the supporting discussion in the paper text; 26.5% of questions are unanswerable by design. Evaluating 18 models, the authors find a wide gap between machines and people: human experts score 89.0%, the best proprietary model scores 78.3%, and the best open-source model scores 61.6%. The result matters because scientists commonly skim a paper by reading its schematic overview, so reliable diagram-grounded answering would be a practical capability for literature assistants, and this benchmark makes that capability measurable for the first time.

What carries the argument

The central object is the benchmark task itself: a schematic diagram with one or two colored bounding boxes highlighting the module or process in question, paired with the paper's main text, a short caption, and an information-seeking question that refers to 'the module highlighted by [color] bounding box.' The colored boxes force the model to resolve the figure visually before it can search the text, blocking the text-only shortcut found in earlier multimodal benchmarks. The evaluation machinery is an LLM-as-judge protocol using GPT-4.1 to assign 0, 0.5, or 1 accuracy scores to free-form answers, applied to both models and human experts with a chain-of-thought prompt adapted from M3SciQA. The unanswerable questions act as a control condition for hallucination and overconfidence.

What would settle it

Have two or more independent human experts score a sample of model outputs from MISS-QA that spans low, medium, and high GPT-4.1 scores, then compare their scores with the judge's; if agreement is low (say Cohen's kappa below 0.7) or the judge systematically favors one model family, the reported 89.0% versus 61.6% gap and the model ranking would not survive.

Watch

Extended reading notes

Core claim

The paper claims that schematic-diagram interpretation—reading the overview figure of a scientific paper and grounding an answer in the paper's text—is a distinct multimodal capability that existing chart-focused benchmarks do not measure. To test it, the authors build MISS-QA, a benchmark of 1,500 expert-annotated question-answer pairs over 465 recent AI papers, with each question tied to a highlighted region of the diagram and one of five information-seeking scenarios: design rationale, implementation details, literature background, experimental results, and others. On this benchmark the authors report a human-expert accuracy of 89.0%, a best proprietary model (o4-mini) at 78.3%, and a best open-source model (Qwen2.5-VL-72B) at 61.6%, with the gap driven partly by overconfidence on the 26.5% of questions that are deliberately unanswerable. An error analysis of 100 cases attributes most failures to difficulty interpreting and contextualizing the schematic diagram and to failure to retrieve the relevant paper context, rather than to pure visual perception.

Load-bearing premise

The reported gap rests on the assumption that the automated GPT-4.1 judge grades free-form answers the way human experts would, since the paper does not report judge-validation against human ratings or inter-annotator agreement, and the human-expert score is based on only two PhD candidates solving 50 examples.

Editorial extensions

If this is right

  • If MISS-QA measures what it claims, no evaluated multimodal model is currently reliable enough to answer information-seeking questions about a paper's overview figure: even the best proprietary model sits about 11 points below human experts, and the best open-source model about 27 points below.
  • Most models fail to say 'I do not know' on unanswerable questions, so deploying them in scientific literature search without refusal calibration would produce confident hallucinations rather than useful uncertainty.
  • Within-family gains (Qwen2.5-VL-72B ahead of Qwen2-VL-72B by 7.4 points; InternVL3-38B ahead of InternVL2.5-38B by 8.9 points) show open-source models are improving on this task, but the improvement is not yet close to closing the human-model gap.
  • The dominant error types point to joint diagram-text grounding as the bottleneck: models fail to interpret and contextualize the highlighted diagram region, or fail to retrieve the paper section that explains it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not run: applying the same highlighted-region protocol to papers outside AI (biology, medicine, engineering) would test whether the measured gap generalizes or is specific to the annotation style and figure conventions of AI papers.
  • A testable design implication is that fine-tuning or prompting models to explicitly produce 'I do not know' on unanswerable items could close part of the gap; the benchmark's confusion matrices provide the exact failure counts needed to estimate the ceiling of such an intervention.
  • Because the paper corpus was released after most open-source pretraining cutoffs, the gap is unlikely to be memorization; a contamination probe that checks whether models recognize paper titles, captions, or section text would turn that inference into a measured fact.
  • If the gap survives human-judge re-scoring, a practical near-term consequence is that literature-search assistants should attach confidence or source-snippet evidence to any answer grounded in a schematic figure, and fall back to text-only retrieval when the question's target region is unclear.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MISS-QA, a benchmark of 1,500 expert-annotated question-answering examples over 465 AI-related arXiv papers, in which models must interpret a schematic diagram, locate relevant information in the paper text, and produce a free-form answer. The authors report a three-stage annotation pipeline with question, answer, and validation annotators, and they evaluate 18 proprietary and open-source multimodal foundation models plus a two-person human expert baseline. Their headline finding is a large gap between human experts (89.0% accuracy) and the best open-source model (Qwen2.5-VL-72B, 61.6% on the test set), with o4-mini the best proprietary model at 78.3%. The paper also analyzes performance on deliberately unanswerable questions and provides an error taxonomy based on 100 sampled errors.

Significance. If the evaluation is trustworthy, MISS-QA fills a clear gap: existing scientific QA benchmarks focus on text-only contexts or on charts and tables, whereas schematic overview diagrams are a distinct and practically important visual modality. The benchmark construction is unusually careful for this type of resource: questions are written by domain experts, colored bounding boxes force models to use the figure rather than textual shortcuts, evidence is aligned by two annotators, a third annotator validates each example, and the source papers are chosen after the training cutoffs of many open-source models. The release of data and code is a concrete strength, and the unanswerable-question design is a useful stress test for hallucination. However, the measured human-model gap and the model rankings currently rest on an unvalidated GPT-4.1 judge and a very small human sample, so the benchmark's quantitative conclusions are not yet calibrated; the resource itself remains valuable regardless of the precise numbers.

major comments (4)
  1. [Section 4.2 / Table 4] The headline gap mixes evaluation splits. The human expert score of 89.0% is measured on 50 examples drawn from testmini (§4.1), while the model scores quoted in §4.2 (o4-mini 78.3, Qwen2.5-VL-72B 61.6) are test-set numbers. A single table therefore cannot be read as comparing human and model performance on the same benchmark split. Please report human accuracy on the test split, or compare all systems on testmini in the same table, and provide a confidence interval for the human score; with 50 examples and 89.0% accuracy, the 95% CI is roughly ±8.8 points.
  2. [Section 4.1, Automated Evaluation System] The GPT-4.1 judge is the sole scoring mechanism for both model outputs and the human expert responses, yet no validation is reported that the judge's 0/0.5/1 scores agree with human ratings on MISS-QA's free-form answers. This is especially important because GPT-4.1 is itself one of the evaluated models (Table 4), so a systematic bias toward or against particular answer styles or model families could shift both the absolute scores and the ranking. Please add a judge-validation study: human-grade a stratified sample of model outputs, including unanswerable cases, report judge-human agreement (e.g., Cohen's kappa or exact agreement), and report whether agreement varies by model family or by answerable versus unanswerable questions.
  3. [Section 4.1, Human Expert Performance Measurement] The 89.0% human figure rests on two PhD candidates solving 50 examples, but the paper reports no inter-annotator agreement, no per-annotator breakdown, and no statement about whether the candidates were independent of the 16-person benchmark construction team. Without this information it is unclear whether the human score reflects expert reading of papers or familiarity with the annotation protocol and the expected answer style. Please report inter-annotator agreement and per-annotator scores, clarify independence from the benchmark construction process, and ideally enlarge the human sample with results stratified by the five information-seeking subsets.
  4. [Section 4.2 / Table 4 ranking differences] Several ranking statements are based on test-set differences that may be within sampling noise, such as GPT-4.1 at 77.8% versus o4-mini at 78.3%, and Qwen2.5-VL-72B at 61.6% versus InternVL3-38B at 60.4%. With 1,000 test examples, differences of one to two percentage points are not shown to be statistically meaningful. Please provide confidence intervals or significance tests for the main model comparisons, or soften the claims that are not statistically supported.
minor comments (5)
  1. [Table 4] The column heading 'Unaswerable' is misspelled and should read 'Unanswerable'.
  2. [Table 3] The sentence 'MISS-QA include 1,500 expert-annotated examples' has a subject-verb agreement error and should read 'MISS-QA includes...'.
  3. [Appendix C, Table 6] The caption says the 'Source' column includes URLs and Hugging Face model names, but the rendered table has no 'Source' column and no URLs; either add the column or correct the caption.
  4. [Section 4.1] The paper states that human responses were evaluated using the authors' automated evaluation protocol, but only the QA prompt is shown in Appendix Figure 9; the judge prompt itself should be included in the appendix or released with the code so that the grading procedure is fully reproducible.
  5. [Limitations] The Limitations section candidly acknowledges that most evaluated models do not disclose pretraining data and that contamination cannot be ruled out; this caveat is important enough that it should also appear next to the leaderboard table rather than only in the Limitations section.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: MISS-QA is an external benchmark and the reported model-human gap is a held-out measurement, not a fitted or derived quantity; the unvalidated GPT-4.1 judge shared by humans and models is a validity caveat, not a circular reduction.

full rationale

MISS-QA is an empirical evaluation, not a derivation. The central claims—that the benchmark exists, that 18 models score as reported, and that human experts score 89.0% versus 61.6% for the best open-source model (Table 4)—are supported by held-out accuracy computed against expert-annotated ground truth. No equation in the paper defines a target quantity in terms of an input quantity, and no parameter is fitted to force the performance gap. The only author-overlap citation that affects the experimental setup is M3SciQA (Li et al., 2024a), used as the source of the CoT prompt and the LLM-as-judge convention; that citation is methodological precedent, not a load-bearing justification of the benchmark's conclusions. Section 4.1 does state that human responses 'were evaluated using our developed automated evaluation protocol,' i.e., the same GPT-4.1 judge used for models, and no human-agreement validation or inter-annotator statistics are reported; this is a measurement-validity limitation that could change the precise gap, but it does not make the gap follow from the benchmark's construction. The paper's own Limitations section also notes that model pretraining-data transparency prevents definitive contamination analysis, which is a stated limitation, not a circular step. Under the hard rules, no quoted reduction (Eq. X = Eq. Y by construction, or fitted parameter renamed as prediction) is present, so no circularity step is recorded.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central measurement rests on assumptions about annotation reliability, judge validity, and question design. None of these are numeric free parameters, but they are domain assumptions that, if false, would change the reported gap.

assumptions (3)
  • domain assumption The 50-example human-expert evaluation approximates true expert performance on the full benchmark.
    Section 4.1: only two PhD candidates solved 50 examples, and their average accuracy (89.0%) is used as the human baseline.
  • domain assumption GPT-4.1 as judge produces accuracy labels that agree with human judgment without systematic bias.
    Section 4.1: the automated evaluation system relies exclusively on GPT-4.1 with scores of 0, 0.5, or 1; no validation of judge agreement with human raters is reported.
  • domain assumption Colored bounding boxes force models to interpret the schematic diagram rather than answer from text alone.
    Section 3.3: the authors claim highlighting prevents text-only shortcuts, but they do not test whether questions are answerable without the diagram; original unmasked questions are retained but not analyzed for leakage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers." pith.science (2026). https://pith.science/paper/I4N2ZMKB

@misc{pith2026250710787,
  author       = {Pith},
  title        = {Pith review of: Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I4N2ZMKB}},
  note         = {Machine review of arXiv:2507.10787}
}
read the original abstract

This paper introduces MISS-QA, the first benchmark specifically designed to evaluate the ability of models to interpret schematic diagrams within scientific literature. MISS-QA comprises 1,500 expert-annotated examples over 465 scientific papers. In this benchmark, models are tasked with interpreting schematic diagrams that illustrate research overviews and answering corresponding information-seeking questions based on the broader context of the paper. We assess the performance of 18 frontier multimodal foundation models, including o4-mini, Gemini-2.5-Flash, and Qwen2.5-VL. We reveal a significant performance gap between these models and human experts on MISS-QA. Our analysis of model performance on unanswerable questions and our detailed error analysis further highlight the strengths and limitations of current models, offering key insights to enhance models in comprehending multimodal scientific literature.

Figures

Figures reproduced from arXiv: 2507.10787 by the authors.

Figure 1
Figure 1. An overview of the MISS-QA benchmark and research questions investigated in this study. systems that can assist scientists in seeking infor￾mation from paper through schematic diagrams. Foundation models have recently demonstrated remarkable capabilities in tackling various com￾plex tasks (Touvron et al., 2023; Jiang et al., 2023; OpenAI, 2024; Zhao et al., 2025a). However, their capabilities in interpreting schemat… view at source ↗
Figure 2
Figure 2. An overview of the MISS-QA benchmark construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example of a question posed within an [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Confusion matrices comparing the model response types (whether answerable) to the ground truth on the test set. MISS-QA are designed to be unanswerable (§3.4). This subset aims to evaluate models’ ability to han￾dle uncertain queries and appropriately respond by indica…
Figure 5
Figure 5. Figure 5: An example of model reasoning error. needed to answer the question but fails to reason correctly with this information, leading to incor￾rect response. Examples are presented in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The interface used by the first annotator for question annotation, where annotators highlight question [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: The interface used by the first annotator to evaluate question answerability and identify supporting [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: The interface used by the second annotator for answer annotation. The annotator begins by entering the [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The CoT prompt used in our experiment [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Querying Multimodal Scientific Papers with AI: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientists

    cs.HC 2026-07 conditional novelty 7.0 of 10

    A study of 115 real AI queries shows BLV scientists use chatbots mainly to access figures and tables (49% of queries) while sighted scientists use them to synthesize methods (56%), and both groups abandon AI tools ove...

  2. AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research

    cs.CL 2025-07 conditional novelty 7.0 of 10

    A benchmark of 1,500 expert-annotated ablation study designs from 807 NLP papers shows frontier LLMs underperform human experts and that LLM-as-a-judge evaluations correlate weakly with human judgments.

Reference graph

Works this paper leans on

40 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan...

    arXiv 2024

  4. [4]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Singh Chaplot, Jessica Chudnovsky, Saurabh Garg, Th \'e ophile Gervet, Soham Ghosh, Am'elie H'eliou, Paul Jacob, Albert Q. Jiang, Timoth \'e e Lacroix, Guillaume Lample, Diego de Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Reddy Mudd...

    arXiv 2024

  5. [5]

    Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.840 L it S earch: A retrieval benchmark for scientific literature search . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15068--15083, Miami, Florida, USA. Association ...

    doi:10.18653/v1/2024.emnlp-main.840 2024

  6. [6]

    Akari Asai and Eunsol Choi. 2021. https://doi.org/10.18653/v1/2021.acl-long.118 Challenges in information-seeking QA : Unanswerable questions and paragraph retrieval . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)...

    work page doi:10.18653/v1/2021.acl-long.118 2021

  7. [7]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. http://arxiv.org/abs/1611.09268 Ms marco: A human generated machine reading comprehension dataset

    arXiv 2018

  8. [8]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. 2024 a . http://arxiv.org/abs/2403.20330 Are we on the right way for evaluating large vision-language models?

    arXiv 2024

Show all 40 references
  1. [9]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024 b . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

    2024 arXiv

  2. [10]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024 c . How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    2024 arXiv

  3. [11]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv ...

    2023 arXiv

  4. [12]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. https://doi.org/10.18653/v1/2021.naacl-main.365 A dataset of information-seeking questions and answers anchored in research papers . In Proceedings of the 2021 Conference of the North Amer...

    2021 doi

  5. [13]

    Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, and Arman Cohan. 2024. https://doi.org/10.18653/v1/2024.findings-acl.951 Unveiling the spectrum of data contamination in language model: A survey from detection to remediation . In Findings of the As...

    2024 doi

  6. [14]

    Gemini. 2024. http://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    2024 arXiv

  7. [15]

    Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, T...

    2023 arXiv

  8. [16]

    Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-In Lee, and Moontae Lee. 2023 a . https://proceedings.mlr.press/v202/lee23n.html QASA : Advanced question answering on scientific articles . In Proceedings of the 40th International Conference on Machine...

    2023

  9. [17]

    Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-In Lee, and Moontae Lee. 2023 b . https://proceedings.mlr.press/v202/lee23n.html QASA : Advanced question answering on scientific articles . In Proceedings of the 40th International Conference on Machine...

    2023

  10. [18]

    Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, and Arman Cohan. 2024 a . https://doi.org/10.18653/v1/2024.findings-emnlp.904 M 3 S ci QA : A multi-modal multi-document scientific QA benchmark for evaluating foundation models . In Findings of the Association for ...

    2024 doi

  11. [19]

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.775 Multimodal A r X iv: A dataset for improving scientific comprehension of large vision-language models . In Proceedings of the 62nd Annual ...

    2024 doi

  12. [20]

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024 c . http://arxiv.org/abs/2403.00231 Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models

    2024 arXiv

  13. [21]

    Wilson, Woosang Lim, and William Yang Wang

    Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, Linda Ruth Petzold, Stephen D. Wilson, Woosang Lim, and William Yang Wang. 2024 d . http://arxiv.org/abs/2407.04903 Mmsci: A dataset for graduate-le...

    2024 arXiv

  14. [22]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Repre...

    2024

  15. [23]

    Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao ...

    2025 arXiv

  16. [24]

    Mistral AI . 2025. Mistral-small-3.1-24b-instruct-2503. https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503. Apache 2.0 License

    2025

  17. [25]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  18. [26]

    OpenAI. 2025 a . https://openai.com/index/o3-o4-mini-system-card-addendum-operator-o3/ Addendum to openai o3 and o4-mini system card: Openai o3 operator

    2025

  19. [27]

    OpenAI. 2025 b . https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api

    2025

  20. [28]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...

    2018 doi

  21. [29]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. http://arxiv.org/abs/1606.05250 Squad: 100,000+ questions for machine comprehension of text

    2016 arXiv

  22. [30]

    Elior Sulem, Jamaal Hay, and Dan Roth. 2022. https://doi.org/10.18653/v1/2022.naacl-main.79 Yes, no or IDK : The challenge of unanswerable yes/no questions . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: H...

    2022 doi

  23. [31]

    Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M

    Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cant \'o n Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes,...

    2023 arXiv

  24. [32]

    Chengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan, and Yilun Zhao. 2025. http://arxiv.org/abs/2506.15569 Sciver: Evaluating foundation models for multimodal scientific claim verification

    2025 arXiv

  25. [33]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 a . http://arxiv.org/abs/2409.12191 Qwen2...

    2024 arXiv

  26. [34]

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. 2024 b . http://arxiv.org/abs/2406.18521 Charxiv: Charting gaps in realistic chart understanding in mul...

    2024 arXiv

  27. [35]

    Zhijian Xu, Yilun Zhao, Manasi Patwardhan, Lovekesh Vig, and Arman Cohan. 2025. http://arxiv.org/abs/2507.02694 Can llms identify critical limitations within scientific research? a systematic evaluation on ai research papers

    2025 arXiv

  28. [36]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. http://arxiv.org/abs/1809.09600 Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    2018 arXiv

  29. [37]

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2024. http://arxiv.org/abs/2409.02813 Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    2024 arXiv

  30. [38]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. 2024. http://arxiv.org/abs/2403.14624 Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?

    2024 arXiv

  31. [39]

    Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, Chengye Wang, Ziyao Shangguan, Zhenwen Liang, Yixin Liu, Chen Zhao, and Arman Cohan. 2025 a . Mmvu: Measuring expert-level multi-discipline video understandi...

    2025

  32. [40]

    Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, Yixin Liu, Charles McGrady, Xiangru Tang, Zihang Wang, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, and Arman Cohan. 2025 b . http://a...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.