Pith. sign in

REVIEW 6 major objections 5 minor 2 cited by

Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science

T0 review · 6 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MatVQA argues that current multimodal AI models can answer only about half of research-level materials-science questions that require reading figures, not just text.

desk verdict A genuinely useful new benchmark and pipeline, but the gold labels aren't shown to require the image, so the headline accuracy numbers don't yet support the central claim. read the letter →

arxiv 2505.18319 v1 pith:O73OWDLS submitted 2025-05-23 cs.CE

Sifan Wu , Huan Zhang , Yizhan Li , Farshid Effaty , Amirreza Ataei , Bang Liu This is my paper
classification cs.CE
keywords MatVQAMArxivAgentmultimodallargelanguagemodelsvisualquestionansweringmaterialsscienceshortcuteliminationstructure-property-performancereasoningscientificbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MatVQA, a benchmark of 1,325 multiple-choice questions that test whether multimodal AI models can reason about materials-science figures at research level, not just read text. The authors argue that existing materials-science QA datasets are mostly text-based and can be solved with shortcuts, so they build an automated pipeline that rewrites questions until the image is genuinely required. On this benchmark the best of 17 models reaches 51.9% accuracy, which the paper reads as a large open gap in visual-scientific reasoning. If correct, MatVQA gives the field an evaluation standard that separates genuine figure-reading reasoning from language and caption guessing, and it scales automatically from new literature.

What carries the argument

MArxivAgent, an automated three-stage pipeline. It parses papers into text and figures, extracts a verifiable reasoning path structured around structure, property, performance, processing, and environment, and generates multiple-choice questions; then it runs an evaluator agent that answers without the image to detect language shortcuts, rewrites the item, and repeats with a caption-only evaluator to detect caption shortcuts. A consistency checker ensures each rewrite preserves the original scientific claim or reasoning path. This two-stage loop is what carries the benchmark's claim to measure visual-scientific reasoning rather than text guessing.

What would settle it

Take the released MatVQA questions, replace every image with a blank placeholder while keeping the stem and options, and run the same models under the benchmark's prompt; if any model scores far above random (25% for four-option items), textual shortcuts remain and the central claim fails.

Watch

Extended reading notes

Core claim

MatVQA claims to be the first benchmark for research-level multimodal reasoning in materials science. It contains 1,325 questions organized into four structure-property-performance tasks: quantitative, comparative, causal, and hypothetical variation. Generated by the MArxivAgent pipeline from recent materials-science papers and 378 unique figures, each item is passed through iterative refinement that removes language shortcuts (answerable from wording) and caption shortcuts (answerable from the figure caption), forcing models to inspect low-level visual features such as diffraction peaks and lattice fringes. Benchmarking 17 open- and closed-source multimodal models under chain-of-thought prompting, the best model scores 51.9%, with ablations showing raw questions are answered with roughly 76–83% accuracy before refinement and roughly 39–52% after, evidence that the shortcut removal meaningfully raises difficulty.

Load-bearing premise

The load-bearing premise is that the iterative rewrites preserve scientifically correct, figure-grounded answers; because only a random 20% of items are human-audited, any ambiguity or leakage that survives would make the model scores misread as visual-scientific reasoning.

Editorial extensions

If this is right

  • If MatVQA measures what it claims, research-level multimodal reasoning in materials science is currently unsolved: the best model is barely above half accuracy, and materials-finetuned models fall below 24%.
  • The automated pipeline makes benchmark construction scalable: the same procedure can be re-run on new papers, which the paper plans to use to expand MatVQA to roughly 12,000 questions.
  • The ablation quantifies that caption shortcuts are the larger leak: removing language shortcuts drops accuracy by about 10–15 percentage points, and additionally removing caption shortcuts drops it by another 18–20 points.
  • Comparing task types shows comparative reasoning is the weakest area and causal reasoning dominates the benchmark, so future model improvement should target joint perception of multiple structures and multi-hop causal chains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same shortcut-elimination recipe could be ported to other image-heavy sciences, such as biology microscopy or medical imaging, wherever captions and wording can leak answers; each field would need its own reasoning-path ontology.
  • The heavy reliance on LLM-generated reasoning paths means part of MatVQA's validity depends on the generator's domain knowledge; an independent human re-judgement of all 1,325 gold answers, not just a random 20%, would directly test for hidden label noise.
  • The tiny quantitative split, only 7 items, makes the 57% quantitative score statistically fragile; expanding that split should be a priority before drawing conclusions about numerical structure-property-performance reasoning.
  • A testable extension: measure whether a model pre-finetuned on MatVQA-style visual question pairs transfers to unseen materials papers; if transfer is strong, the benchmark could become a training resource as well as an evaluation set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper introduces MatVQA, a 1,325-item multiple-choice visual question answering benchmark for materials science, automatically generated from 44 recent arXiv papers via the MArxivAgent pipeline. The questions are organized into four structure-property-performance (SPP) reasoning tasks (causal, comparative, quantitative, hypothetical), derived from extracted reasoning chains, and refined in two stages that iteratively remove language shortcuts and caption shortcuts using LLM evaluators, rewriters, and consistency checkers. The authors benchmark 17 open- and closed-source MLLMs and report that the best model, Claude-3.7-Sonnet, reaches 51.9% overall accuracy, concluding that current MLLMs have a substantial gap in fine-grained visual-scientific reasoning. A random 20% of items is said to have been audited by two materials-science experts. The abstract also claims a comparison with human experts, but no human accuracy numbers appear in the main text.

Significance. If the gold labels are trustworthy, MatVQA would be a useful and scalable evaluation resource for multimodal scientific reasoning in materials science. The paper has clear strengths: the dataset and evaluation code are publicly released; the benchmark spans a wide range of materials subfields; the two-stage shortcut-removal procedure is a sensible response to known textual-bias problems; and the systematic evaluation of 17 models is a useful contribution. However, the central validity claim rests on the correctness and visual groundedness of the gold answers, and that grounding is not established by the evidence currently reported. The human audit is described only qualitatively, the image-only answerability of final items is not checked, the seven-item quantitative split is too small to support per-task accuracy claims, and at least one table contains internally inconsistent numbers. These issues are addressable in revision, but they are load-bearing for the paper's headline claims.

major comments (6)
  1. [§4.3] The human quality audit is described as a random 20% sample reviewed by two materials-science experts for scientific accuracy, logical consistency, and contextual relevance, with 'uniformly high marks,' but no scores, no per-criterion distributions, no inter-annotator agreement, and no resolution procedure for disagreements are reported. Because this audit is the only evidence presented that the machine-generated gold labels are correct, the absence of quantitative audit results is a load-bearing gap. Please report the full audit outcome, including how many items were accepted, revised, or rejected, the actual score ranges, and agreement statistics.
  2. [§4.2 / Algorithm 1] Stage 2 of the refinement loop verifies only that (i) the evaluator can no longer answer correctly from the stem, options, and caption, and (ii) the rewritten question-answer pair follows the same extracted reasoning chain (Algorithm 1, lines 25-30). It does not check that the gold answer is entailed by the image alone, that the image determines a unique option, or that the rewritten options are mutually exclusive under the image. Since Section 4.2 states that Stage 2 'results in larger modification on generated questions,' the risk of semantic drift or ambiguity is real. The reported accuracy drops therefore show that rewriting made the LLMs fail, not that the rewritten questions are genuinely visually grounded. Add an image-only validation: have experts or a carefully controlled protocol answer a sample from the figure alone, without captions or surrounding text, and report the proportion of items with a uniquely determinable correct answer.
  3. [Figure 4] Figure 4 shows the correct answer changing from A in the raw sample to B after language-shortcut removal, with no explanation. If Stage 1 is claimed to preserve semantic equivalence, a changed gold label suggests either the raw label was incorrect or the rewriting altered the scientific content. Please clarify this example and describe how label changes during refinement are handled; otherwise the stability of the gold standard is not established.
  4. [§5.1 / Table 2] The quantitative split contains only 7 items, yet Table 2 reports per-split accuracies for it (e.g., 57.1%, 28.5%) and Section 5.1 draws conclusions such as quantitative items appearing 'easy' and large models outperforming small models by +9.5 pp on this split. With 7 items, a single answer shifts accuracy by about 14 percentage points, so these task-level comparisons are statistically unstable. Either expand the quantitative split or remove per-split quantitative conclusions and report confidence intervals instead.
  5. [Abstract / §5.1] The abstract and introduction claim that a subset of models was compared against human experts, but the main text contains no human accuracy results, no expert protocol, and no human-baseline table. For a benchmark whose purpose is to show that MLLMs perform poorly relative to research-level human reasoning, the human baseline is essential evidence. Add expert performance numbers with a clear protocol (e.g., image-only, no caption, fixed time budget) or remove the claim.
  6. [Table 2] Several rows in Table 2 are internally inconsistent with the split sizes in Table 1. For example, Claude-3.5-Haiku is reported with overall 44.7%, but its split accuracies (Caus 32.9%, Hypo 38.3%, Quan 57.1%, Comp 37.5%) weight to approximately 34.5% given the 950/112/256/7 split; MOL-VL-7B's reported overall of 23.6% is also inconsistent with its split accuracies, which weight to about 28.4%. Please recompute and report corrected overall accuracies, or explain the discrepancy if a different evaluation subset was used.
minor comments (5)
  1. [§4.2, §4.1, §5.1] There are several typographical issues: 'aviod' in Section 4.2, 'consistancy' and 'consistencey' in Section 4.1, 'deatails' and 'splited' in Section 5.1, and 'in varies domain' in the contributions list in Section 1. Please copy-edit the manuscript.
  2. [Figures 2 and 3] The inline text in Figures 2 and 3 is partially garbled, with fragments such as 'BCCX' and '21.79° rotated TBLG' in the diagram boxes. A redrawn, larger version with legible example questions would make the pipeline much easier to follow.
  3. [Appendix C] The error-analysis examples in Figures 6 and 7 show blank or misordered options (e.g., options (2), (3), (4) are empty in the rendered excerpt). Please ensure the full question text and all options are visible in the final version.
  4. [Table 3] Table 3 reports large accuracy drops across refinement stages but no confidence intervals or significance tests. Because the same items are evaluated before and after refinement, a paired test (e.g., McNemar's test) and confidence intervals would strengthen the claim that the drops are systematic rather than noise.
  5. [Table 1(a)] The comparison table lists MacBench without a complete citation; reference [5] gives only a title and no venue, arXiv identifier, or year. Please provide full bibliographic details for all benchmark citations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MatVQA's construction pipeline is explicit and its headline accuracies are measured, not fitted; the LLM-in-the-loop shortcut removal is a design choice, and the gold-label validity concerns are quality risks rather than circular reductions.

full rationale

The paper does not derive a first-principles result from definitions that already contain it. MatVQA is constructed by an explicit pipeline: LLM extraction of reasoning chains from paper text, MCQ generation, and two-stage iterative refinement that removes items answerable without the image or from the caption. The final accuracies (e.g., 51.9% for Claude-3.7-Sonnet) are empirical measurements on the resulting dataset, not quantities fitted or predicted by the pipeline. The refinement loop guarantees only that the LLM evaluators fail on text-only and caption-only versions of the items; it does not fix the image-plus-text accuracy, which is why different models score differently. The statement that solving the final questions 'requires precise, low-level visual scrutiny' is a design claim, not a circular consequence of the method. The main weaknesses are evidential, not circular: Section 4.3 reports 'uniformly high marks' from a 20% human audit without reporting scores or inter-annotator agreement; Section 5.1's introduction promises a comparison 'against human experts and vision-language model baselines' but no such results appear in the paper; and Algorithm 1's Checker_caption validates rewrites only against the caption-derived reasoning path, so gold labels may not be uniquely image-entailed. These are data-validity concerns that should be addressed, but they are not instances of the paper's conclusions being equivalent to its inputs by construction. Self-citations (HoneyComb, HoneyBee, DynaMath) appear only in related-work or method contexts and are not load-bearing for the benchmark's central claim.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The central claim depends on several unverified premises: the faithfulness of LLM-extracted reasoning paths, the invariance of question correctness under rewriting, the representativeness of the four SPP tasks, and the generalizability of a 20% human audit. The dataset and pipeline are public, which provides some external grounding, but the quantitative split is too small to support strong conclusions.

free parameters (5)
  • initial corpus size = 500
    MArxivAgent retrieved 500 materials-science arXiv papers from 2024 (Section 4.1); this arbitrary threshold limits diversity and may introduce selection bias.
  • human audit sampling rate = 20%
    Only a random 20% of MCQs were manually audited (Section 4.3), leaving the other 80% unverified.
  • maximum shortcut-removal iterations = T (unspecified)
    Algorithm 1 loops up to T rewrite iterations; the value of T is not reported, which affects reproducibility and the degree of rewriting.
  • task-specific keyword filter = not enumerated
    The corpus was filtered using keywords such as 'property' and 'structure' (Section 4.1), but the full list is not provided, making the corpus composition hard to replicate.
  • random seed
    No random seed is reported for sampling or model evaluation, so exact numerical replication is not guaranteed.
assumptions (5)
  • domain assumption Gold labels can be derived from the textual reasoning paths extracted from arXiv papers by an LLM.
    The pipeline assumes that an LLM extractor's reasoning chains are faithful to the source paper and constitute correct answers (Section 4.1).
  • ad hoc to paper Removing language and caption shortcuts does not change the scientific content of the question.
    The LLM-based checker enforces semantic consistency and reasoning-path consistency (Algorithm 1), but no independent verification shows that rewritten questions remain answerable from the figure alone.
  • domain assumption The four SPP tasks (causal, comparative, quantitative, hypothetical) capture core scientific reasoning in materials science.
    The paper asserts these tasks cover critical cognitive operations, but provides no empirical justification that they are sufficient or representative (Section 3.2).
  • domain assumption Human audit of a random 20% sample generalizes to the entire dataset.
    Section 4.3 infers dataset quality from a 20% sample without reporting audit scores or inter-annotator agreement.
  • standard math Standard accuracy comparisons without confidence intervals are meaningful.
    The paper treats accuracy point estimates as directly comparable across models, which is a common but statistically weak assumption when sample sizes are small.
invented entities (3)
  • MatVQA dataset independent evidence
    purpose: Benchmark of 1,325 multiple-choice visual QA items for materials science evaluation
    The dataset is released publicly and can be used to test new models, providing an external falsifiable handle.
  • MArxivAgent pipeline independent evidence
    purpose: Automated three-stage pipeline for generating shortcut-free MCQs from arXiv papers
    The pipeline description is detailed enough to reimplement and the released code (claimed) allows external validation.
  • SPP tasks (causal, comparative, quantitative, hypothetical)
    purpose: Taxonomy of structure-property-performance reasoning tasks for benchmarking
    These are conceptual categories introduced by the authors; no independent evidence shows they are the most natural or complete decomposition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science." pith.science (2026). https://pith.science/paper/O73OWDLS

@misc{pith2026250518319,
  author       = {Pith},
  title        = {Pith review of: Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O73OWDLS}},
  note         = {Machine review of arXiv:2505.18319}
}
read the original abstract

The emergence of Multimodal Large Language Models (MLLMs) that integrate vision and language modalities has unlocked new potentials for scientific reasoning, outperforming prior benchmarks in both natural language and coding domains. Current materials science evaluation datasets such as MaScQA and SciQA remain largely text-based and fail to capture the visual and research-level analytic complexity required in materials discovery and design. We introduce MatVQA, a scalable benchmark specifically designed to address this gap. Generated via an automated pipeline, MArxivAgent, from recent materials literature, MatVQA features 1325 questions across four critical structure-property-performance (SPP) reasoning tasks. Uniquely, MatVQA employs an iterative process to eliminate textual shortcuts, compelling MLLMs to perform fine-grained, low-level visual analysis of material imagery (e.g., microscopy, diffraction patterns) integrated with multi-step scientific reasoning. Benchmarking 17 open- and closed-source MLLMs on MatVQA reveals substantial gaps in current multimodal reasoning capabilities. MatVQA benchmark data, along with evaluation code, is publicly available in \href{https://anonymous.4open.science/r/matvqa-1E01}{https://anonymous.4open.science/r/matvqa-1E01/README.md} to catalyze further research in applying MLLMs to complex materials science problems.

Figures

Figures reproduced from arXiv: 2505.18319 by the authors.

Figure 1
Figure 1. Domain attribution for MatVQA MatVQA is built by MArxivAgent, an auto￾mated and verifiable pipeline engineered for the efficient generation of challenging multiple￾choice questions (MCQs) directly from arXiv materials-science papers. This automated con￾struction underpins MatVQA’s inherent scala￾bility, facilitating ongoing expansion and adap￾tation. After generation by advanced LLMs, a random 20% of the MCQs are ve… view at source ↗
Figure 2
Figure 2. Construction Pipeline of MatVQA more MLLM benchmarks contributed to various domains[43, 53, 14, 18, 28]. Furthermore, EMMA BENCH [19] (Enhanced MultiModal reAsoning) targets organic multimodal reasoning across mathe￾matics, physics, chemistry, and coding. EMMA tasks require advanced cross-modal reasoning that cannot be solved by considering each modality independently, thus providing a challenging test suite for MLL… view at source ↗
Figure 3
Figure 3. MArxivAgent Pipeline for MCQ automatically Generation. "Lan. Rem" represents the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Evolution of a sample question through the two-stage shortcut removal process. The figure [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Representative Examples for varies material science domain. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Error sample for Material Knowledge Misunderstanding error type, the response missed [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Error sample for Reasoning-Wrong-Judgement error type. In this example, optical control [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What to Ask Next? Probing the Imaginative Reasoning of LLMs with TurtleSoup Puzzles

    cs.AI 2025-08 unverdicted novelty 6.0 of 10

    TurtleSoup-Bench is a new interactive benchmark showing that LLMs struggle with imaginative reasoning compared to humans.

  2. System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts

    cs.CL 2025-05 conditional novelty 6.0 of 10

    System-1.5 Reasoning lets LLMs reason in latent space with early exits and step-skipping, matching chain-of-thought accuracy at over 20x speedup on GSM8K and StrategyQA.

Reference graph

Works this paper leans on

54 extracted references · 35 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gemini 2.0 flash

    Google AI. Gemini 2.0 flash. https://cloud.google.com/vertex-ai/generative-ai/ docs/models/gemini/2-0-flash, Feb 2025. Accessed May 6, 2025

    work page 2025

  2. [2]

    Gemini 2.5: Our most intelligent ai model.https://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/ , Mar 2025

    Google AI. Gemini 2.5: Our most intelligent ai model.https://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/ , Mar 2025. Ac- cessed May 6, 2025; Experimental Version

    work page 2025

  3. [3]

    Llama 3: Open foundation and instruction-tuned language models

    Meta AI. Llama 3: Open foundation and instruction-tuned language models. https://ai. meta.com/research/llama, 2024. Version 3.2-90b-Vision-Instruct; Accessed May 6, 2025

    work page 2024

  4. [4]

    Mistral small 3.1: The best model in its weight class

    Mistral AI. Mistral small 3.1: The best model in its weight class. https://mistral.ai/ news/mistral-small-3-1 , Mar 2025. Accessed May 6, 2025

    work page 2025

  5. [5]

    Macbench: A multimodal chemistry and materials science benchmark

    Nawaf Alampara, Indrajeet Mandal, Pranav Khetarpal, Hargun Singh Grover, Mara Schilling- Wilhelmi, NM Anoop Krishnan, and Kevin Maik Jablonka. Macbench: A multimodal chemistry and materials science benchmark

    work page

  6. [6]

    Claude 3.5 haiku: Our fastest model for daily tasks

    Anthropic. Claude 3.5 haiku: Our fastest model for daily tasks. https://www.anthropic. com/claude/haiku, Oct 2024. Accessed May 6, 2025

    work page 2024

  7. [7]

    Claude 3.7 sonnet: Frontier reasoning made practical

    Anthropic. Claude 3.7 sonnet: Frontier reasoning made practical. https://www.anthropic. com/news/claude-3-7-sonnet , Feb 2025. Accessed May 6, 2025

    work page 2025

  8. [8]

    The sciqa scientific question answering benchmark for scholarly knowledge

    Sören Auer, Dante AC Barone, Cassiano Bartz, Eduardo G Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, et al. The sciqa scientific question answering benchmark for scholarly knowledge. Scientific Reports, 13(1):7240, 2023

    work page 2023

Show all 54 references
  1. [9]

    Testing and evaluation of health care applications of large language models: a systematic review

    Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA, 2024

    2024

  2. [10]

    Large language model assisted software engineering: prospects, challenges, and a case study

    Lenz Belzner, Thomas Gabor, and Martin Wirsing. Large language model assisted software engineering: prospects, challenges, and a case study. In International Conference on Bridging the Gap between AI and Reality, pages 355–374. Springer, 2023

    2023

  3. [11]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  4. [12]

    M.J. Buehler. Cephalo: Multi-modal vision-language models for bio-inspired materials analysis and design. arXiv, 2024. Accessed May 6, 2025; Related to Cephalo-Llava-v1.6-Mistral-vision- 8b-alph

    2024

  5. [13]

    Microvqa: A multimodal reasoning benchmark for microscopy-based scientific research

    James Burgess, Jeffrey J Nirschl, Laura Bravo-Sánchez, Alejandro Lozano, Sanket Rajan Gupte, Jesus G Galaz-Montoya, Yuhui Zhang, Yuchang Su, Disha Bhowmik, Zachary Coman, et al. Microvqa: A multimodal reasoning benchmark for microscopy-based scientific research. arXiv preprint...

    2025 arXiv

  6. [14]

    Physbench: Benchmarking and enhancing vision-language models for physical world understanding, 2025

    Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world understanding, 2025

    2025

  7. [15]

    Pixtral-12b

    Clarifai. Pixtral-12b. https://clarifai.com/mistralai/completion/models/ pixtral-12b, Oct 2024. Accessed May 6, 2025

    2024

  8. [16]

    Crqbench: A benchmark of code reasoning questions, 2024

    Elizabeth Dinella, Satish Chandra, and Petros Maniatis. Crqbench: A benchmark of code reasoning questions, 2024

    2024

  9. [17]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution, 2024

    2024

  10. [18]

    Chawla, Olaf Wiest, and Xiangliang Zhang

    Kehan Guo, Bozhao Nan, Yujun Zhou, Taicheng Guo, Zhichun Guo, Mihir Surve, Zhenwen Liang, Nitesh V . Chawla, Olaf Wiest, and Xiangliang Zhang. Can llms solve molecule puzzles? a multimodal benchmark for molecular structure elucidation. In A. Globerson, L. Mackey, D. Belgrave, ...

    2024

  11. [19]

    Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark, 2025

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark, 2025

    2025

  12. [20]

    Accurate predictions on small data with a tabular foundation model

    Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025

    2025

  13. [21]

    14 examples of how llms can transform materials science and chemistry: a reflection on a large language model hackathon

    Kevin Maik Jablonka, Qianxiang Ai, Alexander Al-Feghali, Shruti Badhwar, Joshua D Bocarsly, Andres M Bran, Stefan Bringuier, L Catherine Brinson, Kamal Choudhary, Defne Circi, et al. 14 examples of how llms can transform materials science and chemistry: a reflection on a large...

    2023

  14. [22]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  15. [23]

    Llm based biological named entity recognition from scientific literature

    Sung Jae Jung, Hajung Kim, and Kyoung Sang Jang. Llm based biological named entity recognition from scientific literature. In 2024 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 433–435. IEEE, 2024

    2024

  16. [24]

    Physicsassistant: An llm-powered interactive learning robot for physics lab investigations

    Ehsan Latif, Ramviyas Parasuraman, and Xiaoming Zhai. Physicsassistant: An llm-powered interactive learning robot for physics lab investigations. In 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN), pages 864–871. IEEE, 2024

    2024

  17. [25]

    Lab-bench: Measuring capabilities of language models for biology research

    Jon M Laurent, Joseph D Janizek, Michael Ruzo, Michaela M Hinks, Michael J Hammer- ling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. Lab-bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362, 2024

    2024 arXiv

  18. [26]

    Mmsci: A dataset for graduate-level multi-discipline multimodal scientific understanding

    Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al. Mmsci: A dataset for graduate-level multi-discipline multimodal scientific understanding. arXiv preprint arXiv:2407.04903, 2024

    2024 arXiv

  19. [27]

    Micro-bench: A microscopy benchmark for vision-language under- standing

    Alejandro Lozano, Jeffrey Nirschl, James Burgess, Sanket Rajan Gupte, Yuhui Zhang, Alyssa Unell, and Serena Yeung. Micro-bench: A microscopy benchmark for vision-language under- standing. Advances in Neural Information Processing Systems, 37:30670–30685, 2024

    2024

  20. [28]

    Micro-bench: A microscopy benchmark for vision-language understanding

    Alejandro Lozano, Jeffrey Nirschl, James Burgess, Sanket Rajan Gupte, Yuhui Zhang, Alyssa Unell, and Serena Yeung-Levy. Micro-bench: A microscopy benchmark for vision-language understanding. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, ...

    2024

  21. [29]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    2024

  22. [30]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    2022

  23. [31]

    Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine, 2023

    Yizhen Luo, Kai Yang, Massimo Hong, Xing Yi Liu, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine, 2023. Accessed May 6, 2025

    2023

  24. [32]

    Bioinspiredllm: Conversational large language model for the mechanics of biological and bio-inspired materials

    Rachel K Luu and Markus J Buehler. Bioinspiredllm: Conversational large language model for the mechanics of biological and bio-inspired materials. Advanced Science, 11(10):2306724, 2024

    2024

  25. [33]

    Doan, Nam V

    Dung Nguyen Manh, Thang Phan Chau, Nam Le Hai, Thong T. Doan, Nam V . Nguyen, Quang Pham, and Nghi D. Q. Bui. Codemmlu: A multi-task benchmark for assessing code understanding & reasoning capabilities of codellms, 2025

    2025

  26. [34]

    Phi-4-multimodal-instruct

    Microsoft. Phi-4-multimodal-instruct. https://huggingface.co/microsoft/ Phi-4-multimodal-instruct , 2025. Accessed May 6, 2025

    2025

  27. [35]

    Using an llm to help with code understanding

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    2024

  28. [36]

    Large language models in healthcare and medical domain: A review

    Zabir Al Nazi and Wei Peng. Large language models in healthcare and medical domain: A review. In Informatics, volume 11, page 57. MDPI, 2024

    2024

  29. [37]

    Gpt-4o mini: Advancing cost-efficient intelligence, 2024

    OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence, 2024. Accessed May 6, 2025

    2024

  30. [38]

    OpenAI (2024)

    OpenAI. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. Please cite this work as “OpenAI (2024)”; Accessed May 6, 2025

    2024 arXiv

  31. [39]

    A study of generative large language model for medical research and healthcare

    Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, et al. A study of generative large language model for medical research and healthcare. NPJ digital medicine, 6(1):210, 2023

    2023

  32. [40]

    Qwen2.5-vl technical report

    Alibaba Group Qwen Team. Qwen2.5-vl technical report. https://arxiv.org/abs/2502. 13923v1, 2025. arXiv:2502.13923v1 [cs.CV]; Accessed May 6, 2025

    2025 arXiv

  33. [41]

    Llm4mat-bench: benchmarking large language models for materials property prediction

    Andre Niyongabo Rubungo, Kangming Li, Jason Hattrick-Simpers, and Adji Bousso Dieng. Llm4mat-bench: benchmarking large language models for materials property prediction. arXiv preprint arXiv:2411.00177, 2024

    2024 arXiv

  34. [42]

    Honeybee: Progressive instruction finetuning of large language models for materials science

    Yu Song, Santiago Miret, Huan Zhang, and Bang Liu. Honeybee: Progressive instruction finetuning of large language models for materials science. arXiv preprint arXiv:2310.08511, 2023

    2023 arXiv

  35. [43]

    Multimodalqa: Complex question answering over text, tables and images, 2021

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images, 2021

    2021

  36. [44]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. Accessed May 6, 2025

    2024

  37. [45]

    Evaluating the performance and robustness of llms in materials science q&a and property predictions

    Hongchen Wang, Kangming Li, Scott Ramsay, Yao Fehlis, Edward Kim, and Jason Hattrick- Simpers. Evaluating the performance and robustness of llms in materials science q&a and property predictions. arXiv preprint arXiv:2409.14572, 2024

    2024 arXiv

  38. [46]

    Grok-2 beta release

    xAI. Grok-2 beta release. https://x.ai/blog/grok-2, Aug 2024. Accessed May 6, 2025; with Vision Capabilities. 12

    2024

  39. [47]

    Dearing, Jeffrey W

    Angel Yanguas-Gil, Matthew T. Dearing, Jeffrey W. Elam, Jessica C. Jones, Sungjoon Kim, Adnan Mohammad, Chi Thang Nguyen, and Bratin Sengupta. Benchmarking large language models for materials synthesis: the case of atomic layer deposition, 2024

    2024

  40. [48]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024

    2024 arXiv

  41. [49]

    Mascqa: investigating materials science knowledge of large language models

    Mohd Zaki, NM Anoop Krishnan, et al. Mascqa: investigating materials science knowledge of large language models. Digital Discovery, 3(2):313–327, 2024

    2024

  42. [50]

    Chemllm: A chemical large language model

    Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, et al. Chemllm: A chemical large language model. arXiv preprint arXiv:2402.06852, 2024

    2024 arXiv

  43. [51]

    Honeycomb: A flexible llm-based agent system for materials science

    Huan Zhang, Yu Song, Ziyu Hou, Santiago Miret, and Bang Liu. Honeycomb: A flexible llm-based agent system for materials science. arXiv preprint arXiv:2409.00135, 2024

    2024 arXiv

  44. [52]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2024

  45. [53]

    The answer is (X) \

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models, 2025. 13 A Prompts For MArxivAgent All prompts for evaluation used the chain-of-thought pro...

    2025

  46. [54]

    peak shape

    **Relate findings to the options**: - Option (1): Incorrect. The gradients or absorption patterns are not “smooth” or uniform across configurations. - Option (2): Correct. The relative positions and shapes of peaks directly correlate with changes in \( \delta L_z \), showing d...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.