Pith. sign in

REVIEW 3 major objections 5 minor 21 references

Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Best AI model scores 58.5% on visual STEM exam questions; students reach 62.7% and keep the edge when images matter.

desk verdict A useful dataset and a plausible descriptive pattern, but the main interaction claim is untested and Table 5 contains an impossible CI. read the letter →

arxiv 2507.03013 v1 pith:NLFCQ4SW submitted 2025-07-02 cs.CY cs.AI

Aymeric de Chillaz , Anna Sotnikova , Patrick Jermann , Antoine Bosselut This is my paper
classification cs.CYcs.AI
keywords multimodalLLMevaluationSTEMassessmenthuman-AIcomparisonquestiondesignacademicintegrityvisualreasoningpromptingstrategiesuniversityexams
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a dataset of 201 university-level STEM exam questions with images, each annotated for image type, whether the image is essential or merely supplemental, question format, and conceptual complexity, and each paired with historical student performance averaging 546 respondents per question. Against that baseline, the best model with majority voting answers 58.5% of questions correctly, below the student average of 62.7%. The paper's central finding is that students consistently outperform the model precisely where images carry essential information or where questions combine multiple concepts, while model performance also fluctuates more across subjects. The authors conclude that questions can be designed to challenge current AI systems without making them harder for students, by making images crucial and keeping questions concise.

What carries the argument

The load-bearing instrument is the hand-curated, feature-annotated question set: 201 real university STEM questions, each labelled for image type (diagram, line plot, algorithm, picture), image purpose (crucial versus supplemental), question type (single-answer, multiple-answer, and compound questions), and problem complexity (simple versus complex), with historical student accuracy as a human baseline. The comparison protocol uses a fixed set of five prompting strategies—direct zero-shot, chain-of-thought, image-first, text-first, and a two-stage description pipeline—aggregated by majority vote and by maximum score, with exact-match grading for all multiple-choice variants. The annotation allows the authors to attribute model failures to specific features, such as showing that removing supplemental images barely changes model accuracy, which is what turns a benchmark into guidance for question design.

What would settle it

Re-run the benchmark on the same questions in their original language, score multiple-answer questions with partial credit, and have the same instructors' students answer an identical machine-gradable version; if the model's deficit on crucial-image and multi-concept questions disappears or reverses, the design guidance does not survive.

Watch

Extended reading notes

Core claim

The paper claims that current multimodal large language models are systematically weaker than human students on visually grounded STEM reasoning, and that this weakness is predictable from two question features: whether the image is essential and how many concepts must be integrated. On its 201-question benchmark the strongest system answers 58.5% of questions with a majority-vote aggregation of five prompting strategies, against a 62.7% human average; across at least one prompting strategy the model retrieves the correct answer on 75.5% of questions, meaning the limitation is as much about reliably selecting the right answer as about not knowing it. Human performance is nearly flat across image type, image role, question format, and complexity, whereas model accuracy drops markedly on crucial images (56.1% versus 74.0% for supplemental images), on multiple-answer formats, and on complex problems. From this the authors draw a design principle: an assessment that requires a crucial image and multiple interacting concepts is harder for models while imposing no added difficulty on students, so such features can protect take-home exams without penalising learners.

Load-bearing premise

The comparison assumes that historical student scores from real exams, with instructor-specific grading and the original language, measure the same difficulty as exact-match scoring of translated questions given to models; if translation or grading choices shift difficulty, the human advantage on visual questions could be an artifact.

Editorial extensions

If this is right

  • Educators can make take-home exams more AI-resistant by writing questions whose image is essential to solving the problem and whose conditions draw on multiple concepts, without adding measured difficulty for students.
  • A model that answers text-heavy STEM questions well should not be taken as evidence of multimodal competence; on questions where images matter, the best model scores 56.1 percent versus 63.7 percent for students.
  • Because the model succeeds on 75.5 percent of questions with at least one prompting strategy but only 58.5 percent under majority voting, students who sample several answer strategies could still pass, so single-attempt assessment is a weaker safeguard.
  • Subject matter matters more for models than for students: the model is strongest in astronomy, computer science, and microfabrication and weakest in quantum physics, chemistry, and neuroscience, so AI-proofing advice should be subject-aware.
  • Supplemental images confer no meaningful advantage to the model, since removing them leaves performance essentially unchanged, so accessibility-friendly text descriptions do not by themselves make questions AI-answerable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's historical student baseline may not be strictly comparable to model scoring: students answered original-language exams with instructor-specific grading while models saw English translations scored by exact match. We infer that re-evaluating on original-language, partial-credit multiple-answer questions could narrow or widen the reported gaps by subject.
  • The flat student performance across question features suggests the human advantage on visual questions is durable, but a stronger test would give students and models identical, machine-gradable questions with image purpose randomly varied; we infer such matched-pair experiments would sharpen the causal claim about crucial images.
  • Because models retrieve the right answer in at least one prompt on three-quarters of questions, we infer that the practical vulnerability of remote assessments is not lack of knowledge but answer selection; calibration or voting schemes that force consistency may be a better defensive test than harder questions.
  • The design recommendations are correlational, not causal; we infer that prospective testing, writing fresh exam questions that vary image necessity and concept count while controlling content, is the natural next step before institutions rely on these features for integrity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a manually curated dataset of 201 university-level STEM exam questions with images, annotated for image type, image purpose (crucial vs. supplemental), question format, and problem complexity. The authors evaluate four multimodal LLM families under five prompting strategies, aggregate model responses by majority vote and by max, and compare the resulting accuracies with historical student performance from course records (546 respondents per question on average). Their main descriptive findings are that the best model (GPT-4o under majority vote) reaches 58.5% accuracy versus 62.7% for students; that student performance is relatively stable across image and question features while model performance drops on crucial images, diagrams, line plots, MCQMA questions, and complex problems; and that these patterns motivate design recommendations for assessments that are harder for AI without adding difficulty for students.

Significance. If the central claim holds, the paper provides a useful, field-relevant result: multimodal LLMs are measurably weaker than students on visually grounded STEM reasoning, and this weakness is predictable from question features. The dataset release is a genuine asset; manual annotation of image purpose and problem complexity, use of multiple model families and prompting strategies, and grounding in real student performance data are all strengths. The paper also makes a concrete, falsifiable design recommendation. However, the headline interaction claim—that crucial images and multiple concepts challenge models more than students—is not supported by the statistical evidence presented, and one reported confidence interval is internally impossible. The significance of the paper therefore depends on a re-analysis that the current manuscript does not provide.

major comments (3)
  1. [Section 4.3 and Conclusion, Tables 5 and 6] The central claim that crucial images and multiple concepts 'pose a greater challenge for models without increasing difficulty for students' is an interaction claim, but the paper never tests the group-by-feature interaction. The only inferential evidence is the bootstrap 95% confidence intervals in Tables 5 and 6, and for the key comparisons those intervals overlap: model crucial accuracy is 0.561 [0.49, 0.63] versus supplemental 0.740 [0.60, 0.86], while student crucial accuracy is 0.637 [0.60, 0.67] versus supplemental 0.588 [0.51, 0.67]. Overlapping intervals do not establish an interaction, and the descriptive differences could be driven by correlated features such as subject, question length, or format. I request a formal interaction test, for example a logistic mixed-effects regression on item-level predictions with random intercepts for questions, with group (student vs. model) interacted with image purpose and problem complexity. Without such a test, the actionable conclusion in the abstract and conclusion is underdetermined.
  2. [Table 5] Table 5 reports MCQMA model accuracy as 0.457 with a 95% confidence interval of [0.68, 0.84]. This interval does not contain the reported mean and is therefore impossible for a bootstrap CI computed from the same data. This suggests an indexing or transcription error in the very table used to support statements about statistical significance. The authors should correct the table and re-verify all other intervals, since a reader cannot currently assess which of the reported intervals are reliable. This issue directly affects the discussion of MCQMA performance in Section 4.3.
  3. [Section 3, Section 4, and Limitations] The human baseline is historical student accuracy from original course records, while the model is evaluated on English translations of the question text with exact-match grading and no partial credit for MCQMA. The manuscript acknowledges these mismatches only in the Limitations section, but they are load-bearing for the central comparison. If the original exams were in French (Appendix A.1, fields 13–14) or if instructor-specific grading included partial credit, then the student-model accuracy gap on visual and MCQMA questions could be partly an artifact of translation and grading differences rather than a genuine human visual advantage. Please either re-analyze with a more comparable human baseline (e.g., scoring the model with partial credit for MCQMA, or evaluating students on the same translated items) or substantially temper the causal-sounding recommendations in the Conclusion.
minor comments (5)
  1. [Appendix B.3] The selection of the five prompting strategies is based on performance on only 10 questions. This is a small selection set and risks overfitting the prompt choice to those items; the manuscript should acknowledge this explicitly.
  2. [References] The reference for Claude 3.7 Sonnet is incomplete ('Claude 3.7 Sonnet, 2025' without a publisher or technical report identifier), and the SciBench reference appears twice in essentially identical form (Wang et al., 2023 and 2024b). Please clean up the bibliography.
  3. [Limitations] The Limitations section states 'we confirmed statistical significance,' but the only inferential statistics presented are bootstrap confidence intervals, and no formal significance tests appear in the main text or appendices. Please either report the tests or remove the phrase.
  4. [Section 4.2 / Figure 3] The ablation study in Appendix D.2 is described as showing that removing supplemental images 'slightly improves model performance, though the effect is minimal,' but no numerical comparison or confidence interval is provided for that ablation. Adding the numbers would make the claim checkable.
  5. [Section 4.1] The statement that 'all prompting strategies perform similarly' in Figure 9 is based on visual inspection; reporting the per-strategy accuracies and their variability would make the claim more precise.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the claims are direct empirical comparisons with external human baselines and model evaluations; no fitted quantity or self-citation chain defines the reported accuracies.

full rationale

This paper is an empirical measurement study, not a derivation. The central quantities—student accuracy, model accuracy, and their breakdowns by image and question features—are all directly measured: student scores come from historical course records (Section 2) and model scores from running five prompting strategies on the released 201-question dataset with exact-match grading (Section 3). There is no equation fitted to the data whose output is then renamed as a prediction, no parameter calibrated to a subset and then reported as a forecast on a closely related quantity, and no theoretical derivation that assumes its own conclusion. The only self-citations are to Borges et al. (2024), used to support background statements that LLMs perform well on text-based assessments; that citation is not load-bearing for the paper's main visual-comparison claim, and it is an external published result rather than an invoked uniqueness theorem. The selection of five prompting strategies from an initial set of twelve on ten questions (Appendix B.3) is a model-selection choice that could introduce selection bias, but it is not circular: the reported 58.5% accuracy is an evaluation on the full dataset, not the same quantity used to select the strategies. Similarly, the hand-picked thresholds in the Section 4.4 error analysis are descriptive groupings, not fitted inputs that define the headline accuracies. Valid concerns about the absence of an interaction test for the headline claim, overlapping confidence intervals, and the impossible CI in Table 5 are statistical correctness risks, not circularity. Because no load-bearing step reduces by construction to its own inputs, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No model parameters are fitted; all quantities are measured accuracy. The only hand-chosen items are the error-analysis thresholds in Section 4.4 and the selection of five prompting strategies from a 10-question pilot (Appendix B.3), neither of which is a fitted parameter in the main comparisons.

free parameters (1)
  • Error-analysis thresholds = 40% student accuracy and 65% model accuracy
    Section 4.4 uses these cutoffs to partition 59 questions into easy-for-students/hard-for-model and vice versa; they are hand-chosen and not justified, and altering them changes the qualitative error categories.
assumptions (4)
  • domain assumption Historical course-record accuracy is a valid human difficulty baseline.
    Invoked in Section 2 where student performance is used as 'an indicator of problem difficulty'; confounded by instructor, course context, and grading policy.
  • domain assumption English translations used for models are semantically equivalent to the original French questions taken by students.
    Appendix A.1 labels Was_translated=true for French originals, but no translation validation or back-translation checks are reported; this underpins all human-model comparisons.
  • domain assumption Exact-match no-partial-credit grading for models on MCQMA is comparable to student scores from course records.
    Authors acknowledge the mismatch in Limitations, yet still report pooled human-model comparisons without adjusting for partial credit.
  • domain assumption The 201 manually annotated questions are representative enough for subject-level claims.
    Subject sample sizes range from 8 (Biology) to 32 (Astronomy) in Table 1; unmodeled sampling noise is possible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison." pith.science (2026). https://pith.science/paper/NLFCQ4SW

@misc{pith2026250703013,
  author       = {Pith},
  title        = {Pith review of: Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLFCQ4SW}},
  note         = {Machine review of arXiv:2507.03013}
}
read the original abstract

Generative AI systems have rapidly advanced, with multimodal input capabilities enabling reasoning beyond text-based tasks. In education, these advancements could influence assessment design and question answering, presenting both opportunities and challenges. To investigate these effects, we introduce a high-quality dataset of 201 university-level STEM questions, manually annotated with features such as image type, role, problem complexity, and question format. Our study analyzes how these features affect generative AI performance compared to students. We evaluate four model families with five prompting strategies, comparing results to the average of 546 student responses per question. Although the best model correctly answers on average 58.5 % of the questions using majority vote aggregation, human participants consistently outperform AI on questions involving visual components. Interestingly, human performance remains stable across question features but varies by subject, whereas AI performance is susceptible to both subject matter and question features. Finally, we provide actionable insights for educators, demonstrating how question design can enhance academic integrity by leveraging features that challenge current AI systems without increasing the cognitive burden for students.

Figures

Figures reproduced from arXiv: 2507.03013 by the authors.

Figure 1
Figure 1. Example of STEM problems with average model performance (majority vote) compared to average student performance. benchmarks were introduced to assess AI perfor￾mance (Wang et al., 2024b). Recent advances in multimodal large language models (LLMs) have led to extensive efforts in developing image-based exam datasets, particularly in STEM. Anand et al. (2024) introduced a multimodal physics dataset, expanding from 300… view at source ↗
Figure 2
Figure 2. Average model and student accuracy per sub [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. (a) Effect of image properties on model performance aggregated by the majority vote strategy compared [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: (a) Effect of question type on model performance aggregated by the majority vote strategy compared to [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Example of a complex question with supple [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Student accuracy distribution. A.3 Student Performance Dataset difficulty illustrated by student perfor￾mance is presented in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Model performance on the initial set of prompting strategies. Recursive Prompt You are an expert in STEM courses tasked with answering questions with recursive analysis. You will follow the next steps before providing an answer. Step 1: Analyze the Image(s) First - Des…
Figure 8
Figure 8. Figure 8: Model performance with and without supple [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 10
Figure 10. Figure 10: Comparison of student vs model accuracy depending on the question length in characters. D.4 Student vs model accuracy depending on the question length [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 19 canonical work pages

  1. [1]

    Calculus I

    Course_name: • Description: The name or identifier of the course associated with the question. • Example: "Calculus I" , "Physics 101"

    work page

  2. [2]

    Complex” means that the question involves multiple concepts of the subject, while “Simple

    Scibench: Evaluating college-level scientific problem-solving abilities of large language models. Preprint, arXiv:2307.10635. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2024b. Scibench: Evaluating college-level scientific problem-solving abilities of large langu...

    arXiv 2023

  3. [3]

    Question: • Description: The text of the ques- tion, may include LaTeX formatting and placeholders for images

    work page

  4. [4]

    Exercise_name: • Description: The unique exercise id

    work page

  5. [5]

    MCQ" (Multiple Choice Question) –

    Question_type: • Description: The format or type of the question. • Possible Labels: – "MCQ" (Multiple Choice Question) – "MCQMA" (MCQ Multiple Answers) – "Compound" (a non-open-ended question with multiple objectives)

    work page

  6. [6]

    Gold_answer: • Description: The correct answer to the question

    work page

  7. [7]

    Crucial" (Essential for solving the question) –

    Image_purpose: • Description: The role of the image in the context of the question. • Possible Labels: – "Crucial" (Essential for solving the question) – "Supplemental" (Doesn’t provide additional context)

    work page

  8. [8]

    line plot

    Image_type: • Description: The type of images in- cluded in the question. • Possible Labels (Others may be added as we manually label): – "line plot" – "bar plot" – "scatter plot" – "histogram" – "pie chart" – "table" – "image" – "diagram"

    work page

Show all 21 references
  1. [9]

    Question_images: • Description: A list of filenames or identi- fiers for images included in the question

  2. [10]

    Complex" doesn’t necessarily mean that the problem is difficult. It simply means that many conditions are in play. • Possible Labels: –

    Problem_conditions: • Description: The complexity of the con- ditions within the problem. • "Complex" doesn’t necessarily mean that the problem is difficult. It simply means that many conditions are in play. • Possible Labels: – "Simple" (Conditions are straight- forward and n...

  3. [11]

    • Example: 1, 2

    Num_objectives: • Description: The number of sub- questions within the question. • Example: 1, 2

  4. [12]

    Question_length_characters: • Description: The length of the question text is measured in characters

  5. [13]

    • Example: "French"

    Original_language: • Description: The original language of the question before translation. • Example: "French"

  6. [14]

    • Always in "English" because we trans- lated the French ones

    Language: • Description: The language in which the question is written. • Always in "English" because we trans- lated the French ones

  7. [15]

    PNG", "JPEG

    Image_file_type: • Description: The file format of the im- ages used. • Example: "PNG", "JPEG"

  8. [16]

    • Possible Values: true or false

    Was_translated: • Description: Indicates whether the ques- tion was translated from another lan- guage. • Possible Values: true or false

  9. [17]

    Unique answer

    Solution_type: • Description: Indicates whether the ques- tion has a unique correct answer or mul- tiple correct answers. • Possible Labels: – "Unique answer" – "Multiple answers"

  10. [18]

    Only MCQ Letter

    Answer_format: • Description: The expected format of the answer. • Possible Labels: – "Only MCQ Letter" (previously called MCQ) – "Only Numeric Answer" – "Derivation" – "Text" – "Code" – "Calculation"

  11. [19]

    All Independent

    Objective_dependency: • Description: Indicates whether the objec- tives in the question are independent or dependent on previous ones. • Possible Labels: – "All Independent" (Objectives can be solved separately) – "Dependent" (Some objectives rely on answers from previous part...

    2000

  12. [20]

    LaTeX",

    Type_of_text: • Description: The formatting or typeset- ting used in the question text. • Example: "LaTeX", "Plain text" , "XML"

  13. [2023]

    Preprint, arXiv:2305.15074

    Have llms advanced enough? a challenging problem solving benchmark for large language mod- els. Preprint, arXiv:2305.15074. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.