Pith. sign in

REVIEW 2 major objections 4 minor 77 references

Multimodal large language models and physics visual tasks: comparative analysis of performance and costs

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Multimodal AI models scored between 81.5 percent and 21 percent on visual physics questions, and price did not predict performance.

desk verdict A useful, thoroughly documented benchmark of 17 MLLMs on standard physics concept inventories, but the unaddressed contamination risk of public test items keeps it from fully earning the claim that it measures visual reasoning. read the letter →

arxiv 2506.19662 v2 pith:YMSADVMX submitted 2025-06-24 physics.ed-ph

Giulia Polverini , Bor Gregorcic This is my paper · ORCID
classification physics.ed-ph
keywords multimodallargelanguagemodelsphysicseducationconceptinventoriesvisualreasoningcost-performancebenchmarkinggeometricalopticsAImodelselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether commercially available multimodal AI models can handle the kinds of image-based conceptual physics questions that students encounter, and whether the cost of using them is justified. The authors benchmark a range of publicly available models from three major providers on 102 items drawn from four validated concept inventories covering kinematics, electromagnetism, quantum mechanics, and geometrical optics. Their central empirical claim is that performance varies enormously, from 81.5 percent down to 21 percent, and that price is not a reliable guide to quality: some inexpensive models perform close to the best, while one of the most expensive models underperforms mid-tier rivals. The paper argues that educational institutions should select models on measured performance and cost rather than provider reputation or listed prices. It also finds that all models struggle most with geometrical optics, suggesting that the type of visual reasoning required, rather than the physics topic by itself, is what drives difficulty.

What carries the argument

The central mechanism is a standardized, image-based benchmark: screenshots of 102 items from four research-validated concept inventories, including BEMA for electricity and magnetism, TUG-K for kinematics graphs, QMVI for quantum mechanics visualization, and FTGOT for geometrical optics. Each item is submitted to every model through its API under one minimal prompt, with temperature set to 0.7 where possible, ten fresh runs per item, and scoring based only on the final answer letter. This protocol converts the vague question of whether a model understands physics pictures into a comparable performance number. The same runs supply the cost analysis, which tracks input, output, and hidden reasoning tokens and multiplies them by listed per-token prices to give a normalized single-pass cost for the entire benchmark.

What would settle it

Re-run the same 102 items with several alternative minimal prompts, such as no answer-letter instruction, a 'think step by step' cue, or different temperatures, and check whether the top models and the cheap-model advantage persist. If GPT-5 mini's 75 percent collapses or Claude Opus 4 jumps above the mid-tier models under a plausible prompt, the cost-performance ranking is protocol-dependent rather than a stable property of the models.

Watch

Extended reading notes

Core claim

On the paper's own terms, the best multimodal models of mid-2025 already surpass post-instruction university student averages on established concept inventories: GPT-5 reaches 81.5 percent, with o3, Gemini 2.5 Pro, and GPT-5 mini in the 75 to 76 percent range. Every model's score drops sharply on items requiring spatial and geometric interpretation, most notably geometrical optics, where the best score is 51.5 percent. The cost analysis shows a loose, non-proportional relationship between price and accuracy: GPT-5 mini scores 75 percent at roughly $0.27 for one pass over the full benchmark, while Claude Opus 4 scores 57 percent at roughly $4.54 and Gemini 2.5 Pro scores 75.8 percent at roughly $4.68. The paper interprets this as evidence that cheaper models can be sufficiently capable for some educational uses, and that high per-token prices do not guarantee high performance on visual physics tasks.

Load-bearing premise

The entire ranking rests on the assumption that a single minimal prompt, repeated ten times at one temperature, gives a stable and representative estimate of each model's ability on these image-based items; if answers depend heavily on prompt wording or formatting, the reported performance gaps could be artifacts of the protocol.

Editorial extensions

If this is right

  • Institutions deploying AI for multiple-choice physics diagnostics can get near-top accuracy from a model costing less than a dollar for a full 102-item pass, rather than assuming the flagship model is required.
  • Price tier and provider reputation are not reliable proxies for visual physics ability: one of the most expensive models scored roughly 24 percentage points below a model costing about one-seventeenth as much.
  • Free open-weight models scored between 21 and 35 percent, below what the paper considers acceptable for student-facing physics work involving images.
  • Even the best models score below 52 percent on geometrical optics items, so AI support in that domain should be treated as unreliable for now.
  • Benchmarks like this need to be re-run regularly, because model capabilities, pricing, and API availability change quickly.
  • The finding that visual task type matters more than physics topic suggests that future work should analyze which specific visual formats cause failures.
  • A testable extension is to run the same 102 items under several prompt variants and temperatures to map sensitivity; large rank shifts would show that the current single-protocol numbers are protocol-dependent.
  • Because only the final answer letter is scored, a model could reach a high accuracy while generating flawed explanations; a qualitative analysis might rank models differently for tutoring uses where reasoning quality matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the reported scores are lower bounds on what targeted prompting could achieve, particularly for non-reasoning models; if prompt engineering raises cheap models more than reasoning models, the cost-performance advantage could grow.
  • I infer that scoring only the final letter, without analyzing the reasoning text, could hide systematic differences in how models arrive at correct answers, so a qualitative pass might alter the practical ranking for tutoring.
  • I infer that the benchmark protocol could be tested for robustness by repeating the same 102 items with several minimal prompt variants and temperatures; if the rank order changes substantially, the conclusions are protocol-dependent.
  • I infer that the finding that visual format matters more than topic points toward building a physics-specific visual-reasoning benchmark that could predict model performance on unseen diagram types.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper benchmarks 17 multimodal large language models (MLLMs) from Anthropic, Google, and OpenAI on four established, image-based physics concept inventories (TUG-K, BEMA, QMVI, FTGOT), comprising 102 items. Each item was submitted as a screenshot with a minimal prompt, 10 times per model, and responses were scored by the final selected letter. The authors report per-inventory and total accuracy, standard deviations, and standard errors, alongside token-based cost estimates for a single pass through the benchmark. The main empirical findings are: total accuracy ranges from 81.5% (GPT-5) to 21.0% (Gemma 3-4b); performance is substantially higher on BEMA and TUG-K than on QMVI and FTGOT; and cost does not strictly track performance, with GPT-5 mini being a notable low-cost, high-performance outlier. The paper concludes that institutions should select models on the basis of measured benchmark performance and cost rather than provider reputation or list price, and that some cheaper models may be adequate for certain educational tasks.

Significance. If the results are valid, the paper provides a useful, independent, and reusable benchmark for physics educators and administrators choosing among commercial MLLMs. The authors have made their response dataset publicly available on Zenodo (ref. [71]), which is a concrete strength and supports reproducibility. The paper also connects performance to per-use cost in a transparent way, addressing a gap in the physics education literature. However, the central claim that these scores reflect competence on 'physics visual tasks' rests on the assumption that the models actually use the image content rather than memorized text-answer associations from the published concept inventories. The statistical reporting also contains a technical error that undermines the precision of the reported confidence intervals. These issues are fixable but need to be addressed before the practical recommendations can be fully trusted.

major comments (2)
  1. [Section 2.4 and Table 3] The reported SEM is not the standard error of the averaged performance. The authors write that SD is the square root of the sum of item-score variances, and that SEM is SD divided by sqrt(10). For item i with 10 Bernoulli trials and success probability p_i, the item-score variance is p_i(1-p_i)/10, so the computed quantity is sqrt(Σ p_i(1-p_i)/10), while the SEM of the mean across k items is sqrt(Σ p_i(1-p_i)/10)/k. The reported values therefore overstate the uncertainty of the total score by a factor of roughly k (the number of items, e.g., 31 for BEMA). Because Table 3 uses these SEMs to indicate confidence in the performance estimates, the current numbers do not support statements like 'most SEM values below 2.5%' as a statement about the precision of the reported averages. The calculation should be corrected or the quantity relabeled as an aggregate variability measure.
  2. [Section 2.2 and Discussion] The paper does not address the risk that the four concept inventories—especially the widely circulated BEMA and TUG-K—are part of the models' training corpora. Since the text of the questions and answer options is publicly available via PhysPort, the high scores on BEMA (93.2% for GPT-5) and TUG-K (92.3%) could partly reflect memorization of answer-key patterns rather than visual interpretation of the screenshots. The drop to 48.5% on the edited FTGOT items, whose format was modified to remove the four-tier structure, is consistent with this concern. Because the central claim is that the benchmark measures performance on 'physics visual tasks,' the authors should either provide evidence that the visual content is necessary (e.g., a text-only ablation or a set of novel, non-public items) or substantially temper the claim that the scores reflect image-based reasoning. As written, the construct validity of the benchmark for its stated purpose is not established.
minor comments (4)
  1. [Abstract, Table 1, Table 3] The abstract and research framing state that 15 models were benchmarked, but Table 1 lists 17 models and Table 3 includes an additional row for 'Gemini 2.5 Flash (no reasoning),' for 18 rows total. The number should be corrected and the status of the no-reasoning condition clarified as a configuration rather than a separate model.
  2. [Section 2.3] The prompt instructs models to answer with the letter N when no option is correct, but the original concept inventories do not contain an N option. This changes the response space and may interact with model behavior in ways not analyzed; the authors should at least report how often N was selected and whether any model used it disproportionately.
  3. [Section 4] There is a typo in the fourth paragraph of the Discussion: 'praticular' should be 'particular.'
  4. [Figure 2] The figure includes a 'Student' marker with reference to multiple sources [67, 68, 74, 75], but it is not explained how student performance was aggregated across four different instruments with different student populations. The caption should state the source and construction of this reference point.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark and cost analysis are direct empirical measurements on external concept inventories, with no fitted parameters or self-derived predictions.

full rationale

The paper's central claims are empirical measurements, not derivations. Performance is obtained by submitting screenshots of 102 items from four established, externally published concept inventories (BEMA, TUG-K, QMVI, FTGOT) to 15 MLLMs via official APIs, scoring the final answer letter, and averaging per-item percentages (Section 2.4). Costs are computed by multiplying observed token counts by published per-token API prices (Section 2.4, Table 4). The headline results—performance ranging from 81.5% to 21% and the absence of a strict cost-performance proportionality—are read directly from Tables 3 and 4, so no reported number is equivalent by construction to an input assumption. The authors cite their own prior work (e.g., [33], [36], [37], [40]) for background and to justify the choice of 10 repetitions, and they self-publish the dataset [71]; these citations inform protocol choices but do not enter the computation of any score or cost. The acknowledged limitations (minimal prompt, temperature 0.7, final-answer-only scoring, snapshot in time) are validity limitations, not circular steps. The possible memorization of public inventories is a construct-validity threat that would affect interpretation of what the scores measure, but it is not a case of the paper predicting or fitting its own inputs. No equation, fitted parameter, or uniqueness assertion is imported from the authors' prior work to force the conclusions. Therefore the derivation chain is self-contained against external benchmarks and no circularity is found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted; the design choices (temperature, repetition count, reasoning effort) are fixed experimental conditions rather than fitted values. No new physical or theoretical entities are introduced. The four axioms above are the background assumptions the benchmark depends on.

assumptions (4)
  • domain assumption The four selected concept inventories are valid, research-based measures of conceptual understanding and are suitable as submitted screenshots without re-validation.
    Section 2.2 states the inventories are well-validated and almost entirely image-based; the study does not re-validate them in screenshot form.
  • ad hoc to paper The graphical edits to BEMA and FTGOT items preserve the meaning, difficulty, and visual demands of the original items.
    Section 2.3 describes splitting 16 shared-image BEMA items and trimming FTGOT four-tier items, but provides no pilot test or check that the edited items behave like the originals.
  • domain assumption Ten repetitions with a fixed minimal prompt and temperature 0.7 (where possible) produce stable, representative estimates of each model's performance.
    Section 2.3 justifies 10 repetitions based on prior research and fixed prompts to avoid confounding; this assumes prompt sensitivity is not a dominant factor in the rankings.
  • domain assumption Provider-listed token prices and measured token counts reflect the true cost of deploying each model.
    Section 2.4 and Table 4 multiply observed token usage by prices from vendor pages; prices and tokenization can change and are not independently audited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal large language models and physics visual tasks: comparative analysis of performance and costs." pith.science (2026). https://pith.science/paper/YMSADVMX

@misc{pith2026250619662,
  author       = {Pith},
  title        = {Pith review of: Multimodal large language models and physics visual tasks: comparative analysis of performance and costs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YMSADVMX}},
  note         = {Machine review of arXiv:2506.19662}
}
read the original abstract

Multimodal large language models (MLLMs) capable of processing both text and visual inputs are increasingly being explored for uses in physics education, such as tutoring, formative assessment, and grading. This study evaluates a range of publicly available MLLMs on a set of standardized, image-based physics research-based conceptual assessments (concept inventories). We benchmark 15 models from three major providers (Anthropic, Google, and OpenAI) across 102 physics items, focusing on two main questions: (1) How well do these models perform on conceptual physics tasks involving visual representations? and (2) What are the financial costs associated with their use? The results show high variability in both performance and cost. The performance of the tested models ranges from 81.5% to as low as 21%. We also found that expensive models do not always outperform cheaper ones and that, depending on the demands of the context, cheaper models may be sufficiently capable for some tasks. This is especially relevant in contexts where financial resources are limited or for large-scale educational implementation of MLLMs. By providing these analyses, our aim is to inform teachers, institutions, and other educational stakeholders so that they can make evidence-based decisions about the selection of models for use in AI-supported physics education.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 45 canonical work pages

  1. [71]

    Dataset on Zenodo (2025)

    Gregorcic, B., Polverini, G.: Responses of Multimodal Large Language Models on BEMA, TUG-K, QMVI and FTGOT. Dataset on Zenodo (2025). https://doi. org/10.5281/zenodo.15719827

    work page doi:10.5281/zenodo.15719827 2025

  2. [1]

    Education Sciences13(12), 1216 (2023) https://doi.org/10.3390/educsci13121216 EISSN 2227-7102

    Gligorea, I., Cioca, M., Oancea, R., Gorski, A.-T., Gorski, H., Tudorache, P.: Adaptive learning using artificial intelligence in e-learning: A literature review. Education Sciences13(12), 1216 (2023) https://doi.org/10.3390/educsci13121216 EISSN 2227-7102

    doi:10.3390/educsci13121216 2023

  3. [2]

    IEEE Transactions on Learning Technologies17, 12–31 (2024)

    Ahmad, K., Iqbal, W., El-Hassan, A., Qadir, J., Benhaddou, D., Ayyash, M., Al-Fuqaha, A.: Data-driven artificial intelligence in education: A comprehensive review. IEEE Transactions on Learning Technologies17, 12–31 (2024)

    work page 2024

  4. [3]

    Liu, V., Latif, E., Zhai, X.: Advancing education through tutoring systems: A systematic literature review (2025) arXiv:2503.09748 18

    arXiv 2025

  5. [4]

    Computer Science & IT Research Journal4(3), 185–199 (2023)

    Aderibigbe, A.O., Ohenhen, P.E., Nwaobia, N.K., Gidiagba, J.O., Ani, E.C.: Arti- ficial intelligence in developing countries: Bridging the gap between potential and implementation. Computer Science & IT Research Journal4(3), 185–199 (2023)

    work page 2023

  6. [5]

    Language Education & Technology (LET Journal)3(2), 96–109 (2023)

    Boˇ zi´ c, V.: Artificial intelligence as the reason and the solution of digital divide. Language Education & Technology (LET Journal)3(2), 96–109 (2023)

    work page 2023

  7. [6]

    European Journal of Physics45(2), 025701 (2024) https://doi.org/10.1088/1361-6404/ad1420

    Polverini, G., Gregorcic, B.: How understanding large language models can inform the use of chatgpt in physics education. European Journal of Physics45(2), 025701 (2024) https://doi.org/10.1088/1361-6404/ad1420

    doi:10.1088/1361-6404/ad1420 2024

  8. [7]

    Zeng, Z., Chen, P., Liu, S., Jiang, H., Jia, J.: Mr-gsm8k: A meta-reasoning benchmark for large language model evaluation (2023) arXiv:2312.17080

    arXiv 2023

Show all 77 references
  1. [8]

    Xuan, W., Yang, R., Qi, H., Zeng, Q., Xiao, Y., Feng, A., Liu, D., Xing, Y., Wang, J., Gao, F., Lu, J., Jiang, Y., Li, H., Li, X., Yu, K., Dong, R., Gu, S., Li, Y., Xie, X., Juefei-Xu, F., Khomh, F., Yoshie, O., Chen, Q., Teodoro, D., Liu, N., Goebel, R., Ma, L., Marrese-Taylo...

    2025 arXiv

  2. [9]

    Physical Review Physics Education Research21(2), 020157 (2025) https: //doi.org/10.1103/6fmx-bsnl

    Tschisgale, P., Maus, H., Kieser, F., Kroehs, B., Petersen, S., Wulff, P.: Eval- uating gpt- and reasoning-based large language models on physics olympiad problems: Surpassing human performance and implications for educational assess- ment. Physical Review Physics Education Re...

    2025 doi

  3. [10]

    Physics Education59(2), 025010 (2024) https://doi.org/10.1088/1361-6552/ad1fa2

    Yeadon, W., Hardy, T.: The impact of ai in physics education: A comprehensive review from gcse to university levels. Physics Education59(2), 025010 (2024) https://doi.org/10.1088/1361-6552/ad1fa2

    2024 doi

  4. [11]

    Dao, X.-Q., Le, N.-B., Phan, X.-D., Ngo, B.-B., Vo, T.-D.: Evaluation of chatgpt and microsoft bing ai chat performances on physics exams of vietnamese national high school graduation examination (2023) arXiv:2306.04538

    2023 arXiv

  5. [12]

    Kortemeyer, G.: Could an artificial-intelligence agent pass an introductory physics course? Physical Review Physics Education Research19(1), 010132 (2023) https: //doi.org/10.1103/PhysRevPhysEducRes.19.010132

    2023 doi

  6. [13]

    Frenkel, M., Emara, H.: Chatgpt & mechanical engineering: Examining per- formance on the fe mechanical engineering and undergraduate exams (2023) arXiv:2309.15866

    2023 arXiv

  7. [14]

    https://openai.com/chatgpt/overview/

    OpenAI: ChatGPT: Overview. https://openai.com/chatgpt/overview/. Accessed 2025-08-19 (2025)

    2025

  8. [15]

    Physics Education59(4), 045005 (2024) https://doi.org/ 10.1088/1361-6552/ad3d21

    Gregorcic, B., Polverini, G., Sarlah, A.: Chatgpt as a tool for honing teachers’ 19 socratic dialogue skills. Physics Education59(4), 045005 (2024) https://doi.org/ 10.1088/1361-6552/ad3d21

    2024 doi

  9. [16]

    Computers and Education: Artificial Intelligence5, 100318 (2024) https://doi.org/10.1016/j.caeai.2024.100318

    Sirnoorkar, A., Zollman, D., Laverty, J.T., Magana, A.J., Rebello, S., Bryan, L.A.: Student and ai responses to physics problems examined through the lenses of sensemaking and mechanistic reasoning. Computers and Education: Artificial Intelligence5, 100318 (2024) https://doi.o...

    2024

  10. [17]

    Scientific Reports15(1), 17458 (2025) https: //doi.org/10.1038/s41598-025-97652-6

    Kestin, G., Miller, K., Klales, A., Milbourne, T., Ponti, G.: Ai tutoring outper- forms in-class active learning: an rct introducing a novel research-based design in an authentic educational setting. Scientific Reports15(1), 17458 (2025) https: //doi.org/10.1038/s41598-025-97652-6

    2025 doi

  11. [18]

    Frontiers in Education8(2024) https://doi.org/10.3389/feduc.2023.1330486

    Wang, K.D., Burkholder, E., Wieman, C., Salehi, S., Haber, N.: Examining the potential and pitfalls of chatgpt in science and engineering problem-solving. Frontiers in Education8(2024) https://doi.org/10.3389/feduc.2023.1330486

    2024

  12. [19]

    American Journal of Physics91(12), 955–958 (2023) https://doi.org/10.1119/5

    Kumar, T., Kats, M.A.: Chatgpt-4 with code interpreter can be used to solve introductory college-level vector calculus and electromagnetism problems. American Journal of Physics91(12), 955–958 (2023) https://doi.org/10.1119/5. 0182627

    2023 doi

  13. [20]

    Physics Education 58(3), 035027 (2023) https://doi.org/10.1088/1361-6552/acc5cf

    Yeadon, W., Inyang, O.-O., Mizouri, A., Peach, A., Testrow, C.P.: The death of the short-form physics essay in the coming ai revolution. Physics Education 58(3), 035027 (2023) https://doi.org/10.1088/1361-6552/acc5cf

    2023 doi

  14. [21]

    Scientific Reports14, 23285 (2024) https://doi.org/10.1038/s41598-024-73634-y

    Yeadon, W., Peach, A., Testrow, C.: A comparison of human, gpt-3.5, and gpt- 4 performance in a university-level coding course. Scientific Reports14, 23285 (2024) https://doi.org/10.1038/s41598-024-73634-y

    2024 doi

  15. [22]

    Kilde-Westberg, S., Johansson, A., Enger, J.: Generative ai as a lab partner: A case study. Phys. Rev. Phys. Educ. Res., (2025) https://doi.org/10.1103/ ggy1-3kjk

    2025

  16. [23]

    Low, A., Kalender, Z.Y.: Data dialogue with chatgpt: Using code interpreter to simulate and analyse experimental data (2023) arXiv:2311.12415

    2023 arXiv

  17. [24]

    Besta, M., Barth, J., Schreiber, E., Kubicek, A., Catarino, A., Gerstenberger, R., Nyczyk, P., Iff, P., Li, Y., Houliston, S., Sternal, T., Copik, M., Kwa´ sniewski, G., M¨ uller, J., Flis, L., Eberhard, H., Chen, Z., Niewiadomski, H., Hoefler, T.: Reasoning language models: A...

    2025 arXiv

  18. [25]

    Xu, F., Hao, Q., Zong, Z., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., Ouyang, T., Meng, F., Shao, C., Yan, Y., Yang, Q., Song, Y., Ren, S., Hu, X., Li, Y., Feng, J., Gao, C., Li, Y.: Towards large reasoning models: A survey of reinforced reasoning with large language mo...

    2025 arXiv

  19. [26]

    Yoon, D., Kim, S., Yang, S., Kim, S., Kim, S., Kim, Y., Choi, E., Kim, Y., Seo, M.: Reasoning models better express their confidence (2025) arXiv:2505.14489

    2025

  20. [27]

    Zhang, X., Dong, Y., Wu, Y., Huang, J., Jia, C., Fernando, B., Shou, M.Z., Zhang, L., Liu, J.: Physreason: A comprehensive benchmark towards physics- based reasoning (2025) arXiv:2502.12054

    2025 arXiv

  21. [28]

    https://ml-site.cdn-apple

    Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., Farajtabar, M.: The Illusion of Thinking: Understanding the Strengths and Limitations of Reason- ing Models via the Lens of Problem Complexity (2025). https://ml-site.cdn-apple. com/papers/the-illusion-of-thinking.pdf

    2025

  22. [29]

    In: Proceedings of the IEEE International Conference on Big Data (BigData), pp

    Wu, J., Gan, W., Chen, Z., Wan, S., Yu, P.S.: Multimodal large language models: A survey. In: Proceedings of the IEEE International Conference on Big Data (BigData), pp. 2247–2256 (2023). https://doi.org/10.1109/BigData59044.2023. 10386743

    2023

  23. [30]

    Wang, J., Jiang, H., Liu, Y., Ma, C., Zhang, X., Pan, Y., Liu, M., Gu, P., Xia, S., Li, W., Zhang, Y., Wu, Z., Liu, Z., Zhong, T., Ge, B., Zhang, T., Qiang, N., Hu, X., Jiang, X., Zhang, X., Zhang, W., Shen, D., Liu, T., Zhang, S.: A comprehensive review of multimodal large la...

    2024 arXiv

  24. [31]

    Learning and Individual Differences118, 102601 (2025) https://doi.org/10.1016/j.lindif.2024.102601

    Bewersdorff, A., Hartmann, C., Hornberger, M., Se ˘ßler, K., Bannert, M., Kas- neci, E., Kasneci, G., Zhai, X., Nerdel, C.: Taking the next step with generative artificial intelligence: The transformative role of multimodal large language mod- els in science education. Learnin...

    2025

  25. [32]

    (eds.): Multiple Representations in Physics Education vol

    Treagust, D.F., Duit, R., Fischer, H.E. (eds.): Multiple Representations in Physics Education vol. 10. Springer, Cham (2017). https://doi.org/10.1007/ 978-3-319-58914-5

    2017

  26. [33]

    Polverini, G., Gregorcic, B.: Performance of chatgpt on the test of understanding graphs in kinematics. Phys. Rev. Phys. Educ. Res.20, 010109 (2024) https:// doi.org/10.1103/PhysRevPhysEducRes.20.010109

    2024 doi

  27. [34]

    International Journal of Advanced Multi- disciplinary Research and Studies5(1), 7–18 (2025) https://doi.org/10.62225/ 2583049X.2025.5.1.3610

    Bessas, N., Tzanaki, E., Vavougios, D., Plagianakos, V.P.: Comparative analy- sis of chatgpt and gemini; implications for junior high school physics education: Opportunities and ethical challenges. International Journal of Advanced Multi- disciplinary Research and Studies5(1),...

    2025

  28. [35]

    chatgpt vs

    Jiang, Q., Gao, Z., Karniadakis, G.E.: Deepseek vs. chatgpt vs. claude: A com- parative study for scientific computing and scientific machine learning tasks. Theoretical and Applied Mechanics Letters15(2), 100583 (2025) https://doi.org/ 10.1016/j.taml.2025.100583 21

    2025

  29. [36]

    In: Proceedings of the Physics Education Research Conference (PERC), Boston, MA (2024)

    Polverini, G., Gregorcic, B.: Performance of freely available vision-capable chat- bots on the test for understanding graphs in kinematics. In: Proceedings of the Physics Education Research Conference (PERC), Boston, MA (2024). https: //doi.org/10.1119/perc.2024.pr.Polverini

    2024 doi

  30. [37]

    Polverini, G., Melin, J., ¨Onerud, E., Gregorcic, B.: Performance of chatgpt on tasks involving physics visual representations: The case of the brief electricity and magnetism assessment. Phys. Rev. Phys. Educ. Res.21, 010154 (2025) https: //doi.org/10.1103/PhysRevPhysEducRes....

    2025 doi

  31. [38]

    Artificial Intelligence in Physics Courses to Support Active Learning (ICSLT ’24 proceedings), 68–75 (2024) https: //doi.org/10.1145/3678610.3678631

    Robledo-Rella, V., Gonzalez-Nucamendi, A., Neri, L., Garc´ ıa-Castel´ an, R.M.G., Noguez, J., Valverde-Rebaza, J.: Can we trust ai chatbots to teach university physics? a performance comparison of ai chatbots. Artificial Intelligence in Physics Courses to Support Active Learni...

    2024

  32. [39]

    Kortemeyer, G., Babayeva, M., Polverini, G., Widenhorn, R., Gregorcic, B.: Multilingual performance of a multimodal artificial intelligence system on mul- tisubject physics concept inventories. Phys. Rev. Phys. Educ. Res.21, 020101 (2025) https://doi.org/10.1103/98hg-rkrf

    2025 doi

  33. [40]

    Frontiers in Education9, 1452414 (2024) https://doi.org/10.3389/feduc.2024

    Polverini, G., Gregorcic, B.: Evaluating vision-capable chatbots in interpreting kinematics graphs: a comparative study of free and subscription-based models. Frontiers in Education9, 1452414 (2024) https://doi.org/10.3389/feduc.2024. 1452414

    2024 doi

  34. [41]

    Technical report, OECD Publishing (2023)

    Organisation for Economic Co-operation and Development: Education at a Glance 2023: OECD Indicators. Technical report, OECD Publishing (2023). Accessed: 2025-08-19. https://www.oecd.org/en/publications/2023/09/ education-at-a-glance-2023 581c9602.html

    2023

  35. [42]

    PhD thesis, Uppsala University, Department of Physics and Astronomy, Physics Didactics, Uppsala, Sweden (May 2025)

    Winter, J.: Educating pre-service physics teachers in england: The need for knowl- edge transformation. PhD thesis, Uppsala University, Department of Physics and Astronomy, Physics Didactics, Uppsala, Sweden (May 2025). https://uu. diva-portal.org/smash/get/diva2:1945934/FULLT...

    2025

  36. [43]

    Teaching & Learning Inquiry10(2022) https://doi.org/10.20343/ teachlearninqu.10.3

    Paris, B.M.: Instructors’ perspectives of challenges and barriers to providing effec- tive feedback. Teaching & Learning Inquiry10(2022) https://doi.org/10.20343/ teachlearninqu.10.3

    2022

  37. [44]

    (eds.) Constructivist Instruction: Success or Failure?, pp

    Clark, R.E.: How much and what type of guidance is optimal for learning from instruction? In: Tobias, S., Duffy, T.M. (eds.) Constructivist Instruction: Success or Failure?, pp. 158–183. Routledge/Taylor & Francis Group, ??? (2009)

    2009

  38. [45]

    The Educational Forum75(3), 210–227 (2011) https://doi.org/10.1080/00131725.2011.577669 22

    Tierney, R.D., Simon, M., Charland, J.: Being fair: Teachers’ interpretations of principles for standards-based grading. The Educational Forum75(3), 210–227 (2011) https://doi.org/10.1080/00131725.2011.577669 22

    2011

  39. [46]

    Educational Research Quarterly31(3), 39–56 (2008)

    Melnick, S.A., Meister, D.G.: A comparison of beginning and experienced teachers’ concerns. Educational Research Quarterly31(3), 39–56 (2008)

    2008

  40. [47]

    Physical Review Physics Education Research19(2), 020163 (2023) https://doi.org/10.1103/PhysRevPhysEducRes.19.020163

    Kortemeyer, G.: Toward ai grading of student problem solutions in introductory physics: A feasibility study. Physical Review Physics Education Research19(2), 020163 (2023) https://doi.org/10.1103/PhysRevPhysEducRes.19.020163

    2023 doi

  41. [48]

    Physi- cal Review Physics Education Research20(2), 020144 (2024) https://doi.org/10

    Kortemeyer, G., N¨ ohl, J., Onishchuk, D.: Grading assistance for a handwritten thermodynamics exam using artificial intelligence: An exploratory study. Physi- cal Review Physics Education Research20(2), 020144 (2024) https://doi.org/10. 1103/PhysRevPhysEducRes.20.020144

    2024

  42. [49]

    Mok, R., Akhtar, F., Clare, L., Li, C., Ida, J., Ross, L., Campanelli, M.: Using ai large language models for grading in education: A hands-on test for physics (2024) arXiv:2411.13685

    2024

  43. [50]

    Physical Review Physics Education Research21(1), 010126 (2025) https://doi.org/10

    Chen, Z., Wan, T.: Grading explanations of problem-solving process and gen- erating feedback using large language models at human-level accuracy. Physical Review Physics Education Research21(1), 010126 (2025) https://doi.org/10. 1103/PhysRevPhysEducRes.21.010126

    2025

  44. [51]

    Physical Review Physics Education Research20(1), 010152 (2024) https://doi.org/10.1103/PhysRevPhysEducRes.20.010152

    Wan, T., Chen, Z.: Exploring generative ai assisted feedback writing for students’ written responses to a physics conceptual question with prompt engineering and few-shot learning. Physical Review Physics Education Research20(1), 010152 (2024) https://doi.org/10.1103/PhysRevPh...

    2024 doi

  45. [52]

    EPJ Quantum Technology 12(1), 33 (2025) https://doi.org/10.1140/epjqt/s40507-025-00334-5

    Krupp, L., Bley, J., Gobbi, I., Geng, A., M¨ uller, S., Suh, S., Moghiseh, A., Medina, A.C., Bartsch, V., Widera, A., Ott, H., Lukowicz, P., Karolus, J., Kiefer-Emmanouilidis, M.: Llm-generated tips rival expert-created tips in help- ing students answer quantum-computing quest...

    2025 doi

  46. [53]

    Guo, S., Latif, E., Zhou, Y., Huang, X., Zhai, X.: Using generative ai and multi- agents to provide automatic feedback (2024) arXiv:2411.07407

    2024 arXiv

  47. [54]

    Education Sciences15(2), 233 (2025) https://doi.org/10.3390/educsci15020233

    Clark, A.K., Hirt, A., Whitcomb, D., Thompson, W.J., Wine, M., Karvonen, M.: Artificial intelligence in science and mathematics assessment for students with disabilities: Opportunities and challenges. Education Sciences15(2), 233 (2025) https://doi.org/10.3390/educsci15020233

    2025 doi

  48. [55]

    In: 2024 Physics Education Research Conference Proceedings, pp

    El-Adawy, S., MacDonagh, A., Abdelhafez, M.: Exploring large language mod- els as formative feedback tools in physics. In: 2024 Physics Education Research Conference Proceedings, pp. 126–131. American Association of Physics Teachers, Boston, MA (2024). https://doi.org/10.1119/...

    2024 doi

  49. [56]

    Physical Review 23 Physics Education Research21(1), 010136 (2025) https://doi.org/10.1103/ PhysRevPhysEducRes.21.010136

    Kortemeyer, G., N¨ ohl, J.: Assessing confidence in ai-assisted grading of physics exams through psychometrics: An exploratory study. Physical Review 23 Physics Education Research21(1), 010136 (2025) https://doi.org/10.1103/ PhysRevPhysEducRes.21.010136

    2025

  50. [57]

    https://www

    OpenAI: GPT-4o (Omni) Math Tutoring Demo on Khan Academy. https://www. youtube.com/watch?v=ivxzcocyu m. Accessed June 18, 2025 (2024)

    2024

  51. [58]

    https://www.youtube

    Google DeepMind: Math & Physics with AI — Gemini. https://www.youtube. com/watch?v=k4px1vaxaai. Accessed June 18, 2025 (2023)

    2023

  52. [59]

    Zhang, X., Cao, J., You, C.: Counting ability of large language models and impact of tokenization (2024) arXiv:2410.19730

    2024 arXiv

  53. [60]

    Journal of AI and Education (2025)

    Ahmed, F.: The digital divide and ai in education: Addressing equity and acces- sibility. Journal of AI and Education (2025). Lahore University of Management Sciences (LUMS)

    2025

  54. [61]

    https://www.unesco.org/en/articles/ ai-literacy-and-new-digital-divide-global-call-action

    UNESCO: AI literacy and the new Digital Divide – A Global Call for Action. https://www.unesco.org/en/articles/ ai-literacy-and-new-digital-divide-global-call-action. Last update: 28 February 2025 (2024)

    2024

  55. [62]

    https://docs.anthropic.com/en/docs/about-claude/ models/overview

    Anthropic: Models overview. https://docs.anthropic.com/en/docs/about-claude/ models/overview. Accessed: 2025-06-13 (2025)

    2025

  56. [63]

    https://ai.google.dev/gemini-api/docs/models?utm source=chatgpt.com

    Google AI: Gemini models. https://ai.google.dev/gemini-api/docs/models?utm source=chatgpt.com. Accessed: 2025-06-13 (2025)

    2025

  57. [64]

    https://ai.google.dev/gemma/docs

    Google DeepMind: Gemma models overview. https://ai.google.dev/gemma/docs. Last updated: 2025-03-04; Accessed: 2025-06-13 (2025)

    2025

  58. [65]

    https://platform.openai.com/docs/models?utm source= chatgpt.com

    OpenAI: Models. https://platform.openai.com/docs/models?utm source= chatgpt.com. Accessed: 2025-08-19 (2025)

    2025

  59. [66]

    Physical Review Special Topics - Physics Education Research2(1), 010105 (2006) https: //doi.org/10.1103/PhysRevSTPER.2.010105

    Ding, L., Chabay, R., Sherwood, B., Beichner, R.: Evaluating an electricity and magnetism assessment tool: Brief electricity and magnetism assessment. Physical Review Special Topics - Physics Education Research2(1), 010105 (2006) https: //doi.org/10.1103/PhysRevSTPER.2.010105

    2006 doi

  60. [67]

    Research in Science & Technological Education35(2), 238–260 (2017) https://doi.org/10.1080/02635143.2017.1310094

    Kaltakci-Gurel, D., Eryilmaz, A., McDermott, L.C.: Development and application of a four-tier test to assess pre-service physics teachers’ misconceptions about geometrical optics. Research in Science & Technological Education35(2), 238–260 (2017) https://doi.org/10.1080/026351...

    2017

  61. [68]

    American Journal of Physics70(3), 238–251 (2002) https://doi.org/10

    Cataloglu, E., Robinett, R.W.: Testing the development of student conceptual and visualization understanding in quantum mechanics through the undergraduate career. American Journal of Physics70(3), 238–251 (2002) https://doi.org/10. 1119/1.1405509 24

    2002

  62. [69]

    American Journal of Physics62(8), 750–762 (1994) https://doi.org/10.1119/1.17449

    Beichner, R.J.: Testing student interpretation of kinematics graphs. American Journal of Physics62(8), 750–762 (1994) https://doi.org/10.1119/1.17449

    1994 doi

  63. [70]

    TechTarget SearchEnterpriseAI (2025)

    Garn, D.: Understanding the Role of Temperature Set- tings in AI Output. TechTarget SearchEnterpriseAI (2025). https://www.techtarget.com/searchenterpriseai/tip/ Understanding-the-role-of-temperature-settings-in-AI-output

    2025

  64. [72]

    Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Win- ter, C., Hesse, C., Chen, M., Sigler, E.,...

    2020 arXiv

  65. [73]

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models (2022) arXiv:2201.11903

    2022 arXiv

  66. [74]

    Physical Review Physics Education Research 20(1), 010104 (2024) https://doi.org/10.1103/PhysRevPhysEducRes.20.010104

    Wheatley, C., Wells, J., Stewart, J.: Applying module analysis to the brief elec- tricity and magnetism assessment. Physical Review Physics Education Research 20(1), 010104 (2024) https://doi.org/10.1103/PhysRevPhysEducRes.20.010104

    2024 doi

  67. [75]

    Physical Review Physics Education Research 13(2), 020111 (2017) https://doi.org/10.1103/PhysRevPhysEducRes.13.020111

    Zavala, G., Tejeda, S., Barniol, P., Beichner, R.J.: Modifying the test of under- standing graphs in kinematics. Physical Review Physics Education Research 13(2), 020111 (2017) https://doi.org/10.1103/PhysRevPhysEducRes.13.020111

    2017 doi

  68. [76]

    https://platform.openai.com/docs/guides/reasoning

    OpenAI: Reasoning. https://platform.openai.com/docs/guides/reasoning. Accessed: 2025-08-19 (2025)

    2025

  69. [77]

    https://platform.openai.com/docs/guides/ reasoning-best-practices

    OpenAI: Reasoning Best Practices. https://platform.openai.com/docs/guides/ reasoning-best-practices. Accessed: 2025-08-19 (2025) 25

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.