Pith. sign in

REVIEW 4 major objections 4 minor 46 references

PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that current large language models, including reasoning-specialized models, consistently fail to apply fundamental physics principles, scoring between 8.9% and 45.8% accuracy on its new 380-problem PhySense benchmark.

desk verdict PhySense has the right instinct, but the evaluation is load-bearing on an unreleased, unvalidated answer key, and at least one showcased ground truth looks wrong. read the letter →

arxiv 2505.24823 v1 pith:UPPGE623 submitted 2025-05-30 cs.LG cs.AIcs.CL

Yinggan Xu , Yue Liu , Zhiqiang Gao , Changnan Peng , Di Luo This is my paper
classification cs.LGcs.AIcs.CL
keywords physicsreasoningbenchmarkprinciple-basedlargelanguagemodelssymmetrytokenefficiencyoverthinkingscientificevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces PhySense, a 380-problem benchmark of physics questions designed so that human experts can solve each one quickly by applying a fundamental principle such as symmetry, dimensional analysis, conservation, or topology, while brute-force calculation is long and error-prone. Evaluating seven state-of-the-art LLMs under three prompting strategies, the authors report that no model exceeds 45.8% accuracy and most fall far lower, with reasoning models consuming around 10,000 tokens per problem compared to an estimated roughly 100 tokens for an expert. Hints naming the relevant principle and instructions to avoid computation barely move accuracy, which the authors interpret as a systematic deficit in applying principles correctly rather than a failure to recognize when they apply. If the finding holds, it locates a specific gap in current LLM reasoning, namely over-thinking without principle-led simplification, and points to training objectives that reward short, verifiable principle-based derivations.

What carries the argument

The central object is PhySense itself, a set of 380 physics problems built around 19 named problem models, each deliberately constructed so that a single guiding principle yields a short solution while brute-force computation is long. The argument is carried by combining two metrics, accuracy and token usage, across three controlled prompting protocols (zero-shot, hint, and no-computation), so that a model's success is judged not only by its final answer but by whether it reasons with the expected economy and principle-stated derivation.

What would settle it

Evolve the L=100 quantum spin chain of Example 3 exactly or with a well-controlled tensor-network method under the given Hamiltonian from t=0 to t=100 and compute the expectation values ⟨Z39Y40⟩ and ⟨Y90Z91⟩; if they differ, the benchmark's ground-truth choice (b) is wrong, so the accuracy numbers for that problem and any aggregate that includes it are unreliable.

Watch

Extended reading notes

Core claim

PhySense is the claim that current LLMs, including reasoning-specialized models, cannot reliably reproduce the concise, principle-first reasoning that physicists use, even when explicitly prompted with the relevant principle. The benchmark consists of 380 novel, human-curated problems across 19 problem models spanning symmetry, dimensional analysis, limiting cases, conservation laws, and topology, each solvable by a short principle-based argument. Across seven LLMs under zero-shot, hint, and no-computation prompting, average accuracy ranges from 8.9% to 45.8%, and reasoning models generate roughly ten thousand tokens per solution, about two orders of magnitude more than the authors' estimate of expert output. The paper's explanation is that LLMs' primary errors are misapplications of principles, such as invoking the wrong symmetry or failing to identify all relevant symmetries, rather than ignorance of principles, which is why hints give only marginal gains.

Load-bearing premise

The load-bearing assumption is that the benchmark's ground-truth answers are correct, including Example 3's claim that choice (b) follows from time-reversal symmetry; if that answer is wrong, the accuracy scores built on it are unreliable.

Editorial extensions

If this is right

  • If PhySense is a faithful probe, then any LLM claiming expert-level physics reasoning must demonstrate both high accuracy and near-expert token usage on principle-first problems, not just correct final answers.
  • The marginal impact of hint and no-computation prompts implies that post-hoc prompting cannot repair the deficit, so improving principle-based reasoning likely requires training-time changes.
  • The performance variation across the 19 problem models suggests domain-specific weaknesses, such as low accuracy on topological and fermionic problems, that can guide targeted data generation and evaluation.
  • Token counts of roughly 10,000 per problem imply practical cost and latency penalties for scientific use, so the reported gap is not only about correctness but also about efficiency.
  • The benchmark's design offers a reusable template for evaluating whether AI systems reason by parsimonious principles in other expert domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ground-truth answers are validated independently, PhySense could be repurposed as a training signal: reward functions that penalize long token output and reward explicit statements of the governing principle might push models toward expert-like reasoning.
  • The reported token-efficiency gap could partly reflect the API and chain-of-thought scaffolding used in the evaluation rather than a fundamental model limitation; a controlled study using the same interface with human experts would make the roughly 100-token estimate falsifiable.
  • The flagged Example 3 ground truth, which attributes choice (b) to time-reversal symmetry, is a testable linchpin: exact numerical time evolution of the L=100 chain would confirm or refute whether that answer is correct, and any error there would lower the reliability of the affected accuracy scores.
  • The benchmark's heavy emphasis on symmetry-based problems may under-represent other forms of principle-based reasoning, such as order-of-magnitude estimation or asymptotic analysis, so the failure rate on PhySense is not necessarily the full measure of principle reasoning ability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces PhySense, a benchmark of 380 human-curated physics problems in which each item is intended to be solvable by a short, principle-based argument (symmetry, dimensional analysis, conservation laws, topology, etc.) rather than by long computation. The authors evaluate seven LLMs under three prompting schemes (zero-shot, hint, and no-computation), reporting accuracy and token usage per problem category and overall. They conclude that reasoning-focused LLMs outperform non-reasoning models but that all tested models remain far below expert human performance in accuracy and use roughly 10^4 tokens per problem versus an estimated 10^2 for human experts. The main evidence consists of accuracy and token tables (Tables 1 and 2), supported by worked examples in the text and appendix.

Significance. If the benchmark, answer keys, and evaluation code were publicly available and independently validated, PhySense would be a useful addition to physics-reasoning evaluation: it targets a distinct failure mode, short principle-based reasoning, rather than long calculations, and it jointly measures accuracy and computational cost. The paper gives credit for a meaningful problem design and for evaluating current models across multiple prompt types. However, the empirical contribution is currently not reproducible because the dataset and answer key are not released, and the key load-bearing claims rest on an unmeasured human baseline and at least one suspect ground-truth answer. The finding that LLMs perform poorly is partly by construction, since the problems were explicitly curated to be easy for experts and hard for LLMs that do not use the intended principle; this does not invalidate the measurements but should be stated carefully as a property of an adversarially selected benchmark.

major comments (4)
  1. [Section 5.4, Example 3; Appendix A.2] The asserted ground truth (a,b,c) for Example 3 rests on a time-reversal argument for choice (b), but no derivation is supplied and the claim is doubtful for the stated Hamiltonian. The reflection R(j -> 101-j) maps Z39Y40 to Z62Y61, not to Y90Z91; the global spin-flip leaves both sides of (b) invariant but does not relate them; and the antiunitary time-reversal T=K commutes with both H0 and H(t) only in the sense of mapping the state at time t to a state at time -t, which does not imply equality of these expectation values at t=100. Appendix A.2 shows the o4-mini-high model returning {a,c}, which is exactly what spin-flip and reflection imply, and the paper marks this answer wrong solely because it omits (b). Since Table 1's DynCon column and the overall averages are computed against this key, the supporting accuracy numbers are contaminated unless the time-reversal step is actually demonstrated. Because the full answer key is not released, the remaining 379 keys cannot be audited, which makes this the central correctness risk of the paper.
  2. [Section 3; Tables 1 and 3] The central artifact of a benchmark paper, the 380-item problem set and its answer key, is not released. There is no data availability statement, repository link, or evaluation script. The claim in Section 3 that all problems are newly crafted and cannot be found elsewhere is therefore unverifiable, and readers cannot reproduce the accuracies in Table 1 or audit the other ground-truth answers. For this paper to support its conclusions, the full dataset, answer keys, per-problem principle labels, and extraction/evaluation scripts should be made publicly available.
  3. [Section 5.2 and Section 7] The comparison to expert human performance is not measured. The paper states that human physicists solve the same problems in about 10^2 tokens and concludes that all models are 'substantially below expert human performance,' but no human-subject accuracy, no human token measurement, and no experimental protocol are reported. The 10^2 figure appears to be anecdotal. Since the paper's headline claims are the accuracy gap and the token-efficiency gap relative to experts, this baseline is load-bearing; it should be measured or the claims should be restricted to absolute model accuracy and token counts without expert comparison.
  4. [Section 4.2 and Section 5.2] Token usage is reported for all generated answers, including incorrect ones, without conditioning on correctness. As a result, the 'efficiency' metric conflates verbosity with reasoning cost: a model that produces a long wrong solution is counted as inefficient in the same way as a model that produces a long correct solution, while a model that writes a short wrong answer appears efficient. Since the paper's efficiency claim concerns principle-based reasoning, token counts should be reported for correct solutions only, or should be accompanied by accuracy-conditioned efficiency measures.
minor comments (4)
  1. [Throughout] There are several typos and spelling errors, including 'benmark' in Section 1, 'cabability' in Section 3, and 'hundered' and 'reaoning' in Section 5.2; these should be corrected in revision.
  2. [Tables 1 and 2] The number of problems in each category is not stated, and the averaging scheme is not defined. The percentages suggest 20 items per set, but the paper should state explicitly whether the AVG column is a simple average over categories or a per-problem weighted average.
  3. [References] Reference [41] lacks a title and author list, and several other references have inconsistent formatting; the reference list should be brought to a uniform, complete style.
  4. [Appendix D] The prompt specifications in Appendix D contain LaTeX artifacts and irregular spacing that make them hard to read; these listings should be cleaned up for clarity.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the paper reports benchmark measurements rather than deriving results from fitted inputs, and its only self-citations are contextual; the mild selection of hard problems and an unvalidated answer key are validity risks, not circular reductions.

full rationale

PhySense is an empirical benchmark study, not a derivation. The central claim that LLMs score 8.9%-45.8% and use roughly 10^4 tokens is a measurement against a fixed answer key; no model parameter is fitted to the benchmark and then reused as a prediction. The benchmark's design goal is explicitly to be 'easily solvable by experts using guiding principles, yet deceptively difficult for LLMs without principle-first reasoning' (Abstract), which creates a mild selection bias: low LLM accuracy is partially a consequence of hand-picking problems that resist non-principle methods. This is a known limitation of benchmark construction, but it is not a circular derivation because the measured failure is not logically forced—models could still fail or succeed for many reasons, and the paper reports actual API outputs. The only author-overlapping references are [8] (introductory citation for LLM physics applications) and [10] (related-work comparison, TP-Bench); neither is load-bearing for the main claim, and no uniqueness theorem or fitted parameter is imported from them. The possible invalidity of the Example 3 ground truth (choice (b) attributed to time-reversal symmetry without a derivation, Appendix A.2) is a correctness and validation risk for the answer key, not a circularity: the benchmark computes accuracy relative to its own key by definition, which is how benchmarks operate. On the circularity axis, the paper is self-contained and scores 2 only for the mild by-construction selection of hard problems and the contextual self-citation; there is no reduction of a prediction to its input.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark itself rests on the correctness of the curated answers and on the choice of metrics. The 5% numerical tolerance and the anecdotal ~100-token human baseline are hand-set anchors for the evaluation. No new physical entities are introduced.

free parameters (2)
  • Numerical answer tolerance = 5%
    Hand-chosen threshold for comparing numerical answers; affects accuracy scores.
  • Human token baseline = ~100 tokens
    Anecdotal estimate, not measured, used to contrast with LLM token usage.
assumptions (3)
  • domain assumption The 380 curated problems have correct, principle-based ground-truth solutions.
    The benchmark's validity depends on the expert answers being correct; Example 3(b) raises doubt.
  • domain assumption Physics principles such as symmetry, dimensional analysis, and conservation are the appropriate shortcuts for every problem.
    Section 3.1; the entire dataset is built around this premise.
  • domain assumption Token count is a valid proxy for reasoning complexity and principle alignment.
    Section 4.2; they use token usage as the efficiency metric without validation against human reasoning traces.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models." pith.science (2026). https://pith.science/paper/UPPGE623

@misc{pith2026250524823,
  author       = {Pith},
  title        = {Pith review of: PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UPPGE623}},
  note         = {Machine review of arXiv:2505.24823}
}
read the original abstract

Large language models (LLMs) have rapidly advanced and are increasingly capable of tackling complex scientific problems, including those in physics. Despite this progress, current LLMs often fail to emulate the concise, principle-based reasoning characteristic of human experts, instead generating lengthy and opaque solutions. This discrepancy highlights a crucial gap in their ability to apply core physical principles for efficient and interpretable problem solving. To systematically investigate this limitation, we introduce PhySense, a novel principle-based physics reasoning benchmark designed to be easily solvable by experts using guiding principles, yet deceptively difficult for LLMs without principle-first reasoning. Our evaluation across multiple state-of-the-art LLMs and prompt types reveals a consistent failure to align with expert-like reasoning paths, providing insights for developing AI systems with efficient, robust and interpretable principle-based scientific reasoning.

Figures

Figures reproduced from arXiv: 2505.24823 by the authors.

Figure 1
Figure 1. Illustrating how LLMs use lengthy, complex reasoning for physics problems intuitively [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distribution of physics principles in the dataset. Novel problems from human experts Although the underlying concepts in our problem set are widely available online, we have crafted entirely new questions with physicists from top universi￾ties that cannot be found elsewhere, ensuring that LLMs have not been exposed to similar problems. This novelty is essential for testing an LLM’s ability to generalize the applicat… view at source ↗
Figure 3
Figure 3. Average accuracy and token usage for different models. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Zeroshot accuracy of LLMs under three difficulties. We report the model performance in terms of accu￾racy score for each section and an average accuracy over all problems. We quantify model performance using accuracy percentage, calculated for each dis￾tinct problem ca…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 9 canonical work pages

  1. [1]

    A comprehensive survey of scientific large language models and their applications in scientific discovery, 2024

    Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, and Jiawei Han. A comprehensive survey of scientific large language models and their applications in scientific discovery, 2024. URL https://arxiv.org/abs/2406.10833

    arXiv 2024

  2. [2]

    Romera-Paredes, M

    B. Romera-Paredes, M. Barekatain, A. Novikov, and et al. Mathematical discoveries from program search with large language models. Nature, 625:468–475, Jan 2024. doi: 10.1038/ s41586-023-06924-6. URL https://doi.org/10.1038/s41586-023-06924-6

    doi:10.1038/s41586-023-06924-6 2024

  3. [3]

    The ai scientist: Towards fully automated open-ended scientific discovery, 2024

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery, 2024. URL https: //arxiv.org/abs/2408.06292

    arXiv 2024

  4. [4]

    Llm-sr: Scientific equation discovery via programming with large language models,

    Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. Llm-sr: Scientific equation discovery via programming with large language models,

    work page

  5. [5]

    Brenner, and Eun-Ah Kim

    Haining Pan, Nayantara Mudur, William Taranto, Maria Tikhanovskaya, Subhashini Venu- gopalan, Yasaman Bahri, Michael P. Brenner, and Eun-Ah Kim. Quantum many-body physics calculations with large language models. Communications Physics , 8(1):49, De- cember 2025. doi: 10.1038/s42005-025-01956-y. URL https://doi.org/10.1038/ s42005-025-01956-y

    doi:10.1038/s42005-025-01956-y 2025

  6. [6]

    Llm-feynman: Leveraging large language models for universal scientific formula and theory discovery

    Zhilong Song, Minggang Ju, Chunjin Ren, Qiang Li, Chongyi Li, Qionghua Zhou, and Jinlan Wang. Llm-feynman: Leveraging large language models for universal scientific formula and theory discovery. arXiv preprint arXiv:2503.06512, 2025

    arXiv 2025

  7. [7]

    Large physics models: Towards a collaborative approach with large language models and foundation models

    Kristian G Barman, Sascha Caron, Emily Sullivan, Henk W de Regt, Roberto Ruiz de Austri, Mieke Boon, Michael Färber, Stefan Fröse, Faegheh Hasibi, Andreas Ipp, et al. Large physics models: Towards a collaborative approach with large language models and foundation models. arXiv preprint arXiv:2501.05382, 2025

    arXiv 2025

  8. [8]

    Advancing ai-scientist understanding: Making llm think like a physicist with interpretable reasoning.arXiv preprint arXiv:2504.01911, 2025

    Yinggan Xu, Hana Kimlee, Yijia Xiao, and Di Luo. Advancing ai-scientist understanding: Making llm think like a physicist with interpretable reasoning.arXiv preprint arXiv:2504.01911, 2025

    arXiv 2025

Show all 46 references
  1. [9]

    Alex Hofer, Jan Humplik, Atil Iscen, Mithun George Jacob, Deepali Jain, Ryan Julian, Dmitry Kalash- nikov, M

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, Steven Bohez, Konstantinos Bousmalis, Anthony Brohan, Thomas Buschmann, Arunkumar B...

    2025 arXiv

  2. [10]

    Theoretical physics benchmark (tpbench)–a dataset and study of ai reasoning capabilities in theoretical physics

    Daniel JH Chung, Zhiqi Gao, Yurii Kvasiuk, Tianyi Li, Moritz Münchmeyer, Maja Rudolph, Frederic Sala, and Sai Chaitanya Tadepalli. Theoretical physics benchmark (tpbench)–a dataset and study of ai reasoning capabilities in theoretical physics. arXiv preprint arXiv:2502.15815, 2025

    2025 arXiv

  3. [11]

    Stop overthinking: A survey on efficient reasoning for large language models

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025

    2025 arXiv

  4. [12]

    Expert and novice performance in solving physics problems

    Jill Larkin, John McDermott, Dorothea P Simon, and Herbert A Simon. Expert and novice performance in solving physics problems. Science, 208(4450):1335–1342, 1980. ISSN 0036- 8075

    1980

  5. [13]

    Michelene T. H. Chi, Paul J. Feltovich, and Robert Glaser. Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2):121–152, apr 1981. doi: 10.1207/s15516709cog0502_2

    1981 doi

  6. [14]

    Cognitive load during problem solving: Effects on learning

    John Sweller. Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2):257–285, apr 1988. doi: 10.1207/s15516709cog1202_4

    1988 doi

  7. [15]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  8. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  9. [17]

    Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021

    arXiv

  10. [18]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507–2...

    2022

  11. [19]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023

    2023 arXiv

  12. [20]

    Have LLMs advanced enough? a chal- lenging problem solving benchmark for large language models

    Daman Arora, Himanshu Singh, and Mausam. Have LLMs advanced enough? a chal- lenging problem solving benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Process...

    2023 doi

  13. [21]

    Scibench: Evaluating college-level scientific problem-solving abilities of large language models

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635, 2023

    2023 arXiv

  14. [22]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  15. [23]

    Scieval: A multi-level large language model evaluation benchmark for scientific research

    Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19053–19061, 2024. 11

    2024

  16. [24]

    Theoremqa: A theorem-driven question answering dataset

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset. arXiv preprint arXiv:2305.12524, 2023

    2023 arXiv

  17. [25]

    Humanity’s last exam

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025

    2025 arXiv

  18. [26]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402....

    2024 arXiv

  19. [27]

    Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai

    Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. Advances in Neural Information Processing Systems, 37:192...

    2024

  20. [28]

    Using large language model to solve and explain physics word problems approaching human level

    Jingzhe Ding, Yan Cen, and Xinyuan Wei. Using large language model to solve and explain physics word problems approaching human level. arXiv preprint arXiv:2309.08182, 2023

    2023 arXiv

  21. [29]

    Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models

    Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. arXiv preprint arXiv:2502.00334, 2025

    2025 arXiv

  22. [30]

    Phybench: Holistic evaluation of physical perception and reasoning in large language models

    Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, et al. Phybench: Holistic evaluation of physical perception and reasoning in large language models. arXiv preprint arXiv:2504.16074, 2025

    2025 arXiv

  23. [31]

    Physreason: A comprehensive benchmark towards physics-based reasoning

    Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: A comprehensive benchmark towards physics-based reasoning. arXiv preprint arXiv:2502.12054, 2025

    2025 arXiv

  24. [32]

    Curie: Evalu- ating llms on multitask scientific long context understanding and reasoning

    Hao Cui, Zahra Shamsi, Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, Peter Norgaard, Nayantara Mudur, Martyna Plomecka, Paul Raccuglia, et al. Curie: Evalu- ating llms on multitask scientific long context understanding and reasoning. arXiv preprint arXiv:2503.13517, 2025

    2025 arXiv

  25. [33]

    Mm-phyqa: Multimodal physics question-answering with multi-image cot prompting

    Avinash Anand, Janak Kapuriya, Apoorv Singh, Jay Saraf, Naman Lal, Astha Verma, Rushali Gupta, and Rajiv Shah. Mm-phyqa: Multimodal physics question-answering with multi-image cot prompting. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 53–64. Springer, 2024

    2024

  26. [34]

    Feabench: Evaluating language models on multiphysics reasoning ability

    Nayantara Mudur, Hao Cui, Subhashini Venugopalan, Paul Raccuglia, Michael P Brenner, and Peter Norgaard. Feabench: Evaluating language models on multiphysics reasoning ability. arXiv preprint arXiv:2504.06260, 2025

    2025 arXiv

  27. [35]

    Learning to reason with llms, September 2024

    OpenAI. Learning to reason with llms, September 2024. URL https://openai.com/index/ learning-to-reason-with-llms/ . Accessed: 2025-05-12

    2024

  28. [36]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  29. [37]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  30. [38]

    Claude 3.7 sonnet and extended thinking mode

    Anthropic. Claude 3.7 sonnet and extended thinking mode. https://www.anthropic.com/ news/claude-3-7-sonnet , February 2025. Accessed: 2025-05-10

    2025

  31. [39]

    Qwq: Reflect deeply on the boundaries of the unknown, November 2024

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://qwenlm.github.io/blog/qwq-32b-preview/ . Accessed: 2025-05-12. 12

    2024

  32. [40]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. https://openai.com/index/ o3-o4-mini-system-card/ , April 2025. Accessed: 2025-05-10

    2025

  33. [41]

    URL https://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/

    Google DeepMind, March 2025. URL https://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/

    2025

  34. [42]

    Introducing gpt-4.1 in the api, April 2025

    OpenAI. Introducing gpt-4.1 in the api, April 2025. URL https://openai.com/index/ gpt-4-1/

    2025

  35. [43]

    Claude 3.7 sonnet and claude code, February 2025

    Anthropic. Claude 3.7 sonnet and claude code, February 2025. URL https://www. anthropic.com/news/claude-3-7-sonnet . Accessed: 2025-05-12

    2025

  36. [44]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

  37. [46]

    Zero-shot

    URL https://arxiv.org/abs/2412.19437. 13 A Analysis of Several Examples A.1 An example of LLM failing to apply principles For some problems, even if the LLM is forced to use principle, it fails to figure out the correct way to use it. In the following example, Gemini-2.5 Pro f...

    arXiv

  38. [2025]

    URL https://arxiv.org/abs/2404.18400

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.