Pith. sign in

REVIEW 3 major objections 4 minor 5 cited by

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UGPhysics, a bilingual benchmark of 5,520 undergraduate physics problems, reports that the best current LLM, OpenAI-o1-mini, reaches only 49.78% accuracy, far below its math performance.

desk verdict Valuable bilingual undergraduate physics benchmark, but Table 5 contradicts the headline leaderboard claim — fix the reporting, not the data. read the letter →

arxiv 2502.00334 v4 pith:E57R7TF6 submitted 2025-02-01 cs.CL cs.AI

Xin Xu , Qiyun Xu , Tong Xiao , Tianhao Chen , Yuchen Yan , Jiaxin Zhang , Shizhe Diao , Can Yang
show 1 more author
Yang Wang
This is my paper · ORCID
classification cs.CLcs.AI
keywords undergraduatephysicsbenchmarkLLMreasoningbilingualevaluationanswerjudgmentpipelineMARJskillsdataleakagedetectionlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces UGPhysics, a benchmark of 5,520 undergraduate-level physics problems in English and Chinese, designed to test whether large language models can reason through college physics rather than merely recall facts. Across 31 LLMs, the highest accuracy is 49.78%, scored by OpenAI-o1-mini, and 15 of the 31 models fall below 20%. The authors argue that physics reasoning, especially mathematical derivation embedded in physical contexts, is substantially harder for LLMs than abstract math, and that math-specialized training transfers only weakly to physics. They also present MARJ, a two-stage answer-judging pipeline, as a reliable evaluation method for open-ended physics answers.

What carries the argument

The benchmark dataset itself is the central object: 5,520 problems in Chinese translated into English, each with a problem, solution, and ground-truth answer annotated with one of seven answer types and one of four reasoning skills. The MARJ pipeline is the evaluation mechanism: a rule-based first stage handles numerical values (with a relative error tolerance of $10^{-2}$), expressions and equations (after stripping physical constants), intervals, true/false, and multiple choice, and a model-based second stage uses GPT-4o to judge anything the rules flag as false. This two-stage design lets the evaluation keep numerical precision while accommodating physics answers that are equivalent but not literally identical, such as different units or omitted physical constants.

What would settle it

Score a fresh random subset of several hundred UGPhysics model answers by human physics experts and compare against MARJ, stratifying by answer type (numerical value, expression, equation, interval, compound); if agreement on the expression, equation, and interval types falls well below 98%, the reported 49.78% headline and per-model rankings are not stable.

Watch

Extended reading notes

Core claim

The paper's central claim is that UGPhysics measures a capability gap: current LLMs solve undergraduate physics problems at a level far below their mathematics performance. The benchmark comprises 5,520 problems across three domains (Mechanics & Thermodynamics, Electromagnetism, Modern Physics), 13 subjects, and 59 topics, with six atomic answer types plus one compound type and four physics-reasoning skill labels. Evaluation of 31 LLMs places OpenAI-o1-mini at 49.78% overall accuracy, DeepSeek-R1-Distill-Llama-70B at 40.17%, and most models below 20%. The paper further claims that math-specialized LLMs show only marginal gains over general-purpose models, that o1-like reasoning models are the most promising open-source direction, and that models' main failure modes are flawed reasoning, knowledge deficiency, and incorrect application rather than calculation errors.

Load-bearing premise

The headline accuracies rest on MARJ's 98% agreement with human annotations from just 100 randomly sampled problems; if the rule-based tolerances or the GPT-4o fallback misjudge harder answers outside that sample, the reported model scores and rankings could shift.

Editorial extensions

If this is right

  • Any LLM claiming competence in undergraduate physics should be tested on UGPhysics; the current best score of 49.78% sets a low bar that most models do not approach.
  • Math-specialized training does not transfer to physics: math-specialized LLMs gain only 0.14 to 3.3 percentage points over their general-purpose counterparts, so physics-specific corpora will be needed.
  • O1-like reasoning models are the strongest open-source category, but the best open model (DeepSeek-R1-Distill-Llama-70B, 40.17%) still trails OpenAI-o1-mini by about 10 points.
  • Physics reasoning errors are dominated by flawed reasoning, knowledge deficiency, and wrong application, so improving calculation or chain-of-thought length alone is unlikely to close the gap.
  • MARJ's 98% agreement with human judges on a 100-example sample suggests rule-plus-model judging can reliably grade open-ended physics answers, making large-scale physics evaluation feasible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: if physics-specific fine-tuning data are what the benchmark implies, then models trained on a UGPhysics-derived corpus should improve on the same answer types and reasoning skills more than math-trained models of equal size.
  • Because UGPhysics is text-only and the paper excludes image-based problems, extending the benchmark to multi-modal physics problems could reveal whether visual reasoning changes the current model ranking.
  • The skill labels allow a finer diagnostic: models that score well on Knowledge Recall but poorly on Math Derivation suggest knowledge access is not the bottleneck; training and prompting should target derivation-with-physical-meaning.
  • The 100-example human validation is too small to rule out answer-judgment errors on rarer answer types such as equations and intervals; an independent scored subset stratified by answer type would be a cheap way to test the headline accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. UGPhysics introduces a bilingual (English/Chinese) benchmark of 5,520 undergraduate-level physics problems, sourced from seven Chinese exercise books, covering 13 subjects, 59 topics, seven answer types, and four physics-reasoning skill labels. The paper describes a two-stage Model-Assistant Rule-based Judgment (MARJ) pipeline for answer grading, reports leakage screening, and evaluates 31 (or 33, depending on table rows) LLMs, with the headline result that OpenAI-o1-mini achieves the highest overall accuracy of 49.78%. The paper also presents fine-grained analyses by subject, skill, language, token budget, and error type, concluding that current LLMs are substantially weaker at undergraduate physics reasoning than at mathematics.

Significance. If the benchmark and the MARJ scoring are reliable, UGPhysics fills a real gap: most existing physics benchmarks are small, multiple-choice, or high-school level, whereas UGPhysics is large, bilingual, text-only, and includes diverse answer types and fine-grained subject/topic coverage. The public release of data and code, the explicit leakage-screening procedure, and the human validation of MARJ are concrete strengths. However, the paper contains an internal contradiction in its headline result, and the evaluation pipeline has validation and token-budget caveats that prevent full confidence in the reported rankings as printed.

major comments (3)
  1. [Abstract, Section 4.2, Table 5] The abstract and Section 4.2 claim that OpenAI-o1-mini achieves the highest overall accuracy (49.78%) and that only two models surpass the 40% overall accuracy. Table 5's 'Newly-added Results' row reports DeepSeek-R1 at 56.34% overall (EN 57.16, ZH 55.53), which exceeds o1-mini by 6.56 percentage points; with DeepSeek-R1-Distill-Llama-70B at 40.17%, at least three models surpass 40%. The text also says 'Our evaluation of 31 leading LLMs,' but Table 5 contains 33 rows if the newly-added rows are counted. As printed, the central ranking claim is internally inconsistent: if DeepSeek-R1 and Phi-4 are not part of the main evaluation, the table and text must say so explicitly, and if they are, the abstract, Section 1 key finding 1, and Section 4.2 must be revised.
  2. [Section 5.2, Section 3.3] The reliability of MARJ rests on a human validation of only 100 of the 5,520 problems, with 98% agreement reported as a point estimate. With 100 binary comparisons, the 95% confidence interval is approximately 93% to 99.7%, and no per-model standard errors or confidence intervals are reported for the accuracies in Table 5 (at N=5,520, a 49.78% accuracy has a standard error around 0.67%). The rule-based stage uses hand-chosen tolerances such as 1e-2 relative error for numerical values, and the model-based fallback uses GPT-4o, which is itself one of the evaluated models. Because every headline accuracy is mediated by this scoring pipeline, a misjudgment rate higher than 2% on the unvalidated answer types (expression, equation, interval, compound) would directly change the reported accuracies and rankings.
  3. [Section 5.6, Tables 8 and 9] The token-budget analysis shows that the reported o1-like model results are partly artifacts of the 8,192-token limit: 19.16% of DeepSeek-R1-Distill-Llama-70B generations and 44.4% of DeepSeek-R1-Distill-Qwen-7B generations are clipped, and extending the limit to 16,384 raises DeepSeek-R1-Distill-Llama-70B from 40.17% to 41.77% and QwQ-32B-Preview from 37.34% to 38.90%. The paper uses the 8,192 numbers for the main results while acknowledging the effect only in Section 5.6. This does not overturn the o1-mini first-place result, but it does affect the claim that DeepSeek-R1-Distill-Llama-70B is clearly second, and the main table should either use the larger budget or report both budgets consistently.
minor comments (4)
  1. [Title page] The affiliation line contains a typo: 'The Universify of Science and Technology of China' should be 'University of Science and Technology of China.'
  2. [Table 8] The table header says '16382' while Section 5.6 and Table 9 use '16384'; the header should be corrected to 16,384.
  3. [Section 5.4] The leakage-detection description says the authors 'randomly chose K positions for extracting 5-grams' but never specifies the value of K, and Table 6 reports results for only a subset of LLMs without explaining why that subset was selected.
  4. [Algorithm 1, Section 5.2] For EX and EQ answers, the algorithm checks whether 'gt equals a' as an equation or expression, but the text does not specify how symbolic equivalence is decided; Section 5.2 mentions SymPy only in passing, so the exact equality procedure should be stated for reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: UGPhysics is anchored to external exercise-book ground truth, and the minor GPT-4o judge/annotator self-involvement and prior-work self-citation are not load-bearing for the headline accuracy claims.

full rationale

UGPhysics is anchored to external ground truth: problems are collected from published undergraduate physics exercise books, answers are the books' definitive answers, and the central reported numbers are the accuracies of independent LLMs judged against those fixed answers. There is no fitted parameter that is later relabeled as a prediction, and no headline quantity is defined in terms of the models being evaluated. The MARJ pipeline combines a rule-based stage with hand-set tolerances and a GPT-4o fallback judge, and GPT-4o is itself one of the evaluated models; this is a legitimate bias concern but not a circular derivation, because correctness is still compared with external golden answers and the pipeline was independently checked against 100 human annotations at 98% agreement. The authors cite their own UGMathBench for answer-type taxonomy and evaluation-prompt conventions, but that citation is not load-bearing: the seven answer types are an annotation scheme rather than evidence for the ranking of models. The internal inconsistency between the 49.8% headline and the 'Newly-added Results' row showing DeepSeek-R1 at 56.34% is a consistency/correctness defect, not a case of a result reducing to its own inputs. No circular step satisfying the quote-and-reduction standard was found, so the circularity score is low.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark's scores rest on external source books and hand-built evaluation rules rather than on fitted physical models. No new physical entities are introduced. The key burden is the reliability of MARJ's fallback judgments and the representativeness of the source problems.

free parameters (4)
  • NV relative-error tolerance epsilon = 1e-2
    Section 3.3 and Algorithm 1 accept numerical answers whose scientific-notation base differs from ground truth by less than this threshold. The threshold is chosen by hand and changes measured accuracy for borderline answers.
  • Maximum output token limits = 4,096 default; 8,192 for o1-mini and o1-like models; 16,384 in extended runs
    Token clipping affects completion completeness, and Section 5.6 reports high clip ratios for several open-source o1-like models. Extended budgets change accuracies, so the limit is a consequential evaluation parameter.
  • Decoding temperature = 0 for most models; 1 for o1-mini
    Appendix B.3 states greedy decoding with temperature 0, but o1-mini is restricted to temperature 1 by the API, introducing non-determinism for the headline model.
  • K in n-gram leakage detection = not specified
    Section 5.4 says K positions are randomly chosen but never reports K, so the contamination rates in Table 6 are not reproducible.
assumptions (5)
  • domain assumption The seven USTC exercise books are an accurate and authoritative source of undergraduate physics with correct ground-truth answers.
    Section 3.2 describes sourcing and manual review, but no independent check of source answer correctness is presented; every accuracy number inherits any errors in these books.
  • domain assumption OCR conversion and manual correction preserve the exact mathematical and physical content of the original problems.
    Section 3.2 says PDFs are converted with Mathpix and manually corrected; transcription errors would change both questions and ground-truth answers.
  • domain assumption The Chinese-to-English translation preserves semantics, notation, and answer equivalence.
    Section 3.2 states problems are translated to English; EN and ZH are treated as comparable evaluation tracks in Table 5.
  • domain assumption GPT-4o's fallback judgments in MARJ are reliable for answers not matched by rules.
    Section 5.2 validates MARJ on 100 examples; with 5,520 problems, the judge-model assumption extends beyond the validated sample, and GPT-4o is also an evaluated model.
  • domain assumption 5-gram overlap detection is a valid leakage measure.
    Section 5.4 follows Xu et al. 2024a and reports very low contamination, but K is unspecified and n-gram overlap only catches verbatim memorization, not paraphrased leakage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models." pith.science (2026). https://pith.science/paper/E57R7TF6

@misc{pith2026250200334,
  author       = {Pith},
  title        = {Pith review of: UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E57R7TF6}},
  note         = {Machine review of arXiv:2502.00334}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities in solving complex reasoning tasks, particularly in mathematics. However, the domain of physics reasoning presents unique challenges that have received significantly less attention. Existing benchmarks often fall short in evaluating LLMs' abilities on the breadth and depth of undergraduate-level physics, underscoring the need for a comprehensive evaluation. To fill this gap, we introduce UGPhysics, a large-scale and comprehensive benchmark specifically designed to evaluate UnderGraduate-level Physics (UGPhysics) reasoning with LLMs. UGPhysics includes 5,520 undergraduate-level physics problems in both English and Chinese, covering 13 subjects with seven different answer types and four distinct physics reasoning skills, all rigorously screened for data leakage. Additionally, we develop a Model-Assistant Rule-based Judgment (MARJ) pipeline specifically tailored for assessing answer correctness of physics problems, ensuring accurate evaluation. Our evaluation of 31 leading LLMs shows that the highest overall accuracy, 49.8% (achieved by OpenAI-o1-mini), emphasizes the necessity for models with stronger physics reasoning skills, beyond math abilities. We hope UGPhysics, along with MARJ, will drive future advancements in AI for physics reasoning. Codes and data are available at https://github.com/YangLabHKUST/UGPhysics .

Figures

Figures reproduced from arXiv: 2502.00334 by the authors.

Figure 1
Figure 1. An overall illustration of UGPhysics. The top part repre￾sents the hierarchical physics domains and subjects. The bottom part showcases one concrete example. 1. Introduction “Physics is the foundation of all the natural sci￾ences.” — Max Planck Physics forms the foundation for natural sciences (Planck, 1949; Hawking, 1988; Giancoli, 2000), and physics prob￾lem solving constitutes a significant aspect of reasoning fo… view at source ↗
Figure 2
Figure 2. The distribution of overall accuracy across subjects, and physics reasoning skills. (a) The overall accuracy of different subjects averaged across 8 strong LLMs listed in Figure (b). Each bar consists of several segments with colors indicating their corresponding reasoning skills. (b) The overall accuracy of reasoning skills, averaged across all subjects. Only 8 strong LLMs are included for brevity. “KR”: Knowledge … view at source ↗
Figure 3
Figure 3. Performance in different languages, sorted by the differ￾ence of EN - ZH. 5.3. Error Analysis To gain deeper insights into the performance of LLMs, we select 100 incorrect answers generated by OpenAI-o1- mini and have these errors annotated by human evaluators to determine failure reasons. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Word Cloud of Topics in UGPhysics 18 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy

    cs.AI 2026-02 reject novelty 6.0 of 10

    A new benchmark of 2,703 automatically generated multimodal questions for scanning probe microscopy, plus a modified F1 metric that penalizes over-selection and labels model 'personalities'.

  2. DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A unified visual benchmark of six physics environments shows current VLMs, including frontier models, mostly fail at interactive physical reasoning and cannot convert descriptive physics knowledge into reliable control.

  3. Superstudent intelligence in thermodynamics

    cs.CE 2025-06 conditional novelty 6.0 of 10

    OpenAI's o3 model scored higher than all 90 students on a real university thermodynamics exam, zero-shot, including problems with graphical output.

  4. PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models

    cs.LG 2025-05 reject novelty 6.0 of 10

    A new benchmark of 380 principle-based physics problems shows that state-of-the-art LLMs struggle to apply symmetry, conservation, and dimensional-analysis shortcuts, achieving under 50 percent average accuracy with h...

  5. PhyX: Does Your Model Have the "Wits" for Physical Reasoning?

    cs.AI 2025-05 conditional novelty 6.0 of 10

    PhyX is a new 3,000-question visual physics benchmark; the best AI model tested scores 45.8 percent, well below the 75.6 to 78.9 percent of a small human student sample.

Reference graph

Works this paper leans on

77 extracted references · 24 canonical work pages · cited by 5 Pith papers

  1. [1]

    Yi: Open foundation models by 01.ai, 2024

    01-AI, :, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang, S., Yang, S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P., Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundation models by 01.ai, 2024

    work page 2024

  2. [2]

    G., et al

    Arora, D., Singh, H. G., et al. Have llms advanced enough? a challenging problem solving benchmark for large language models. arXiv preprint arXiv:2305.15074, 2023

    arXiv 2023

  3. [3]

    D., McAleer, S., Jiang, A

    Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics. ArXiv preprint, abs/2310.10631, 2023. URL https://arxiv.org/abs/2310.10631

    arXiv 2023

  4. [4]

    Phyre: A new benchmark for physical reasoning

    Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L., and Girshick, R. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  5. [5]

    C., Jiang, A., Li, J., Lipkin, B., Qina, Z., Rasul, K., Shen, Z., Soletskyi, R., and Tunstall, L

    Beeching, E., Huang, S. C., Jiang, A., Li, J., Lipkin, B., Qina, Z., Rasul, K., Shen, Z., Soletskyi, R., and Tunstall, L. Numinamath 7b cot. https://huggingface.co/AI-MO/NuminaMath-7B-CoT, 2024

    work page 2024

  6. [6]

    Reasoning about physical commonsense in natural language, 2019

    Bisk, Y., Zellers, R., Le Bras, R., Gao, J., and Choi, Y. Reasoning about physical commonsense in natural language, 2019

    work page 2019

  7. [7]

    and Xiao, J

    Cao, J. and Xiao, J. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics, pp.\ 1511--1520, 2022

    work page 2022

  8. [8]

    P., and Lin, L

    Chen, J., Tang, J., Qin, J., Liang, X., Liu, L., Xing, E. P., and Lin, L. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2105.14517, 2021

    arXiv 2021

Show all 77 references
  1. [9]

    Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv preprint, abs/2211.12588, 2022. URL https://arxiv.org/abs/2211.12588

    2022 arXiv

  2. [10]

    Theoremqa: A theorem-driven question answering dataset

    Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., and Xia, T. Theoremqa: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 7889--7901, 2023

    2023

  3. [11]

    Do not think that much for 2+ 3=? on the overthinking of o1-like llms

    Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., Song, L., Liu, Q., Zhou, M., Zhang, Z., et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187, 2024

    2024 arXiv

  4. [12]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168

    2021 arXiv

  5. [13]

    X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y

    Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR, abs/2401.06066...

    2024 arXiv

  6. [14]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    2024

  7. [15]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., D...

    2025 arXiv

  8. [16]

    R., Guo, S., Valko, M., Lillicrap, T., Jimenez Rezende, D., Bengio, Y., Mozer, M

    Didolkar, A., Goyal, A., Ke, N. R., Guo, S., Valko, M., Lillicrap, T., Jimenez Rezende, D., Bengio, Y., Mozer, M. C., and Arora, S. Metacognitive capabilities of llms: An exploration in mathematical problem solving. Advances in Neural Information Processing Systems, 37: 0 1978...

    2024

  9. [17]

    Using large language model to solve and explain physics word problems approaching human level

    Ding, J., Cen, Y., and Wei, X. Using large language model to solve and explain physics word problems approaching human level. arXiv preprint arXiv:2309.08182, 2023

    2023 arXiv

  10. [18]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  11. [19]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q., Li, L., Ma, C., Chen, L., Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024

    2024 arXiv

  12. [20]

    Giancoli, D. C. Physics for Scientists and Engineers. Prentice Hall, 2000

    2000

  13. [21]

    A Brief History of Time: From the Big Bang to Black Holes

    Hawking, S. A Brief History of Time: From the Big Bang to Black Holes. Bantam Books, 1988

    1988

  14. [22]

    L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., et al

    He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024

    2024 arXiv

  15. [23]

    Teaching problem solving through cooperative grouping

    Heller, P., Keith, R., and Anderson, S. Teaching problem solving through cooperative grouping. part 1: Group versus individual problem solving. American Journal of Physics, 60 0 (7): 0 627--636, 1992

    1992

  16. [24]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  17. [25]

    Measuring mathematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. ArXiv preprint, abs/2103.03874, 2021. URL https://arxiv.org/abs/2103.03874

    2021 arXiv

  18. [26]

    E-eval: A comprehensive chinese k-12 education evaluation benchmark for large language models

    Hou, J., Ao, C., Wu, H., Kong, X., Zheng, Z., Tang, D., Li, C., Hu, X., Xu, R., Ni, S., et al. E-eval: A comprehensive chinese k-12 education evaluation benchmark for large language models. arXiv preprint arXiv:2401.15927, 2024

    2024 arXiv

  19. [27]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Fu, Y., et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  20. [28]

    Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai

    Huang, Z., Wang, Z., Xia, S., Li, X., Zou, H., Xu, R., Fan, R.-Z., Ye, L., Chern, E., Ye, Y., et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. arXiv preprint arXiv:2406.12753, 2024 b

    2024 arXiv

  21. [30]

    P., Anand, A., Dharmadhikari, A., Marathe, A., and Shah, R

    Jaiswal, R., Jain, D., Popat, H. P., Anand, A., Dharmadhikari, A., Marathe, A., and Shah, R. R. Improving physics reasoning in large language models using mixture of refinement agents. arXiv preprint arXiv:2412.00821, 2024 b

    2024 arXiv

  22. [31]

    and Waterman, D

    Klahr, P. and Waterman, D. A. Applications of Artificial Intelligence to Physics. AAAI Press, 1986

    1986

  23. [32]

    Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al

    Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  24. [33]

    H., McDermott, J., Simon, D

    Larkin, J. H., McDermott, J., Simon, D. P., and Simon, H. A. Expert and novice performance in solving physics problems. Science, 208 0 (4450): 0 1335--1342, 1980

    1980

  25. [34]

    Solving quantitative reasoning problems with language models

    Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35: 0 3843--3857, 2022

    2022

  26. [35]

    cmmlu: Measuring massive multitask language understanding in chinese

    Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., and Baldwin, T. cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212, 2023

    2023 arXiv

  27. [36]

    Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark

    Liu, H., Zheng, Z., Qiao, Y., Duan, H., Fei, Z., Zhou, F., Zhang, W., Zhang, S., Lin, D., and Chen, K. Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark. arXiv preprint arXiv:2405.12209, 2024

    2024 arXiv

  28. [37]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 0 2507--2521, 2022

    2022

  29. [38]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  30. [39]

    Sciagent: Tool-augmented language models for scientific reasoning

    Ma, Y., Gou, Z., Hao, J., Xu, R., Wang, S., Pan, L., Yang, Y., Cao, Y., Sun, A., Awadalla, H., et al. Sciagent: Tool-augmented language models for scientific reasoning. arXiv preprint arXiv:2402.11451, 2024

    2024 arXiv

  31. [40]

    McDermott, L. C. and Redish, E. F. Resource letter on physics education research. American Journal of Physics, 67 0 (9): 0 755--767, 1999

    1999

  32. [41]

    and Zelinski, D

    Mendelson, E. and Zelinski, D. E. The Physics Problem Solver. 1984

    1984

  33. [42]

    Mathstral

    Mistral. Mathstral. https://mistral.ai/news/mathstral/, 2023. Accessed: 2024-09-23

    2023

  34. [43]

    Mistral large 2

    Mistral. Mistral large 2. = https://mistral.ai/news/mistral-large-2407/ , 2024 a

    2024

  35. [44]

    The future of ai: Trends and predictions

    Mistral. The future of ai: Trends and predictions. https://mistral.ai/news/mistral-nemo/, 2024 b . Accessed: September 29, 2024

    2024

  36. [45]

    mistralai/mistral-small-instruct-2409

    Mistral. mistralai/mistral-small-instruct-2409. https://huggingface.co/mistralai/Mistral-Small-Instruct-2409, 2024 c

    2024

  37. [46]

    Ministral model card, 2024

    MistralAI. Ministral model card, 2024. URL https://huggingface.co/mistralai/Ministral-8B-Instruct-2410

    2024

  38. [47]

    Mudur, N., Cui, H., Venugopalan, S., Raccuglia, P., Brenner, M., and Norgaard, P. C. Feabench: Evaluating language models on real world physics reasoning ability. In NeurIPS 2024 Workshop on Open-World Agents, 2024

    2024

  39. [48]

    Hello gpt-4o

    OpenAI . Hello gpt-4o. https://openai.com/index/hello-gpt-4o/, 2024 a

    2024

  40. [49]

    Learning to reason with llms

    OpenAI . Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/, 2024 b . Accessed: 2024-09-23

    2024

  41. [50]

    Physics reasoner: Knowledge-augmented reasoning for solving physics problems with large language models

    Pang, X., Hong, R., Zhou, Z., Lv, F., Yang, X., Liang, Z., Han, B., and Zhang, C. Physics reasoner: Knowledge-augmented reasoning for solving physics problems with large language models. arXiv preprint arXiv:2412.13791, 2024

    2024 arXiv

  42. [51]

    Scientific Autobiography and Other Papers

    Planck, M. Scientific Autobiography and Other Papers. 1949

    1949

  43. [52]

    Varbench: Robust language model benchmarking through dynamic variable perturbation

    Qian, K., Wan, S., Tang, C., Wang, Y., Zhang, X., Chen, M., and Yu, Z. Varbench: Robust language model benchmarking through dynamic variable perturbation. arXiv preprint arXiv:2406.17681, 2024

    2024 arXiv

  44. [53]

    Qwq-32b-preview

    QwQ-Team. Qwq-32b-preview. = https://qwenlm.github.io/blog/qwq-32b-preview/ , 2024

    2024

  45. [54]

    Redish, E. F. Teaching Physics with the Physics Suite. Wiley, 2003

    2003

  46. [55]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  47. [56]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  48. [57]

    Skywork-o1 model card, 2024

    Skywork. Skywork-o1 model card, 2024. URL https://huggingface.co/Skywork/Skywork-o1-Open-Llama-3.1-8B

    2024

  49. [58]

    Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap

    Srivastava, S., PV, A., Menon, S., Sukumar, A., Philipose, A., Prince, S., Thomas, S., et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402.19450, 2024

    2024 arXiv

  50. [59]

    Mathscale: Scaling instruction tuning for mathematical reasoning

    Tang, Z., Zhang, X., Wan, B., and Wei, F. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884, 2024

    2024 arXiv

  51. [60]

    Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving

    Tong, Y., Zhang, X., Wang, R., Wu, R., and He, J. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. arXiv preprint arXiv:2407.13690, 2024

    2024 arXiv

  52. [61]

    Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data

    Toshniwal, S., Du, W., Moshkov, I., Kisacanin, B., Ayrapetyan, A., and Gitman, I. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024

    2024 arXiv

  53. [62]

    Self-consistency improves chain of thought reasoning in language models

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. ArXiv preprint, abs/2203.11171, 2022. URL https://arxiv.org/abs/2203.11171

    2022 arXiv

  54. [63]

    R., Zhang, S., Sun, Y., and Wang, W

    Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635, 2023 a

    2023 arXiv

  55. [64]

    R., Duan, J., Fox, D., and Srinivasa, S

    Wang, Y. R., Duan, J., Fox, D., and Srinivasa, S. Newton: Are large language models capable of physical reasoning? arXiv preprint arXiv:2310.07018, 2023 b

    2023 arXiv

  56. [65]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 0 24824--24837, 2022

    2022

  57. [66]

    F., and Gardner, M

    Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017

    2017 arXiv

  58. [67]

    Livebench: A challenging, contamination-free llm benchmark

    White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314, 2024

    2024 arXiv

  59. [68]

    Benchmarking benchmark leakage in large language models

    Xu, R., Wang, Z., Fan, R.-Z., and Liu, P. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824, 2024 a

    2024 arXiv

  60. [69]

    Can we verify step by step for incorrect answer detection? arXiv preprint arXiv:2402.10528, 2024 b

    Xu, X., Diao, S., Yang, C., and Wang, Y. Can we verify step by step for incorrect answer detection? arXiv preprint arXiv:2402.10528, 2024 b

    2024 arXiv

  61. [70]

    Can llms solve longer math word problems better? arXiv preprint arXiv:2405.14804, 2024 c

    Xu, X., Xiao, T., Chao, Z., Huang, Z., Yang, C., and Wang, Y. Can llms solve longer math word problems better? arXiv preprint arXiv:2405.14804, 2024 c

    2024 arXiv

  62. [71]

    Ugmathbench: A diverse and dynamic benchmark for undergraduate-level mathematical reasoning with large language models

    Xu, X., Zhang, J., Chen, T., Chao, Z., Hu, J., and Yang, C. Ugmathbench: A diverse and dynamic benchmark for undergraduate-level mathematical reasoning with large language models. arXiv preprint arXiv:2501.13766, 2025

    2025 arXiv

  63. [72]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024 a

    2024 arXiv

  64. [73]

    Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024 b

    2024 arXiv

  65. [74]

    A careful examination of large language model performance on grade school arithmetic

    Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Slack, D., Lyu, Q., et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332, 2024

    2024 arXiv

  66. [75]

    Evaluating the performance of large language models on gaokao benchmark

    Zhang, X., Li, C., Zong, Y., Ying, Z., He, L., and Qiu, X. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474, 2023

    2023 arXiv

  67. [76]

    Processbench: Identifying process errors in mathematical reasoning

    Zheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J., and Lin, J. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559, 2024

    2024 arXiv

  68. [77]

    Agieval: A human-centric benchmark for evaluating foundation models

    Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023

    2023 arXiv

  69. [78]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.