{"id":"4a54006e-7c00-4ba8-a0b4-d8c7c51a328d","arxiv_id":"2502.00334","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"UGPhysics is a new bilingual benchmark of 5,520 undergraduate physics problems; the strongest tested LLM, OpenAI o1-mini, reaches only 49.8% accuracy.","lead":"The paper introduces UGPhysics, a set of 5,520 undergraduate physics problems in Chinese and English, and uses it to test 31 AI language models. The best model, OpenAI o1-mini, answers fewer than half correctly, which suggests current AI systems are much weaker at physics than at math.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5's 'Newly-added Results' row shows DeepSeek-R1 at 56.34% overall, directly contradicting the abstract and Section 4.2 claim that OpenAI-o1-mini's 49.78% is the highest; the stated top performance and 'only two models above 40%' are false as printed.","rationale":"The single most load-bearing condition for the central claim is that the reported top accuracy is correct. The paper's own Table 5 contradicts it: DeepSeek-R1 appears with 56.34% overall under 'Newly-added Results,' yet the abstract and Section 4.2 assert that OpenAI-o1-mini's 49.78% is the highest. The reader's concern about MARJ reliability is legitimate but second-order: a 2% judge error on 100 samples could shift scores slightly, whereas the table shows a six-point gap in the opposite direction. This is an internal inconsistency, not a dispute with external consensus, and it can be checked directly from released artifacts. The benchmark and MARJ may still be valuable, but the headline conclusion and key findings require correction or explicit scoping. I would keep the reader's CONDITIONAL verdict and add a mandatory requirement to reconcile or remove the contradictory row.","tokens_in":32026,"tokens_out":4970,"duration_ms":46966,"concrete_test":"Recompute the Overall Average columns in Table 5 from the per-language rows for all listed models, including DeepSeek-R1 and Phi-4, and compare with the abstract's '49.8% highest' and Section 4.2's 'only two models surpass 40%.' If DeepSeek-R1's 56.34% reproduces, update the abstract, key findings, and ranking statements; if it is intentionally outside the official 31-model evaluation, state that exclusion explicitly in the table and text. The released code and results at the GitHub repository should make this check straightforward.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central result is the ranking claim that OpenAI-o1-mini achieves the highest accuracy (49.78%), with DeepSeek-R1-Distill-Llama-70B second at 40.17%. However, in the same Table 5, under 'Newly-added Results,' DeepSeek-R1 is listed with 56.34% overall (EN 57.16, ZH 55.53), surpassing o1-mini by 6.56 points, and Phi-4 is also added. Section 4.2 states that 'only two models surpass the 40% overall accuracy,' but Table 5 contains at least three such models if DeepSeek-R1 is included. The abstract, intro key findings, and Section 4.2 were not updated to reflect this row. Either DeepSeek-R1 is part of the evaluated set and the headline is incorrect, or it is excluded and the table and text must say so explicitly. Because the claimed top performance is the paper's headline conclusion, this internal contradiction is more load-bearing than the MARJ validation sample size. It does not necessarily invalidate the benchmark itself, but the paper as printed cannot support its main claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"UGPhysics introduces a bilingual (English/Chinese) benchmark of 5,520 undergraduate-level physics problems, sourced from seven Chinese exercise books, covering 13 subjects, 59 topics, seven answer types, and four physics-reasoning skill labels. The paper describes a two-stage Model-Assistant Rule-based Judgment (MARJ) pipeline for answer grading, reports leakage screening, and evaluates 31 (or 33, depending on table rows) LLMs, with the headline result that OpenAI-o1-mini achieves the highest overall accuracy of 49.78%. The paper also presents fine-grained analyses by subject, skill, language, token budget, and error type, concluding that current LLMs are substantially weaker at undergraduate physics reasoning than at mathematics.","tokens_in":32238,"tokens_out":5578,"duration_ms":58504,"significance":"If the benchmark and the MARJ scoring are reliable, UGPhysics fills a real gap: most existing physics benchmarks are small, multiple-choice, or high-school level, whereas UGPhysics is large, bilingual, text-only, and includes diverse answer types and fine-grained subject/topic coverage. The public release of data and code, the explicit leakage-screening procedure, and the human validation of MARJ are concrete strengths. However, the paper contains an internal contradiction in its headline result, and the evaluation pipeline has validation and token-budget caveats that prevent full confidence in the reported rankings as printed.","major_comments":[{"comment":"The abstract and Section 4.2 claim that OpenAI-o1-mini achieves the highest overall accuracy (49.78%) and that only two models surpass the 40% overall accuracy. Table 5's 'Newly-added Results' row reports DeepSeek-R1 at 56.34% overall (EN 57.16, ZH 55.53), which exceeds o1-mini by 6.56 percentage points; with DeepSeek-R1-Distill-Llama-70B at 40.17%, at least three models surpass 40%. The text also says 'Our evaluation of 31 leading LLMs,' but Table 5 contains 33 rows if the newly-added rows are counted. As printed, the central ranking claim is internally inconsistent: if DeepSeek-R1 and Phi-4 are not part of the main evaluation, the table and text must say so explicitly, and if they are, the abstract, Section 1 key finding 1, and Section 4.2 must be revised.","section":"Abstract, Section 4.2, Table 5"},{"comment":"The reliability of MARJ rests on a human validation of only 100 of the 5,520 problems, with 98% agreement reported as a point estimate. With 100 binary comparisons, the 95% confidence interval is approximately 93% to 99.7%, and no per-model standard errors or confidence intervals are reported for the accuracies in Table 5 (at N=5,520, a 49.78% accuracy has a standard error around 0.67%). The rule-based stage uses hand-chosen tolerances such as 1e-2 relative error for numerical values, and the model-based fallback uses GPT-4o, which is itself one of the evaluated models. Because every headline accuracy is mediated by this scoring pipeline, a misjudgment rate higher than 2% on the unvalidated answer types (expression, equation, interval, compound) would directly change the reported accuracies and rankings.","section":"Section 5.2, Section 3.3"},{"comment":"The token-budget analysis shows that the reported o1-like model results are partly artifacts of the 8,192-token limit: 19.16% of DeepSeek-R1-Distill-Llama-70B generations and 44.4% of DeepSeek-R1-Distill-Qwen-7B generations are clipped, and extending the limit to 16,384 raises DeepSeek-R1-Distill-Llama-70B from 40.17% to 41.77% and QwQ-32B-Preview from 37.34% to 38.90%. The paper uses the 8,192 numbers for the main results while acknowledging the effect only in Section 5.6. This does not overturn the o1-mini first-place result, but it does affect the claim that DeepSeek-R1-Distill-Llama-70B is clearly second, and the main table should either use the larger budget or report both budgets consistently.","section":"Section 5.6, Tables 8 and 9"}],"minor_comments":[{"comment":"The affiliation line contains a typo: 'The Universify of Science and Technology of China' should be 'University of Science and Technology of China.'","section":"Title page"},{"comment":"The table header says '16382' while Section 5.6 and Table 9 use '16384'; the header should be corrected to 16,384.","section":"Table 8"},{"comment":"The leakage-detection description says the authors 'randomly chose K positions for extracting 5-grams' but never specifies the value of K, and Table 6 reports results for only a subset of LLMs without explaining why that subset was selected.","section":"Section 5.4"},{"comment":"For EX and EQ answers, the algorithm checks whether 'gt equals a' as an equation or expression, but the text does not specify how symbolic equivalence is decided; Section 5.2 mentions SymPy only in passing, so the exact equality procedure should be stated for reproducibility.","section":"Algorithm 1, Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The internal contradiction between Table 5's newly-added DeepSeek-R1 result and the abstract/Section 4.2 headline is the main blocker; it must be resolved before the paper can support its central claim. The MARJ validation and token-budget concerns are also serious but fixable with clearer presentation and additional analysis. The benchmark itself is a useful contribution if those issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"UGPhysics is a genuinely useful resource: 5,520 undergraduate physics problems in Chinese and English, spanning 13 subjects with seven answer types and four skill labels, all screened for leakage. That breadth is new, and the MARJ judging pipeline is a reasonable answer to the real problem of evaluating symbolic and numerical physics answers with physical constants and equivalent quantities. The 98% agreement with human ratings on 100 sampled problems is encouraging, and the error analysis (flawed reasoning, knowledge deficiency, wrong application) is informative. This is a solid benchmark paper.\n\nThe paper has a load-bearing internal contradiction, though. Table 5 includes a 'Newly-added Results' block with DeepSeek-R1 at 56.34% overall (EN 57.16, ZH 55.53), which beats the 49.78% for OpenAI-o1-mini. The abstract, intro key findings, and Section 4.2 all state o1-mini is highest and that only two models surpass 40%. Those statements are false as printed. Either DeepSeek-R1 is part of the evaluated set and the headline is wrong, or it was added later and the text was never updated. The benchmark's value does not depend on which model tops the chart, but the main claim of the paper does, and right now it is internally inconsistent.\n\nOther soft spots are smaller. The human validation covers only 100 of 5,520 problems; MARJ uses hand-chosen tolerances (e.g. 1e-2 relative error for numerical values) that could misjudge expression, equation, or interval answers outside that sample; GPT-4o serves as both fallback judge and evaluated model. No confidence intervals are reported. Open-source o1-like models are capped at 8,192 tokens and show high clip ratios (19–52%), so part of their accuracy gap may be an artifact of token truncation. None of these kill the benchmark, but they should be acknowledged and preferably addressed with error bars and a sensitivity analysis.\n\nMy bottom line: the resource is worth having, and the evaluation methodology is basically sound. The internal contradiction is a genuine reporting error, not a flaw in the underlying data. This deserves peer review, not rejection, but the authors need to reconcile Table 5 with the narrative and add a short statement about the newly added models. For everyday use, the dataset and MARJ pipeline are valuable, but I would not quote the leaderboard numbers in any paper until the revision clears this up.","headline":"Valuable bilingual undergraduate physics benchmark, but Table 5 contradicts the headline leaderboard claim — fix the reporting, not the data.","tokens_in":32826,"tokens_out":2033,"would_cite":true,"duration_ms":20712,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UGPhysics, a bilingual benchmark of 5,520 undergraduate physics problems, reports that the best current LLM, OpenAI-o1-mini, reaches only 49.78% accuracy, far below its math performance.","keywords":["undergraduate physics benchmark","LLM physics reasoning","bilingual evaluation","answer judgment pipeline","MARJ","physics reasoning skills","data leakage detection","large language models"],"falsifier":"Score a fresh random subset of several hundred UGPhysics model answers by human physics experts and compare against MARJ, stratifying by answer type (numerical value, expression, equation, interval, compound); if agreement on the expression, equation, and interval types falls well below 98%, the reported 49.78% headline and per-model rankings are not stable.","tokens_in":31782,"feed_emoji":"⚛️","tokens_out":6888,"duration_ms":61104,"temperature":0.7,"pith_summary":"The paper introduces UGPhysics, a benchmark of 5,520 undergraduate-level physics problems in English and Chinese, designed to test whether large language models can reason through college physics rather than merely recall facts. Across 31 LLMs, the highest accuracy is 49.78%, scored by OpenAI-o1-mini, and 15 of the 31 models fall below 20%. The authors argue that physics reasoning, especially mathematical derivation embedded in physical contexts, is substantially harder for LLMs than abstract math, and that math-specialized training transfers only weakly to physics. They also present MARJ, a two-stage answer-judging pipeline, as a reliable evaluation method for open-ended physics answers.","feed_headline":"Best LLM tops out at 49.8% on undergraduate physics","feed_subtitle":"A 5,520-problem benchmark in English and Chinese shows 31 LLMs lag far behind their math performance.","key_machinery":"The benchmark dataset itself is the central object: 5,520 problems in Chinese translated into English, each with a problem, solution, and ground-truth answer annotated with one of seven answer types and one of four reasoning skills. The MARJ pipeline is the evaluation mechanism: a rule-based first stage handles numerical values (with a relative error tolerance of $10^{-2}$), expressions and equations (after stripping physical constants), intervals, true/false, and multiple choice, and a model-based second stage uses GPT-4o to judge anything the rules flag as false. This two-stage design lets the evaluation keep numerical precision while accommodating physics answers that are equivalent but not literally identical, such as different units or omitted physical constants.","core_discovery":"The paper's central claim is that UGPhysics measures a capability gap: current LLMs solve undergraduate physics problems at a level far below their mathematics performance. The benchmark comprises 5,520 problems across three domains (Mechanics & Thermodynamics, Electromagnetism, Modern Physics), 13 subjects, and 59 topics, with six atomic answer types plus one compound type and four physics-reasoning skill labels. Evaluation of 31 LLMs places OpenAI-o1-mini at 49.78% overall accuracy, DeepSeek-R1-Distill-Llama-70B at 40.17%, and most models below 20%. The paper further claims that math-specialized LLMs show only marginal gains over general-purpose models, that o1-like reasoning models are the most promising open-source direction, and that models' main failure modes are flawed reasoning, knowledge deficiency, and incorrect application rather than calculation errors.","pith_inferences":["A testable extension: if physics-specific fine-tuning data are what the benchmark implies, then models trained on a UGPhysics-derived corpus should improve on the same answer types and reasoning skills more than math-trained models of equal size.","Because UGPhysics is text-only and the paper excludes image-based problems, extending the benchmark to multi-modal physics problems could reveal whether visual reasoning changes the current model ranking.","The skill labels allow a finer diagnostic: models that score well on Knowledge Recall but poorly on Math Derivation suggest knowledge access is not the bottleneck; training and prompting should target derivation-with-physical-meaning.","The 100-example human validation is too small to rule out answer-judgment errors on rarer answer types such as equations and intervals; an independent scored subset stratified by answer type would be a cheap way to test the headline accuracy."],"forward_implications":["Any LLM claiming competence in undergraduate physics should be tested on UGPhysics; the current best score of 49.78% sets a low bar that most models do not approach.","Math-specialized training does not transfer to physics: math-specialized LLMs gain only 0.14 to 3.3 percentage points over their general-purpose counterparts, so physics-specific corpora will be needed.","O1-like reasoning models are the strongest open-source category, but the best open model (DeepSeek-R1-Distill-Llama-70B, 40.17%) still trails OpenAI-o1-mini by about 10 points.","Physics reasoning errors are dominated by flawed reasoning, knowledge deficiency, and wrong application, so improving calculation or chain-of-thought length alone is unlikely to close the gap.","MARJ's 98% agreement with human judges on a 100-example sample suggests rule-plus-model judging can reliably grade open-ended physics answers, making large-scale physics evaluation feasible."],"supporting_citations":[{"why":"Supplies OlympiadBench and the 12% judgment error rate for physics answers that motivates the MARJ design.","marker":"(He et al., 2024)"},{"why":"OlympicArena is used as a comparison benchmark and as a template for rule-plus-model evaluation and zero-shot prompting.","marker":"(Huang et al., 2024b)"},{"why":"UGMathBench provides the underlying answer-type taxonomy and evaluation-prompt conventions that UGPhysics adapts.","marker":"(Xu et al., 2025)"},{"why":"MATH is the benchmark used to contrast o1-mini's high math accuracy with its 49.78% physics score.","marker":"(Hendrycks et al., 2021)"},{"why":"GPQA is the graduate-level benchmark used to contextualize GPT-4o's 38.67% accuracy on UGPhysics.","marker":"(Rein et al., 2023)"},{"why":"Omni-MATH offers the model-based judging prompt that MARJ's second stage adapts and serves as a math performance comparison.","marker":"(Gao et al., 2024)"},{"why":"o1-mini is the model whose 49.78% accuracy anchors the paper's headline result.","marker":"(OpenAI, 2024b)"}],"fun_headline_variants":["Physics wall: best LLM scores 49.8% on new undergrad test","UGPhysics benchmark: no LLM breaks 50% on undergrad physics","Undergrad physics stumps LLMs, best accuracy 49.8%","LLMs lag in physics reasoning, top score 49.8% on UGPhysics","New physics benchmark: 5,520 problems, best LLM at 49.8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracies rest on MARJ's 98% agreement with human annotations from just 100 randomly sampled problems; if the rule-based tolerances or the GPT-4o fallback misjudge harder answers outside that sample, the reported model scores and rankings could shift.","fun_headline_variants_meta":{"raw":{"variants":["Physics wall: best LLM scores 49.8% on new undergrad test","UGPhysics benchmark: no LLM breaks 50% on undergrad physics","Undergrad physics stumps LLMs, best accuracy 49.8%","LLMs lag in physics reasoning, top score 49.8% on UGPhysics","New physics benchmark: 5,520 problems, best LLM at 49.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00126,"raw_usage":{"total_tokens":5166,"prompt_tokens":958,"completion_tokens":4208,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":4098}},"tokens_in":574,"tokens_out":4208,"duration_ms":27966,"temperature":1.0,"reasoning_tokens":4098,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:23:22.168604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score a fresh random subset of several hundred UGPhysics model answers by human physics experts and compare against MARJ, stratifying by answer type (numerical value, expression, equation, interval, compound); if agreement on the expression, equation, and interval types falls well below 98%, the reported 49.78% headline and per-model rankings are not stable.","supporting_citations":[],"review_version":1}