{"id":"d031c6b5-b377-431b-bc72-0404cdb3214f","arxiv_id":"2507.04766","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new physics benchmark with static and dynamically varied numeric problems shows top LLMs solve at most 43 percent of the static set and drop sharply when problem constants change.","lead":"This paper introduces ABench-Physics, a benchmark of 500 graduate- and Olympiad-level physics problems that require exact numerical answers, plus a variation engine that changes the numbers in 100 problems to test generalization. It evaluates several top AI models and finds they solve at most 43 percent of the static questions and lose about 22.5 percent when the numbers change.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 22.5% dynamic drop in Table 2 conflates a stricter all-or-nothing scoring rule with sensitivity to perturbation: even a perfectly robust model would show a p - p^3 decline when moving from one variant to requiring all three.","rationale":"The reader's verdict is CONDITIONAL because the dynamic-subset conclusion rests on an untested assumption about numeric perturbation and noisy all-or-nothing scoring. I agree that this is the right area, but the more specific and more fundamental problem is that the two accuracy numbers being differenced are not commensurable: one is a single-variant probability, the other is a conjunction over three variants. No amount of variant-equivalence or 'same physical model' checking can fix the fact that p - p^3 > 0 even for a perfectly robust model. The paper needs to report per-variant accuracy; once that is available, the dynamic claim can be evaluated properly. The static Phy_A result (43% top accuracy) remains a valid benchmark result and supports the existence of a gap, so this is not a rejection. I would keep the verdict at CONDITIONAL, but with the requirement switched from 'validate variant equivalence' to 'report per-variant accuracy and corrected Delta.' The reader's weakest_assumption partially overlaps: it noted all-or-nothing noise and variant equivalence, but did not identify the p versus p^3 confound.","tokens_in":8059,"tokens_out":8996,"duration_ms":105763,"concrete_test":"Using the released ABench-Physics repository and the three variants per Phy_B problem from footnote 2, evaluate each of the 15 models on each variant separately. Compute (a) mean per-variant accuracy across all variants, (b) all-or-nothing Dynamic Acc, and (c) Delta_corrected = reference-variant accuracy - mean-per-variant accuracy, with bootstrap 95% confidence intervals over the 100 problems. If Delta_corrected is statistically indistinguishable from zero, or if Dynamic Acc is consistent with p_avg^3 under the observed per-variant accuracies, the reported 22.5% decline is a scoring artifact rather than evidence of perturbation sensitivity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 and the abstract use the 22.5% average decline (Table 2) as the main evidence that LLMs rely on memorization. But 'Static Acc' and 'Dynamic Acc' are not measured on the same scale: Static Acc credits a model for solving one reference variant of a Phy_B problem, while Dynamic Acc credits it only if it solves all three released variants (footnote 2). If p is the per-variant success probability and errors are independent, the expected Dynamic Acc is p^3, so the expected decline is p - p^3: at p=0.75 the expected decline is 0.33, and at p=0.79 it is 0.30. The observed declines (0.19-0.28) fall in this range. In fact, every observed Dynamic Acc exceeds p_ref^3, which is consistent with the non-reference variants being easier or with positive error correlation; in neither case does Delta isolate sensitivity to numerical perturbation. The paper never reports mean per-variant accuracy, so the central drop, the 'RL outperforms SFT on dynamic questions' claim, and the memorization conclusion are confounded by a scoring artifact until that number is provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ABench-Physics is a benchmark for LLM physical reasoning consisting of 500 numerical-answer problems: Phy_A contains 400 static problems screened to be hard for current models, and Phy_B contains 100 parameterized problems with an automatic variation engine that perturbs numerical constants in the LaTeX solutions. The evaluation protocol uses a 1% relative-error tolerance, and for Phy_B a model is credited only if it solves all released variants. Across 15 models, the best Phy_A accuracy is 43.0%; the paper reports an average 22.5% drop from Phy_B reference-variant accuracy to all-variants dynamic accuracy and interprets this as evidence of reliance on memorization and of RL models generalizing better than SFT models. The paper's central quantitative claims are compromised by a scoring mismatch between the static and dynamic Phy_B metrics and by missing uncertainty quantification.","tokens_in":8301,"tokens_out":7278,"duration_ms":81533,"significance":"The resource itself addresses a real gap: static multiple-choice benchmarks are vulnerable to contamination, and dynamic numerical variants are a sensible way to probe robustness. The dataset appears carefully constructed, with manual annotation, independent verification, paraphrasing, and unique numerical answers, and the evaluation covers a broad set of models. If the experimental claims were supported by per-variant accuracy, the benchmark could be a useful community diagnostic. As written, the headline '22.5% drop' and the RL/SFT reversal are not established because they are confounded by the all-or-nothing scoring rule.","major_comments":[{"comment":"The headline dynamic decline is computed under two different scoring rules. In Table 2, 'Stactic Acc' (sic) for Phy_B credits a model for solving one reference variant, while 'Dynamic Acc' credits it only if all three released variants are solved. For any per-variant success probability p, the expected all-variants accuracy is p^3 under independent errors, so a model totally insensitive to perturbation would still show an expected decline of p - p^3 (about 0.33 at p = 0.75). The observed declines, 0.19–0.28, are actually below that benchmark, consistent with positive error correlation or with non-reference variants being easier; in no case does Δ isolate sensitivity to numerical perturbation. The paper never reports average per-variant accuracy on Phy_B, so the central memorization conclusion is not currently supported. Please report per-variant accuracies for each variant and compare static and dynamic performance under a matched scoring rule, with confidence intervals.","section":"§3.3, §4.1, Table 2"},{"comment":"The 'RL Outperforms SFT on Dynamic Questions' subsection is not supported by the reported numbers. Dynamic Acc is a nonlinear function of per-variant accuracy (p^3 under independence), so small differences in per-variant performance can produce the observed ranking reversals, and the correlation structure across variants is unknown. Moreover, Table 2 has no error bars or significance tests on a 100-question subset, and the systems are not explicitly labeled by training paradigm. This claim should either be removed or supported with per-variant accuracies for each group and a statistical comparison.","section":"§4.1"},{"comment":"Phy_A was explicitly constructed by 'experimental screening that identified cases where models tend to produce long responses with low accuracy.' Thus the 43.0% top accuracy on Phy_A is an estimate for an adversarially selected set, not an unbiased estimate of performance on graduate- or Olympiad-level physics. The abstract and §4.1 present the 43% as evidence of a general 'substantial gap'; this framing should be softened or supplemented with human performance or with an unscreened sample of similar problems. The selection rule is a legitimate benchmark-design choice, but it must be reflected in the strength of the claims.","section":"§3.1"}],"minor_comments":[{"comment":"The column header 'Stactic' is a typo, and the text should clarify that the 'Static Acc' column refers to Phy_B reference variants, not to Phy_A, to avoid the ambiguity in the running text.","section":"Table 2"},{"comment":"The opening sentence says 'four widely used large language models' but Table 2 reports fifteen systems; update the sentence accordingly.","section":"§4.1"},{"comment":"The fact that only three variants per Phy_B problem are released in the preprint is important for interpreting the all-variants score and should be stated in the main text (§3.1 or §3.3) rather than in a footnote.","section":"Footnote 2"},{"comment":"The reference 'OpenAI. Learning to reason with llms, Spetember 2024b' contains a typo ('Spetember') and the OpenAI o3 system-card reference is incomplete; also the title line 'AB ENCH -PHYSICS' has a formatting artifact.","section":"References"},{"comment":"Figure 1 has no error bars or confidence intervals and largely repeats Table 2; consider adding uncertainty information or removing the figure.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a potentially useful contribution, but the current version's headline analyses are not yet publishable. The requested per-variant accuracy table and matched-scoring comparison are straightforward and should resolve the main concern; the RL/SFT subsection should be rewritten or dropped. I do not see the issue as unfixable, and the resource itself is worth serious consideration after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe static part of this benchmark is the real contribution: 400 graduate/Olympiad numerical physics problems with strict 1% tolerance and human-verified answers. Top models land below 43% on Phy_A. That is a clean, sobering measurement and worth having in the literature.\n\nThe dynamic Phy_B idea is also genuinely new in physics — a variation engine that perturbs numeric constants while keeping the physical model fixed. But the headline number built on it, the 22.5% average drop from static to dynamic accuracy, is largely an artifact of how the two scores are defined. Static Acc appears to credit solving one reference variant; Dynamic Acc requires all three variants correct. If per-variant accuracy is p, a perfectly robust model would score p^3 on the dynamic subset, so the expected drop is p - p^3. At p=0.75 that is 0.33, bigger than most observed drops in Table 2. The paper never reports mean per-variant accuracy, so the drop does not isolate sensitivity to numeric perturbation. Until that breakdown is provided, the 'memorization' conclusion and the 'RL outperforms SFT on dynamic questions' claim are not supported.\n\nThe remaining issues are softer: only 100 problems with three variants each, no confidence intervals or significance tests, and Phy_A was explicitly screened for questions where models give long, wrong answers — a selection bias worth noting but not disqualifying. I also could not verify the GitHub repo; no commit hash is given.\n\nCredit where due: the construction pipeline (OCR, manual annotation, human re-solving, paraphrasing) is transparent, and the related work is positioned correctly. The static benchmark is usable as-is and deserves serious refereeing. The dynamic claims need a per-variant accuracy table and an expected p^3 baseline before they can be trusted.\n\nRecommendation: send to peer review, with a request for those missing numbers. If they come back clean, this becomes a solid benchmark paper.\n\nBest,","headline":"Static benchmark is useful; the dynamic 22.5% drop mostly reflects the all-variants scoring rule, not memorization.","tokens_in":8862,"tokens_out":3891,"would_cite":true,"duration_ms":40749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ABench-Physics shows that state-of-the-art LLMs solve under half of graduate-level physics problems and lose 22.5 points on average when only the numbers change.","keywords":["physical reasoning evaluation","dynamic benchmark","numerical perturbation","memorization vs generalization","graduate-level physics","large language models","reinforcement learning robustness"],"falsifier":"A decisive control would compare two ways of generating Phy_B variants: one where constants are rescaled by powers of ten, making arithmetic harder but leaving the physics the same, and one where constants are changed so the problem crosses into a different physical regime. If model accuracy drops sharply in the first condition too, the 22.5-point decline is at least partly an arithmetic effect, and the memorization conclusion would need revision; if it drops only in the second, the benchmark cleanly isolates physical modeling.","tokens_in":7855,"feed_emoji":"⚛️","tokens_out":11055,"duration_ms":109158,"temperature":0.7,"pith_summary":"ABench-Physics is a benchmark built to test whether LLMs genuinely reason about physics or just match familiar solution patterns. It pairs a fixed set of 400 graduate- and Olympiad-level numerical problems with a dynamic subset of 100 problems whose numerical constants are automatically perturbed, and it credits a model on the dynamic subset only if it solves every variant of a problem. The paper's central claim is that current state-of-the-art models fail this test: the strongest model solves only about 43% of the static problems, and moving from unperturbed to perturbed variants lowers every model's accuracy, by 22.5 percentage points on average. The authors read this drop as evidence that models lean on superficial pattern matching rather than stable algebraic and physical modeling, and that reinforcement-trained models are somewhat more robust to such perturbations than instruction-tuned ones. If this is right, static benchmarks, especially multiple-choice or single-number ones, overstate LLM physics ability.","feed_headline":"LLMs solve under 43% of graduate physics problems","feed_subtitle":"When problem numbers change, accuracy drops another 22.5 points on average—a sign of memorized patterns.","key_machinery":"The load-bearing mechanism is the parameter variation engine: it takes a Phy_B problem's ground-truth LaTeX solution and automatically regenerates fresh variants by changing the embedded numerical constants, while leaving the physical setup and solution structure intact. Coupled with the all-or-nothing scoring rule, credit only if every variant is solved, this turns a single problem into a probe of whether a model can redo the same physical derivation under shifted numbers. The 1% relative-error tolerance and explicit unit and significant-figure specifications make the numerical answers machine-gradable, and the static Phy_A set supplies a high-difficulty ceiling against which the dynamic drop is measured.","core_discovery":"The paper introduces ABench-Physics, a two-part benchmark for LLM physical reasoning. Phy_A contains 400 fixed, high-difficulty problems drawn from graduate and Olympiad sources; Phy_B contains 100 parameterised problems together with an automatic engine that regenerates variants by perturbing the numerical constants inside the ground-truth LaTeX solutions. Every answer is a number, scored with a 1% relative-error tolerance, and a Phy_B problem counts as solved only if the model answers all of its variants correctly. Evaluated across a range of frontier systems, the paper reports that the strongest model reaches 43.0% on Phy_A, that no model approaches human-level performance on the static set, and that every system loses accuracy on the dynamic variant task, with an average drop of 22.5 percentage points and the largest drop near 28 points. Because the semantic structure of each problem is unchanged, the paper attributes the dynamic drop to reliance on memorized solution patterns rather than robust physical modeling, and notes that RL-trained reasoning models show smaller relative losses and sometimes overtake SFT models that had scored higher on the static set.","pith_inferences":["A sharper test of the memorization explanation would separate arithmetic sensitivity from physics sensitivity: e.g., perturb constants by powers of ten to change computation without changing the physics, versus small changes that cross a physical regime boundary, and compare the two accuracy drops.","Because the preprint releases only three variants per Phy_B problem, the all-or-nothing score is coarse at the per-problem level; evaluating with more variants per template would tighten the estimate of how much of the drop is memorization versus arithmetic.","The same parameter-perturbation design could transfer to other quantitative disciplines, such as chemistry, circuits, or engineering mechanics, where numerical constants carry much of a problem's identity and memorization is easy to mask.","If future models close the dynamic gap, training on procedurally varied physics problems or reinforcement learning against variant-solved rewards would be the natural mechanism, though the paper does not test training interventions."],"forward_implications":["If the benchmark's results hold, current LLMs are far from reliable graduate-level or Olympiad physics: even the best tested system solves only about 43% of the static problems.","A static-only evaluation score materially overstates physical reasoning ability, since every tested model scores lower on the numerically perturbed variants of the same problems.","The average 22.5-point dynamic drop gives a quantitative target for robustness work: models need to retain accuracy when problem constants change without altering the physical model.","RL-trained reasoning models appear to generalize across numerical perturbations better than SFT instruction-tuned models, suggesting a training-direction hypothesis the benchmark can track over time.","The dynamic variation engine and all-variants metric provide a reusable template for contamination-resistant physics evaluation."],"supporting_citations":[{"why":"Supplies the strongest tested baseline, whose 43.0% Phy_A accuracy defines the performance ceiling the paper reports.","marker":"(Google DeepMind, 2025)"},{"why":"The o3-family baselines contribute the largest observed dynamic drop (about 28 points), supporting the fragility finding.","marker":"(OpenAI, 2025b)"},{"why":"DeepSeek-R1 is both the rewrite engine used to paraphrase question scenarios during construction and an RL-tuned baseline in the evaluation.","marker":"(DeepSeek Team, 2025)"},{"why":"UGPhysics provides the prior university-level physics benchmark and evaluation protocol that ABench-Physics extends with dynamic variants and strict numeric tolerance.","marker":"(Xu et al., 2025b)"},{"why":"The contamination survey is cited to justify why static benchmarks overstate ability and why dynamic evaluation is needed.","marker":"(Deng et al., 2024)"},{"why":"The memorization analysis supports the paper's interpretation that dynamic variants separate memorization from genuine generalization.","marker":"(Xie et al., 2025)"}],"fun_headline_variants":["LLMs fail dynamic physics: 22.5-point drop when numbers change","Physics benchmark: LLMs memorize, not reason, under variation","ABench-Physics: LLMs get 43% on static, less on variants","New physics test shows LLMs struggle with problem variation","LLMs' physics reasoning breaks when problem numbers shift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central conclusion assumes that changing only the numbers inside a problem's LaTeX solution leaves the physical model and reasoning required to solve it unchanged, so that a lower score on the variants isolates memorization rather than arithmetic difficulty, rounding, or formatting failures.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail dynamic physics: 22.5-point drop when numbers change","Physics benchmark: LLMs memorize, not reason, under variation","ABench-Physics: LLMs get 43% on static, less on variants","New physics test shows LLMs struggle with problem variation","LLMs' physics reasoning breaks when problem numbers shift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1344,"prompt_tokens":975,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":279}},"tokens_in":591,"tokens_out":369,"duration_ms":4963,"temperature":1.0,"reasoning_tokens":279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:39:28.505998+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive control would compare two ways of generating Phy_B variants: one where constants are rescaled by powers of ten, making arithmetic harder but leaving the physics the same, and one where constants are changed so the problem crosses into a different physical regime. If model accuracy drops sharply in the first condition too, the 22.5-point decline is at least partly an arithmetic effect, and the memorization conclusion would need revision; if it drops only in the second, the benchmark cleanly isolates physical modeling.","supporting_citations":[],"review_version":1}