{"id":"c5562fe1-36d5-4194-bb60-a72a25570500","arxiv_id":"2506.20729","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On TPBench physics problems, a SymPy-augmented stepwise verifier selected more correct LLM solutions than majority vote or LLM self-grading, raising Gemini 2.5 Pro Level 5 accuracy from 29.3% to 54.5%, though gains are inconsistent and statistically untested.","lead":"The authors compared test-time scaling methods on TPBench physics problems and introduced a SymPy-based stepwise verifier that selects LLM solutions by checking each calculation symbolically. The method improved Level 5 accuracy with Gemini 2.5 Pro from 29.3% single-attempt to 54.5%, but the gains are inconsistent across models and difficulty levels and lack error bars.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed significant outperformance is not statistically supported: with 11 Level-5 problems, the headline 29.3% to 54.5% gain is about 3 problems, and no significance tests or per-problem data are reported.","rationale":"The reader's conditional verdict is appropriate. My stress test agrees with the reader that the auto-verification oracle assumption is fragile, but I consider the small-sample statistical reliability to be the single most load-bearing issue for the headline comparative claim, because the paper's own numbers are internally inconsistent across levels and models and no error bars appear. The Level-5 Gemini-2.5-Pro improvement, the paper's flagship result, is about three problems out of eleven and fails a simple significance check. The missing manual-grading baseline promised in Section 4.6 is a concrete, locatable omission that compounds the uncertainty: the verifier's own accuracy is unvalidated, so we cannot attribute the observed gains to symbolic verification as opposed to additional LLM deliberation. The case studies in Appendices C and D do provide genuine evidence that SymPy checks can catch algebraic errors, which supports the method's plausibility, but they do not establish that the end-to-end accuracy differences are significant. The paper should keep the CONDITIONAL verdict, with the added condition that the authors supply per-problem results and appropriate significance or confidence analyses, or temper the 'significant outperformance' claim to a preliminary observation.","tokens_in":19452,"tokens_out":7512,"duration_ms":79557,"concrete_test":"Request the per-problem success/failure vectors for Tables 1-3 and the auto-verification outputs, then run a paired permutation test (or McNemar test) comparing SymPy Verifier against Majority Vote and against Single Attempt for each level and model. If the 95% confidence interval for the Level-5 Gemini-2.5-Pro difference includes 0, or the two-sided p-value exceeds 0.05, the 'significantly outperforms' claim is unsupported. As a secondary check, perform the promised manual-grading precision/recall study of the SymPy verifier described in Section 4.6 and report the confusion matrix.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the SymPy verifier 'significantly outperforms' existing methods rests on accuracy differences computed over only 11-14 problems per level (Section 4.1). For the headline Level-5 Gemini-2.5-Pro result (Table 1), 54.5% vs 29.3% corresponds to roughly 6 vs 3.2 correct problems out of 11; a one-sided binomial test against p0=0.293 gives p approximately 0.065, so conventional significance is not reached. The results are also inconsistent: on Gemini-2.5-Pro Level 4, SymPy Verifier (71.4%) is below Majority Vote (78.6%); on Gemini-2.0-Flash Level 5 they tie at 9.1%; and on o4-mini-high Level 4 they tie at 71.4%. No confidence intervals, resampling tests, or per-problem success/failure vectors are provided, so the comparison cannot be properly paired or assessed for noise. In addition, Section 4.6 promises a manual-grading baseline for the verifier ('quantifies its accuracy against a manually graded baseline') that is never reported; only usage statistics and capability tables appear. Without this, the verifier's precision and recall are unknown, and the source of the Level-5 gain (symbolic checking vs. extra LLM deliberation vs. chance) is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates a range of test-time scaling methods on the TPBench theoretical-physics dataset (levels 3-5) and on a 20-problem AIME subset, and proposes a novel 'SymPy-augmented weak verifier' that selects among N=50 candidate solutions by having an LLM agent verify each derivation step with SymPy scripts. The central claim is that this symbolic verifier significantly outperforms single-attempt, majority-vote, simple weak-verifier, and sequential multi-round baselines, approaching the Best-of-N upper bound. Experiments are run with Gemini 2.5 Pro, Gemini 2.0 Flash, and o4-mini-high, and the paper includes two detailed case studies showing the verifier catching real algebraic errors.","tokens_in":1793,"tokens_out":1684,"duration_ms":56543,"significance":"If the headline claim were established, the paper would make a useful contribution: it is one of the few systematic comparisons of test-time scaling on research-level physics problems, and the appended case studies (Apps. C and D) give reproducible SymPy scripts and show the verifier catching concrete, load-bearing algebraic errors (a spurious factor of sigma in truncated Gaussian moments, and a missing a_e factor in the imaginary part of a phase integral). The prompts and pipeline details are described with enough specificity to be reimplemented, which is a real strength. However, the empirical evidence base is too small to support the 'significantly outperforms' claim as stated, and the promised manual-grading validation of the verifier is absent. The contribution is therefore a promising method plus an honest account of its failure modes, but not yet a demonstrated significant improvement.","major_comments":[{"comment":"The headline comparison is not statistically supported. Level 5 contains only 11 problems, so the claimed improvement from 29.3% to 54.5% is roughly 3 to 6 correct problems; even under an idealized one-sided binomial test with p0=0.293, the p-value is about 0.065, and this ignores per-problem heterogeneity and the fact that the baseline is an average over 50 attempts. No confidence intervals, resampling tests, or per-problem success/failure vectors are provided, so the comparison cannot be assessed for noise or paired structure. Please report per-problem outcomes and appropriate paired tests, or temper the significance claims.","section":"Section 4.1 and Table 1"},{"comment":"The text promises that this section 'quantifies its accuracy against a manually graded baseline', but no manual-grading results, precision/recall figures, or confusion-matrix numbers are actually reported; only usage statistics (Table 5) and a capability list (Table 6) appear. This is load-bearing because without such a baseline the source of the Level-5 gain is unidentified: it could come from symbolic checking, from the extra LLM deliberation during step-by-step grading, or from chance. Please include the promised manual-grading evaluation.","section":"Section 4.6"},{"comment":"The claimed superiority of the SymPy verifier is inconsistent across model-difficulty combinations. On Gemini 2.5 Pro Level 4 the SymPy verifier (71.4%) is below Majority Vote (78.6%); on Gemini 2.0 Flash Level 5 the two methods tie at 9.1%; and on o4-mini-high Level 4 they tie at 71.4%. Given sample sizes of 11-14 problems, these discrepancies are exactly the kind of noise the paper needs to rule out before claiming a significant, consistent improvement in Sections 4.3 and 4.4.","section":"Tables 1-3"},{"comment":"All accuracy numbers inherit the TPBench auto-verification pipeline as ground truth, but the paper does not validate that this numeric test-case oracle agrees with expert physics grading. Since every method comparison in Tables 1-4 is filtered through this oracle, a systematic disagreement (a wrong derivation passing or a correct derivation failing the numeric tests) would change the rankings; at minimum this assumption should be stated explicitly and its limitations discussed.","section":"Section 3.1"},{"comment":"The claim that 'accuracy improves with an increasing number of attempts for all difficulty levels' and that the SymPy verifier 'consistently outperforms' Majority Vote is presented without error bars or uncertainty quantification, and the underlying level sizes are 11-14 problems. The 'saturation' interpretation is also unsupported without a plateau criterion. Please show per-point uncertainty or otherwise qualify these scaling claims.","section":"Section 4.4 and Figure 2"}],"minor_comments":[{"comment":"The caption and the body text describe different bar colors; the caption says red bars are unique attempts and blue bars are correct attempts, while Section 4.2 refers to unique solutions as blue bars and to a best-case 'orange bars' that the caption does not define. The figure and caption must be reconciled.","section":"Figure 1"},{"comment":"The paper says results cover levels 3 to 5, but Tables 1 and 3 report only levels 4 and 5. Please either add the Level-3 rows or explain their omission for those models.","section":"Tables 1-3"},{"comment":"The text reports '54.4%' for the Gemini 2.5 Pro Level-5 SymPy verifier, while Table 1 shows 54.5%; this is a minor typo but should be corrected.","section":"Section 4.3"},{"comment":"Algorithm 2 line 11 selects {s_i | V_i = V_max}, whereas Section 3.5 defines S_best using a tolerance delta = 0.05 (i.e., V_i >= max V_j - delta). The algorithm and text should be made consistent.","section":"Algorithm 2"},{"comment":"The paper uses 'theoretical Best-of-N' for what is actually the empirical maximum over the 50 sampled solutions; this is an empirical upper bound for the sample, not a theoretical quantity, and the terminology should be adjusted.","section":"Section 4.3"},{"comment":"The paper mentions 'recall failures' of the verifier, but no recall or precision numbers are given anywhere; the promise of Section 4.6 (see major comment 2) would resolve this gap.","section":"Section 4.6"},{"comment":"The token-cost plot (Figure 3) and the scaling plot (Figure 2) would be more informative with per-method total cost or a cost-accuracy frontier, since the paper motivates test-time scaling as a compute-efficient alternative; the current comparison mixes different per-sample budgets.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the benchmark TPBench is authored by a largely overlapping group (Chung et al., including several of the current authors) and the paper uses the TPBench auto-verifier as ground truth. This is not evidence of fabrication, but it is a mild self-benchmarking concern that should be visible to readers; the paper should state this connection explicitly. The main issue is statistical: with 11-14 problems per level, the central comparative claim is not yet established, and the promised manual-grading validation is missing. Both are fixable in revision within the paper's scope, so I do not recommend rejection. The venue fit is reasonable for an empirical methods paper, though the framing should not overstate the significance of differences of one or two problems."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: genuinely useful application of known test-time scaling tricks to a hard physics benchmark, with a step-wise SymPy verifier that demonstrably catches real algebraic errors. But the headline claim of significant gains over baselines does not survive contact with the per-level sample sizes: 11-14 problems per level, differences of a couple of problems, no error bars, no paired tests, and inconsistent ordering across models (e.g., Gemini-2.5-Pro Level 4 gives SymPy 71.4% vs Majority Vote 78.6%). A one-sided binomial on the Level-5 29.3% to 54.5% change gives p about 0.065, so conventional significance is not reached. What is actually new: a comparison of standard parallel and sequential scaling methods on TPBench, and a verifier that asks an LLM to check each derivation step with SymPy before selecting among candidates. The case studies in Appendices C and D are the real evidence. SymPy catches a missing factor of sigma in the halo-bias truncated Gaussian moments and a missing a_e in the Bogoliubov phase integral. That is credible and useful, and it shows the method has concrete value beyond a vague LLM-as-judge approach. The soft spots are mostly statistical and structural. The tiny per-level sample sizes make the central claim fragile, and the paper never reports confidence intervals, resampling tests, or per-problem outcomes. Section 4.6 explicitly promises a manually graded baseline for verifier accuracy that never appears; the section only gives usage statistics and capability tables. The auto-verifier from TPBench is taken as ground truth without validation against expert grading, and since the same group built TPBench, there is a mild self-benchmarking flavor, though selection itself is driven by SymPy outputs rather than the auto-verifier. No code or data are shipped, so the pipeline is not reproducible as described. These are addressable, not fatal. The authors also honestly list verifier limitations on tensors and path integrals, which is to their credit. Who this is for: people building LLM-plus-symbolic tools for physics, and benchmark developers who care about evaluation methodology. I would send it to a serious referee because the core idea is plausible and the case studies are genuine, but I would demand the missing statistical analysis, the manual grading baseline, and code/data release before publication.","headline":"Useful empirical study of test-time scaling on a hard physics benchmark, with a SymPy step-verifier that catches real errors, but headline gains are not statistically supported at these sample sizes and a promised manual-grading baseline is missing.","tokens_in":742,"tokens_out":1557,"would_cite":false,"duration_ms":35057,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A SymPy-augmented step verifier nearly doubles LLM accuracy on theoretical physics problems.","keywords":["test-time scaling","symbolic verification","SymPy","large language models","theoretical physics reasoning","TPBench","weak verifier","parallel sampling"],"falsifier":"Have expert physicists independently grade the generated solution pools from Tables 1-4 and compare their verdicts with the TPBench auto-verifier's pass/fail labels; any nontrivial disagreement would rescale every method comparison reported in the paper.","tokens_in":19182,"feed_emoji":"🧮","tokens_out":8757,"duration_ms":82357,"temperature":0.7,"pith_summary":"The paper asks whether test-time scaling lessons from math benchmarks transfer to advanced theoretical physics, and finds a partial answer: parallel sampling helps, but the bottleneck is picking the correct solution out of the pool. An LLM asked to grade its own solutions barely improves on a single attempt, while a step-wise verifier that re-runs each calculation with SymPy raises Gemini 2.5 Pro accuracy on TPBench Level 5 from 29.3% to 54.5%, close to the 63.6% Best-of-N oracle. Sequential multi-round reasoning gives little or no gain on hard problems. The paper concludes that domain-specific symbolic verification, not generic self-grading, is what unlocks test-time scaling for physics.","feed_headline":"SymPy step checks lift physics LLM accuracy to 54.5%","feed_subtitle":"On TPBench level 5, the symbolic verifier beats majority vote and nears the 63.6% oracle ceiling.","key_machinery":"The load-bearing object is the step-wise symbolic weak verifier: a SymPy-augmented grading agent that numbers every derivation step in a candidate solution, generates a SymPy script for each calculable step, executes it, and compares the output against the written step before assigning a binary score. It prunes functionally equivalent candidates by evaluating each proposed function on five test inputs, then keeps the solutions whose average verification score lies within a tolerance of 0.05 of the maximum, resolving ties with pairwise LLM comparisons. The tool catches concrete algebraic and calculus errors, such as a Gaussian-moment integral off by a factor of $\\sigma$ and a phase-integral imaginary part missing a factor of $a_e$, which are exactly the errors that sink LLM self-grading.","core_discovery":"The central discovery is that correctness detection, not solution generation, is the limiting factor when scaling LLM inference on theoretical physics problems. Across repeated samples, a correct answer is often present in the pool, but simple self-grading picks badly because LLMs approve flawed derivations. The paper's remedy is a weak verifier that decomposes each candidate solution into steps, writes a SymPy script for roughly 83% of those steps, executes it in a sandbox, and compares the symbolic output to the step's claim. With scores averaged over ten verifier passes and a pairwise tie-break, this method selects the correct solution well enough to approach the Best-of-N upper bound, whereas the same pipeline with an LLM-only grader does not beat single-attempt accuracy. On AIME the symbolic verifier ties majority vote, suggesting the benefit is largest when calculations are algebraically heavy.","pith_inferences":["A natural extension is to feed the same step-wise SymPy checks back into solution generation, letting the model repair a step as it is written rather than only selecting among finished candidates.","Replacing the TPBench auto-verifier with expert human grading on a subset would calibrate every reported accuracy, since the current oracle is judge, jury, and ground truth.","Because the verifier's power comes from reducing derivations to executable symbolic computation, the same recipe is likely to transfer to other computation-heavy sciences where answers reduce to algebra.","The near-oracle selection rate implies the remaining gap to Best-of-N is a verifier-precision problem, so improving tensor- and path-integral verification may raise accuracy more than adding more samples."],"forward_implications":["On TPBench, parallel sampling paired with the SymPy verifier is a stronger use of test-time compute than sequential multi-round reasoning, which shows little or no gain at level 5.","A simple LLM self-grader does not beat single-attempt accuracy in the same pipeline, consistent with the paper's evidence that LLMs are poor at spotting their own calculation errors.","The symbolic verifier's advantage is larger on TPBench than on the AIME subset, where it ties majority vote, indicating that step-wise symbolic checks matter most for calculation-heavy physics derivations.","Accuracy of both majority vote and the SymPy verifier improves with the number of parallel attempts and saturates near 50 samples, matching known inference-scaling saturation behavior.","The verifier's known blind spots, such as tensor manipulation in general relativity and advanced path integrals, limit how much selection quality it can recover on those problem types."],"supporting_citations":[{"why":"Supplies the TPBench dataset and the auto-verification pipeline that judges every solution.","marker":"Chung et al. (2025)"},{"why":"Defines the sample-verify-scale framework that both weak-verifier strategies adapt.","marker":"Zhao et al. (2025)"},{"why":"Establishes repeated-sampling scaling and motivates the Best-of-N oracle upper bound.","marker":"Brown et al. (2024)"},{"why":"Introduces the sequential budget-forcing method whose limitations the paper documents.","marker":"Muennighoff et al. (2025)"},{"why":"Shows LLMs cannot reliably correct their own reasoning, explaining the simple verifier's failures.","marker":"Huang et al. (2024)"},{"why":"Provides the inference-scaling saturation curves that the paper's N-scaling results mirror.","marker":"Wu et al. (2025)"}],"fun_headline_variants":["Symbolic step checks beat majority vote on physics LLM","Physics LLM gets 54.5% via symbolic step verification","Symbolic verifier closes gap to oracle on physics reasoning","Correctness detection is the bottleneck in physics LLM scaling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All accuracy numbers rest on the TPBench auto-verifier being a faithful oracle, marking a solution correct only when its submitted Python function passes predefined numeric tests, and the paper never checks that these verdicts agree with expert physics grading.","fun_headline_variants_meta":{"raw":{"variants":["Symbolic step checks beat majority vote on physics LLM","Physics LLM gets 54.5% via symbolic step verification","Symbolic verifier closes gap to oracle on physics reasoning","Correctness detection is the bottleneck in physics LLM scaling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000769,"raw_usage":{"total_tokens":3371,"prompt_tokens":871,"completion_tokens":2500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":2431}},"tokens_in":487,"tokens_out":2500,"duration_ms":18809,"temperature":1.0,"reasoning_tokens":2431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:43:49.679263+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have expert physicists independently grade the generated solution pools from Tables 1-4 and compare their verdicts with the TPBench auto-verifier's pass/fail labels; any nontrivial disagreement would rescale every method comparison reported in the paper.","supporting_citations":[],"review_version":1}