{"id":"8940c7b0-241d-48a6-9ece-7502413be2ba","arxiv_id":"2507.21130","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"INTEGRALBENCH is a 317-problem benchmark for definite integrals with dual symbolic and numerical ground truth, on which nine LLMs score between 22 and 56 percent.","lead":"INTEGRALBENCH is a new benchmark of 317 advanced definite-integral problems, each with a symbolic answer, a numerical answer, and a human difficulty rating. Nine language models were tested, and even the strongest solved only about half of the symbolic problems, with accuracy dropping sharply on harder problems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §3.1 symbolic-validation protocol is self-referential and appears to produce inflated symbolic accuracies (PASS@3(Sym) > PASS@3(Num) for all models), so the headline symbolic baselines are not trustworthy until independently re-verified.","rationale":"The paper is best read as a benchmark-plus-baseline submission: the contribution is the dataset and the evaluation framework. The dataset construction is described with manual checks, source attribution, difficulty annotations, and a cost breakdown; those parts are plausible and useful. The weakest link is the evaluation protocol for symbolic answers because it determines the headline numbers that the paper advertises. The reader's weakest_assumption already identified the self-referential symbolic verification, and I agree. I would sharpen it: the observed pattern in Table 2 (symbolic PASS uniformly exceeding numerical PASS) is exactly what one expects when the symbolic check is a self-consistency test rather than a correctness test, since a model can simply repeat its own decimal answer as the 'symbolic' expression. This makes the concern concrete and directly connected to the reported baselines. The independent re-check I propose would settle whether the symbolic accuracies are inflated; if they are, the symbolic baselines and Finding 1 need revision, while the numerical baselines and dataset may remain usable after re-validation. That is why I keep the reader's CONDITIONAL verdict rather than moving to REJECT: the issue is fixable and does not, on the evidence currently in hand, invalidate the numerical part of the benchmark. But the condition is not optional: the paper must re-run symbolic verification against independent ground truth before the claimed symbolic accuracies can be cited.","tokens_in":18931,"tokens_out":8504,"duration_ms":93788,"concrete_test":"Take all PASS@3 symbolic outputs for Qwen3-235B-A22B and DeepSeek-R1. For each, compute the numerical value of the model's symbolic answer with mpmath at dps=30 and compare it to a high-precision independent numerical integration of the original integral (e.g., mpmath.quad or Mathematica NIntegrate), not to the model's numerical_answer. Where feasible, also use SymPy/Mathematica to check the model's expression against the dataset ground-truth symbolic expression. Count how many outputs that the paper counts as symbolic PASS fail this independent comparison; if the count is nonzero, the §3.1 validation is producing false positives and the reported symbolic accuracies are inflated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is the symbolic-validation rule in §3.1: a symbolic answer is declared correct when LLM-generated code evaluates it to within 1e-6 of the model's own numerical_answer, not to the ground truth. This does not verify symbolic correctness at all. A model can satisfy the check with a wrong expression s whose numerical value equals its wrong decimal n; the degenerate case is putting the same decimal string in both JSON fields, which the evaluator then treats as a constant. The paper's own Table 2 shows PASS@3(Sym) > PASS@3(Num) for all nine models. Under the stated self-consistency rule, that is exactly the expected signature of models submitting numeric constants or mutually consistent wrong answers as their 'symbolic' answer. Consequently, the headline symbolic accuracies (e.g., Qwen3-235B-A22B 56.15%) are not measurements of symbolic integral-solving ability. The numerical ground truth is also produced by LLM-generated scripts rather than an independent CAS, but the symbolic-validation flaw alone is sufficient to cast doubt on the central empirical claim about symbolic baselines. Related Work's criticism of 'circular evaluation' in previous benchmarks is thus ironically applicable to §3.1 itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"INTEGRALBENCH introduces a new benchmark of 317 graduate-level definite integral problems with symbolic and numerical ground truth, manual difficulty ratings, and source attributions. The paper describes a construction pipeline based on OCR, manual annotation, and LLM-assisted parameter instantiation, then evaluates nine LLMs under PASS@3 and ALL@3 metrics. The main empirical findings are that larger models tend to perform better, that Qwen3-235B-A22B achieves the highest accuracies (50.16% numerical, 56.15% symbolic PASS@3), that performance declines with annotated difficulty, and that inference-time scaling curves show model-specific plateaus. The paper also provides a failure-mode analysis and a cost breakdown for dataset construction.","tokens_in":19222,"tokens_out":4411,"duration_ms":47275,"significance":"If the benchmark and its evaluation protocol were sound, INTEGRALBENCH would be a useful focused resource: it has a moderate number of challenging problems, manual difficulty annotations, source metadata, a publicly released dataset, a transparent cost breakdown, and an interesting failure-mode taxonomy. The manual curation effort and the decision to separate symbolic from numerical accuracy are genuinely valuable design choices. However, the central evaluation protocol for symbolic answers is self-referential, and the numerical ground truth is not independently verified, so the headline symbolic accuracies and the symbolic-versus-numerical comparison are not currently trustworthy. The paper would be significant after the validation chain is reworked and the numbers are recomputed; as it stands, the empirical contribution is not reliable.","major_comments":[{"comment":"The symbolic-answer verification rule compares LLM-generated Python evaluation of the model's symbolic answer against the same model's own numerical answer, rather than against the dataset ground truth. This is a self-consistency check, not a correctness check. A model that emits the same decimal string in both the \"answer\" and \"numerical_answer\" JSON fields will pass the check even when both are wrong, since the evaluator will evaluate that constant string. The uniform pattern in Table 2, where PASS@3(Sym) exceeds PASS@3(Num) for all nine models, is exactly the signature one expects from this circular rule: the symbolic score is inflated because it only requires agreement with the model's own numerical output. The headline symbolic baselines, such as Qwen3-235B-A22B at 56.15% PASS@3(Sym), are therefore not measurements of symbolic integration ability. The fix is to validate the model's symbolic answer against the independently computed ground-truth numerical value (or to check symbolic equivalence with a CAS such as SymPy or Mathematica).","section":"Section 3.1, Answer validation"},{"comment":"The numerical ground truth is derived from LLM-generated Python scripts that evaluate the symbolic ground truth, rather than from independent numerical integration or a computer algebra system. As described in Section 2.2, DeepSeek V3 generates scripts that substitute parameter values and evaluate the analytical expressions, and the numerical ground truth is the output of those scripts. No independent cross-check with an established CAS or adaptive quadrature is reported. Since the numerical ground truth is the reference for all numerical-accuracy claims and for the proposed fix to the symbolic-validation protocol, this is a load-bearing gap. The paper must provide independent numerical ground truth, at least for a substantial random sample, before the reported accuracies can be considered verified.","section":"Section 2.2, Ground truth solutions"},{"comment":"All accuracy comparisons are reported as point estimates with no confidence intervals or significance tests. With 317 problems and N=3 attempts, adjacent rankings can easily be within sampling noise: for example, DeepSeek-R1 and Doubao are both reported at 45.43% PASS@3(Num), and the gap between Qwen3 and DeepSeek-R1 in numerical PASS@3 is only 4.73 percentage points. The claims in Sections 3.2-3.4 about which models are better and about the shape of inference-time scaling curves are therefore stronger than the evidence supports. Bootstrap confidence intervals or per-model standard errors should be reported, and the comparative statements should be softened in their absence.","section":"Table 2, Sections 3.2-3.4"},{"comment":"The paper criticizes prior benchmarks for relying on \"circular evaluation and multi-solution consistency checks,\" but the symbolic-validation protocol in Section 3.1 is itself a multi-solution consistency check between two outputs of the same model. Recognizing this internal inconsistency is important not only for presentation but also for the framing of the contribution: the paper should acknowledge that its own symbolic metric is a consistency metric and either replace it with a ground-truth-based check or explicitly restrict the symbolic claims to self-consistency.","section":"Section 5, Related Work"}],"minor_comments":[{"comment":"The claim that \"correct symbolic solutions\" are paired with incorrect numerical evaluations is not supported by the validation protocol, because symbolic correctness was never checked against ground truth. The observed Sym > Num gap should not be interpreted as \"strong symbolic reasoning but weak numerical computation\" without fixing the validation rule first.","section":"Section 3.5, Failure mode analysis"},{"comment":"The legend of the inference-time scaling figure includes only five of the nine evaluated models; please clarify whether the other four were omitted and, if so, why, or include all models for consistency with the rest of the evaluation.","section":"Figure 6, Section 3.4"},{"comment":"The cost table labels the API subtotal as \"$5.00\" even though the rows sum to $0.83 + $4.17; consider clarifying that this excludes the cost of the DeepSeek calls used for symbolic verification during evaluation, if that is not already included.","section":"Section 2.3, Table 1"},{"comment":"The abstract and conclusion describe INTEGRALBENCH as a \"rigorous evaluation framework.\" Given the circular validation chain described in Section 3.1, this descriptor is currently overstated; the rigor claim should be re-earned after the validation protocol is repaired.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The circularity in Section 3.1 is severe enough that the symbolic accuracy numbers should not be cited without re-verification, and the numerical ground truth also lacks an independent cross-check. However, the dataset construction effort, manual annotation, and public release suggest the paper can be repaired: the authors should re-run evaluation against independent ground truth and either remove the symbolic-consistency metric or replace it with a genuine symbolic-equivalence check. I would lean toward major revision rather than rejection, because the empirical claims are clearly intended to be load-bearing and the necessary corrections are within the scope of a substantial revision. The paper's own critique of circular evaluation in prior work makes this oversight especially visible, and the authors should address it head-on."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset is real and fills a gap, but the symbolic validation as written is uninformative, so the headline symbolic numbers in Table 2 shouldn't be taken at face value. The fix is straightforward, though: check model symbolic answers against ground truth with an independent CAS, rerun the numbers, and report the corrected table.\n\nWhat's genuinely new: 317 graduate-level definite integrals with both symbolic and numerical ground truths, manual difficulty ratings, source attribution, and a public dataset. The construction pipeline is honestly described, and I appreciate the explicit dollar-cost table and the failure-mode catalog. The evaluation runs nine current models with a standard prompt and reports PASS@3/ALL@3. That part is solid scaffolding.\n\nThe soft spots are the usual benchmark-construction hazards, plus one bigger one. The big one is §3.1's symbolic verification. A symbolic answer is declared correct when LLM-generated code evaluates it to within 1e-6 of the model's own numerical answer—not the ground truth. That is a self-consistency check, not a correctness check. A model that emits a wrong expression and a matching wrong decimal passes; the degenerate case is putting the same decimal in both JSON fields. The fact that every model's PASS@3(Sym) exceeds its PASS@3(Num) is exactly the pattern you'd expect from that loophole. So the headline '56.15% symbolic' for Qwen3 is not a measurement of symbolic integration ability. The numerical ground truths are also derived from the symbolic answers via LLM-generated code, not from an independent CAS, so the symbolic and numerical pipelines are not independent.\n\nSmaller issues: the intro promises a 'novel term-rewriting method' for variations, but Section 2 describes parameter instantiation, not term rewriting—that claim should be fixed or substantiated. And with N=3 per problem and no confidence intervals, the reported model rankings are not statistically grounded.\n\nOverall: the dataset artifact has value, the paper is clearly written, and the flaws are fixable. But the reported baselines need re-verification before anyone should use them. I'd send it to a serious referee with a clear demand: recompute all ground truths with an independent CAS and re-score symbolic answers against those ground truths. If the authors do that, the corrected Table 2 would be a useful benchmark for the LLM-math crowd. As-is, it's a solid dataset in need of a validation overhaul.","headline":"Real dataset, broken symbolic-validation rule; headline symbolic scores are self-consistency scores, not correctness scores.","tokens_in":19677,"tokens_out":2470,"would_cite":false,"duration_ms":26075,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces INTEGRALBENCH, a 317-problem benchmark of graduate-level definite integrals with paired symbolic and numerical ground truth, and reports baseline results for nine LLMs, with the best model reaching 56.15% symbolic…","keywords":["definite integrals","LLM evaluation","symbolic computation","mathematical reasoning","benchmark dataset","difficulty annotation","numerical verification","PASS@3 metrics"],"falsifier":"Take a sample of INTEGRALBENCH symbolic failures, have an independent symbolic computation engine check each model's expression for equivalence with the ground-truth symbolic answer, and recompute symbolic PASS@3; if the independently checked accuracy is materially lower than the reported ~56%, the dual-verification scoring overstates symbolic capability. A cheaper probe: feed the verifier a deliberately wrong closed form whose decimal value was chosen to match the ground truth to 10 digits and see whether it passes.","tokens_in":18787,"feed_emoji":"🧮","tokens_out":9222,"duration_ms":85601,"temperature":0.7,"pith_summary":"INTEGRALBENCH is a 317-problem benchmark of graduate-level definite integrals, each with a symbolic answer, a numerical value, a manual difficulty rating, and a source attribution. The paper's central claim is that this dual ground truth plus difficulty stratification makes definite integrals a measurable test bed for LLM mathematical reasoning, and that current models are far from solving it: the best of nine evaluated models, Qwen3-235B-A22B, reaches 56.15% symbolic PASS@3 and 50.16% numerical PASS@3. The paper also establishes that accuracy falls sharply as difficulty rises across all models, validating the difficulty annotations, and that a 32B model (QwQ) outperforms several larger models, arguing that architecture matters as much as scale.","feed_headline":"Best LLM solves 56% of graduate-level integrals","feed_subtitle":"A 317-problem benchmark with symbolic and numeric ground truth shows where nine models succeed and stall.","key_machinery":"The load-bearing machinery is the dual-verification evaluation pipeline: a numerical check that compares the model's decimal to ground truth with error below $10^{-6}$, and a symbolic check in which another LLM writes Python/mpmath code to evaluate the model's symbolic answer and the answer is accepted only if that computed value matches the model's own numerical answer within the same tolerance. The benchmark construction pipeline is equally central: bounding-box annotation of problem images, OCR to LaTeX with manual verification, and parameter instantiation for integrals containing free variables, with LLM-assisted random sampling and manual confirmation. Difficulty ratings from 1 to 5 are manually assigned and then validated by the observed difficulty-accuracy correlation. The metrics PASS@3 and ALL@3 convert generation randomness into peak-ability and consistency measurements.","core_discovery":"On its own terms, the paper's contribution is a focused evaluation framework for definite integrals. It claims that no existing benchmark offers enough challenging integrals, integral-specific metrics, or difficulty gradation, and that INTEGRALBENCH fills that gap with 317 verified problems collected from one graduate textbook and three competition series. Each problem stores the integral body, unsimplified and simplified symbolic answers, a numerical answer, difficulty, and source. The evaluation protocol generates three independent attempts per model and reports PASS@3 and ALL@3; numerical answers are checked against ground truth within $10^{-6}$, and symbolic answers are checked by generating code that evaluates the expression and comparing that value with the model's own numerical answer. The reported baseline is the strongest evidence: Qwen3-235B-A22B at 56.15% symbolic PASS@3, with all models declining sharply on difficulty 4-5 problems and exhibiting a consistent symbolic-over-numerical accuracy gap.","pith_inferences":["The symbolic verification step implicitly assumes that the LLM-generated evaluator code is faithful to the symbolic expression; an independent check with a computer algebra system would convert the reported symbolic accuracy into a measure of true symbolic equivalence, and would likely yield lower numbers.","Because many problems contain free variables, the parameter-instantiation scheme can regenerate fresh variants of the same integral, offering a built-in contamination-resistant re-testing procedure the paper does not demonstrate.","The large drop from PASS@3 to ALL@3 (for example Qwen3 symbolic 56.15% vs 38.17%) suggests that peak performance is partly a sampling artifact; comparing PASS@16 would likely reorder models whose success variance differs.","The failure modes documented—truncation, circular repetition, and JSON format violations—look like general structured-output failures, so the same evaluation harness could be applied to other computation-heavy math tasks beyond integrals."],"forward_implications":["Future LLM work on definite integrals can be compared against a fixed, public 317-problem set with a clear pass threshold, making claims about integration skill reproducible.","Because all nine models degrade sharply on difficulty 4-5 problems, the difficulty ratings can be used to target model weaknesses rather than just report averages.","The 32B QwQ outperforming larger models means parameter count alone is a poor predictor of symbolic integration ability, so evaluations should report architecture and training details alongside size.","The consistent symbolic-over-numerical gap indicates models can derive expressions they cannot evaluate reliably, which motivates tool-augmented pipelines where external evaluators finish the computation.","Inference-time scaling plateaus imply that there is a model-specific token budget beyond which additional reasoning yields little accuracy gain, so token efficiency should be reported alongside accuracy."],"supporting_citations":[{"why":"Prior MATH benchmark; establishes that existing math benchmarks contain few challenging integrals, motivating the new benchmark.","marker":"Hendrycks et al., 2021b"},{"why":"GSM8K as representative elementary math benchmark; the contrast defines the intended graduate-level difficulty.","marker":"Cobbe et al., 2021b"},{"why":"MathVista, used as an example of multimodal math evaluation that lacks integral-specific metrics.","marker":"Lu et al., 2024"},{"why":"Affinity propagation clustering; the method used to show the dataset spans 56 diverse problem clusters.","marker":"Frey & Dueck, 2007"},{"why":"Qwen3 technical report; the model that achieves the reported best baseline.","marker":"Yang et al., 2025"},{"why":"QwQ-32B report; the 32B model whose strong showing supports the architecture-over-size claim.","marker":"Team, 2025"},{"why":"PutnamBench; prior use of functional variations to limit contamination, which INTEGRALBENCH's parameter instantiation extends.","marker":"Tsoukalas et al., 2024"}],"fun_headline_variants":["INTEGRALBENCH: best LLM hits 56% on definite integrals","LLMs on definite integrals: benchmark shows 56% ceiling","Definite integral benchmark: best model passes 56%","INTEGRALBENCH reveals LLM gaps on definite integrals","Benchmarking LLMs: definite integrals stump best at 56%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A symbolic answer is judged correct when an automatically generated program evaluates it to the same decimal as the model's own numerical answer; if a wrong expression and a wrong decimal happen to agree, the answer is scored correct even though the symbolic claim is false.","fun_headline_variants_meta":{"raw":{"variants":["INTEGRALBENCH: best LLM hits 56% on definite integrals","LLMs on definite integrals: benchmark shows 56% ceiling","Definite integral benchmark: best model passes 56%","INTEGRALBENCH reveals LLM gaps on definite integrals","Benchmarking LLMs: definite integrals stump best at 56%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001198,"raw_usage":{"total_tokens":4871,"prompt_tokens":811,"completion_tokens":4060,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":3967}},"tokens_in":427,"tokens_out":4060,"duration_ms":25740,"temperature":1.0,"reasoning_tokens":3967,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:11:04.112348+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of INTEGRALBENCH symbolic failures, have an independent symbolic computation engine check each model's expression for equivalence with the ground-truth symbolic answer, and recompute symbolic PASS@3; if the independently checked accuracy is materially lower than the reported ~56%, the dual-verification scoring overstates symbolic capability. A cheaper probe: feed the verifier a deliberately wrong closed form whose decimal value was chosen to match the ground truth to 10 digits and see whether it passes.","supporting_citations":[],"review_version":1}