{"id":"2820a084-de11-46f6-a7b4-4b520c409852","arxiv_id":"2505.23701","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A disentangled evaluation of math word problems finds arithmetic computation, not abstract formulation, is the main bottleneck for LLMs without chain-of-thought, and chain-of-thought mainly helps computation.","lead":"This paper shows that large language models often fail math word problems because of arithmetic mistakes, not because they cannot understand the problem structure. It introduces separate tests for problem formulation and calculation, which could change how AI reasoning is measured.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Computation sub-task strips away word-problem context; the claimed computation bottleneck may reflect format mismatch, and conjunctive bottleneck is not behaviorally tested.","rationale":"The reader's weakest assumption—that the task variants cleanly isolate the two sub-skills and that the Original task is their conjunction—is exactly where the central claim is least secure. My concern sharpens this into a specific confound: the Arithmetic Computation condition removes the word-problem narrative, while the abstraction conditions retain it. This changes not just the skill being measured but also the available surface cues, making it unclear whether the observed computation deficit is a genuine bottleneck in Original-style reasoning or an artifact of how the computation probe is presented. The paper's own SVAMP results (Appendix A.3), where computation can outperform abstraction on one-step problems, reinforce that task complexity and format interact with the skill ranking; the GSM8K finding may be driven by the fraction-heavy, decontextualized expressions rather than by computation per se. The mechanistic analyses are internally consistent and suggestive, but they use simple synthetic templates (Appendix B.1), not the GSM8K/SVAMP items, so they cannot by themselves validate the conjunctive bottleneck on the actual benchmark. I am not claiming the conclusion is false; the product-rule and contextualized-computation tests I propose could confirm it cleanly. The absence of released code/data is a reproducibility concern but not the load-bearing one. Because the reader already assigned a conditional verdict and my concern is the same load-bearing premise, I recommend no change to that verdict; the conditionality should remain until the suggested tests are run.","tokens_in":19718,"tokens_out":6530,"duration_ms":73349,"concrete_test":"On the same GSM8K/SVAMP item sets, add a Contextualized Arithmetic Computation condition: show the original story, then ask 'Using the expression E = 12 × (50/60), what is the final answer?' and compare accuracy to the paper's bare 'What is E?' condition. Separately, apply sympy to each model's Numerical Abstraction output to compute O_pred (accuracy of generating a correct expression, assuming perfect arithmetic), and compare O_pred to the reported Original accuracy O. If O is far below O_pred, computation is the bottleneck; if O is close to O_pred or the contextualized computation accuracy rises to abstraction-level, the paper's bottleneck attribution is an artifact of decontextualized computation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract: final-answer accuracy is 'overwhelmingly bottlenecked' by computation) rests on comparing abstraction accuracy against arithmetic-computation accuracy. But the two sub-tasks differ in more than the targeted skill: the abstraction variants are posed as word problems with a narrative (Table 1), while the Arithmetic Computation variant is a bare expression ('What is the value of 12 × (50/60)?'), stripped of all story context. Lower accuracy on the bare-expression task therefore need not indicate a computation deficit in the Original setting, where the same arithmetic appears inside a narrative that may cue the relevant operations and numbers. The paper never verifies the conjunctive model: it does not check whether Original accuracy O is close to P(correct abstraction)×P(correct computation), nor whether a model that outputs the correct expression in Numerical Abstraction would have produced the correct final answer if its arithmetic were perfect. The mechanistic abstract-then-compute evidence (Sec. 5) is based on 1–2 step synthetic templates, not on the GSM8K/SVAMP items used for the bottleneck claim, so it does not close this gap. Consequently the headline attribution is conditional on a task-design assumption that is plausible but untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a disentangled evaluation framework for LLM performance on math word problems, separating abstract formulation (mapping a word problem to an expression) from arithmetic computation (evaluating an expression). Using GSM8K and SVAMP with Llama-3 and Qwen2.5 models (1B–32B) in zero-shot no-CoT settings, the authors report that abstraction accuracy (numerical and symbolic) is consistently higher than arithmetic-computation accuracy, leading them to conclude that final-answer accuracy is \"overwhelmingly bottlenecked\" by computation rather than reasoning. They further report that CoT prompting improves computation most and abstraction least. A mechanistic section on synthetic 1–2 step problems uses logit attribution and activation patching to argue for an abstract-then-compute mechanism, localizing abstraction to L13–14 and computation to L15–18 in Llama-3 8B, and showing that symbolic abstractions can be transferred across surface forms. The authors argue for adopting disentangled evaluation and reinterpreting poor no-CoT performance as largely arithmetic failure. The manuscript acknowledges limitations concerning language coverage, two-stage decomposition, single-step interpretability, and model scale, though one limitation statement contradicts the reported experiments.","tokens_in":19946,"tokens_out":5921,"duration_ms":57362,"significance":"If the behavioral claim holds, the paper makes a useful methodological contribution: final-answer accuracy conflates two distinct skills, and CoT's benefit may be largely computational rather than reasoning-oriented. The scope is a strength: seven model sizes across two model families and two datasets, with a consistent pattern. The mechanistic section is also creative, particularly the cross-prompt patching experiments showing that symbolic abstractions can be transferred into concrete problems and alter the final answer. The paper does not fit parameters and relies on direct causal interventions, which are more informative than correlational probes. The evaluation pipeline is described in detail, including sympy-based validation and a GPT-4o-mini symbolic evaluator with 94% human agreement. However, the central \"bottleneck\" inference depends on a task-design assumption that is not directly validated: that the three subtasks cleanly isolate abstraction and computation, and that the original task is their conjunction. Because that assumption is load-bearing, the headline contribution is currently conditional rather than fully established.","major_comments":[{"comment":"The claim that models are 'overwhelmingly bottlenecked' by computation rests on the assumption that Numerical Abstraction is a pure test of abstraction and Arithmetic Computation is a pure test of computation. This is not obviously the case. For GSM8K items, producing the expression 12 × (50/60) in the Numerical Abstraction subtask requires the model to perform unit-conversion arithmetic internally (50/60); a wrong internal arithmetic step is scored as an abstraction error. Conversely, the Arithmetic Computation subtask presents the bare expression ('What is the value of 12 × (50/60)?'), removing the narrative that, in the Original setting, may cue the operation and the operands. Lower accuracy on the bare-expression task therefore need not indicate a computation deficit in the Original setting. I would like to see a computation variant embedded in the original narrative (e.g., the word problem followed by 'compute 12 × (50/60)') and an abstraction task that does not require arithmetic (e.g., multiple-choice operator/quantity identification), to determine whether the observed ordering survives.","section":"Section 4 (Framework), Table 1"},{"comment":"The paper's own phrasing is conditional ('if deriving the final answer ... depends on these two skills conjunctively'), but the conclusion is stated unconditionally, and the conjunctive model is never tested. No analysis compares Original accuracy against the product of abstraction and computation accuracies, examines whether abstraction and computation errors are independent, or performs an intervention such as feeding the model's own correct expression into a calculator and measuring how often the final answer would become correct. Without this, the marginal ordering of subtask accuracies does not identify the bottleneck of the joint task; correlated errors or a failure mode in which a wrong abstraction also corrupts computation could overturn the attribution. The mechanistic evidence in Section 5 is generated on 1–2 step synthetic templates (Appendix B.1), not on the GSM8K/SVAMP items used for the behavioral claim, so it does not close this gap. Please add behavioral tests of the conjunctive model on the same datasets.","section":"Section 4.1, Figure 3"},{"comment":"The interpretation that CoT 'primarily aids computation' is vulnerable to the same format mismatch. The Arithmetic Computation subtask uses a bare expression, and CoT may improve performance there simply by making the model more likely to write out intermediate arithmetic steps, but the comparison to the abstraction subtasks does not control for the different output formats and instructions. Additionally, the Discussion mentions 'possible data leakage' in the Original CoT gains without quantifying or controlling it; since the conclusion about CoT relies on the difference between Original and subtask gains, leakage in the Original setting would directly affect the attribution. Please include a format-matched control and report results with at least one alternative CoT prompt template to assess prompt sensitivity.","section":"Section 4.2, Table 2"}],"minor_comments":[{"comment":"The text says 'due to compute constraints, we analyze models up to 12B parameters,' but the experiments include Qwen2.5 32B (e.g., Table 2 and Figure 3). This is contradictory and should be corrected.","section":"Section 7 (Limitations)"},{"comment":"Accuracy values are reported as point estimates without confidence intervals or sample sizes. Since the main claim concerns the ordering of two accuracies, binomial confidence intervals (or at least per-point sample counts) would substantially strengthen the comparison.","section":"Figures 3, 9–11"},{"comment":"Equation (1) defines LD* but the subsequent effect formula (Eq. 2) uses LDo, LDp, and LDc without connecting the notation. Please align the notation or remove the unused definition.","section":"Section 5.1.1, Eqs. (1)–(2)"},{"comment":"The paper repeatedly writes 'SV AMP' with a space; the standard dataset name is SVAMP. Also, Table 1's columns would be clearer if they explicitly distinguished model inputs from model outputs.","section":"Throughout"},{"comment":"Symbolic expression evaluation relies on GPT-4o-mini with 94% agreement on 120 samples; please report the eight disagreement cases and include a sensitivity analysis (e.g., evaluating with sympy where possible) so readers can assess the impact of evaluator error on the abstraction accuracies.","section":"Section A.2"}],"recommendation":"major_revision","confidential_remarks":"This is a promising empirical paper, but the central claim is currently conditional on an untested task-design assumption. I would be willing to revise upward if the authors add a behavioral test of the conjunctive model on GSM8K/SVAMP and a narrative-context computation control. The manuscript is within the scope of the journal; I have no concerns about citation practices beyond the self-contradictory limitation statement noted in the minor comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The core contribution is a clean disaggregation: instead of final-answer accuracy alone, they measure abstraction (expression generation) and computation separately on GSM8K and SVAMP, and they show across Llama-3 and Qwen2.5 (1B-32B) that no-CoT models are much better at producing the right expression than at evaluating it. That is a real finding and it undercuts the lazy reading that low accuracy equals poor reasoning. The CoT comparison is also informative: CoT mostly lifts computation, not abstraction. The mechanistic part is careful—logit attribution, activation patching, cross-prompt transfer on synthetic 1-2 step problems—and it does demonstrate an abstract-then-compute sequence in simple cases, with transferable operator representations. Now the soft spots. The biggest is the Arithmetic Computation condition. It strips the story entirely: \"What is the value of 12 × (50/60)?\" vs. the word problem. So the gap between abstraction and computation could partly be a format effect—models may compute better when the relevant quantities are surfaced in a narrative. The paper's own language is honest in Section 4.1: \"if final-answer accuracy depends on these two skills conjunctively.\" But the abstract drops the conditional and asserts the bottleneck as fact. That is a step too far without testing the conjunctive model directly. A quick check would be to compare Original accuracy to P(abstraction)×P(computation), or to embed the arithmetic in narrative scaffolding. Also, the mechanistic evidence is on synthetic templates, not on the GSM8K/SVAMP items, so it does not close that gap. And no code or data are released yet, which makes the GPT-4o-mini symbolic eval and the sympy checks harder to verify and extend. None of this kills the paper. The disentangled framework is valuable, and the main pattern—abstraction ahead of computation—holds up across two datasets and several model families. I would send it out, with the expectation that the authors tighten the bottleneck claim, add a conjunctive test, and release artifacts.","headline":"A genuinely useful disentangled evaluation showing that no-CoT math errors are mostly arithmetic, not abstraction, though the headline bottleneck claim leans on an untested conjunctive assumption and a format confound.","tokens_in":640,"tokens_out":786,"would_cite":true,"duration_ms":30025,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On math word problems, a model's final-answer accuracy is bottlenecked by arithmetic computation, not by abstract formulation, so final-answer scores understate its reasoning.","keywords":["math word problems","abstract formulation","arithmetic computation","disentangled evaluation","chain-of-thought","mechanistic interpretability","activation patching","LLM reasoning"],"falsifier":"Run the same four task variants on a distractor-heavy benchmark where the arithmetic is single-step and trivial; if final-answer accuracy tracks symbolic-abstraction accuracy instead of arithmetic-computation accuracy, the claimed bottleneck is not general. A second check is to compute per-item error correlations between the abstraction and computation variants; if the same problems fail both more often than independence predicts, the conjunctive attribution is an artifact of the evaluation design.","tokens_in":1815,"feed_emoji":"🧮","tokens_out":2196,"duration_ms":77089,"temperature":0.7,"pith_summary":"The paper claims that final-answer accuracy on math word problems conflates two distinct skills: turning the story into a mathematical expression (abstraction) and evaluating that expression (arithmetic computation). By testing each skill alone with task variants, it finds that without chain-of-thought, models are much better at abstraction than at computation, so low final-answer scores are mostly arithmetic errors, not reasoning failures. It also finds that chain-of-thought helps mainly by improving computation, with little effect on abstraction. Mechanistic evidence from a single forward pass supports an 'abstract-then-compute' ordering: the model first encodes the operation, then moves operands, then computes. If this is right, benchmark scores based only on final answers systematically understate what models can grasp about a problem.","feed_headline":"LLM math failures are arithmetic, not reasoning","feed_subtitle":"Disentangled tests show abstraction beats computation, so chain-of-thought gains come mostly from calculating.","key_machinery":"The load-bearing device is a disentangled evaluation design with four task variants: the Original task (question to number), Arithmetic Computation (expression to number), Numerical Abstraction (question to numerical expression), and Symbolic Abstraction (question to variable expression). The design assumes that solving the Original task requires both sub-skills conjunctively, so comparing error rates across variants attributes failures to one skill. A second device is the single-forward-pass mechanistic analysis using logit attribution (projecting hidden states onto the vocabulary to track when operator and operand tokens become readable), activation patching (replacing a layer's hidden state in a corrupted run with the clean state to measure causal recovery), and cross-prompt patching (transplanting abstraction states from symbolic into numerical problems). These localize the named 'abstract-then-compute' mechanism: abstraction at middle layers, operand transfer slightly later, and computation at a later MLP layer.","core_discovery":"The central claim is that final-answer accuracy alone gives a misleading picture of LLM math reasoning because it couples abstraction with computation, and the paper demonstrates this in two complementary ways. Behaviorally, on zero-shot GSM8K and SVAMP, Llama-3 and Qwen2.5 models (1B-32B) without CoT show lower error rates on numerical and symbolic abstraction tasks than on arithmetic computation tasks; for instance, symbolic abstraction accuracy is 45.7% for Llama-3 8B and 76.8% for Qwen2.5 32B, far above final-answer accuracy. Adding CoT produces large gains on computation (average +58.7 accuracy points on the arithmetic-computation variant) and small gains on abstraction (average +6.7 points on symbolic abstraction). Mechanistically, logit attribution and activation patching on simplified one- and two-step problems show that in a single forward pass the model first captures the abstract operation around middle layers (e.g., L13-14 in Llama-3 8B), then transfers operands, then performs the calculation around a later layer (e.g., L18). Cross-prompt patching shows that an abstraction encoded at those layers can be transplanted from a symbolic problem into a numerical problem with different operands, and the model composes the transplanted operation with the new numbers to produce a different final answer.","pith_inferences":["Editorial extension: if computation is the bottleneck, tool use or program-aided decoding should raise final-answer accuracy more than additional chain-of-thought prompting; the paper's framework predicts this but does not test it.","Editorial extension: the same four-variant design could be applied to distractor-heavy benchmarks to test directly whether observed accuracy drops are abstraction errors or computation errors, sharpening the paper's reinterpretation of earlier perturbation results.","Editorial extension: the abstract-then-compute circuit was localized only in single-pass, one- or two-step generation; extending causal patching to multi-token CoT could reveal whether CoT's computation aid reuses the same late-layer arithmetic circuit or a different one.","Editorial extension: if abstraction representations are surface-form invariant, then adversarial rephrasing that changes wording without changing structure should leave abstraction accuracy high and computation accuracy unchanged; this behavioral prediction follows from the transfer finding but is not tested here."],"forward_implications":["Final-answer-only leaderboards should be read as measuring arithmetic execution at least as much as reasoning; a low number is not by itself evidence of a reasoning deficit.","Chain-of-thought's value on math word problems is mostly as a computation aid, so using CoT to diagnose or improve reasoning may target the wrong sub-skill.","Reported performance drops on perturbed problem variants may stem from arithmetic brittleness rather than failure to generalize the underlying structure.","The abstract-then-compute ordering implies that models can perform a form of reasoning in a single forward pass without explicit reasoning steps, so improving arithmetic components may raise final-answer accuracy more than further reasoning prompting.","Disentangled evaluation should accompany final-answer metrics when assessing model reasoning, because the two sub-skills can move independently."],"supporting_citations":[{"why":"Supplies the GSM8K dataset and its final-answer format, which is the primary benchmark for the disentangled evaluation.","marker":"Cobbe et al. 2021"},{"why":"Supplies the SVAMP dataset with paired expressions, enabling symbolic and numerical abstraction variants.","marker":"Patel et al. 2021"},{"why":"Defines chain-of-thought prompting, whose gains the paper re-attributes mainly to computation.","marker":"Wei et al. 2022"},{"why":"Provides the logit-attribution technique used to read operator and operand information from hidden states.","marker":"Belrose et al. 2023"},{"why":"Provides the activation-patching methodology and best practices used for causal localization.","marker":"Zhang and Nanda 2023"},{"why":"Provides a patchscopes-style inspection framework that supports cross-prompt patching experiments.","marker":"Ghandeharioun et al. 2024"},{"why":"A prior study of symbolic perturbations whose 'reasoning decline' interpretation the paper challenges.","marker":"Mirzadeh et al. 2025"},{"why":"Motivates the finer-grained decomposition of math problem-solving into cognitive sub-skills.","marker":"Opedal et al. 2024"}],"fun_headline_variants":["LLM math errors: arithmetic, not reasoning","Chain of thought aids calculation, not abstraction","Abstraction beats computation in LLM math","Math reasoning intact; arithmetic is the limit","CoT mostly fixes arithmetic, not abstraction"],"cache_read_input_tokens":22656,"weakest_assumption_plain":"The evaluation assumes that the four task variants cleanly separate abstraction from computation and that the original task requires both; if writing an expression still uses calculation, or if abstraction and computation errors happen together, the paper's attribution of failures and of CoT gains could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["LLM math errors: arithmetic, not reasoning","Chain of thought aids calculation, not abstraction","Abstraction beats computation in LLM math","Math reasoning intact; arithmetic is the limit","CoT mostly fixes arithmetic, not abstraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1334,"prompt_tokens":1035,"completion_tokens":299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":231}},"tokens_in":651,"tokens_out":299,"duration_ms":3174,"temperature":1.0,"reasoning_tokens":231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:38:39.988232+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four task variants on a distractor-heavy benchmark where the arithmetic is single-step and trivial; if final-answer accuracy tracks symbolic-abstraction accuracy instead of arithmetic-computation accuracy, the claimed bottleneck is not general. A second check is to compute per-item error correlations between the abstraction and computation variants; if the same problems fail both more often than independence predicts, the conjunctive attribution is an artifact of the evaluation design.","supporting_citations":[],"review_version":1}