Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On math word problems, a model's final-answer accuracy is bottlenecked by arithmetic computation, not by abstract formulation, so final-answer scores understate its reasoning.

desk verdict A genuinely useful disentangled evaluation showing that no-CoT math errors are mostly arithmetic, not abstraction, though the headline bottleneck claim leans on an untested conjunctive assumption and a format confound. read the letter →

arxiv 2505.23701 v1 pith:W3REE26Y submitted 2025-05-29 cs.CL

Ziling Cheng , Meng Cao , Leila Pishdad , Yanshuai Cao , Jackie Chi Kit Cheung This is my paper
classification cs.CL
keywords mathwordproblemsabstractformulationarithmeticcomputationdisentangledevaluationchain-of-thoughtmechanisticinterpretabilityactivationpatchingLLMreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that final-answer accuracy on math word problems conflates two distinct skills: turning the story into a mathematical expression (abstraction) and evaluating that expression (arithmetic computation). By testing each skill alone with task variants, it finds that without chain-of-thought, models are much better at abstraction than at computation, so low final-answer scores are mostly arithmetic errors, not reasoning failures. It also finds that chain-of-thought helps mainly by improving computation, with little effect on abstraction. Mechanistic evidence from a single forward pass supports an 'abstract-then-compute' ordering: the model first encodes the operation, then moves operands, then computes. If this is right, benchmark scores based only on final answers systematically understate what models can grasp about a problem.

What carries the argument

The load-bearing device is a disentangled evaluation design with four task variants: the Original task (question to number), Arithmetic Computation (expression to number), Numerical Abstraction (question to numerical expression), and Symbolic Abstraction (question to variable expression). The design assumes that solving the Original task requires both sub-skills conjunctively, so comparing error rates across variants attributes failures to one skill. A second device is the single-forward-pass mechanistic analysis using logit attribution (projecting hidden states onto the vocabulary to track when operator and operand tokens become readable), activation patching (replacing a layer's hidden state in a corrupted run with the clean state to measure causal recovery), and cross-prompt patching (transplanting abstraction states from symbolic into numerical problems). These localize the named 'abstract-then-compute' mechanism: abstraction at middle layers, operand transfer slightly later, and computation at a later MLP layer.

What would settle it

Run the same four task variants on a distractor-heavy benchmark where the arithmetic is single-step and trivial; if final-answer accuracy tracks symbolic-abstraction accuracy instead of arithmetic-computation accuracy, the claimed bottleneck is not general. A second check is to compute per-item error correlations between the abstraction and computation variants; if the same problems fail both more often than independence predicts, the conjunctive attribution is an artifact of the evaluation design.

Watch

Extended reading notes

Core claim

The central claim is that final-answer accuracy alone gives a misleading picture of LLM math reasoning because it couples abstraction with computation, and the paper demonstrates this in two complementary ways. Behaviorally, on zero-shot GSM8K and SVAMP, Llama-3 and Qwen2.5 models (1B-32B) without CoT show lower error rates on numerical and symbolic abstraction tasks than on arithmetic computation tasks; for instance, symbolic abstraction accuracy is 45.7% for Llama-3 8B and 76.8% for Qwen2.5 32B, far above final-answer accuracy. Adding CoT produces large gains on computation (average +58.7 accuracy points on the arithmetic-computation variant) and small gains on abstraction (average +6.7 points on symbolic abstraction). Mechanistically, logit attribution and activation patching on simplified one- and two-step problems show that in a single forward pass the model first captures the abstract operation around middle layers (e.g., L13-14 in Llama-3 8B), then transfers operands, then performs the calculation around a later layer (e.g., L18). Cross-prompt patching shows that an abstraction encoded at those layers can be transplanted from a symbolic problem into a numerical problem with different operands, and the model composes the transplanted operation with the new numbers to produce a different final answer.

Load-bearing premise

The evaluation assumes that the four task variants cleanly separate abstraction from computation and that the original task requires both; if writing an expression still uses calculation, or if abstraction and computation errors happen together, the paper's attribution of failures and of CoT gains could be an artifact.

Editorial extensions

If this is right

  • Final-answer-only leaderboards should be read as measuring arithmetic execution at least as much as reasoning; a low number is not by itself evidence of a reasoning deficit.
  • Chain-of-thought's value on math word problems is mostly as a computation aid, so using CoT to diagnose or improve reasoning may target the wrong sub-skill.
  • Reported performance drops on perturbed problem variants may stem from arithmetic brittleness rather than failure to generalize the underlying structure.
  • The abstract-then-compute ordering implies that models can perform a form of reasoning in a single forward pass without explicit reasoning steps, so improving arithmetic components may raise final-answer accuracy more than further reasoning prompting.
  • Disentangled evaluation should accompany final-answer metrics when assessing model reasoning, because the two sub-skills can move independently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if computation is the bottleneck, tool use or program-aided decoding should raise final-answer accuracy more than additional chain-of-thought prompting; the paper's framework predicts this but does not test it.
  • Editorial extension: the same four-variant design could be applied to distractor-heavy benchmarks to test directly whether observed accuracy drops are abstraction errors or computation errors, sharpening the paper's reinterpretation of earlier perturbation results.
  • Editorial extension: the abstract-then-compute circuit was localized only in single-pass, one- or two-step generation; extending causal patching to multi-token CoT could reveal whether CoT's computation aid reuses the same late-layer arithmetic circuit or a different one.
  • Editorial extension: if abstraction representations are surface-form invariant, then adversarial rephrasing that changes wording without changing structure should leave abstraction accuracy high and computation accuracy unchanged; this behavioral prediction follows from the transfer finding but is not tested here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a disentangled evaluation framework for LLM performance on math word problems, separating abstract formulation (mapping a word problem to an expression) from arithmetic computation (evaluating an expression). Using GSM8K and SVAMP with Llama-3 and Qwen2.5 models (1B–32B) in zero-shot no-CoT settings, the authors report that abstraction accuracy (numerical and symbolic) is consistently higher than arithmetic-computation accuracy, leading them to conclude that final-answer accuracy is "overwhelmingly bottlenecked" by computation rather than reasoning. They further report that CoT prompting improves computation most and abstraction least. A mechanistic section on synthetic 1–2 step problems uses logit attribution and activation patching to argue for an abstract-then-compute mechanism, localizing abstraction to L13–14 and computation to L15–18 in Llama-3 8B, and showing that symbolic abstractions can be transferred across surface forms. The authors argue for adopting disentangled evaluation and reinterpreting poor no-CoT performance as largely arithmetic failure. The manuscript acknowledges limitations concerning language coverage, two-stage decomposition, single-step interpretability, and model scale, though one limitation statement contradicts the reported experiments.

Significance. If the behavioral claim holds, the paper makes a useful methodological contribution: final-answer accuracy conflates two distinct skills, and CoT's benefit may be largely computational rather than reasoning-oriented. The scope is a strength: seven model sizes across two model families and two datasets, with a consistent pattern. The mechanistic section is also creative, particularly the cross-prompt patching experiments showing that symbolic abstractions can be transferred into concrete problems and alter the final answer. The paper does not fit parameters and relies on direct causal interventions, which are more informative than correlational probes. The evaluation pipeline is described in detail, including sympy-based validation and a GPT-4o-mini symbolic evaluator with 94% human agreement. However, the central "bottleneck" inference depends on a task-design assumption that is not directly validated: that the three subtasks cleanly isolate abstraction and computation, and that the original task is their conjunction. Because that assumption is load-bearing, the headline contribution is currently conditional rather than fully established.

major comments (3)
  1. [Section 4 (Framework), Table 1] The claim that models are 'overwhelmingly bottlenecked' by computation rests on the assumption that Numerical Abstraction is a pure test of abstraction and Arithmetic Computation is a pure test of computation. This is not obviously the case. For GSM8K items, producing the expression 12 × (50/60) in the Numerical Abstraction subtask requires the model to perform unit-conversion arithmetic internally (50/60); a wrong internal arithmetic step is scored as an abstraction error. Conversely, the Arithmetic Computation subtask presents the bare expression ('What is the value of 12 × (50/60)?'), removing the narrative that, in the Original setting, may cue the operation and the operands. Lower accuracy on the bare-expression task therefore need not indicate a computation deficit in the Original setting. I would like to see a computation variant embedded in the original narrative (e.g., the word problem followed by 'compute 12 × (50/60)') and an abstraction task that does not require arithmetic (e.g., multiple-choice operator/quantity identification), to determine whether the observed ordering survives.
  2. [Section 4.1, Figure 3] The paper's own phrasing is conditional ('if deriving the final answer ... depends on these two skills conjunctively'), but the conclusion is stated unconditionally, and the conjunctive model is never tested. No analysis compares Original accuracy against the product of abstraction and computation accuracies, examines whether abstraction and computation errors are independent, or performs an intervention such as feeding the model's own correct expression into a calculator and measuring how often the final answer would become correct. Without this, the marginal ordering of subtask accuracies does not identify the bottleneck of the joint task; correlated errors or a failure mode in which a wrong abstraction also corrupts computation could overturn the attribution. The mechanistic evidence in Section 5 is generated on 1–2 step synthetic templates (Appendix B.1), not on the GSM8K/SVAMP items used for the behavioral claim, so it does not close this gap. Please add behavioral tests of the conjunctive model on the same datasets.
  3. [Section 4.2, Table 2] The interpretation that CoT 'primarily aids computation' is vulnerable to the same format mismatch. The Arithmetic Computation subtask uses a bare expression, and CoT may improve performance there simply by making the model more likely to write out intermediate arithmetic steps, but the comparison to the abstraction subtasks does not control for the different output formats and instructions. Additionally, the Discussion mentions 'possible data leakage' in the Original CoT gains without quantifying or controlling it; since the conclusion about CoT relies on the difference between Original and subtask gains, leakage in the Original setting would directly affect the attribution. Please include a format-matched control and report results with at least one alternative CoT prompt template to assess prompt sensitivity.
minor comments (5)
  1. [Section 7 (Limitations)] The text says 'due to compute constraints, we analyze models up to 12B parameters,' but the experiments include Qwen2.5 32B (e.g., Table 2 and Figure 3). This is contradictory and should be corrected.
  2. [Figures 3, 9–11] Accuracy values are reported as point estimates without confidence intervals or sample sizes. Since the main claim concerns the ordering of two accuracies, binomial confidence intervals (or at least per-point sample counts) would substantially strengthen the comparison.
  3. [Section 5.1.1, Eqs. (1)–(2)] Equation (1) defines LD* but the subsequent effect formula (Eq. 2) uses LDo, LDp, and LDc without connecting the notation. Please align the notation or remove the unused definition.
  4. [Throughout] The paper repeatedly writes 'SV AMP' with a space; the standard dataset name is SVAMP. Also, Table 1's columns would be clearer if they explicitly distinguished model inputs from model outputs.
  5. [Section A.2] Symbolic expression evaluation relies on GPT-4o-mini with 94% agreement on 120 samples; please report the eight disagreement cases and include a sensitivity analysis (e.g., evaluating with sympy where possible) so readers can assess the impact of evaluator error on the abstraction accuracies.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity: the bottleneck and CoT claims are empirical comparisons under an explicitly flagged conjunctive assumption; the only self-citation is non-load-bearing.

full rationale

The central claims do not reduce to their inputs by construction. The bottleneck claim in Section 4.1 follows from comparing independently measured accuracies on the abstraction and computation subtasks, and the paper explicitly flags the conjunctive premise with 'Suppose' in the Section 4 Framework and 'if' in Section 4.1, rather than treating it as a definitional identity. The CoT-attribution claim in Section 4.2 uses measured accuracy deltas and is not a fitted parameter renamed as a prediction. The abstract-then-compute mechanism in Section 5 is supported by external interventions (logit attribution, activation patching, and cross-prompt patching) on separately constructed templates, not by an imported uniqueness theorem or ansatz smuggled in via self-citation. The only self-citation, Cheng et al. (2025), appears in a list of causal-patching methods in Related Work and is not load-bearing. The paper's own Limitations section flags the two-stage decomposition and the single-step scope of interpretability, which identify plausible threats to validity (e.g., the Arithmetic Computation variant strips narrative context) but are not circular reasoning. With no fitted value masquerading as a prediction and no equation equating the conclusion to the input, there is no significant circularity; the score reflects only the presence of one minor, non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; all evaluations use fixed zero-shot and CoT prompts. The main commitments are the decomposition assumption, the validity of the symbolic abstraction measure, and the representativeness of the synthetic interpretability dataset.

assumptions (3)
  • domain assumption Mathematical problem solving decomposes into two independent skills, abstraction and computation, composed conjunctively.
    Section 4 Framework: solving the original task is modeled as the conjunction of the two sub-skills; separate subtask accuracies are compared to infer bottlenecks. The independence and conjunction are not directly tested.
  • domain assumption The symbolic abstraction variant, with variables replacing numbers, measures abstraction without requiring computation.
    Section 3 and Table 1. The model must output an expression; the assumption is that this does not invoke arithmetic computation. Internal computation could still occur.
  • domain assumption The interpretability dataset of 3,600 simple 1-2 step problems is representative of the abstraction mechanism in general math word problems.
    Appendix B.1; the authors restrict to problems where models perform adequately, and Llama-3 8B only reaches 16.5% accuracy on two-operator problems, so the mechanism may not generalize.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation." pith.science (2026). https://pith.science/paper/W3REE26Y

@misc{pith2026250523701,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W3REE26Y}},
  note         = {Machine review of arXiv:2505.23701}
}
read the original abstract

Final-answer-based metrics are commonly used for evaluating large language models (LLMs) on math word problems, often taken as proxies for reasoning ability. However, such metrics conflate two distinct sub-skills: abstract formulation (capturing mathematical relationships using expressions) and arithmetic computation (executing the calculations). Through a disentangled evaluation on GSM8K and SVAMP, we find that the final-answer accuracy of Llama-3 and Qwen2.5 (1B-32B) without CoT is overwhelmingly bottlenecked by the arithmetic computation step and not by the abstract formulation step. Contrary to the common belief, we show that CoT primarily aids in computation, with limited impact on abstract formulation. Mechanistically, we show that these two skills are composed conjunctively even in a single forward pass without any reasoning steps via an abstract-then-compute mechanism: models first capture problem abstractions, then handle computation. Causal patching confirms these abstractions are present, transferable, composable, and precede computation. These behavioural and mechanistic findings highlight the need for disentangled evaluation to accurately assess LLM reasoning and to guide future improvements.

Figures

Figures reproduced from arXiv: 2505.23701 by the authors.

Figure 1
Figure 1. Left (Disentangled evaluation framework): Final-answer accuracy obscures reasoning ability due to conflating abstract formulation and arithmetic computation. Right (Abstract-then-Compute Mechanism in Llama-3 8B): (a) Residual stream at the last token position shows that models first capture problem abstraction (L13-14), followed by computation (L18). (b) Same as (a), but one critical layer output is patched with a d… view at source ↗
Figure 2
Figure 2. Distribution of problem characteristics by [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Model zero-shot without CoT performance on GSM8K. (i) Models exhibit much better abstraction performance (Symbolic and Numerical) than in actually computing the expressions (Arithmetic Computation). (ii) Final-answer accuracy in the Original setting may provide a misleading picture of models’ reasoning ability, possibly due to arithmetic limitations. the Original setting may substantially mislead a model’s underlyin… view at source ↗
Figures from the paper (26 more)
Figure 4
Figure 4. Figure 4: Overview of interpretability methods probing the abstract-then-compute mechanism in simple math [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Visualizations of internal computations at last token position in Llama-3 8B for addition math word [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Cross-patching results for Llama-3 8B with corresponding clean and corrupted run. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Generate-then-validate pipeline: We use API calls to obtain abstract question-answer-substitution triplets from the concrete question-solution pair from GSM-8K, then validate them against gold answer using sympy. Triplets that fail this check are manully reviewer and c…
Figure 8
Figure 8. Figure 8: Experiment configurations for the ablation [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Model zero-shot without CoT performance on SVAMP [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Model zero-shot with and without CoT performance on GSM8K. A.C.: Arithmetic Computation; N.A.: Numerical Abstraction; O.: Original; S.A.: Symbolic Abstraction. ’q’: ’l’, ’r’: ’c’, ’s’: ’v’, ’t’: ’j’, ’u’: ’m’, ’v’: ’t’, ’w’: ’o’, ’x’: ’u’, ’y’: ’p’, ’z’: ’b’, ’Z’: ’f’…
Figure 11
Figure 11. Figure 11: Model zero-shot with and without CoT performance on SVAMP. A.C.: Arithmetic Computation; N.A.: Numerical Abstraction; O.: Original; S.A.: Symbolic Abstraction. Subset Example Data (+, −) (+) [name] owns x stuffed animals. A relative sends them y more stuffed animals. …
Figure 12
Figure 12. Figure 12: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]
Figure 21
Figure 21. Figure 21: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]
Figure 22
Figure 22. Figure 22: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p022_22.png]
Figure 23
Figure 23. Figure 23: Llama-3 8B cross-prompt patching for symbolic abstraction results: First row: patching symbolic logic to concrete addition; Second row: patching symbolic logic to concrete subtraction; Third row: patching symbolic logic to concrete multiplication; Fourth row: patching…
Figure 24
Figure 24. Figure 24: Qwen-7b cross-prompt patching for symbolic abstraction results: First row: patching symbolic logic to concrete addition; Second row: patching symbolic logic to concrete subtraction; Third row: patching symbolic logic to concrete multiplication; Fourth row: patching sy…
Figure 25
Figure 25. Figure 25: Qwen-14b cross-prompt patching for symbolic abstraction results: First row: patching symbolic logic to concrete addition; Second row: patching symbolic logic to concrete subtraction; Third row: patching symbolic logic to concrete multiplication; Fourth row: patching s…
Figure 26
Figure 26. Figure 26: Cross-prompt patching results for numerical abstraction. First row: results for Llama-3 8B with corresponding clean and corrupted run. Second row: results for Qwen2.5 7B with corresponding clean and corrupted run. Third row: results for Qwen2.5 14B with corresponding …
Figure 27
Figure 27. Figure 27: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p025_27.png]
Figure 28
Figure 28. Figure 28: Visualizations of internal computations at last token position in [PITH_FULL_IMAGE:figures/full_fig_p026_28.png]
Figure 29
Figure 29. Figure 29: Two-operation cross-prompt patching for symbolic abstraction results: First row & Second row: patching symbolic logic to concrete problems for Qwen 2.5 7B. Third row & Fourth row: patching symbolic logic to concrete problems for Qwen 2.5 14B [PITH_FULL_IMAGE:figures/…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Stochastic Chameleons: Irrelevant Context Hallucinations Reveal Class-Based (Mis)Generalization in LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    LLMs systematically combine abstract category cues from a query with features from irrelevant context, causing structured answer flips, a behavior the authors call class-based (mis)generalization.

Reference graph

Works this paper leans on

40 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. https://doi.org/10.18653/v1/N19-1245 M ath QA : Towards interpretable math word problem solving with operation-based formalisms . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics:...

    doi:10.18653/v1/n19-1245 2019

  2. [2]

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112

    arXiv 2023

  3. [3]

    Haozhe Chen, Carl Vondrick, and Chengzhi Mao. 2024. https://proceedings.mlr.press/v235/chen24ao.html S elf IE : Self-interpretation of large language model embeddings . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 7373--7388. PMLR

    work page 2024

  4. [4]

    Ziling Cheng, Meng Cao, Marc-Antoine Rondeau, and Jackie Chi Kit Cheung. 2025. https://arxiv.org/abs/2505.22630 Stochastic chameleons: Irrelevant context hallucinations reveal class-based (mis)generalization in llms . Preprint, arXiv:2505.22630

    work page Pith review arXiv 2025

  5. [5]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  6. [6]

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764--10799. PMLR

    2023

  7. [7]

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06102

    arXiv 2024

  8. [8]

    Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. 2023. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969

    arXiv 2023

Show all 40 references
  1. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. https://doi.org/10.18653/v1/2024.acl-long.211 O lympiad B ench: A challenging benchmark for promoting AG...

    2024 doi

  3. [11]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021

  4. [12]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  5. [13]

    Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. https://doi.org/10.18653/v1/N16-1136 MAWPS : A math word problem repository . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational L...

    2016 doi

  6. [14]

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. arXiv preprint arXiv:2402.19255

    2024 arXiv

  7. [15]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. https://doi.org/10.18653/v1/P17-1015 Program induction by rationale generation: Learning to solve and explain algebraic word problems . In Proceedings of the 55th Annual Meeting of the Association for Computational ...

    2017 doi

  8. [16]

    Vera Liao, Alexandra Olteanu, and Ziang Xiao

    Yu Lu Liu, Su Lin Blodgett, Jackie Cheung, Q. Vera Liao, Alexandra Olteanu, and Ziang Xiao. 2024. https://doi.org/10.18653/v1/2024.acl-long.861 ECBD : Evidence-centered benchmark design for NLP . In Proceedings of the 62nd Annual Meeting of the Association for Computational Li...

    2024 doi

  9. [17]

    Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. https://openreview.net/forum?id=-h6WAS6eE4 Locating and editing factual associations in GPT . In Advances in Neural Information Processing Systems

    2022

  10. [18]

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2024. https://doi.org/10.18653/v1/2024.naacl-long.281 Language models implement simple W ord2 V ec-style vector arithmetic . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computatio...

    2024 doi

  11. [19]

    Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975--984

    2020

  12. [20]

    Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. https://openreview.net/forum?id=AjXkRZIvjB GSM -symbolic: Understanding the limitations of mathematical reasoning in large language models . In The Thirteenth Intern...

    2025

  13. [21]

    Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.392 LILA : A unified benchmark for mathematical reasoning...

    2022 doi

  14. [22]

    Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. 2025. https://openreview.net/forum?id=O9YTt26r2P Arithmetic without algorithms: Language models solve math with a bag of heuristics . In The Thirteenth International Conference on Learning Representations

    2025

  15. [23]

    nostalgebraist. 2020. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens interpreting gpt: the logit lens

    2020

  16. [24]

    Andreas Opedal, Alessandro Stolfo, Haruki Shirakami, Ying Jiao, Ryan Cotterell, Bernhard Sch \"o lkopf, Abulhair Saparov, and Mrinmaya Sachan. 2024. Do language models exhibit the same cognitive biases in problem solving as human learners? arXiv preprint arXiv:2401.18070

    2024 arXiv

  17. [25]

    OpenAI . 2024. Gpt-4o mini: Advancing cost-efficient intelligence. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  18. [26]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. https://doi.org/10.18653/v1/2021.naacl-main.168 Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling...

    2021 doi

  19. [27]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Sch \"a rli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210--31227. PMLR

    2023

  20. [28]

    Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically solving number word problems by semantic parsing and reasoning. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1132--1142

    2015

  21. [29]

    Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2025. https://openreview.net/forum?id=w6nlcS8Kkn To cot or not to cot? chain-of-thought helps mainly on math and symbolic re...

    2025

  22. [30]

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. https://openreview.net/forum?id=NpsVSN6o4ul Interpretability in the wild: a circuit for indirect object identification in GPT -2 small . In The Eleventh International Conference on L...

    2023

  23. [31]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  24. [32]

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, and 25 others. 2024. https://api...

    2024 arXiv

  25. [33]

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. 2025. https://openreview.net/forum?id=Tn5B6Udq3E Physics of language models: Part 2.1, grade-school math and the hidden reasoning process . In The Thirteenth International Conference on Learning Representations

    2025

  26. [34]

    Fred Zhang and Neel Nanda. 2023. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042

    2023 arXiv

  27. [35]

    Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Vishnu Raja, Charlotte Zhuang, Dylan Z Slack, Qin Lyu, Sean M. Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. 2024 a . https://openreview.net/forum?id=RJZRhMzZzH A carefu...

    2024

  28. [36]

    Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. 2024 b . Interpreting and improving large language models in arithmetic calculation. arXiv preprint arXiv:2409.01659

    2024 arXiv

  29. [37]

    Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474

    2023 arXiv

  30. [38]

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.149 AGIE val: A human-centric benchmark for evaluating foundation models . In Findings of the Association for...

    2024 doi

  31. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.