Pith. sign in

REVIEW 3 major objections 6 minor 33 references

KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chain-of-thought math reasoning is causal, not decoration: the paper argues that intermediate mathematical expressions in LLM reasoning traces are genuine mediators of the final answer, and that models internally favor the same derivation…

desk verdict Solid dataset and a consistently supported narrow mediation claim, but the rank test for internal graph realization uses a mismatched baseline and the causal framing overshoots; worth a serious referee with revisions. read the letter →

arxiv 2507.11408 v2 pith:XOHHC2MO submitted 2025-07-15 cs.CL cs.AI

Soumadeep Saha , Akshay Chaturvedi , Saptarshi Saha , Utpal Garain , Nicholas Asher This is my paper
classification cs.CLcs.AI
keywords chain-of-thoughtreasoningcausalgraphsmathematicalLLMinterpretabilityattentionsuppressionpathsmediationKisMATHdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that chain-of-thought traces are not arbitrary text but encode a fine-grained causal directed acyclic graph linking the question to the answer through intermediate mathematical expressions. It presents an automatic graph-extraction algorithm and a dataset of 1671 mathematics problems, then uses attention suppression across 15 open-weight models to show that removing reasoning-node tokens sharply increases answer uncertainty. It also shows that the models assign higher probability to the derivation paths the graphs identify than to equal-length random paths. The authors take these results as evidence that reasoning nodes are causal contributors to the answer, which they call constitutive of reasoning, and that models implicitly realize graph-like structure. If correct, graph-aligned interventions offer a more principled way to test and steer LLM reasoning than random perturbations.

What carries the argument

The carrying object is the Causal CoT Graph (CCGraph), a directed acyclic graph whose nodes are mathematical expression spans parsed from the question, reasoning trace, and answer, and whose edges connect expressions that share a parse-tree component or exact string match. Construction starts at the answer node, recursively matches earlier spans, prunes nodes with no path to a question node, and reverses edges. Two probes drive the analysis: attention suppression, which zeroes out the influence of reasoning-node tokens across all layers and heads to model their absence, and path probability, which multiplies token-level probabilities along an R-path and compares its rank against equal-length random paths. These two operations turn the abstract hypothesis that intermediate math matters into measurable entropy shifts and rank distributions.

What would settle it

Run attention suppression on CCGraph reasoning nodes versus equal-length sequences of math tokens that are not on the graph, matched for token frequency and position in the same trace; if the entropy shifts are statistically indistinguishable, the mediation effect is not specific to the graph-aligned structure. Alternatively, compute R-path ranks against random paths that contain only math tokens drawn from the same trace; if the 100th-percentile spike disappears, the path-emphasis result reflects arithmetic predictability rather than implicit graph realization.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that causal CoT graphs extracted from solution traces capture dependencies that matter to the models. Suppressing attention to reasoning-node tokens raises the entropy of the first answer token substantially across all 15 tested models, with reported p-values below 1e-12, so the answer distribution is causally sensitive to those intermediate expressions. In addition, the probability of a graph-aligned reasoning path ranks at or near the 100th percentile against random paths of the same length, indicating that models favor the exact derivation chains the graphs identify. The authors interpret the mediation result as a necessary condition for reasoning and the path-probability result as evidence that structures similar to the graphs are internally realized.

Load-bearing premise

Attention suppression is assumed to faithfully implement the counterfactual that the suppressed reasoning tokens were absent, and if zeroing those attention weights pushes the models off-distribution, the large entropy shift is not necessarily a causal effect.

Editorial extensions

If this is right

  • Answer distributions of LLMs are causally tied to intermediate mathematical expressions, so chain-of-thought traces play a functional role rather than being post-hoc decoration.
  • Graph-aligned interventions, such as suppressing a single reasoning node, give a controlled way to change or probe an answer, far more precise than random token perturbation.
  • CCGraphs provide a scalable automatic annotation method for mathematical reasoning traces, enabling dataset-scale studies of reasoning structure and comparison across models.
  • The two observed regimes in path-probability ranks, exponential and bell-shaped, tie a model's internal uncertainty along reasoning paths to exploration behavior and pass@k performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same extraction procedure could be extended beyond parsed math expressions to symbolic or logical expressions in code, formal proofs, or scientific derivations, turning the graph test into a general probe for compositional reasoning.
  • If the mediation finding is robust, inference-time interventions that upweight graph-aligned reasoning tokens, or training objectives that reward them, might improve answer reliability in multi-step mathematics.
  • The random-path baseline in the path-probability experiment compares math-token spans to prose-token spans; matching random paths to math tokens of similar frequency would separate arithmetic predictability from graph alignment.
  • The bell-shaped regime suggests a testable scheme: deliberately inserting high-entropy fork tokens could increase rollout diversity and pass@k, provided the causal interpretation of those forks is correct.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Causal CoT Graphs (CCGraphs), automatically extracted from LLM-generated reasoning traces, and introduces KisMATH, a dataset of 1,671 mathematical reasoning problems (GSM8K, MATH500, AIME) paired with such graphs. Using attention suppression and path-probability rank tests across 15 open-weight LLMs, it claims (i) that reasoning nodes in CCGraphs are causal mediators of the final answer, and (ii) that LLMs assign higher probability to CCGraph-identified reasoning paths than to random token paths, suggesting that the models internally realize structures similar to the graphs. It also identifies two behavioral regimes in path-probability distributions and connects them to exploration behavior in sampling.

Significance. The manuscript's main strengths are its scale and reproducibility: it constructs graphs for 1,671 problems and tests 15 models, far exceeding prior annotated datasets of 10–30 traces, and it publicly releases data, code, and prompts. The mediation analysis in Section 5.1 is consistently supported across models, and the framing of indirect effect as a necessary condition for reasoning is a useful conceptual contribution. However, the second headline claim (ii) rests on a rank test whose baseline is not matched for token type, and the causal reading of attention suppression is imported from prior work without on-model diagnostics. These issues weaken the strongest conclusions, but they are addressable within the manuscript's scope and do not undermine the dataset or the mediation finding.

major comments (3)
  1. [Section 5.3, Eq. (5)–(6), Figures 4/8] The random-path baseline is token-type mismatched. R paths are sequences of parsed mathematical expressions, while random paths are defined as equal-length token sequences from the same trace 'avoiding CCGraph nodes'; since CCGraph nodes constitute essentially all parseable mathematical content on question-to-answer dependency paths, the random-path tokens are almost exclusively natural-language tokens. Eq. (5) multiplies conditional token probabilities, and the two token classes have different systematic predictability: math tokens are often formulaic and reuse numerals/variables from the question, while prose tokens include connectives and function words. The observed 100th-percentile rank spike could therefore reflect arithmetic-token predictability rather than alignment with the specific CCGraph paths. Because claim (ii) in the abstract directly rests on this test, a matched control is required—for example, random paths with identical math-token density drawn from the same trace, or a shuffle of math tokens that are not part of R paths.
  2. [Section 4, Eqs. (2)–(3), used in Sections 5.1–5.2 and 6.1] The attention-suppression intervention assumes that zeroing attention from a token set across all layers and heads implements the counterfactual 'these tokens were absent.' The paper cites Bogdan et al. (2025) for the assertion that this does not induce out-of-distribution behavior, but that validation was performed on a different set of models; no such diagnostic is provided for the 15 open-weight LLMs studied here. If the intervention pushes the model into an OOD activation regime, the large entropy increases in Table 2 and Figure 3 would be expected regardless of the suppressed tokens' causal role. The authors should add a control, such as suppressing an equal number of randomly chosen tokens not on the CCGraph, or comparing against substitution with placeholders, to support the causal interpretation.
  3. [Section 3.1, Algorithm 1 and MATCH rule] The edge criterion 'their parse trees share a common node' is permissive; any shared numeral or symbol can create an edge, and the resulting graph is not validated against human-annotated dependencies, in contrast to the small-scale graphs of Tan (2023), Lee et al. (2025), and Bogdan et al. (2025). The manual intervention for approximately 10% of trivial graphs, also described in the Limitations, is a potential source of bias since it is not given as a fully automated rule. Because R paths are selected from these graphs and feed the main rank test, an independent validation of CCGraph edges, or a sensitivity analysis using a stricter MATCH rule, is needed to rule out that the rank spike reflects general math-token predictability rather than the specific graph structure.
minor comments (6)
  1. [Title/Abstract/body] The dataset name appears as 'KisMA TH' in the title, abstract, and several places in the body due to a spacing artifact; please correct to 'KisMATH' throughout.
  2. [Section 5.3, Eq. (6)] The definition of rank_M(R) is a fraction between 0 and 1, but the text reports '100th percentile'; state explicitly how rank values map to percentiles (e.g., rank = 1.0 corresponds to the 100th percentile).
  3. [Section 3.1, Algorithm 1] The EXPAND procedure terminates when the context is fully composed of question nodes, but it is not clear whether question-to-question edges are ever added before that point; a small worked example of the graph-construction steps would improve clarity.
  4. [Section 6.1, Table 3] The multiple-comparison correction used to set α' = 2.2e-4 is not described; please state the method (e.g., Bonferroni) in the text or caption.
  5. [Figure 5] The two panels in Figure 5 lack a clear legend distinguishing DeepSeek R1 32B from Qwen3 32B; add direct labels or split the panels explicitly.
  6. [Appendix C, Figure 9(a)] The GSM8K wrong-label example uses the bracket notation 'Ground Truth:150 [240]' without explanation; clarify that 240 is the corrected ground truth.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CCGraph construction is independent of the probed models, and both headline results are empirical measurements with no fitted parameter or self-citation chain forcing them.

full rationale

CCGraphs are extracted from OpenAI o3 traces by Algorithm 1: spans are parsed with SymPy and edges are added only when two parsed expressions match exactly or share a parse-tree node, expanding recursively from the answer to question nodes. This construction does not use the 15 open-weight models, their answer entropies, or their token probabilities, so the later measurements are not definitional echoes of the graph builder. The mediation experiments (Sections 5.1 and 5.2) apply attention suppression from Bogdan et al. (2025) to pre-defined graph nodes and record entropy shifts; no parameter is fitted from the entropy data to define the nodes. The R-path realization test (Section 5.3) computes token-level conditional probabilities (Eqs. 4-5) and compares R paths against random paths; a 100th-percentile spike is an empirical outcome that could have failed to occur. The paper does depend on two assumptions that are not circular: that attention suppression is a valid counterfactual intervention without out-of-distribution artifacts (cited to Bogdan et al. 2025), and that random paths avoiding CCGraph nodes are a matched baseline. The second assumption is questionable—CCGraph nodes are mostly mathematical expressions while non-CCGraph tokens are largely natural language, so the rank contrast may partly reflect token-type predictability rather than graph alignment—but this is a validity/correctness concern, not a circular reduction: the high rank of R paths is not entailed by the definitions alone. The authors' self-citations (Asher and Lascarides 2003; Sharma et al. 2025) are used as background and contrast and are not load-bearing. Accordingly, no circular step meets the quoted-reduction bar.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claims rest on hand-chosen structural choices (the MATCH rule for edges, top-k path selection, the random-path baseline, manual repair rules) and on prior-work assertions (Bogdan et al. 2025 that attention suppression is a faithful counterfactual intervention) rather than on fitted numeric constants. No parameters are fit to the 15 models being probed; the graphs come from o3 traces. The invented entity is the CCGraph itself, an analytic construct whose validation uses the same intervention machinery as the claims it supports.

free parameters (3)
  • k (number of longest Q-to-A paths analyzed) = 5 (GSM8K), 10 (MATH500, AIME)
    Section 3.2: 'we selected the top-k longest Q;A paths'; k chosen by hand per split.
  • MATCH rule for edges = exact string match or shared SymPy parse-tree node
    Algorithm 1: defines which expressions are causally linked; chosen by hand, no empirical justification.
  • Random-path baseline construction = same token count as R path, tokens avoiding CCGraph nodes
    Section 5.3: baseline controls for path length but not token type (math vs prose), which may drive the rank result.
assumptions (5)
  • domain assumption Attention suppression (Eqs. 2-3) validly implements the counterfactual absence of the suppressed tokens without problematic OOD behavior
    Section 4, 5.1: the entire mediation analysis rests on this, citing Bogdan et al. (2025) rather than demonstrating it for the 15 models tested.
  • domain assumption String or parse-tree overlap between expressions indicates a causal dependency
    Section 3.1, Algorithm 1: CCGraph edges are defined by MATCH, then treated as 'fine-grained causal' structure in Section 4.
  • domain assumption o3-generated traces are valid stand-ins for reasoning structure when probing other models
    Section 3: graphs are extracted from o3 rollouts and used to score and condition all 15 models; assumes the structure transfers.
  • ad hoc to paper Manual intervention on ~10% trivial graphs does not bias the graphs
    Section 3.1 and Limitations: 88 of ~40K vertices and 71 of ~300K edges edited; rule stated, but the effect on downstream entropy and rank analyses is not quantified.
  • domain assumption Token-level probability product (Eq. 5) is a meaningful proxy for path likelihood
    Section 5.3: the paper acknowledges it is not the exact transition probability because marginalization is intractable, then uses the product anyway.
invented entities (1)
  • Causal CoT Graph (CCGraph)
    purpose: Represents fine-grained causal dependencies between expressions in a reasoning trace; the object of all analyses.
    An internal analytic construct whose only validation (attention suppression, path-probability rank) uses the same intervention and scoring machinery as the claims it supports; no external falsifiable benchmark outside the paper is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?." pith.science (2026). https://pith.science/paper/XOHHC2MO

@misc{pith2026250711408,
  author       = {Pith},
  title        = {Pith review of: KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOHHC2MO}},
  note         = {Machine review of arXiv:2507.11408}
}
read the original abstract

Chain-of-thought (CoT) traces have been shown to improve performance of large language models on a plethora of reasoning tasks, yet there is no consensus on the mechanism by which this boost is achieved. To shed more light on this, we introduce Causal CoT Graphs (CCGraphs), which are directed acyclic graphs automatically extracted from reasoning traces that model fine-grained causal dependencies in language-model outputs. A collection of 1671 mathematical reasoning problems from MATH500, GSM8K, and AIME, together with their associated CCGraphs, has been compiled into our dataset -- KisMATH. Our detailed empirical analysis with 15 open-weight LLMs shows that (i) reasoning nodes in the CCGraphs are causal contributors to the final answer, which we argue is constitutive of reasoning; and (ii) LLMs emphasize the reasoning paths captured by the CCGraphs, indicating that the models internally realize structures similar to our graphs. KisMATH enables controlled, graph-aligned interventions and opens avenues for further investigation into the role of CoT in LLM reasoning.

Figures

Figures reproduced from arXiv: 2507.11408 by the authors.

Figure 1
Figure 1. Example of extracted causal graph and paths. (Left) An example of a (simplified) CoT causal graph (CCGraph) extracted from the GSM8K dataset. Reasoning nodes are highlighted in blue, edges are in gray. (Right) An R path (see Eq. 1), i.e., a simple path from question to answer (solid line) and a random path (dashed line). find that: (i) mathematical expressions in rea￾soning traces are effective mediators between a q… view at source ↗
Figure 2
Figure 2. Examples of R paths from the MATH500 and AIME splits of the KisMATH dataset. Nodes on the R path (qˆα ❀ rˆ(i1) ❀ . . . ❀ rˆ(iµ) ❀ aˆ) are highlighted (see Eq. 1). 5 Experiments 5.1 Is CCGraph a Mediator? We first test whether the reasoning nodes in the CC￾Graph serve as mediators for the final answer by performing attention suppression on all reasoning nodes in a CCGraph for a problem. Our results, summarized in [P… view at source ↗
Figure 3
Figure 3. Do reasoning path interventions affect the answer? We find that when attentions corresponding to tokens in an R path are suppressed, the entropy of the distribution of the answer (H(PA)) increases significantly, i.e., uncertainty over the answer is significantly increased. The figure also reports results of the 2-sample KS test, showing high values of Kolmogorov distance (DKS) and high statistical significance (p < … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Are LLMs aware of implicit structures in reasoning? We compare the probability associated with reasoning paths (see Eq. 5) with the probability of a random path through the reasoning response (e.g [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 6
Figure 6. Figure 6: High entropy along R paths drive exploration. The graph presents performance (pass@k-accuracy) of the two models (DeepSeek R1 32B, Qwen3 32B) with varying number of sam￾ple rollouts (k). The DeepSeek R1 32B model has higher uncertainty (lower probability transitions fo…
Figure 7
Figure 7. Figure 7: Results with additional splits for experiment in Section [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Results with additional splits for experiment in Section [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Dataset samples. (Top) Examples of annotation errors in GSM8K. (Bottom) Examples from MATH500 and AIME [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: System prompt for experiments. We employed 5-shot CoT prompts, alongside general instructions for all experiments in our study. The examples were chosen at random and the reasoning demonstrations were created manually. There are minor variations in the prompt template…
Figure 11
Figure 11. Figure 11: Examples of R paths from the GSM8K split of the KisMATH dataset [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 8 canonical work pages

  1. [1]

    Fazl Barez, Tung-Yu Wu, Iván Arcuschin, Michael Lan, Vincent Wang, Noah Siegel, Nicolas Collignon, Clement Neo, Isabelle Lee, Alasdair Paren, Adel Bibi, Robert Trager, Damiano Fornasiere, John Yan, Yanai Elazar, and Yoshua Bengio. 2025. https://arxiv.org/abs/2025.02v2 Chain-of-thought is not explainability . Preprint, alphaXiv:2025.02v2

    work page 2025

  2. [2]

    Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy

    Paul C. Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. 2025. https://arxiv.org/abs/2506.19143 Thought anchors: Which llm reasoning steps matter? Preprint, arXiv:2506.19143

    arXiv 2025

  3. [3]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

  4. [4]

    DeepSeek-AI. 2025. https://api-docs.deepseek.com/news/news250528 Deepseek-r1-0528 release

    work page 2025

  5. [5]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement lea...

    arXiv 2025

  6. [6]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. https://arxiv.org/abs/2407.21783 The Llama 3...

    arXiv 2024

  7. [7]

    Subbarao Kambhampati. 2024. https://doi.org/10.1111/nyas.15125 Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15–18

    doi:10.1111/nyas.15125 2024

  8. [8]

    Subbarao Kambhampati, Kaya Stechly, and Karthik Valmeekam. 2025. https://doi.org/10.1111/nyas.15339 (how) do reasoning models reason? Annals of the New York Academy of Sciences, 1547(1):33–40

    doi:10.1111/nyas.15339 2025

Show all 33 references
  1. [9]

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish...

    2023 arXiv

  2. [10]

    Jinu Lee, Sagnik Mukherjee, Dilek Hakkani-Tur, and Julia Hockenmaier. 2025. https://arxiv.org/abs/2506.02532 Reasoningflow: Semantic structure of complex reasoning traces . Preprint, arXiv:2506.02532

    2025 arXiv

  3. [11]

    Patil, Matei Zaharia, Joseph E

    Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. https://arxiv.org/abs/2502.07374 Llms can easily learn to reason from demonstrations structure, ...

    2025 arXiv

  4. [12]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. https://arxiv.org/abs/2305.20050 Let's Verify Step by Step . Preprint, arXiv:2305.20050

    2023 arXiv

  5. [13]

    Meta. 2024 a . https://ai.meta.com/blog/meta-llama-3-1 Introducing L lama 3.1: O ur most capable models to date

    2024

  6. [14]

    Meta. 2024 b . https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_3/ Llama 3.3 M odel C ards and P rompt formats

    2024

  7. [15]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, and 244 oth...

    2024 arXiv

  8. [16]

    OpenAI. 2025. https://openai.com/index/openai-o1-mini-advancing-cost-efficient-reasoning/ Introducing OpenAI o3 and o4-mini

    2025

  9. [17]

    Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.882 Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning . In Findings of the Association for Computational Linguistics: E...

    2024 doi

  10. [18]

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2024. https://arxiv.org/abs/2412...

    2024 arXiv

  11. [19]

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. 2025. https://arxiv.org/abs/2506.10947 Spurious rewards: Rethinking...

    2025 arXiv

  12. [20]

    Kaya Stechly, Karthik Valmeekam, Atharva Gundawar, Vardhan Palod, and Subbarao Kambhampati. 2025. https://arxiv.org/abs/2505.13775 Beyond semantics: The unreasonable effectiveness of reasonless intermediate tokens . Preprint, arXiv:2505.13775

    2025 arXiv

  13. [21]

    Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2024. https://openreview.net/forum?id=kPBEAZU5Nm Chain of thoughtlessness? an analysis of cot in planning . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  14. [22]

    Juanhe (TJ) Tan. 2023. https://doi.org/10.18653/v1/2023.blackboxnlp-1.12 Causal abstraction for chain-of-thought reasoning in arithmetic word problems . In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 155--168, Singapor...

    2023 doi

  15. [23]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, ...

    2025 arXiv

  16. [24]

    Hemish Veeraboina. 2023. https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024 Aime problem set 1983-2024

    2023

  17. [25]

    Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023 a . https://doi.org/10.18653/v1/2023.acl-long.153 Towards understanding chain-of-thought prompting: An empirical study of what matters . In Proceedings of the 61st Annual Meeting of t...

    2023 doi

  18. [26]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.147 Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models . In Proceedings of the 61st Annua...

    2023 doi

  19. [27]

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025. https://arxiv.org/abs/2506.01939 Beyond the 80/20 Rule: ...

    2025 arXiv

  20. [28]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  21. [29]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qw...

    2025 arXiv

  22. [30]

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. https://arxiv.org/abs/2504.13837 Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? Preprint, arXiv:2504.13837

    2025 arXiv

  23. [31]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. https://openreview.net/forum?id=5NTt8GFjUHkr Automatic chain of thought prompting in large language models . In The Eleventh International Conference on Learning Representations

    2023

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.