Pith. sign in

REVIEW 5 major objections 5 minor 56 references

Reasoning Can Hurt the Inductive Abilities of Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that chain-of-thought prompting can hurt inductive rule learning: on four games with hidden rules, reasoning models often underperform direct-answer models, and targeted interventions restore accuracy without retraining.

desk verdict A creative benchmark and a plausible failure taxonomy, but the headline causal claim is confounded by comparing different model families rather than reasoning mode within the same model. read the letter →

arxiv 2505.24225 v1 pith:3YNXB6NU submitted 2025-05-30 cs.CV cs.AIcs.CL

Haibo Jin , Peiyan Zhang , Man Luo , Haohan Wang This is my paper
classification cs.CVcs.AIcs.CL
keywords chain-of-thoughtreasoninginductivelargemodelshiddenruleinferencegame-baseddiagnosticserroranalysisinterventionsLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests a common assumption: that chain-of-thought (CoT) prompting, used by large reasoning models (LRMs), improves inductive reasoning. In four controlled games with hidden human-defined rules, models that reason step by step often do worse than models that answer directly. The paper explains this with a theory of reasoning as belief updating, where each step can add noise or misdirection, and proves the expected error is U-shaped in reasoning depth. It then shows that interventions at the decomposition, solving, and summarization stages improve inductive accuracy without retraining. If right, this means more reasoning is not automatically better; what matters is the structure and reliability of each step.

What carries the argument

The load-bearing object is the belief-update recursion $e_k = (1-\gamma_k\alpha_k)e_{k-1} - \gamma_k\varepsilon_k$, with $\alpha_k$ the task alignment of the current sub-question, $\varepsilon_k$ the answer-generation noise, and $\gamma_k$ the step-size weight. Unrolling it gives the expected error $E(N) = b_0 \prod_{i=1}^N (1-\gamma_i\bar\alpha)^2 + \sigma^2 \sum_{i=1}^N \gamma_i^2 \prod_{j=i+1}^N (1-\gamma_j\bar\alpha)^2 + \Delta(N)$, whose bias term shrinks with depth while the variance terms grow. The paper proves (Theorem 4.1) that this makes $E(N)$ U-shaped with a unique minimizer $N^\star$, so reasoning has an optimal length. The same decomposition motivates the interventions: structured decomposition templates raise $\alpha_k$, worked examples that avoid numeric extrapolation reduce $\varepsilon_k$, and a 1000-token budget caps $N$ near $N^\star$.

What would settle it

Take a random sample of the model-induced rules from all eight models and have human annotators judge semantic equivalence to the ground truth, then compare human judgments with the GPT-4o majority vote; if agreement is low, the reported accuracy numbers and the reasoning-versus-nonreasoning gap are not established.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that reasoning steps can amplify error rather than reduce it. The authors model chain-of-thought as a sequence of ask-and-answer operations on a belief state $m_k$, where each step receives evidence $g_k = \alpha_k(y^\star - m_{k-1}) + \varepsilon_k$; the belief error $e_k = m_k - y^\star$ follows $e_k = (1-\gamma_k\alpha_k)e_{k-1} - \gamma_k\varepsilon_k$. From this recursion they derive three failure modes: incorrect sub-task decomposition (low or negative alignment $\alpha_k$), incorrect sub-task solving (noise $\varepsilon_k$), and incorrect final summarization (a stopping depth $N$ that misses the unique optimum $N^\star$). They prove the expected squared error is U-shaped in $N$, giving a formal sense in which deeper reasoning eventually hurts. Empirically, solving errors dominate, with math overuse as the most common subtype, and structured interventions targeting the three modes improve accuracy, especially on special rules.

Load-bearing premise

The load-bearing premise is that GPT-4o can reliably judge whether a model-induced rule means the same thing as the ground-truth rule; no human validation of this judge is reported, and GPT-4o is itself one of the eight evaluated models.

Editorial extensions

If this is right

  • The default assumption that chain-of-thought prompting always helps inductive tasks is false: on hidden-rule games, reasoning models consistently underperform direct-answer models.
  • Guiding the structure of reasoning—decomposition, solving, and summarization—can improve inductive accuracy without retraining, with the combined intervention giving the largest gains on special rules.
  • Reasoning length should be treated as a tunable resource: token budgets and early stopping can reduce error by preventing over-reasoning past the optimal depth.
  • Improving sub-task alignment is the most reliable lever: the theory says any increase in positive alignment strictly lowers expected error, while negative alignment guarantees divergence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the GPT-4o judge is less reliable than assumed, the headline gap between reasoning and non-reasoning models could shrink or shift; a human-validated judge would settle this.
  • Editorial inference: the U-shape result suggests a deployment recipe: measure performance at several reasoning depths and pick the stopping point empirically, since the optimum $N^\star$ depends on task noise, which varies by domain.
  • Editorial inference: the math-overuse finding supports a testable extension—supplying models with symbolic, non-numeric priors or explicitly banning arithmetic on suits and piece positions may transfer the observed gains to other inductive tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces four controlled, diagnostic game-based induction tasks (chess, Texas Hold'em, dice, blackjack) with hidden human-defined rules, and compares eight LLMs/LRMs on rule induction. It reports that LRMs often underperform non-reasoning LLMs on special rules, proposes a belief-update error recursion that decomposes reasoning failures into breakdown, solving, and summarization errors, derives a U-shaped expected-error theorem, and designs prompt interventions targeting these failure modes. The task designs are creative and the failure taxonomy is plausible, but the central causal claim is confounded by model identity, the evaluation rests on an unvalidated GPT-4o judge, and the main theorem has a proof gap.

Significance. If the empirical finding survives a controlled same-model comparison, it would be an important challenge to the default assumption that more reasoning steps always help rule induction. The paper's strengths include controlled rule-induction tasks spanning four game domains, an explicit theoretical model with derived error recursion rather than fitted parameters, and prospective evaluation of structured interventions. The failure-mode taxonomy (decomposition/solving/summarization) is practically actionable. However, the load-bearing evaluation and the theoretical theorem require repair before the claims can be accepted.

major comments (5)
  1. [Section 3.3, Fig. 2] The abstract's causal claim that CoT reasoning degrades inductive performance is not established, because the LRM-versus-LLM comparison in Fig. 2 jointly varies model identity, architecture, training data, model size, and release date. For example, GPT-o3 versus GPT-4o and DeepSeek-R1 versus DeepSeek-V3 differ in more than reasoning mode, so reasoning mode is not an independent variable. A same-model baseline is needed: for the locally run models (DeepSeek-R1, QwQ), compare CoT against a direct-answer or reasoning-disabled condition for the same model; without such a contrast, the data only show that certain LRMs perform worse than certain LLMs, not that reasoning per se hurts.
  2. [Section 3.1, Implementation Details] All accuracy numbers are produced by GPT-4o judging semantic equivalence between the model-induced rule and the ground-truth rule, yet no human validation of this judge is reported and GPT-4o is itself one of the eight evaluated models. This creates a potential evaluation bias favoring GPT-4o's own rule formulations, and the claimed numerical gaps in Fig. 2 and Fig. 3 could change under an independent judge. The authors should report a human-annotated subset with judge accuracy and inter-annotator agreement, and/or use an independent judge or a deterministic rule-checking procedure for the main comparisons.
  3. [Section 4.1, Eq. (5); Appendix F.4, Theorem F.1] Theorem 4.1 is not proven as stated. In Appendix F.4, Step 2 asserts that V(N) diverges because sum_i gamma_i = infinity, but this is false: for gamma_i = 1/i the variance term in Eq. (12) tends to 0, so the expected error need not have an increasing arm and the claimed U-shape can fail. In addition, the theorem assumes Delta(N) is non-decreasing without deriving this from Eq. (5); Appendix F.3 only says Delta(N) >= 0. The proof's 'exactly one sign flip' uniqueness claim also does not follow from the component monotonicity. The theorem should be restated with additional assumptions that guarantee variance growth (e.g., liminf gamma_i > 0) and supplied with a rigorous proof, and the U-shape should be checked empirically.
  4. [Table 1] Table 1 is internally inconsistent: the header lists six categories (Breakdown, Solving, Summary, Hallucinated Rule, Overgeneralization, Math Overuse), but each row contains only five data columns, and the counts in the Solving column do not equal the sum of the three subtype columns. Since this table is the empirical basis for the Section 4.3 claims that Solving Errors dominate and that Math Overuse is the most frequent subtype, the table must be corrected and the percentages recomputed before those claims can be evaluated.
  5. [Section 5.2, Fig. 3] The intervention results are reported without error bars or significance tests; the text states that SR accuracy increases by 20-40% under the combined intervention, but it is unclear whether this is an average over models, rules, or games, and how much variance exists across transcripts. Report per-condition means with confidence intervals or paired statistical tests comparing the combined intervention against unstructured CoT, and ideally also against a same-model direct-answer baseline.
minor comments (5)
  1. [Abstract and Section 1] There are grammatical errors: 'We investigate this assumption with creating' should be 'by creating', and 'Our work address' should be 'Our work addresses'.
  2. [Fig. 2 and Fig. 3] The bar charts do not show error bars or sample-size labels; please add the number of transcripts or rule instances per bar, since the accuracy values are point estimates.
  3. [Section 3.1] The justification for using GPT-4o as a judge cites MT-Bench [35], but MT-Bench validates preference judging for chat responses, not semantic equivalence of induced rules; this citation is insufficient and should be replaced or supplemented.
  4. [Section 5.1] The solving-stage intervention is said to follow Kuo et al. [42], but that reference is a jailbreak paper on hijacking chain-of-thought safety reasoning; it does not appear to support the claim about non-numeric worked examples and should be replaced or removed.
  5. [Section 4.3] The text reports that failure traces were independently reviewed by two annotators, but no agreement statistic is provided; please report Cohen's kappa or a similar measure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central comparison and theory are not derived from their own outputs by construction.

full rationale

The paper's central empirical comparison (LRMs vs. non-reasoning LLMs) is an observational benchmark result, not a quantity fitted from the data and then relabeled as a prediction. The theoretical framework in Section 4 is derived from explicit assumptions (evidence model Eq. (2), belief update Eq. (3), error recursion Eq. (4)), and Theorem 4.1's U-shaped error curve follows analytically from those assumptions rather than from the experimental accuracy values. The failure-mode taxonomy is presented as a modeling decomposition, and the Section 5 interventions are tested prospectively on the benchmark tasks with no fitted parameters being reused as evaluation scores. The GPT-4o-as-judge design and the cross-model LRM/LLM comparison raise validity concerns such as potential judge bias and confounding of model identity with reasoning mode, but these are evaluation-bias and experimental-design issues, not circular derivations; no step in the paper reduces to its own inputs by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's 'theory' is a stochastic approximation model with free parameters that are never estimated. The central empirical claim depends mainly on the GPT-4o judge assumption; the U-shape theorem additionally depends on the unproved monotonicity of Delta(N). No new particles, forces, or physical entities are introduced.

free parameters (4)
  • alpha_bar (mean question alignment)
    Theoretical parameter in the belief update model (Eq. 2), not estimated from data; the U-shape theorem assumes it lies in (0,1).
  • sigma^2 (answer noise variance)
    Noise variance in the evidence model (Eq. 2), assumed constant and not fitted.
  • gamma_i (integration weights)
    Step-size sequence in the belief update (Eq. 3), assumed in (0,1) with infinite sum; not estimated.
  • b0 (initial squared error)
    Initial belief error norm, symbolic in the derivations; not estimated.
assumptions (5)
  • domain assumption The reasoning process follows the linear belief update mk = mk-1 + gamma_k * gk (Eq. 3).
    This is an assumed model of how chain-of-thought updates a hypothesis, not derived from data.
  • domain assumption Evidence at each step is gk = alpha_k * (y* - mk-1) + epsilon_k with epsilon_k ~ N(0, sigma^2 I) (Eq. 2).
    This generative model for sub-task evidence is postulated in Section 4.1.
  • domain assumption alpha_k are i.i.d. with mean alpha_bar and variance tau^2; gamma_k satisfy 0 < gamma_k <= gamma_bar < 1 and sum gamma_k = infinity.
    These distributional and summability assumptions are required for the proof of the U-shape theorem (Appendix F.4).
  • ad hoc to paper The extra variance term Delta(N) is non-decreasing in N (Eq. 5 and Appendix F.4).
    Delta(N) is introduced to collect the discrepancy from averaging over alpha, but its monotonicity is asserted without proof; the theorem depends on it.
  • domain assumption GPT-4o can reliably judge semantic equivalence between an induced rule and the ground-truth rule (Section 3.1).
    All accuracy numbers depend on this judge, yet no human validation or inter-judge agreement is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning Can Hurt the Inductive Abilities of Large Language Models." pith.science (2026). https://pith.science/paper/3YNXB6NU

@misc{pith2026250524225,
  author       = {Pith},
  title        = {Pith review of: Reasoning Can Hurt the Inductive Abilities of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3YNXB6NU}},
  note         = {Machine review of arXiv:2505.24225}
}
read the original abstract

Large Language Models (LLMs) have shown remarkable progress across domains, yet their ability to perform inductive reasoning - inferring latent rules from sparse examples - remains limited. It is often assumed that chain-of-thought (CoT) prompting, as used in Large Reasoning Models (LRMs), enhances such reasoning. We investigate this assumption with creating four controlled, diagnostic game-based tasks - chess, Texas Hold'em, dice games, and blackjack - with hidden human-defined rules. We find that CoT reasoning can degrade inductive performance, with LRMs often underperforming their non-reasoning counterparts. To explain this, we present a theoretical framework that reveals how reasoning steps can amplify error through three failure modes: incorrect sub-task decomposition, incorrect sub-task solving, and incorrect final answer summarization. Based on our theoretical and empirical analysis, we introduce structured interventions that adapt CoT generation according to our identified failure types. These interventions improve inductive accuracy without retraining. Our findings suggest that effective (CoT) reasoning depends not only on taking more steps but also on ensuring those steps are well-structured.

Figures

Figures reproduced from arXiv: 2505.24225 by the authors.

Figure 1
Figure 1. Examples illustrating inductive reasoning on gameplay transcripts. (a) Games begin with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Inductive accuracy on normal rules (NRs) and special rules (SRs) across four games. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Inductive rule accuracy across different intervention strategies and models for each game [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 23 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [3]

    The role of deductive and inductive reasoning in large language models

    Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq- Neng Hwang, Serge Belongie, and Lei Li. The role of deductive and inductive reasoning in large language models. arXiv preprint arXiv:2410.02892, 2024

    arXiv

  3. [4]

    The next decade in ai: four steps towards robust artificial intelligence

    Gary Marcus. The next decade in ai: four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177, 2020

    arXiv 2002

  4. [5]

    Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models

    Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. arXiv preprint arXiv:2406.02061, 2024

    arXiv 2024

  5. [6]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  6. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  7. [8]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    arXiv 2024

  8. [9]

    Advancing reasoning in large language models: Promising methods and ap- proaches

    Avinash Patil. Advancing reasoning in large language models: Promising methods and ap- proaches. arXiv preprint arXiv:2502.03671, 2025

    arXiv 2025

Show all 56 references
  1. [10]

    Unveiling the impact of coding data instruction fine-tuning on large language models reasoning

    Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang, Lichang Chen, William Yang Wang, and Linda Ruth Petzold. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume ...

    2025

  2. [11]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022

    2022 arXiv

  3. [12]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:11809–11822, 2023

    2023

  4. [13]

    Dcr: Divide-and-conquer reasoning for multi-choice question answering with llms

    Zijie Meng, Yan Zhang, Zhaopeng Feng, and Zuozhu Liu. Dcr: Divide-and-conquer reasoning for multi-choice question answering with llms. arXiv preprint arXiv:2401.05190, 2024. 10

    2024 arXiv

  5. [14]

    When more is less: Understanding chain-of-thought length in llms

    Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms. arXiv preprint arXiv:2502.07266, 2025

    2025 arXiv

  6. [15]

    An examination on the effectiveness of divide-and-conquer prompting in large language models

    Yizhou Zhang, Lun Du, Defu Cao, Qiang Fu, and Yan Liu. An examination on the effectiveness of divide-and-conquer prompting in large language models. arXiv preprint arXiv:2402.05359, 2024

    2024 arXiv

  7. [16]

    Towards revealing the mystery behind chain of thought: a theoretical perspective

    Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36:70757–70798, 2023

    2023

  8. [17]

    Chain of thought empowers transformers to solve inherently serial problems

    Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint arXiv:2402.12875, 1, 2024

    2024 arXiv

  9. [18]

    A the- oretical understanding of chain-of-thought: Coherent reasoning and error-aware demonstration

    Yingqian Cui, Pengfei He, Xianfeng Tang, Qi He, Chen Luo, Jiliang Tang, and Yue Xing. A the- oretical understanding of chain-of-thought: Coherent reasoning and error-aware demonstration. arXiv preprint arXiv:2410.16540, 2024

    2024 arXiv

  10. [19]

    Understanding chain-of-thought in llms through information theory

    Jean-Francois Ton, Muhammad Faaiz Taufiq, and Yang Liu. Understanding chain-of-thought in llms through information theory. arXiv preprint arXiv:2411.11984, 2024

    2024 arXiv

  11. [20]

    What happened in llms layers when trained for fast vs

    Ming Li, Yanhong Li, and Tianyi Zhou. What happened in llms layers when trained for fast vs. slow thinking: A gradient perspective. arXiv preprint arXiv:2410.23743, 2024

    2024 arXiv

  12. [21]

    Physics of language models: Part 2.1, grade-school math and the hidden reasoning process

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. In The Thirteenth International Conference on Learning Representations, 2024

    2024

  13. [22]

    Wiley interdisciplinary reviews: cognitive science

    D Collerton and UP Mosimann. Wiley interdisciplinary reviews: cognitive science. 2010

    2010

  14. [23]

    Properties of inductive reasoning

    Evan Heit. Properties of inductive reasoning. Psychonomic bulletin & review, 7:569–592, 2000

    2000

  15. [24]

    Wilt: A multi-turn, memorization-robust inductive logic benchmark for llms

    Eryk Banatt, Jonathan Cheng, Skanda Vaidyanath, and Tiffany Hwu. Wilt: A multi-turn, memorization-robust inductive logic benchmark for llms. arXiv preprint arXiv:2410.10998, 2024

    2024 arXiv

  16. [25]

    Mirage: Evaluating and explaining inductive reasoning process in language models

    Jiachun Li, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. Mirage: Evaluating and explaining inductive reasoning process in language models. arXiv preprint arXiv:2410.09542, 2024

    2024 arXiv

  17. [26]

    Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks

    Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, et al. Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks. arXiv preprint arXiv:2410.06526, 2024

    arXiv

  18. [27]

    Logicvista: Multimodal llm logical reasoning benchmark in visual contexts

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973, 2024

    2024 arXiv

  19. [28]

    Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations

    Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias B Khalil. Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. arXiv preprint arXiv:2305.18354, 2023

    2023 arXiv

  20. [29]

    Mir-bench: Benchmarking llm’s long-context intelligence via many-shot in-context inductive reasoning

    Kai Yan, Zhan Ling, Kang Liu, Yifan Yang, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. Mir-bench: Benchmarking llm’s long-context intelligence via many-shot in-context inductive reasoning. arXiv preprint arXiv:2502.09933, 2025

    2025

  21. [30]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  22. [31]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  23. [32]

    Grok-2 beta release, 2024

    xAI Team. Grok-2 beta release, 2024

    2024

  24. [33]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  25. [34]

    Grok 3 beta — the age of reasoning agents, 2025

    xAI Team. Grok 3 beta — the age of reasoning agents, 2025

    2025

  26. [35]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  27. [36]

    Seemingly plausible distractors in multi- hop reasoning: Are large language models attentive readers? arXiv preprint arXiv:2409.05197, 2024

    Neeladri Bhuiya, Viktor Schlegel, and Stefan Winkler. Seemingly plausible distractors in multi- hop reasoning: Are large language models attentive readers? arXiv preprint arXiv:2409.05197, 2024

    2024 arXiv

  28. [37]

    Hopping too late: Exploring the limitations of large language models on multi-hop queries

    Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. arXiv preprint arXiv:2406.12775, 2024

    2024 arXiv

  29. [38]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems , 36:46534–46594, 2023

    2023

  30. [39]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  31. [40]

    Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965, 2023

    2023

  32. [41]

    The impact of reasoning step length on large language models.arXiv preprint arXiv:2401.04925, 2024

    Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models.arXiv preprint arXiv:2401.04925, 2024

    2024 arXiv

  33. [42]

    H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking

    Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen. H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash think...

    2025 arXiv

  34. [43]

    Harnessing the reasoning economy: A survey of efficient reasoning for large language models

    Rui Wang, Hongru Wang, Boyang Xue, Jianhui Pang, Shudong Liu, Yi Chen, Jiahao Qiu, Derek Fai Wong, Heng Ji, and Kam-Fai Wong. Harnessing the reasoning economy: A survey of efficient reasoning for large language models. arXiv preprint arXiv:2503.24377, 2025

    2025 arXiv

  35. [44]

    difference of one

    Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aware llm reasoning. arXiv preprint arXiv:2412.18547, 2024. 12 A Details for Judging A.1 Examples of Judging To illustrate the evaluation process, we provide examples of GPT-4o judg...

    2024 arXiv

  36. [45]

    Question–alignment noise (αk): cognitive or pragmatic imperfections in how the sub-question is framed. A well -posed question ( αk ≈ 1) elicits evidence that points almost exactly along the residual error, whereas a misguided question ( αk ≈ 0) yields a useless tangent, and a ...

  37. [46]

    Evidence model

    Answer-generation noise (εk): stochasticity in the LLM’s sampling, retrieval errors, hallu- cinations, token truncation, or memory decay. Evidence model. Conditioned on the history up to step k−1, the Answer action produces the evidence vector gk = αk|{z} alignment y⋆ − mk−1 +...

  38. [47]

    The deterministic component αk(y⋆ − mk−1) is collinear with the current error vector

    Directionality. The deterministic component αk(y⋆ − mk−1) is collinear with the current error vector. Its amplitude encodes how well the sub -question aligns with the unknown residual structure

  39. [48]

    The random component εk is isotropic and unbiased, reflecting that answer noise does not systematically drift the belief in any preferred direction

    Zero-mean perturbations. The random component εk is isotropic and unbiased, reflecting that answer noise does not systematically drift the belief in any preferred direction. Belief–integration weight γk. Upon receiving gk the agent chooses how aggressively to incorporate it. T...

  40. [49]

    E(N ) is strictly decreasing for N < N⋆ and strictly increasing for N > N⋆, with N ⋆ := min N ≥ 0 : E(N +1) > E(N )

  41. [50]

    Consequently, E(N ) is U-shaped when plotted against reason- ing depth N

    the minimiser N ⋆ is unique. Consequently, E(N ) is U-shaped when plotted against reason- ing depth N. Proof. Write (12) as E(N ) = A(N ) + V (N ) + ∆(N ) with ( A(N ) := b0 P (N )2, V (N ) := σ2PN i=1 γ2 i P (i+1, N)2, where P (a, b) := Qb j=a(1 − γj ¯α) and we abbreviate P (...

  42. [51]

    There exists a finiteN ⋆ α such that Eα(N +1) < Eα(N ) for N < N⋆ α and Eα(N +1) > Eα(N ) for N ≥ N ⋆ α

  43. [52]

    19 Proof

    The minimiser N ⋆ α is unique, hence Eα(N ) is U-shaped in N. 19 Proof. Define Pα(a, b) := Qb j=a(1 − αγj) and abbreviate Pα(1, N) by Pα(N ). Decompose Eα(N ) = Aα(N ) + Vα(N ) with Aα(N ) = b0 Pα(N )2, V α(N ) = σ2 NX i=1 γ2 i Pα(i+1, N)2. (i) Monotonicity of Aα. Because 0 < ...

  44. [53]

    The partial derivative of Eα,γ(N ) with respect to α is strictly negative: ∂ ∂α Eα,γ(N ) < 0, ∀ (α, γ) ∈ (0, 1)2, N ≥ 1

    Alignment is always beneficial. The partial derivative of Eα,γ(N ) with respect to α is strictly negative: ∂ ∂α Eα,γ(N ) < 0, ∀ (α, γ) ∈ (0, 1)2, N ≥ 1

  45. [54]

    Integration weight exhibits a trade-off. For fixed α the map γ 7→ Eα,γ(N ) is U-shaped: there exists a unique γ⋆ N ∈ (0, 1) solving ∂ ∂γ Eα,γ(N ) = 0 , (15) such that Eα,γ(N ) is strictly decreasing on (0, γ⋆ N ) and strictly increasing on (γ⋆ N , 1). Proof. (a) Monotonicity i...

  46. [55]

    Here 0 < ρ(α) < 1 and ∂ ∂α Eα(N ) < 0, ∀N ≥ 1

    Positive alignment 0 < α < γ−1 . Here 0 < ρ(α) < 1 and ∂ ∂α Eα(N ) < 0, ∀N ≥ 1. Increasing α always lowers the error

  47. [56]

    Then ρ = 1 and E0(N ) = b0 + N σ2γ2

    Zero alignment (α = 0). Then ρ = 1 and E0(N ) = b0 + N σ2γ2. Bias never decays; variance grows linearly with depth

  48. [57]

    angular displace- ment

    Negative alignment −γ−1 < α <0 . Now ρ(α) > 1 and Eα(N ) = b0 ρ2N + σ2γ2ρ2N −1 ρ2−1 , which increases strictly with N. More- over ∂ ∂α Eα(N ) > 0; making α less negative reduces error. Proof. Differentiate (16) wrt. α via ρ(α): ∂Eα ∂α = −γ ∂Eα ∂ρ . A direct calculation gives ∂...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.