Pith. sign in

REVIEW 3 major objections 4 minor 30 references

What Does Chain-of-Thought Contribute at Probe Time? Evidence for Local Co-Occurrence Activation

T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The probe-time benefit of chain-of-thought rationales comes mainly from short-range word co-occurrence, not from sentence-level logical order.

desk verdict A clean probe-time perturbation study showing CoT gains are mostly local, but the abstract overclaims and the causal interpretation needs hedging. read the letter →

arxiv 2605.26795 v2 pith:TP5JB7FG submitted 2026-05-26 cs.AI

Xiang Wang , Wei Wei This is my paper · ORCID
classification cs.AI
keywords chain-of-thoughtprobe-timelocalco-occurrencelexicalactivationn-gramperturbationgap-recoveryrationalerobustnessprompt
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish what a fixed chain-of-thought rationale actually does to a language model when it is read at answer time. Across multiple datasets, model families, and scales, the authors find that randomly shuffling the rationale's sentences barely changes accuracy, while globally scrambling all its words still outperforms the no-rationale baseline. Preserving only contiguous two- or three-token windows recovers most of the remaining gap toward the intact rationale. The paper concludes that the probe-time gain is driven by lexical activation—the presence of domain-relevant words—plus short-range token co-occurrence, rather than by the logical derivation order. If true, this changes how we think about chain-of-thought: its value at prediction time may be local priming, not faithful reasoning.

What carries the argument

The central tool is the n-block shuffle perturbation together with the gap-recovery statistic GR(n). A rationale is partitioned into non-overlapping contiguous blocks of n tokens; the blocks are randomly permuted while token order inside each block is preserved. GR(n) = (Acc(n) - Acc(WS)) / (Acc(CoT) - Acc(WS)), where WS is the fully shuffled word bag (n=1) and CoT is the intact rationale. The critical window n* is the smallest n with GR(n) > 0.5; across most settings n* = 2–3. This statistic isolates the contribution of local structure by measuring how much of the gap between a word bag and the full text is closed when only n-token adjacency is retained.

What would settle it

Build a rationale whose correct answer depends on a chain of co-reference spanning more than three tokens (e.g., a multi-step proof where an intermediate result from step 1 is used in step 10), shuffle it in 3-token blocks, and measure probe accuracy. If accuracy collapses despite all the tokens being present, the local co-occurrence claim is falsified; if accuracy remains high, the claim survives.

Watch

Extended reading notes

Core claim

The paper's central claim is that, at probe time, the usefulness of a fixed chain-of-thought rationale is carried by two complementary textual properties: the vocabulary it injects (lexical activation) and short-range token adjacency within windows of two to three words. To quantify this, the authors define a gap-recovery statistic GR(n): the fraction of the accuracy gap between a fully word-shuffled rationale and the intact rationale that is recovered when the rationale is divided into random contiguous blocks of n tokens and the blocks are shuffled. In almost every configuration, GR crosses 0.5 already at n=2 or 3, meaning that more than half of the structured-text advantage is recovered b

Load-bearing premise

The argument treats accuracy on shuffled text as a direct measure of what the unperturbed rationale contributes locally; if the probe model switches from following the derivation to a lexical-heuristic strategy when global coherence is destroyed, then the small-window recovery does not prove that the intact rationale's benefit is local.

Editorial extensions

If this is right

  • Randomly shuffling the sentences of a fixed rationale changes accuracy very little, so the global order of reasoning steps is not the main source of probe-time benefit.
  • A word-scrambled rationale still beats direct input-output prompting, so lexical activation alone accounts for a large share of the gain.
  • Preserving only two- or three-word contiguous windows recovers most of the remaining gap to full chain-of-thought in many settings.
  • Explicit answer declarations and even the correct answer value can be removed from the rationale with little loss, ruling out answer-copying as the primary mechanism.
  • The small-window recovery is specific to rationale-style text: topic-matched reference passages give near-zero gains, and ordinary question text needs much larger windows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the local co-occurrence account is right, chain-of-thought rationales could be aggressively compressed to short local phrase fragments before injection, with little loss at probe time—a testable extension for prompt-compression work.
  • The paper's asymmetric-strategy concern: accuracy on shuffled text might reflect a fallback to lexical heuristics rather than the intact rationale's local structure; this could be tested by probing internal representations to see whether shuffled and intact rationales engage the same computation circuits.
  • The claim predicts that tasks whose correct answers genuinely require long-range dependencies (e.g., multi-step variable substitutions spanning many tokens) should show larger n*; constructing such rationales would provide a direct stress test.
  • The locality finding connects to broader debates about whether chain-of-thought is a faithful derivation or a distributional priming mechanism; it suggests faithfulness at the sentence level may matter less than token-level associations for prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies chain-of-thought (CoT) prompting from a probe-time perspective: a fixed generated rationale is injected into a probe model's context, and the text is perturbed to determine which textual properties affect the final answer. Across multiple model configurations, datasets, and scales, the authors find that (i) sentence-shuffled rationales perform nearly as well as full CoT, (ii) word-shuffled rationales substantially outperform the no-rationale baseline, and (iii) preserving non-overlapping n-token windows (with n≈2–3) recovers most of the remaining gap between word shuffle and full CoT. Control experiments rule out explicit answer copying, generic topical text, and full grammatical structure as primary explanations. The authors conclude that the probe-time benefit of CoT arises primarily from lexical activation and short-range token co-occurrence rather than sentence-level logical derivation.

Significance. If the conclusion holds, it would significantly challenge the common assumption that CoT rationales help because they encode a global logical derivation, with implications for prompt compression, CoT faithfulness, and mechanistic interpretability. The study has notable strengths: a matched generator–probe design, pooling across three seeds with paired McNemar tests, replication across four configurations and multiple datasets (including open-ended generation), and a well-designed battery of controls (answer stripping, answer-value removal, Wikipedia injection, question-stem sweeps, concept compression). The central quantitative anchor—GR(n) crossing 0.5 at small n—is a clean, falsifiable summary statistic. However, the causal interpretation of GR(n) is subject to an unaddressed strategy-shift confound, and the reported results lack uncertainty quantification; both issues are load-bearing for the paper's main claim.

major comments (3)
  1. [§2.2–2.3, Conclusion] The interpretation of GR(n) as measuring the causal contribution of local windows to full-CoT performance assumes that the probe model uses the same reading strategy on intact and n-block-shuffled rationales. The SS result (Fig. 1) and the question-stem control (§3.3) do not rule out an asymmetric strategy shift: when global coherence is destroyed, the model may fall back to a lexical/local heuristic that it does not use on coherent CoT. In that case, high GR(2) for Config D shows only that local fragments suffice for a different strategy, not that full CoT's benefit arises from those windows. This is load-bearing for the central claim. I suggest an instance-level test: compute the per-example agreement (contingency table/correlation) between correctness under full CoT and under the n-block condition; if the same examples are solved in both, strategy invariance is supported. Alternativel
  2. [§2.2 and Abstract] The n-gram protocol partitions the sequence into non-overlapping blocks of n tokens and shuffles the blocks. Thus for n=2, only pairs (w1,w2), (w3,w4), … are preserved; boundary bigrams such as (w2,w3) are destroyed. The condition therefore does not preserve 'all adjacent two-token windows' as the abstract states; it preserves only a strict subset. The reported n* values (e.g., GR(2)=0.56 for Config D) may depend on this particular partition, and shifting the partition offset could change the recovery curve. Please clarify this in the main text and report sensitivity to the block offset, or use a perturbation that preserves all n-token co-occurrences (e.g., a constrained token shuffle maintaining within-distance order). This is essential for a precise interpretation of 'n=2–3 windows recover most of the gain.'
  3. [§2.3, Table 2] The paper reports GR(n) point estimates and the n* threshold without any measure of uncertainty. With 1,500 paired examples per condition, the ratios are noisy, and some values exceed 1 (e.g., Table 2, Config E, MMLU-Pro: GR(8)=1.07, GR(20)=1.16), which is inconsistent with a bounded recovery measure and suggests sampling error. The claim that sentence shuffle 'has little effect' is not established by a non-significant McNemar test; failure to reject a difference is not evidence of equivalence. Please provide bootstrap confidence intervals for GR(n), report n* with its uncertainty (e.g., a distribution over bootstrap resamples), and run an equivalence test (e.g., TOST with a prespecified minimal effect size) for SS vs. CoT. This is necessary to support the quantitative, cross-setting claims in Sections 2.3 and 4.
minor comments (4)
  1. [Figure 2 caption] The caption lists n∈{1,2,3,5,8,SS} but the x-axis also includes n=12 and n=20. Please make the caption consistent with the plotted window sizes.
  2. [Table 1] The Word-Shuffled row appears to contain a stray hyphen at the beginning; check the typesetting.
  3. [§3.1] The sentence 'we also attempted to randomly mask some tokens (Mask 30%/50%) ... but the results suggest that this is likely not the case' is vague. Report the actual accuracies or remove the claim, as it is currently unsupported by any number or figure.
  4. [§2.3] The choice of GR(n)>0.5 as the threshold defining n* is arbitrary. A brief justification or a robustness check (e.g., 0.4 or 0.6) would strengthen the claim that n*=2–3 is a stable finding.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GR(n) is a descriptive gap-recovery statistic, n* is empirically read off the data, and nothing load-bearing is imported from self-citations.

full rationale

The paper's central quantitative object is the gap-recovery statistic GR(n) = (Acc(n) - Acc(WS)) / (Acc(CoT) - Acc(WS)), and the critical window n* is defined as the smallest n with GR(n) > 0.5. This is a descriptive normalization of measured accuracies, not a fitted parameter being used to predict a closely related quantity. The claim that 'the observed gains appear to arise primarily from lexical activation and short-range token co-occurrence rather than sentence-level logical derivation' is a direct summary of the intervention outcomes: WS preserves only token inventory, n-block shuffles preserve only local windows, and SS destroys sentence order. There is no equation in which the conclusion is substituted back into the definition of a variable; the smallness of n* is an empirical finding, not a construction. The paper also does not rely on self-citations: references such as [1], [2], [16], [17] are external prior work, and no author-overlapping citation is used as load-bearing evidence for the LCA interpretation. The controls in §3.3 (Wikipedia injection, question-stem sweep) are genuinely independent checks rather than restatements of the main condition. The skeptic's concern that shuffled inputs may induce a different model reading strategy is an interpretive assumption about what the perturbation measures, not a case where the paper defines X in terms of Y or fits a parameter and then calls it a prediction. Therefore no circular step is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to data in the classical sense; the only hand-chosen numerical convention is the GR>0.5 threshold. No invented entities: 'Local Co-occurrence Activation (LCA)' is an interpretive label for the measured perturbation pattern, not a postulated mechanism with independent falsifiable handles. The load-bearing assumptions are the perturbation-invariance interpretation and the adequacy of the stripping/scoring protocol.

free parameters (1)
  • GR>0.5 threshold defining n* = GR(n) > 0.5 (convention)
    Hand-chosen convention (§2.2) for 'recovers most of the gap'; the empirical content is where the crossing falls, so this is a measurement convention rather than a fitted model parameter. Reported n* values vary 2-8 across Table 2.
assumptions (4)
  • domain assumption Perturbation invariance: accuracy on n-block-shuffled rationale text measures the causal contribution of the preserved local structure in the unperturbed rationale.
    §2.2-2.3: GR(n) is interpreted as 'how much of the CoT gain is recovered'. If the probe model switches strategy on destroyed text, the inference about full CoT fails. §3.3 controls test specificity but not this asymmetry.
  • domain assumption The answer-declaration regex stripping removes leakage without removing the signal under study.
    §2.1 and A.4: 3-11% of tokens stripped; a misspecified stripper could remove the local triggers the paper claims carry the gain. Tail-sweep experiment (§3.1c) partially guards this.
  • domain assumption First-token logprob, softmax-normalized over valid option letters, is an adequate accuracy measure for probe-time CoT.
    A.3; open-ended GSM8K/MATH500 replication (§4.3) and the q-stem control mitigate scoring artifacts.
  • domain assumption Non-overlapping block partitioning (aligned from the first token) is representative; results do not depend on partition alignment or block-boundary effects.
    A.4: at n=2 only about half of adjacent token pairs are preserved (pairs at block boundaries are not), so 'n=2 works' is tested on a random half of bigrams; alignment-dependent boundary effects are not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Does Chain-of-Thought Contribute at Probe Time? Evidence for Local Co-Occurrence Activation." pith.science (2026). https://pith.science/paper/TP5JB7FG

@misc{pith2026260526795,
  author       = {Pith},
  title        = {Pith review of: What Does Chain-of-Thought Contribute at Probe Time? Evidence for Local Co-Occurrence Activation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TP5JB7FG}},
  note         = {Machine review of arXiv:2605.26795}
}
read the original abstract

Chain-of-thought (CoT) prompting enhances large language model performance, yet what drives these gains remains unclear. We study this question from a probe-time perspective: holding CoT rationales fixed, we test which textual properties matter for the final prediction. Across multiple datasets and model configurations, we find that randomizing the order of rationale sentences has little effect on accuracy, suggesting that the global order of reasoning steps is not the main source of the probe-time benefit. Moreover, even when the words in a rationale are randomly reordered, performance remains well above the no-rationale baseline, indicating that the rationale's words remain useful even without their original order. Restoring only short-range word order further improves performance and brings it substantially closer to full CoT. In most settings, much of this local-order gain is already obtained with three-word windows. Control experiments rule out explicit answer copying, simple lexical cues, generic topical context, and general robustness to shuffling as the main explanations. Mechanistic analyses further show that short-window gains are largely formed in early-to-middle model layers, with answer-relevant evidence concentrated in local text spans. Together, these findings support a local co-occurrence activation (LCA) interpretation: the probe-time benefit of fixed rationales arises mainly from the words they contain and short-range word co-occurrences.

Figures

Figures reproduced from arXiv: 2605.26795 by the authors.

Figure 1
Figure 1. Probe-time accuracy under five conditions (IO, CoT, sentence-shuffle SS, word-shuffle WS, and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. N-gram gap-recovery curves GR(n) across all four configurations and three datasets (n ∈ {1, 2, 3, 5, 8, SS}). Each line corresponds to one model configuration; the shaded region marks GR<0.5. The transition from word-bag (n=1) to short local windows is steep and consistent across configurations. 2.3 From word bags to short-range co-occurrence [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Answer-stripping (a), lexical attribution (b), and tail-sweep (c) experiments. Removing explicit answer [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Concept-compression and local-structure experiments (Config D). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Wikipedia control (a) and question-stem control (b). Topic-matched Wikipedia text produces near-zero [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 11 linked inside Pith

  1. [1]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  2. [2]

    Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    2022

  3. [3]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    arXiv 2022

  4. [4]

    Show your work: Scratchpads for intermediate computation with language models

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. 2021

    2021

  5. [5]

    Chain-of-thoughts prompting with language models for accurate math problem-solving

    Sze Ching Evelyn Fung, Man Fai Wong, and Chee Wei Tan. Chain-of-thoughts prompting with language models for accurate math problem-solving. In2023 IEEE MIT Undergraduate Research Technology Conference (URTC), pages 1–5. IEEE, 2023

    2023

  6. [6]

    Towards better chain-of-thought prompting strategies: A survey.arXiv preprint arXiv:2310.04959, 2023

    Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. Towards better chain-of-thought prompting strategies: A survey.arXiv preprint arXiv:2310.04959, 2023. 9 What Makes Chain-of-Thought Work at Probe Time?

    arXiv 2023

  7. [7]

    Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.Advances in Neural Information Processing Systems, 36:74952–74965, 2023

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.Advances in Neural Information Processing Systems, 36:74952–74965, 2023

    2023

  8. [8]

    Faithful logical reasoning via symbolic chain-of-thought

    Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logical reasoning via symbolic chain-of-thought. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13326–13365, 2024

    2024

Show all 30 references
  1. [9]

    Measuring chain-of-thought monitorability through faithfulness and verbosity.arXiv preprint arXiv:2510.27378, 2025

    Austin Meek, Eitan Sprejer, Iván Arcuschin, Austin J Brockmeier, and Steven Basart. Measuring chain-of-thought monitorability through faithfulness and verbosity.arXiv preprint arXiv:2510.27378, 2025

    2025

  2. [10]

    Walk the talk? measuring the faithfulness of large language model explanations

    Katie Matton, Robert Ness, John Guttag, and Emre Kiciman. Walk the talk? measuring the faithfulness of large language model explanations. InInternational Conference on Learning Representations, volume 2025, pages 73212–73277, 2025

    2025

  3. [11]

    Faithcot- bench: Benchmarking instance-level faithfulness of chain-of-thought reasoning.arXiv preprint arXiv:2510.04040, 2025

    Xu Shen, Song Wang, Zhen Tan, Laura Yao, Xinyu Zhao, Kaidi Xu, Xin Wang, and Tianlong Chen. Faithcot- bench: Benchmarking instance-level faithfulness of chain-of-thought reasoning.arXiv preprint arXiv:2510.04040, 2025

    2025

  4. [12]

    In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  5. [13]

    Towards revealing the mystery behind chain of thought: a theoretical perspective.Advances in Neural Information Processing Systems, 36:70757– 70798, 2023

    Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective.Advances in Neural Information Processing Systems, 36:70757– 70798, 2023

    2023

  6. [14]

    Iteration head: A mechanistic study of chain-of-thought.Advances in Neural Information Processing Systems, 37:109101–109122, 2024

    Vivien Cabannes, Charles Arnal, Wassim Bouaziz, Alice Yang, Francois Charton, and Julia Kempe. Iteration head: A mechanistic study of chain-of-thought.Advances in Neural Information Processing Systems, 37:109101–109122, 2024

    2024

  7. [15]

    How does chain of thought think? mechanistic interpretability of chain-of-thought reasoning with sparse autoencoding

    Xi Chen, Aske Plaat, and Niki van Stein. How does chain of thought think? mechanistic interpretability of chain-of-thought reasoning with sparse autoencoding. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 30297–30305, 2026

    2026

  8. [16]

    Text and patterns: For effective chain of thought, it takes two to tango

    Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07686, 2022

    2022 arXiv

  9. [17]

    Towards understanding chain-of-thought prompting: An empirical study of what matters

    Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    2023

  10. [18]

    Let’s think dot by dot: Hidden computation in transformer language models.arXiv preprint arXiv:2404.15758, 2024

    Jacob Pfau, William Merrill, and Samuel R Bowman. Let’s think dot by dot: Hidden computation in transformer language models.arXiv preprint arXiv:2404.15758, 2024

    2024 arXiv

  11. [19]

    Chain-of-thought is not explainability.Preprint, alphaXiv, page v1, 2025

    Fazl Barez, Tung-Yu Wu, Iván Arcuschin, Michael Lan, Vincent Wang, Noah Siegel, Nicolas Collignon, Clement Neo, Isabelle Lee, Alasdair Paren, et al. Chain-of-thought is not explainability.Preprint, alphaXiv, page v1, 2025

    2025

  12. [20]

    Chain-of-thought reasoning in the wild is not always faithful.arXiv preprint arXiv:2503.08679, 2025

    Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. Chain-of-thought reasoning in the wild is not always faithful.arXiv preprint arXiv:2503.08679, 2025

    2025 arXiv

  13. [21]

    Measuring faithfulness in chain-of-thought reasoning

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023

    2023 arXiv

  14. [22]

    Diagnosing memorization in chain-of-thought reasoning, one token at a time

    Huihan Li, You Chen, Siyuan Wang, Yixin He, Ninareh Mehrabi, Rahul Gupta, and Xiang Ren. Diagnosing memorization in chain-of-thought reasoning, one token at a time. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 3158–3180, 2025

    2025

  15. [23]

    Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales?Advances in Neural Information Processing Systems, 37:123846–123910, 2024

    Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales?Advances in Neural Information Processing Systems, 37:123846–123910, 2024

    2024

  16. [24]

    Badchain: Backdoor chain-of-thought prompting for large language models.arXiv preprint arXiv:2401.12242, 2024

    Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. Badchain: Backdoor chain-of-thought prompting for large language models.arXiv preprint arXiv:2401.12242, 2024

    2024 arXiv

  17. [25]

    Preemptive answer “attacks” on chain-of-thought reasoning

    Rongwu Xu, Zehan Qi, and Wei Xu. Preemptive answer “attacks” on chain-of-thought reasoning. InFindings of the Association for Computational Linguistics: ACL 2024, pages 14708–14726, 2024

    2024

  18. [26]

    Evaluating and improving robustness in large language models: a survey and future directions.arXiv preprint arXiv:2506.11111, 2025

    Kun Zhang, Le Wu, Kui Yu, Guangyi Lv, and Dacao Zhang. Evaluating and improving robustness in large language models: a survey and future directions.arXiv preprint arXiv:2506.11111, 2025. 10 What Makes Chain-of-Thought Work at Probe Time?

    2025 arXiv

  19. [27]

    Rethinking chain-of- thought from the perspective of self-training.arXiv preprint arXiv:2412.10827, 2024

    Zongqian Wu, Baoduo Xu, Ruochen Cui, Mengmeng Zhan, Xiaofeng Zhu, and Lei Feng. Rethinking chain-of- thought from the perspective of self-training.arXiv preprint arXiv:2412.10827, 2024

    2024 arXiv

  20. [28]

    Why think step by step? reasoning emerges from the locality of experience.Advances in Neural Information Processing Systems, 36:70926–70947, 2023

    Ben Prystawski, Michael Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience.Advances in Neural Information Processing Systems, 36:70926–70947, 2023

    2023

  21. [29]

    Large lan- guage models are in-context semantic reasoners rather than symbolic reasoners.arXiv preprint arXiv:2305.14825, 2023

    Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. Large lan- guage models are in-context semantic reasoners rather than symbolic reasoners.arXiv preprint arXiv:2305.14825, 2023

    2023 arXiv

  22. [30]

    the answer is

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 13358–13376, 2023. A Full Exp...

    2023

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.