Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Unveiling Confirmation Bias in Chain-of-Thought Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that chain-of-thought reasoning in large language models is systematically distorted by confirmation bias: a model's pre-existing answer belief changes both the reasoning it generates and how much it relies on that…

desk verdict A useful two-stage framework for belief effects on chain-of-thought, but the key stage-two metric is partly arithmetic and the causal confirmation-bias claim outruns the correlational evidence. read the letter →

arxiv 2506.12301 v1 pith:5ISWYZZJ submitted 2025-06-14 cs.LG cs.AI

Yue Wan , Xiaowei Jia , Xiang Lorraine Li This is my paper
classification cs.LGcs.AI
keywords chain-of-thoughtconfirmationbiaslargelanguagemodelsinternalbeliefsentropyreasoningfaithfulnessmultiple-choiceQAzero-shotprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chain-of-thought prompting is meant to make large language models reason step by step, but this paper argues that a model's pre-existing belief about the answer—approximated by the probability it assigns to each answer choice when asked directly, before any reasoning—systematically distorts the whole process. The authors factor CoT into a rationale-generation stage and a rationale-guided prediction stage, then correlate belief strength (measured as the normalized entropy of the zero-shot answer distribution) with rationale attributes and with how faithfully the final answer follows the rationale. Across six multiple-choice datasets and three open-weight 7B-8B models, strong beliefs are linked to shorter, more self-confirming rationales and to final predictions that lean on the prior instead of the rationale. The paper reads these patterns as confirmation bias in LLMs and uses them to explain why CoT helps far more on symbolic and mathematical tasks than on commonsense reasoning tasks.

What carries the argument

The argument is carried by a two-stage factorization of chain-of-thought, $P(A,R|Q)=P(A|Q,R)P(R|Q)$, into which an unobserved belief $B$ is inserted as a conditioning variable, so both stages become belief-dependent: $P(A,R|Q,B)=P(A|Q,R,B)P(R|Q,B)$. The measurable proxy for $B$ is the normalized entropy of zero-shot answer-choice probabilities, and its strength is additionally characterized by empirical difficulty, the log-probability gap between the best non-correct choice and the correct answer. A stratified correlation analysis—grouping questions into equal-width entropy or difficulty bins and then correlating within or between those groups—is the mechanism that connects $B$ to rationale attributes and stage-wise performance while controlling for confounding factors.

What would settle it

Take questions where a model's zero-shot top-1 answer is wrong with low entropy; the paper predicts these are rarely corrected by CoT. A decisive experiment randomly prepends a counter-priming sentence stating the initial answer is likely wrong before the model reasons: if correction rates do not rise, the apparent confirmation bias may be a difficulty artifact; if they do, the belief-strength mechanism is confirmed.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that confirmation bias is pervasive in LLM chain-of-thought reasoning and acts at both stages of the CoT process. At the generation stage ($Q \to R$), strong beliefs—low entropy in the zero-shot answer distribution—yield rationales that are shorter, more focused on justifying the initially preferred answer, more explicitly conclusive, and less likely to weigh alternative choices; they are also more likely to reach an intermediate conclusion identical to the model's original prediction. At the prediction stage ($QR \to A$), reliance on the rationale's content weakens as belief strength increases, so even a highly informative rationale often cannot override a confident prior. Cross-dataset and cross-model comparisons then explain CoT's uneven gains by how vulnerable each task and model is to this bias: implicit-knowledge tasks like commonsense are most vulnerable, while explicit-knowledge and formal tasks like AQuA are least.

Load-bearing premise

The analysis assumes that zero-shot answer probabilities faithfully measure the internal belief that later shapes reasoning, and that majority-vote extraction of the intermediate answer correctly identifies what the rationale supports; if either fails, the measured correlations could be artifacts of question difficulty or option-count effects.

Editorial extensions

If this is right

  • If the paper is right, end-to-end CoT accuracy conflates genuine reasoning with prior belief: some reasoning gains are just the model re-stating what it already believed.
  • Confidently wrong answers are the hardest case: sufficient and informative rationales will still fail to override a strong prior, so better prompts alone may not fix the error.
  • The known pattern that CoT helps mainly on math and symbolic tasks is explained as lower vulnerability to confirmation bias, not merely task formality.
  • Explicitly supplying the knowledge a question requires (the StrategyQA vs StrategyQA+F comparison) weakens beliefs and increases CoT gains, suggesting a concrete way to reduce the bias.
  • When one model generates a rationale and another uses it, the executor's strong contradictory belief can block the rationale from changing its answer; cross-model debiasing is only partly effective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct causal test is implied but not run: prompting the model to argue against its zero-shot answer before generating the CoT should reduce consistency with the prior and improve accuracy on confidently wrong questions, which would separate causation from correlation.
  • The entropy proxy could be replaced by a memorization-based measure of belief, as the authors suggest; such a measure would clarify whether confirmation bias comes from training-data familiarity rather than from answer confidence.
  • The framework suggests a cheap pre-hoc diagnostic: measure the entropy distribution of a model on a task before choosing whether to apply CoT, since tasks and models with too many strong incorrect priors are unlikely to benefit.
  • The analysis is limited to multiple-choice questions with white-box token probabilities, but the same two-stage logic could be applied to open-ended generation using an ensemble-generated answer pool, at the cost of added noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes that confirmation bias, modeled as the effect of an unobserved internal belief B on reasoning, explains how and when chain-of-thought prompting helps or hurts LLM performance. Belief strength is operationalized as the normalized entropy of zero-shot answer probabilities P(A_i|Q), and CoT is decomposed into a reasoning-generation stage (Q -> R) and a reasoning-guided prediction stage (QR -> A). The authors measure rationale attributes (length, relevance, explicitness, informativeness, sufficiency, etc.), correlate those attributes with the entropy proxy across Mistral-7B, Llama3-8B, and OLMo2-7B on six datasets, and report that strong beliefs are associated with shorter, more self-confirming rationales and with reduced reliance on the rationale at the answer-prediction stage. They also propose a task vulnerability ordering and a cross-model debiasing experiment. The paper concludes that confirmation bias is pervasive in LLM CoT reasoning and that stronger beliefs are harder to debias.

Significance. If the central findings survive the methodological concerns below, this would be a useful contribution: the two-stage decomposition of CoT into Q -> R and QR -> A is a clean analytic device, the stratified correlation analysis is more careful than naive pooling, and the cross-model debiasing experiment in Table 2 is a genuinely interesting intervention-style test. The release of code supports reproducibility. However, the headline claim is currently stronger than the evidence: the design is correlational, one of the two stage-wise metrics shares its baseline distribution with the belief proxy by construction, and no significance tests or confidence intervals are reported. The contribution is therefore moderate rather than definitive, and the framing as evidence of confirmation bias needs substantial support before publication.

major comments (4)
  1. [3.2, Table S3, Figures 5 and 6] The stage-two metric Informativeness is definitionally tied to the belief-strength proxy. Informativeness is defined as log P(A_inter|Q,R) - log P(A_inter|Q), while the Entropy used as the belief-strength proxy is computed from the same P(A_i|Q) distribution. For a low-entropy question in which A_inter is the model's top option, P(A_inter|Q) is large by construction, so Informativeness is mechanically depressed regardless of what the rationale R contains. The headline stage-two result in Figures 5 and 6 therefore rests at least partly on an arithmetic artifact rather than on how the model uses the rationale. The authors should re-estimate the stage-two correlations after conditioning on P(A_inter|Q), for example by stratifying or residualizing on P(A_inter|Q), or should replace the P(A_inter|Q) baseline with an estimate obtained independently of the entropy computation, such as the same model's probability under a different prompt or another model's probability. Without such a check, the claim that strong beliefs reduce reliance on the rationale is not established.
  2. [3.1 and Section 8] The central construct B, the internal belief, is proxied by zero-shot P(A_i|Q), and Section 8 explicitly concedes that this proxy is not validated. Under this proxy, entropy conflates belief strength with question difficulty, option-count effects, and surface-form biases; the normalization C = log(n) attempts to handle option count but no evidence is given that the correction makes entropy comparable across datasets or models. The entire correlation analysis inherits this assumption. I request a validation study: report calibration of P(A_i|Q), compare the entropy proxy with an alternative confidence measure (for example, verbalized confidence, sampling-based consistency, or probabilities under perturbed questions), and check that the stage-one correlations survive when question difficulty or option count is controlled. This is load-bearing because the confirmation-bias interpretation requires that low entropy measures a strong latent preference rather than an easy or lexically distinctive question.
  3. [5, Figures 2 through 6] No significance tests, confidence intervals, or effect sizes are reported anywhere in the correlation analysis. The text repeatedly uses phrases such as 'significantly influence' (Section 5.1) and 'strong correlations' (Section 5.2), but the heatmaps in Figures 5 and 6 do not display the underlying correlation coefficients, sample sizes, or uncertainty. The paper should report per-stratum correlation coefficients with bootstrap confidence intervals or p-values, state the number of questions in each entropy group, and correct for multiple comparisons across the eight rationale attributes. This is necessary to support the claim that belief strength has a statistically reliable association with CoT behavior rather than a set of suggestive but unquantified patterns.
  4. [5.3 and Appendix A.1] The vulnerability ordering CommonsenseQA > SocialIQA >> PIQA = StrategyQA > StrategyQA+F >> AQuA is asserted from task characteristics and then used to explain CoT improvements, but only Mistral-7B's improvement pattern matches it; the deviations of Llama3-8B and OLMo2-7B are reconciled post hoc through entropy and LSE differences. As presented, the ordering is unfalsifiable. The authors should either derive the ordering from an independent, pre-specified measure of confirmation-bias vulnerability or test it quantitatively, for example by computing a rank correlation between predicted vulnerability and CoT improvement across tasks and models, rather than invoking belief differences after the fact.
minor comments (5)
  1. [1, 6.2, 8] There are several typos and wording errors: 'undersanding' in Section 1, 'closet' for 'closest' in Section 6.2, and 'walkaround' for 'workaround' in Section 8.
  2. [Figures 2, 3, 5, 6, 7] The heatmaps and line plots would be much easier to evaluate if they included numeric correlation values or color scales with labeled ranges, and if the red dashed circles referenced in the text for Figures 2 and 3 were actually shown or explained in the captions.
  3. [Table 2] The columns labeled 'Strong', 'Neural', and 'Weak' appear to mean 'Neutral' for the middle column, and the table lacks sample sizes and standard errors for the reported averages.
  4. [3.3] The description of the grouping procedure is unclear: 'equal-width internal' should be 'equal-width intervals', and the criterion that the average number of data points per group is less than 1% should be stated explicitly with the resulting number of groups for each dataset.
  5. [A.2.2] The extraction of A_inter relies on majority voting among four LLMs, but the paper does not report the agreement rate among the voters or any measure of extraction reliability; such information would help the reader assess the noise in the stage-one and stage-two metrics.

Circularity Check

1 steps flagged · score 6.0 of 10

Stage-two evidence is partly definitional: Informativeness subtracts log P(Ainter|Q), the same distribution used to define belief strength, so low-entropy/high-confidence questions have mechanically reduced Informativeness.

  1. self definitional [Section 3.1 (belief-strength entropy) and Section 3.2 / Table S3 (Informativeness); used in Figure 5 and Section 5.2]
    "We leverage the entropy of P(Ai|Q) as the measure, where a lower entropy corresponds to higher confidence: −1/C Σ_i P(Ai|Q) log P(Ai|Q). [Table S3:] Informativeness ... Formulation: log P(Ainter|Q,R)/P(Ainter|Q)."

    The belief-strength proxy is the normalized entropy of the distribution P(A|Q), while the stage-two metric Informativeness is log P(Ainter|Q,R) − log P(Ainter|Q). The subtracted term is exactly the log probability of the answer under the same distribution that defines the entropy. For questions where Ainter is the model's favored option, low entropy forces P(Ainter|Q) to be large, so Informativeness is mechanically depressed regardless of what the rationale R contains. Thus the reported entropy–Informativeness correlation (Figure 5) and the inference that strong beliefs reduce reliance on the rationale (Section 5.2) are partly a restatement of the belief proxy rather than an independent measure of rationale usage.

full rationale

The paper is not globally circular: the stage-one results (entropy versus Length, Relevance, RelevanceNeg, Explicitness, Sufficiency, ConsistencyInter) do not share the entropy term with the belief proxy, and the cross-model debiasing experiment (Table 2) is an independent behavioral test that does not reduce to the definition of P(A|Q). However, the QR→A stage leans on Informativeness, which shares the direct-answer probability P(A|Q) with the belief-strength definition. The positive entropy–Informativeness correlation used as stage-two evidence is partly arithmetic: with Ainter equal to the model's top option, low entropy means a large denominator in the PMI, lowering Informativeness even if the rationale is unused. The intra-group entropy-stratified analysis (Figure 6b) is less directly contaminated, but it still uses Informativeness as the outcome, so the shared-term confound does not disappear entirely. Section 8 also concedes that the entropy proxy is not validated and that a better metric is future work; that is a limitation rather than a circular step, but it amplifies the concern that the stage-two correlation may rest on the proxy definition. No load-bearing self-citation chain or imported uniqueness theorem is present. Because one evidence stream in the central two-stage claim reduces partly by construction while meaningful independent evidence remains, score 6 is appropriate.

Assumptions & free parameters 0 free parameters · 6 assumptions · 1 invented entities

The central analysis rests on the proxy assumption for internal beliefs, the reliability of LLM-extracted intermediate answers, and a post hoc vulnerability ranking. The probability factorization itself is standard. No free parameters are fitted to data, but the analytic choices (entropy normalization, group counts, selection criteria in cross-model experiments) introduce judgment calls that are not validated externally.

assumptions (6)
  • ad hoc to paper Zero-shot direct QA probability P(A_i|Q) approximates the unobserved internal belief B.
    Section 3.1 states B is internal and unobserved, then uses P(A_i|Q) as the proxy; all confirmation-bias correlations inherit this assumption.
  • ad hoc to paper Entropy of P(A|Q), normalized by log of option count, measures belief strength comparably across datasets and models.
    Section 3.1 defines the normalized entropy; cross-dataset comparisons in Section 5.3 rely on this comparability.
  • domain assumption A_inter, extracted by majority vote of four advanced LLMs, is the correct answer supported by the rationale.
    Section 3.2 and A.2.2 use GPT-4o-mini, Llama-3.3-70B, Claude-3.5-Sonnet, and DeepSeek-V3; errors in this extraction propagate to ConsistencyInter, PerformanceInter, and Informativeness.
  • domain assumption Confirmation bias as defined for human cognition transfers to LLM token-level behavior.
    Section 2 borrows the psychological construct (Nickerson 1998) and applies it to LLMs without demonstrating that the same cognitive mechanism underlies the observed statistics.
  • ad hoc to paper The dataset vulnerability ordering CommonsenseQA > SocialIQA > PIQA ≈ StrategyQA > StrategyQA+F > AQuA is valid.
    Section 5.3 and A.1 assert this ranking from reasoning-type intuitions; it is not measured, yet it is used to explain the CoT improvement pattern.
  • standard math The CoT factorization P(A,R|Q)=P(A|Q,R)P(R|Q) is valid.
    This is the chain rule of probability, used in Section 2 to decompose CoT into two stages; it is uncontroversial.
invented entities (1)
  • Internal belief state B
    purpose: Unobserved latent variable invoked to explain why strong initial preferences skew reasoning generation and answer prediction.
    B is never directly measured; only the proxy P(A|Q) is observed. The causal role of B is inferred from correlations, so the construct has no falsifiable handle outside the paper's proxy and results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unveiling Confirmation Bias in Chain-of-Thought Reasoning." pith.science (2026). https://pith.science/paper/5ISWYZZJ

@misc{pith2026250612301,
  author       = {Pith},
  title        = {Pith review of: Unveiling Confirmation Bias in Chain-of-Thought Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ISWYZZJ}},
  note         = {Machine review of arXiv:2506.12301}
}
abstract

Chain-of-thought (CoT) prompting has been widely adopted to enhance the reasoning capabilities of large language models (LLMs). However, the effectiveness of CoT reasoning is inconsistent across tasks with different reasoning types. This work presents a novel perspective to understand CoT behavior through the lens of \textit{confirmation bias} in cognitive psychology. Specifically, we examine how model internal beliefs, approximated by direct question-answering probabilities, affect both reasoning generation ($Q \to R$) and reasoning-guided answer prediction ($QR \to A$) in CoT. By decomposing CoT into a two-stage process, we conduct a thorough correlation analysis in model beliefs, rationale attributes, and stage-wise performance. Our results provide strong evidence of confirmation bias in LLMs, such that model beliefs not only skew the reasoning process but also influence how rationales are utilized for answer prediction. Furthermore, the interplay between task vulnerability to confirmation bias and the strength of beliefs also provides explanations for CoT effectiveness across reasoning tasks and models. Overall, this study provides a valuable insight for the needs of better prompting strategies that mitigate confirmation bias to enhance reasoning performance. Code is available at \textit{https://github.com/yuewan2/biasedcot}.

Figures

Figures reproduced from arXiv: 2506.12301 by the authors.

Figure 1
Figure 1. A typical Venn diagram of confirmation bias in cognitive psychology, using the example of a com￾monsensical question. The agent reinforces its internal beliefs and skews its reasoning process towards "making music", while overlooking other relevant facts of play￾ing guitar. Notes that the internal belief is unobserved but plays a huge role in decision making. examined both the nature of reasoning problems (Sprague e… view at source ↗
Figure 2
Figure 2. Shift in PerformanceE2E from direct to CoT prediction in relation of entropy and empirical difficulty. tion, we measure whether the prediction aligns with AInter (i.e., the intermediate answer extracted from the rationale), regardless of whether it matches A∗ . This serves as the stage-two accuracy (i.e., PerformanceInter) of the model’s ability to faithfully follow the rationale. 5 Results [PITH_FULL_IMAGE:figures… view at source ↗
Figure 4
Figure 4. Correlation trends of base entropy (proxy for model’s internal beliefs) with CoT Length, Consistency [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Correlation of Entropy, proxy for strength [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Correlation analysis of the role of B in the second reasoning stage of P(A|Q, R, B), using behav￾iors of Mistral-7B on CommonsenseQA [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Comprehensive comparison of the question-answering entropy distribution from [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 11 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Nishant Balepur, Shramay Palta, and Rachel Rudinger. 2024. https://doi.org/10.18653/v1/2024.findings-acl.604 It`s not easy being wrong: Large language models struggle with process of elimination reasoning . In Findings of the Association for Computational Linguistics: ACL 2024, pages 10143--10166, Bangkok, Thailand. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.findings-acl.604 2024

  4. [4]

    Guangsheng Bao, Hongbo Zhang, Cunxiang Wang, Linyi Yang, and Yue Zhang. 2025. https://aclanthology.org/2025.coling-main.524/ How likely do LLM s with C o T mimic human reasoning? In Proceedings of the 31st International Conference on Computational Linguistics, pages 7831--7850, Abu Dhabi, UAE. Association for Computational Linguistics

    2025

  5. [5]

    Oliver Bentham, Nathan Stringham, and Ana Marasović. 2024. https://arxiv.org/abs/2402.14897 Chain-of-thought unfaithfulness as disguised accuracy . Preprint, arXiv:2402.14897

    arXiv 2024

  6. [6]

    Vincent Berthet, Predrag Teovanovi \' c , and Vincent de Gardelle. 2024. https://doi.org/10.1038/s41598-024-78053-7 A common factor underlying individual differences in confirmation bias . Scientific Reports, 14(1):27795

    work page doi:10.1038/s41598-024-78053-7 2024

  7. [7]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

    2020

  8. [8]

    Antoine Bosselut, Ronan Le Bras, and Yejin Choi. 2020. https://api.semanticscholar.org/CorpusID:226222402 Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering . In AAAI Conference on Artificial Intelligence

    work page 2020

Show all 41 references
  1. [9]

    Qi Cheng, Michael Boratko, Pranay Kumar Yelugam, Tim O ' Gorman, Nalini Singh, Andrew McCallum, and Xiang Li. 2024. https://doi.org/10.18653/v1/2024.acl-long.29 Every answer matters: Evaluating commonsense with probabilistic measures . In Proceedings of the 62nd Annual Meeting...

    2024 doi

  2. [10]

    DeepSeek-AI. 2024. https://arxiv.org/abs/2412.19437 Deepseek-v3 technical report . Preprint, arXiv:2412.19437

    2024 arXiv

  3. [11]

    Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. 2023. https://openreview.net/forum?id=qHrADgAdYu Towards revealing the mystery behind chain of thought: A theoretical perspective . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  4. [12]

    Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. 2023. https://arxiv.org/abs/2305.17306 Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance . Preprint, arXiv:2305.17306

    2023 arXiv

  5. [13]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies . Transactions of the Association for Computational Linguistics (TACL)

    2021

  6. [14]

    Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. https://openreview.net/forum?id=xYlJRpzZtsY ROSCOE : A suite of metrics for scoring step-by-step reasoning . In The Eleventh International Confere...

    2023

  7. [15]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  8. [16]

    Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.564 Surface form competition: Why the highest probability answer isn`t always right . In Proceedings of the 2021 Conference on Empirical Methods in Natu...

    2021 doi

  9. [17]

    Pattichis, and Douglas B

    Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis, and Douglas B. Kell. 2017. https://arxiv.org/abs/1712.09923 What do we need to build explainable ai systems for the medical domain? Preprint, arXiv:1712.09923

    2017 arXiv

  10. [18]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. 2023. https://arxiv.org/abs/2310.06825 Mistral 7b . Preprint, arXiv:2310.06825

    2023 arXiv

  11. [19]

    Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, and Xiang Ren. 2023. https://doi.org/10.18653/v1/2023.acl-long.392 Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationale...

    2023 doi

  12. [20]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA. Curran Associates Inc

    2022

  13. [21]

    Bowman, and Ethan Perez

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish...

    2023 arXiv

  14. [22]

    Jiachun Li, Pengfei Cao, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. 2025. https://arxiv.org/abs/2405.18915 Towards better chain-of-thought: A reflection on effectiveness and faithfulness . Preprint, arXiv:2405.18915

    2025 arXiv

  15. [23]

    Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.693 Think twice before trusting: Self-detection for large language models through comprehensive answer reflection . In Findings of the Associatio...

    2024 doi

  16. [24]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. https://doi.org/10.18653/v1/P17-1015 Program induction by rationale generation: Learning to solve and explain algebraic word problems . In Proceedings of the 55th Annual Meeting of the Association for Computational ...

    2017 doi

  17. [25]

    Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L

    Ryan Liu, Jiayi Geng, Addison J. Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L. Griffiths. 2024. https://arxiv.org/abs/2410.21333 Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse . Preprint, arXiv:2410.21333

    2024 arXiv

  18. [26]

    Aman Madaan, Katherine Hermann, and Amir Yazdanbakhsh. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.101 What makes chain-of-thought prompting effective? a counterfactual study . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1448--1535...

    2023 doi

  19. [27]

    Raymond S Nickerson. 1998. Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2):175--220

    1998

  20. [28]

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022. https://openreview.net/forum?id=HBlx2idbkbq Show your work: Scratchpads for int...

    2022

  21. [29]

    Team OLMo, Pete Walsh, Luca Soldaini, et al. 2025. https://arxiv.org/abs/2501.00656 2 olmo 2 furious . Preprint, arXiv:2501.00656

    2025 arXiv

  22. [30]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  23. [31]

    OpenRouter

    Inc. OpenRouter. 2025. https://openrouter.ai Openrouter.ai

    2025

  24. [32]

    Letitia Parcalabescu and Anette Frank. 2024. https://doi.org/10.18653/v1/2024.acl-long.329 On measuring faithfulness or self-consistency of natural language explanations . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2024 doi

  25. [33]

    Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.622 R e CE val: Evaluating reasoning chains via correctness and informativeness . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

    2023 doi

  26. [34]

    Zhenting Qi, Mingyuan MA, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2025. https://openreview.net/forum?id=6aHUmotXaw Mutual reasoning makes smaller LLM s stronger problem-solver . In The Thirteenth International Conference on Learning Representations

    2025

  27. [35]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. https://doi.org/10.18653/v1/D19-1454 Social IQ a: Commonsense reasoning about social interactions . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9...

    2019 doi

  28. [36]

    Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2025. https://openreview.net/forum?id=w6nlcS8Kkn To cot or not to cot? chain-of-thought helps mainly on math and symbolic re...

    2025

  29. [37]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  30. [38]

    Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023. https://doi.org/10.18653/v1/2023.acl-long.153 Towards understanding chain-of-thought prompting: An empirical study of what matters . In Proceedings of the 61st Annual Meeting of the ...

    2023 doi

  31. [39]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processi...

    2022

  32. [40]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 arXiv

  33. [41]

    Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. https://openreview.net/forum?id=WZH7099tgfM Least-to-most prompting enables complex reasoning in large language mode...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.