Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting

Dan Shapiro; Ethan Mollick; Lennart Meincke; Lilach Mollick

REVIEW 4 major objections 5 minor 1 cited by

Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chain-of-thought prompting, the 'think step by step' technique, is not a universal accuracy booster: on a PhD-level benchmark it yields small and inconsistent gains for ordinary models, often cuts perfect-accuracy scores, and gives…

desk verdict Useful measurements of CoT on 2025 models, but the headline gain is mostly an artifact of a no-thinking baseline; the Default condition tells the more honest story. read the letter →

arxiv 2506.07142 v1 pith:3YYKFB5O submitted 2025-06-08 cs.CL cs.AI

Lennart Meincke , Ethan Mollick , Lilach Mollick , Dan Shapiro This is my paper

classification cs.CLcs.AI

keywords chain-of-thoughtpromptingLLMevaluationGPQADiamondpromptengineeringreasoningmodelsresponselatencyanswervariabilitycost-aware

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This report tries to establish that Chain-of-Thought prompting (asking a language model to 'think step by step' before answering) is not a universal way to make AI more accurate. Testing five ordinary models and three models with built-in reasoning on the 198-question GPQA Diamond benchmark, with 25 trials per question and prompt, the paper finds that a simple CoT prompt modestly raises average accuracy for ordinary models but also increases answer variability, sometimes turning previously correct answers into wrong ones. For dedicated reasoning models, the same prompt produces at best marginal accuracy gains while substantially increasing response time and token use. If correct, the practical conclusion is that CoT should be chosen conditionally, weighing accuracy gains against latency, cost, and reliability needs.

What carries the argument

The carrying instrument is a three-way prompt contrast repeated on every question: Direct ('Answer directly without any explanation or thinking'), Step-by-step ('Think step by step'), and Default (no suffix, no formatting constraint). Each of the 198 GPQA Diamond questions is run 25 times under each condition, giving 4,950 runs per prompt per model, and scored four ways: average accuracy across all trials, and per-question thresholds requiring 25/25, 23/25, or 13/25 correct. This setup lets the authors separate average gains from reliability losses, since a model can improve on average while losing perfect-accuracy questions. The comparison against Default is what reveals that many models already perform CoT internally, which is why explicit prompting adds little.

What would settle it

Run the same protocol with an unconstrained baseline that does not forbid reasoning, such as 'Answer the question' with no formatting constraint, and compare it against 'Think step by step'; if the average gain over Direct shrinks to near zero while the latency penalty remains, the reported CoT benefit is largely an artifact of the Direct prompt suppressing default reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that explicit CoT prompting has decreasing and conditional value. For non-reasoning models, a 'Think step by step' prompt generally improved average accuracy over the 'answer directly' condition, with significant gains for Gemini 2.0 Flash (0.135) and Claude Sonnet 3.5 (0.117), but the strict 25/25 perfect-accuracy metric often declined, sharply for Gemini 2.0 Flash (-0.131) and Gemini Pro 1.5 (-0.172). When compared to an unprompted default in which the model chooses how to answer, CoT gains were much smaller and usually insignificant, because many models already reason step by step unprompted. For reasoning models (o3-mini, o4-mini, Gemini 2.5 Flash), explicit CoT changed average accuracy by no more than about plus or minus 0.03, with two models gaining slightly and one losing slightly, while response time rose 20 to 80 percent, or 10 to 20 seconds.

Load-bearing premise

The load-bearing assumption is that telling a model to 'answer directly without any explanation or thinking' is a neutral baseline; if that instruction suppresses reasoning the model would otherwise do, the CoT gains may measure the cost of the baseline rather than a benefit of CoT.

Editorial extensions

If this is right

For non-reasoning models, 'think step by step' can lift average accuracy but may lower the chance of a perfect 25/25 score, so users with zero-tolerance tasks should not assume CoT helps.
For reasoning models, explicit CoT adds little or no accuracy while increasing response time by 20 to 80 percent, so the added latency is often not justified.
Many current non-reasoning models reason step by step even when not asked, meaning an explicit CoT prompt frequently duplicates behavior the model would exhibit anyway.
Prompting a model to 'answer directly without explanation' can suppress useful internal reasoning and may hurt performance on hard questions.
Choosing whether to use CoT should depend on the model, the difficulty of the task, the accuracy threshold required, tolerance for variability, and acceptable latency and cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The Direct-versus-CoT comparison likely overstates CoT's benefit: the Direct prompt forbids the model's default reasoning, so a neutral prompt that neither asks nor forbids thinking would be a fairer baseline and would probably shrink the measured gains.
For high-volume applications, the 35 to 600 percent latency increase for non-reasoning models and 20 to 80 percent for reasoning models suggests a cost-aware strategy: reserve CoT for hard questions or use short reasoning budgets rather than applying it uniformly.
If reliability is the goal, repeated sampling with majority voting may address CoT's increased variability more directly than a single CoT answer, although this report does not test that combination.
Since generic CoT prompt variants produced nearly identical results, further gains are more likely to come from task-specific reasoning scaffolds or model-specific tuning than from rewording the step-by-step instruction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a solid measurement report that gives the field a useful data point—explicit CoT prompting adds little for current reasoning models and only small, model-dependent gains for non-reasoning models—but the abstract overstates the case by leaning on a baseline that tells the model not to think. The paper's own Default condition mostly fails to show significant gains.

What is actually new: repeated trials (25 per question) on 2025 frontier models, with bootstrap permutation tests and threshold-based reliability metrics. The inclusion of a Default condition (no prompt, no formatting constraint) is a genuine strength, because it partially controls for the fact that many non-reasoning models already produce CoT-like traces unprompted. The cost and latency findings are directly relevant to practitioners. No fitted parameters drive the conclusions; every effect is a direct measurement against a fixed benchmark. The citation pattern is fine—the self-citation to Report 1 is not load-bearing for the CoT results.

The soft spots are real but fixable. First, the main CoT-versus-baseline comparison uses a Direct prompt that says "Answer directly without any explanation or thinking." That suppresses the model's default reasoning, so part of the measured CoT gain may come from handicapping the control, not from CoT itself. The paper's own Table S4 shows that against a natural Default baseline, only Gemini Flash 2.0 and GPT-4o have significant average gains; Sonnet 3.5 and GPT-4o-mini are null. The abstract's "generally improves average performance" is therefore too strong. Second, the Discussion's claim that "answer only" prompting is likely to harm non-reasoning models is not isolated, because Direct conflates answer-only with "without thinking." Third, Tables S3–S5 say questions were considered perfect at ≥100 successes, but the design uses 25 trials; that is an internal inconsistency that will confuse readers. Fourth, raw data and code are not released, which limits reproducibility checks.

None of these flaws sink the core finding. The measurements are what they are, and the qualitative conclusion—CoT is a conditional, cost-sensitive technique, not a universal accuracy booster—holds up. The audience is applied users and researchers who want current-model evidence on prompt effects, not readers looking for a new mechanism.

Recommendation: yes, send it to peer review, but as a short applied benchmarking report, not as a conceptual advance. The authors should be asked to re-center the abstract on the Default comparison, correct the table notes, and release data or at least per-question scores. With those changes, the paper would be a clean, citeable data point.

Referee Report

4 major / 5 minor

Summary. This manuscript reports a large-scale benchmarking study of simple Chain-of-Thought (CoT) prompting on the 198-question GPQA Diamond dataset, covering five non-reasoning models (Sonnet 3.5, Gemini 2.0 Flash, GPT-4o-mini, GPT-4o, Gemini Pro 1.5) and three reasoning models (o3-mini, o4-mini, Gemini Flash 2.5), with 25 trials per question and condition (4,950 runs per condition per model). The authors compare three prompt variants — “Answer directly without any explanation or thinking” (Direct), “Think step by step” (CoT), and no suffix with no formatting constraint (Default) — across four metrics (average proportion correct plus per-question 100%, 90%, and 51% thresholds). Paired bootstrap permutation tests show that CoT raises average accuracy relative to Direct for most non-reasoning models while often decreasing the 100%-correct rate, increases latency by 35–600% for these models, and yields only marginal accuracy changes (RD ≈ 0.03 in either direction) for reasoning models at substantially higher latency. The paper concludes that CoT is a conditional, cost-sensitive technique whose value depends on model type, baseline behavior, and the accuracy metric, and that many recent models already perform CoT-like reasoning unprompted.

Significance. If the empirical patterns hold, this report makes a useful practical contribution: falsifiable, parameter-free measurements of a widely adopted prompting technique on a standardized benchmark, with per-model effect sizes, confidence intervals, and cost data practitioners can act on. The manuscript’s strengths include direct measurement against an external benchmark with no fitted parameters, paired bootstrap permutation tests with 5,000 replicates, transparent reporting of both beneficial and harmful effects (including declines on the 100%-correct metric), explicit acknowledgement of limitations (single benchmark, limited model set, simple prompt variants), and a robustness check in Figure S3 showing that more sophisticated CoT prompt variants yield negligible differences. The finding that recent models self-generate CoT-like reasoning by default, and the quantification of latency and token costs of explicit CoT prompting for reasoning models, are the most novel and practically relevant elements.

major comments (4)

[How we Prompt the AI; Figure 2; Tables S3–S4] The main CoT-versus-baseline comparison is built on a prompt that actively suppresses default model behavior, conflating two distinct questions. The Direct condition instructs the model to “Answer directly without any explanation or thinking. Just provide the answer,” and the Step-by-Step condition only appends “Think step by step,” so the statistically significant average gains in Table S3 (Sonnet 3.5 RD = 0.117, Gemini Flash 2.0 RD = 0.135, both p < .001) measure CoT against a baseline told not to think rather than against natural usage. The manuscript’s own Default condition, which removes both the formatting constraint and the no-thinking instruction, shows significant average CoT gains for only two of the four models reported in Table S4 (Gemini Flash 2.0 RD = 0.062, p < .001; GPT-4o RD = 0.069, p = 0.003), with null results for Sonnet 3.5 (RD = -0.019, p = 0.189) and GPT-4o-mini (RD = 0.004, p = 0.728), and Gemini Pro 1.5 is missing from both the text and Table S4. Consequently, the abstract’s claim that CoT “generally improves average performance” for non-reasoning models, and the Discussion’s claim that “answer only” prompting is “likely to harm” such models (which cannot be separated from the “without thinking” instruction), are stronger than the evidence supports. The Discussion’s assertion that models were “fairly consistent in their responses to CoT” is also difficult to square with the heterogeneous 100%-metric RDs in Table S3, which range from +0.101 (Sonnet 3.5) to -0.172 (Gemini Pro 1.5). Please reframe the principal claims as baseline-dependent, report the Default condition for all five non-reasoning models, and lead the practical recommendations with the Default comparisons if the intended reference point is ordinary chatbot use.
[Abstract; Discussion and Conclusion] The qualifier “particularly if the model does not inherently engage in step-by-step processing by default” is never operationalized. The only evidence for spontaneous CoT reasoning is a single worked example in Table S1 (GPT-4o) plus a general statement in “How we Prompt the AI”; no per-model measure of default reasoning behavior (such as the fraction of Default-condition responses containing reasoning tokens before the answer, or blind coding of a sample of Default outputs) is reported. Without such a measure, the subset claim that CoT helps most when models do not already reason step by step is post-hoc and unfalsifiable within this report. Please add a quantitative per-model measure of spontaneous reasoning and test whether CoT gains correlate with it, or explicitly label the claim as a hypothesis for future work.
[Tables S2–S5] The trial-count and threshold documentation is internally inconsistent and needs reconciliation. Table S2 compares “100% Correct” at 25 trials (all 25/25) with “100% Correct” at 100 trials (all 100/100), which are different estimands by construction; the observed difference (0.185 [0.162, 0.207] versus 0.147 [0.136, 0.162]) therefore cannot support the note’s conclusion that 25 trials provide “estimation precision and statistical power comparable to 100 trials.” Additionally, the notes to Tables S3, S4, and S5 state that “Questions were considered 'perfect' at ≥100 successes,” which is impossible under the stated 25-trial design and contradicts the definitions in “How we Benchmark the AI” (thresholds of 25/25, 23/25, and 13/25). Because the reported 100%-metric risk differences (e.g., Gemini Flash 2.0 RD = -0.131 and Gemini Pro 1.5 RD = -0.172 in Table S3) are load-bearing for the claim that CoT trades average gains for increased answer variability, please state explicitly which thresholds produced each reported value and correct the table notes accordingly.
[Tables S3–S5; Results] The manuscript performs roughly 48 hypothesis tests across Tables S3–S5 (five non-reasoning models in Table S3, four in Table S4, three reasoning models in Table S5, each across four metrics) and reports raw p-values with no multiple-comparison correction or false-discovery-rate control. Several claims in the Results section rest on p-values that would not survive a per-table Bonferroni correction: GPT-4o at 51% (p = 0.042), Gemini Pro 1.5 Average (p = 0.014), o3-mini Average (p = 0.024, Table S5), and Flash 2.5 Average (p = 0.005). The headline effects (Sonnet 3.5 and Gemini Flash 2.0 Average, both p < .001) are robust, so the direction of the main findings is probably unaffected, but the text should apply a correction, report q-values, or explicitly state which conclusions depend on uncorrected marginal p-values.

minor comments (5)

[Figures S1–S2] The figure legends report N = 197 per condition (and 196 for Flash 2.5), while the main text states that the GPQA Diamond set comprises 198 questions and that each condition involves 4,950 runs per model; please reconcile these numbers or state that one question was excluded from the timing analyses.
[Results] Please define the abbreviation “RD” (risk difference) at its first use in the Results section, and fix the small textual errors in the first results paragraph (“GPT-4o’s saw” should be “GPT-4o saw,” and the passage “Google-proof)” contains a stray closing parenthesis).
[Table S1] The example responses in Table S1 are truncated mid-sentence (for instance, the Step-by-Step column breaks off during the option analysis), and the Default column’s final answer is given only as “C.” despite the preceding hedged discussion; please provide complete or clearly marked excerpts so readers can verify the qualitative claim about default CoT behavior.
[References] The “Think step by step” instruction is the zero-shot CoT phrasing popularized by Kojima et al. (2022), whereas the cited Wei et al. (2022) work is few-shot CoT; please cite Kojima et al. as well, and also cite the source for the “Answer directly without any explanation or thinking” instruction if it follows an established protocol.
[How we Benchmark the AI] The sentence stating that PASS@N and CONSENSUS are “not useful” for benchmarking real-world applications is a strong normative claim presented without references or supporting analysis; please add a brief justification or soften the wording.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all quantitative claims are direct measurements on the external GPQA Diamond benchmark, with no fitted parameters whose values encode the conclusions.

full rationale

This paper reports direct empirical comparisons of Chain-of-Thought prompting against two baseline conditions on the GPQA Diamond benchmark. No parameter is fitted to a subset of data and then renamed as a prediction; every reported risk difference is a measured outcome computed from the paper's own trials (25 per question per condition) against an external benchmark. The only self-citation is to the authors' Report 1 for 'rating measures,' but these metrics are simply definitions (100% Correct, 90% Correct, 51% Correct, Average Rating) and are not load-bearing for the CoT comparisons; the CoT effects are computed independently from the trial data. The concern that the 'Direct' prompt actively suppresses default reasoning is an experimental-design critique about baseline choice, not a circularity: the CoT-versus-Direct comparison is a real measurement, and the paper also reports the CoT-versus-Default comparison with mixed results, so the central claim is not forced by construction. The qualifier that CoT helps 'particularly if the model does not inherently engage in step-by-step processing by default' is not operationalized with a per-model measure, but that is a lack of direct support, not circularity. The claim that answer-only prompting is 'likely to harm' non-reasoning models is an interpretation of the Direct-condition results rather than a mathematically derived conclusion. No equation in the paper reduces to its own input, and no load-bearing step invokes an author-defined uniqueness theorem. The findings are self-contained measurements that could in principle falsify the paper's claims, so the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No coefficients are fitted. The paper's quantitative claims are direct measurements over defined prompt conditions; the main design choices are the success thresholds, the trial count, and the choice of a single benchmark.

free parameters (2)

Per-question success thresholds = 25/25, 23/25, 13/25 correct
Researcher-chosen criteria for counting a question as reliable. These thresholds determine the qualitative split between average accuracy gains and strict-metric declines.
Trial count per question = 25
Design choice validated only for GPT-4o mini in the Direct condition via Table S2; the precision of all reported effects depends on it.

assumptions (3)

domain assumption GPQA Diamond is a valid, Google-proof measure of hard graduate-level reasoning.
The report uses GPQA Diamond as the sole benchmark and takes its external validity from Rein et al. (2024).
domain assumption The 25-trial power analysis on GPT-4o mini Direct generalizes to every model, prompt condition, and threshold metric.
The paper states the correlation between 25 and 100 trials is high but may be task dependent; this generalization is load-bearing for interpreting p-values on small risk differences.
ad hoc to paper The Direct prompt suppresses CoT while the Default prompt allows it, and the qualitative response inspection reliably classifies model behavior.
The interpretation of CoT gains relies on these prompt-behavior assumptions, supported only by example conversations in Table S1.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting." pith.science (2026). https://pith.science/paper/3YYKFB5O

@misc{pith2026250607142,
  author       = {Pith},
  title        = {Pith review of: Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3YYKFB5O}},
  note         = {Machine review of arXiv:2506.07142}
}

read the original abstract

This is the second in a series of short reports that seek to help business, education, and policy leaders understand the technical details of working with AI through rigorous testing. In this report, we investigate Chain-of-Thought (CoT) prompting, a technique that encourages a large language model (LLM) to "think step by step" (Wei et al., 2022). CoT is a widely adopted method for improving reasoning tasks, however, our findings reveal a more nuanced picture of its effectiveness. We demonstrate two things: - The effectiveness of Chain-of-Thought prompting can vary greatly depending on the type of task and model. For non-reasoning models, CoT generally improves average performance by a small amount, particularly if the model does not inherently engage in step-by-step processing by default. However, CoT can introduce more variability in answers, sometimes triggering occasional errors in questions the model would otherwise get right. We also found that many recent models perform some form of CoT reasoning even if not asked; for these models, a request to perform CoT had little impact. Performing CoT generally requires far more tokens (increasing cost and time) than direct answers. - For models designed with explicit reasoning capabilities, CoT prompting often results in only marginal, if any, gains in answer accuracy. However, it significantly increases the time and tokens needed to generate a response.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Tacit Coordination of Large Language Models
cs.GT 2026-01 conditional novelty 6.0 of 10

Across 20+ open-source LLMs, tacit coordination in focal-point games is often at or above human levels, with systematic failures on cultural and numerical salience that culture prompts partially fix.

Reference graph

Works this paper leans on

5 extracted references · 4 canonical work pages · cited by 1 Pith paper

[1]

think step by step

Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting Lennart Meincke 1,2 , Ethan Mollick 1 , Lilach Mollick 1 , Dan Shapiro 1,3 1 Generative AI Labs, The Wharton School of Business, University of Pennsylvania 2 WHU–Otto Beisheim School of Management 3 Glowforge Summary This is the second in a series of short reports that seek ...

work page 2022
[3]

easy" questions that the model would otherwise get right, harming performance on the

GPQA Diamond performance across reasoning models comparing an immediate answer with Chain-of-Thought. Note. The error bars show 95% confidence intervals for individual proportions. For detailed statistical comparisons between conditions, see Table S5. Discussion and Conclusion Our findings indicate that Chain-of-Thought prompting is not a universally opti...

work page 2025
[4]

Miller E (2024) Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

https://papers.ssrn.com/abstract=5165270. Miller E (2024) Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. (November

work page 2024
[5]

Think step by step

http://arxiv.org/abs/2411.00640. Rein D, Hou BL, Stickland AC, Petty J, Pang RY, Dirani J, Michael J, Bowman SR (2024) GPQA: A Graduate-Level Google-Proof Q&A Benchmark. First Conference on Language Modeling . Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi EH, Le QV, Zhou D (2022) Chain-of-thought prompting elicits reasoning in large language ...

arXiv 2024
[2025]

This is our strictest condition for tasks where there is no room for errors

and report the following metrics: ● 100% Correct: In this condition, we require the AI to get the answer correct on all trials (25/25) of a question to count that question as successful. This is our strictest condition for tasks where there is no room for errors. ● 90% Correct: We relax the 100% condition and allow a few errors per question, but 23/25 ans...

work page 2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

think step by step

Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting Lennart Meincke 1,2 , Ethan Mollick 1 , Lilach Mollick 1 , Dan Shapiro 1,3 1 Generative AI Labs, The Wharton School of Business, University of Pennsylvania 2 WHU–Otto Beisheim School of Management 3 Glowforge Summary This is the second in a series of short reports that seek ...

work page 2022

[2] [3]

easy" questions that the model would otherwise get right, harming performance on the

GPQA Diamond performance across reasoning models comparing an immediate answer with Chain-of-Thought. Note. The error bars show 95% confidence intervals for individual proportions. For detailed statistical comparisons between conditions, see Table S5. Discussion and Conclusion Our findings indicate that Chain-of-Thought prompting is not a universally opti...

work page 2025

[3] [4]

Miller E (2024) Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

https://papers.ssrn.com/abstract=5165270. Miller E (2024) Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. (November

work page 2024

[4] [5]

Think step by step

http://arxiv.org/abs/2411.00640. Rein D, Hou BL, Stickland AC, Petty J, Pang RY, Dirani J, Michael J, Bowman SR (2024) GPQA: A Graduate-Level Google-Proof Q&A Benchmark. First Conference on Language Modeling . Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi EH, Le QV, Zhou D (2022) Chain-of-thought prompting elicits reasoning in large language ...

arXiv 2024

[5] [2025]

This is our strictest condition for tasks where there is no room for errors

and report the following metrics: ● 100% Correct: In this condition, we require the AI to get the answer correct on all trials (25/25) of a question to count that question as successful. This is our strictest condition for tasks where there is no room for errors. ● 90% Correct: We relax the 100% condition and allow a few errors per question, but 23/25 ans...

work page 2024