Asking LLMs to Verify First is Almost Free Lunch

Quanming Yao; Shiguang Wu

REVIEW 4 major objections 4 minor 30 references

A simple 'verify first, then solve' prompt reliably outperforms standard chain-of-thought.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:58 UTC pith:LHQHAQOS

load-bearing objection The verification-first trick is genuinely novel and plausibly useful, but the paper's evidence is much thinner than its abstract claims, and the missing error bars and missing flagship result are real problems. the 4 major comments →

arxiv 2511.21734 v2 pith:LHQHAQOS submitted 2025-11-21 cs.CL cs.AI

Asking LLMs to Verify First is Almost Free Lunch

Shiguang Wu , Quanming Yao This is my paper

classification cs.CL cs.AI

keywords verification-first promptingchain-of-thoughttest-time scalingLLM reasoningprompt engineeringreverse reasoningcritical thinkingself-correction

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a single, content-free prompt modification—asking the model to verify a candidate answer, even a random or trivial one, before solving—improves reasoning accuracy across diverse models and benchmarks at minimal extra compute. The gain is attributed to the verification instruction itself, not to the provided answer, because verification elicits a reverse reasoning path that complements forward chain-of-thought and induces critical thinking. The authors further introduce Iter-VF, a Markovian loop that iterates verify-then-solve using the model's previous answer, and report that it beats existing test-time scaling strategies under equal token budgets. The payoff for a broad audience: a parameter-free, training-free, task-agnostic intervention that also works on closed, thought-hidden commercial models, with a headline result of 94.9% accuracy on the hard GPQA-Diamond benchmark.

Core claim

The paper's central claim is that the verification-first instruction is the active ingredient: providing a candidate answer A′—trivial '1' for math, a random shuffled option for multiple choice, or the model's own previous output for open-ended tasks—and asking the model to verify it before solving triggers a 'reverse reasoning' path that narrows the logical search space and reduces egocentric overconfidence. The authors demonstrate consistent gains over zero-shot CoT across models from 1B to 70B parameters, at 20–50% extra output tokens, and show that iterating the process (Iter-VF) outperforms self-correction, progressive-hint prompting, self-consistency, and best-of-N under limited test-t

What carries the argument

The central object is the VF prompt template: 'I guess the answer is A′, possibly wrong. First verify my answer to see if it satisfies the question, then think step by step to find the correct answer.' Its work is to split the model's output into a cheap verification phase and a generation phase; the verification phase is argued to be logically easier and to produce a reverse reasoning path that restricts the output distribution, while framing A′ as external sheds the model's egocentric bias. Iter-VF turns this into a Markovian process by feeding only the previous extracted answer back into the same template, discarding long histories that cause context overflow and error accumulation in sel

Load-bearing premise

The claim that VF is an 'almost free lunch' depends on the placeholder answer being nearly irrelevant: if the content of the random/trivial answer materially changes performance, users would need prior knowledge to choose a good one, and the method would lose its minimal-prior-knowledge advantage.

What would settle it

A controlled experiment on one large model and a fixed benchmark (e.g., GSM8K) that runs VF with many arbitrary placeholder answers—'1', '2025', 'π', 'none of the above', a random option, a false but plausible value—and measures the accuracy spread. If the spread across these placeholders approaches or exceeds the average VF-over-CoT gap (roughly 3–5 points on GSM8K), the content-independence premise fails. The paper's Figure 5 already shows a ~2-point gap between random and false options on GPQA; a broader sweep would settle whether this is noise or signal.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

VF prompting with a random or trivial answer yields consistent accuracy improvements over standard chain-of-thought across six open-weight model families from 1B to 72B parameters, with only 20–50% additional output tokens.
Iter-VF outperforms sequential and parallel test-time scaling baselines (Self-Correction, PHP, Self-Consistency, Best-of-N) under equal token budgets on MATH500.
On open-ended coding and API tasks, VF prompting with a previously generated answer beats both pass@2 sampling and self-correction at the same compute cost.
Because VF only modifies the input instruction, it remains effective on thought-hidden commercial services where reflection-based strategies are inapplicable; the paper reports a ~30% relative error reduction on GPQA-Diamond.
The provided answer's content has limited influence: random, trivial, and even false answers yield similar performance, while true answers help more—evidence that the verification process itself does the work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the effect is truly instruction-driven, a broader family of cheap prompt manipulations—such as asking the model to 'act as a skeptical reviewer' or to 'restate the problem from the answer backwards'—may yield similar gains without any placeholder answer; this is a direct testable extension.
The Markovian reset in Iter-VF suggests that the benefit may come less from verification than from forcing a fresh reasoning attempt with a target to check; a control that simply repeats 'start over, using the previous answer as a hint' would isolate the verification component.
Given that true answers help much more than random ones, VF could be combined with cheap answer heuristics or small verifiers to produce better placeholders, potentially pushing accuracy further at negligible cost.
The 20–50% token overhead means 'free lunch' is approximate; plotting accuracy vs. tokens for VF against self-consistency at larger budgets would delineate the efficient frontier and clarify when to switch strategies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The verification-first trick is genuinely novel and plausibly useful, but the paper's evidence is much thinner than its abstract claims, and the missing error bars and missing flagship result are real problems.

read the letter

The core idea is simple and, as far as I know, not in the cited literature: give the model a random or trivial candidate answer and ask it to verify that before solving. The two examples in Figure 2 are striking - the same model that fails with plain CoT gets the right answer when told to check a wrong guess first. That is a real empirical phenomenon, and the paper earns credit for noticing it and for testing it across a wide range of model sizes and benchmarks. The Iter-VF comparison against self-correction and self-consistency is also a reasonable way to frame the contribution. But the evidence as presented does not support the headline. The abstract's 94.9% GPQA-Diamond result with Gemini-3-Pro-Preview appears nowhere in the experiments. That is not a minor omission; it is the flagship claim. The plots in Figure 4 are single-run point estimates with no error bars, no seeds, no decoding settings, and no code. On GPQA-Diamond, which has only 198 questions, a 2.5-point gap is about five questions - well within binomial noise. The sensitivity analysis in Figure 5 shows that the choice of the provided answer matters more than the text admits: true answers help a lot, and random versus false options differ by 1.7 points. The claim that 'providing different answers with minimal prior knowledge would not effect the final performance a lot' is asserted without any statistical test. And because VF outputs 20-50% more tokens, there is no token-matched CoT baseline; the gain could be partly just extra compute. That undercuts the 'almost free lunch' framing. There is also a citation problem: the paper cites Baker, Gill, and Solovay (1975) to support the idea that verifying is easier than generating. That paper is about relativization of P versus NP and says nothing about verification being easier. It is a wrong reference, and for a paper that grounds its method in cognitive science, that matters. Still, the central idea is worth taking seriously. The fix is not a rewrite; it is a replication package and a more honest presentation of uncertainty. I would send this to peer review, but I would expect the reviewers to demand code, multiple seeds, error bars, and a token-matched control. If the effect survives that scrutiny, this is a genuinely useful prompting method. If it does not, it is a reminder that single-run point estimates can fool us. I would not cite it in my own work yet, but I would bring it to a reading group to get other opinions on whether the trick is as robust as it looks.

Referee Report

4 major / 4 minor

Summary. The paper introduces Verification-First (VF) prompting: the user supplies a candidate answer A′ (possibly random or trivial) and asks the LLM to verify it before generating a solution. It then proposes Iter-VF, a sequential test-time-scaling method that cycles VF with the model's previous answer. The manuscript reports experiments on GSM8K, MATH500, GPQA-Diamond, HumanEval, MBPP, API-Bank, and thought-hidden commercial APIs, and claims that VF consistently outperforms standard CoT with modest token overhead, that Iter-VF outperforms existing TTS strategies, and that VF yields a new SOTA 94.9% on GPQA-Diamond with Gemini-3-Pro-Preview. The central empirical claims, however, are not currently supported at the required level of rigor: the evaluations are single-run point estimates, the GPQA 'random option' condition is confounded, there is no token-matched CoT baseline, and the abstract's flagship Gemini result is absent from the experimental section.

Significance. If the empirical claims survive re-analysis, the practical contribution is real and valuable: a zero-training, zero-few-shot prompt modification that can be applied to black-box APIs across many task families. The Iter-VF algorithm is clearly specified, and the Markovian design is a plausible improvement over accumulating-context self-correction. The paper is not circular: the evaluation is an external benchmark comparison, and the cognitive-science narrative does not assume the conclusion. However, the current evidence is insufficient: no code, seeds, or error bars are provided; the GPQA 'random option' condition appears to leak true answers; the 20–50% extra token budget is not controlled; and the abstract's SOTA number is unauditable. These issues must be addressed before the consistency and 'free lunch' claims can be accepted.

major comments (4)

[§4.1, Figure 5] The GPQA-Diamond 'random option' condition is confounded by true-answer leakage. On a four-choice benchmark, a uniformly random option is the true answer 25% of the time. Using the paper's own Figure 5 values (random 44.9%, false 43.2%, true 50.0%), the expected accuracy of a random option is 0.75×43.2 + 0.25×50.0 = 44.9, exactly the reported value. Thus the difference between random and false is fully attributable to the chance of providing the correct answer, not to the verification instruction. The main GPQA result and the §4.1 sensitivity claim ('would not effect ... a lot') are not established; report a false-only condition or condition on the random draw.
[§4.1, Figures 4–5] No standard errors, seeds, decoding settings, or code are reported; every comparison is a single-run point estimate. On GPQA-Diamond (198 questions), CoT 42.4% vs VF-random 44.9% is a 2.5-point gap (~5 questions) with a binomial standard error around 3.5 points, so the headline 'VF consistently outperforms CoT' is within sampling noise on this benchmark (and the same holds for random-vs-false). Multi-seed or bootstrap intervals are required for the central consistency claim, together with the exact sampling protocol for the random option.
[§4.1, Table 1] There is no token-matched CoT control. Table 1 shows VF consumes 20–50% more output tokens (MATH500 1109.6 vs 808.3; GPQA 901.8 vs 739.3; GSM8K 533.6 vs 365.6), so the observed gain over CoT could be an extra-compute effect rather than the verification instruction. Add CoT baselines with matched token budgets (e.g., longer reasoning instructions, output-token limit, or two CoT calls) and report accuracy per output token. This is necessary to support the 'almost free lunch' framing.
[Abstract; §4.3.2] The flagship claim — 'a new SOTA 94.9% accuracy on GPQA-Diamond with Gemini-3-Pro-Preview' and 'reduces its errors by ~30% relatively' — appears nowhere in the experiments. Table 3 reports only GPT-5 Nano/Mini on MATH500/GPQA-Diamond, with no Gemini results, no 94.9% number, and no protocol. A headline quantitative claim must be reproducible from the main text; either add the Gemini experiment with full details or remove the claim from the Abstract.

minor comments (4)

[Global] Typos and formatting: 'V erify' in the title, 'Electonic Engineering', 'generalizaibility', and the duplicated sentence in §4.1 ('We provide trivial answer "1" ...'). Also, 'effect' is used as a verb where 'affect' is intended.
[Abstract; §2.2; §3.1] The mechanism claims ('pruning the LLM's output distribution', 'reverse reasoning path', 'overcoming egocentrism') are plausible narratives but are not directly tested. Consider framing them as hypotheses or adding an ablation, e.g., a prompt with a neutral candidate-answer instruction that does not ask for verification.
[§4.2, Figure 6] It is unclear how output-token budgets are accumulated for parallel methods (Self-Consistency, Best-of-N) versus sequential methods. Specify whether the x-axis is total tokens across all sampled paths and how many samples correspond to each budget point.
[Algorithm 1] The pseudocode requires an initial answer A0 but marks it optional; the text's first-iteration branch (VF(Q,A0) or CoT(Q)) should be reflected in the algorithm.

Circularity Check

0 steps flagged

No circular derivation; the claims are empirical benchmark comparisons and no fitted parameter is relabeled as a prediction.

full rationale

The paper's central claims are empirical: VF prompting with random answers is compared to standard CoT and other TTS methods on external benchmarks (GSM8K, MATH500, GPQA-D, HumanEval, MBPP, API-Bank). No parameter is fitted from a subset of data and then reported as a prediction; no equation is defined in terms of the quantity it is supposed to derive; and no load-bearing argument reduces to a self-citation. The cited prior work is used as baselines or background. Concerns about missing error bars, single-run point estimates, and the abstract's unreported 94.9% Gemini figure affect evidence quality, not circularity. Under the required standard, no specific circular step can be exhibited.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

No free parameters are fitted to data in the usual sense; the listed items are hand-chosen inputs/hyperparameters the claim depends on. Axioms are the informal cognitive-transfer assumptions that do the explanatory work but are not directly measured.

free parameters (2)

Initial random/trivial answer A′ = A′=1 (GSM8K/MATH500), random or 'Option B' (GPQA); '1' or '2025' for sensitivity (Figure 5)
Hand-chosen placeholder; the method's generality claim rests on the answer content not mattering much, but Figure 5 shows 43.2 vs 44.9 (GPQA, false vs random) and 78.6 vs 90.6 (MATH500, '2025' vs true) — so the choice carries some signal.
Iteration budget B in Iter-VF = varied per budget (Figures 6), no reported rule
Hyperparameter controlling test-time compute; results depend on where the stopping point lies.

axioms (3)

domain assumption Verifying a candidate answer is easier/cheaper than generating the correct answer
Invoked in §1 to justify VF; cited to Baker et al. 1975, which is actually about relativized P vs NP and does not establish ease of verification — a miscitation. Also not empirically demonstrated for LLMs beyond the paper's own results.
domain assumption Human cognitive constructs ('egocentrism', 'critical thinking') transfer to LLM generation behavior
§2.2 bases the mechanism on Piaget/Brookfield; no LLM-specific evidence that the model 'overcomes egocentrism' — this is an informal analogy.
domain assumption LLMs follow the two-stage instruction (verify then generate) rather than ignoring the verification request
The whole method assumes the instruction is honored; the paper does not measure verification-following rates except in illustrative examples (Figure 2).

pith-pipeline@v1.3.0-alltime-deepseek · 164 in / 11812 out tokens · 144034 ms · 2026-08-03T20:58:49.372441+00:00 · methodology

0 comments

read the original abstract

To enhance the reasoning capabilities of Large Language Models (LLMs) without high costs of training, nor extensive test-time sampling, we introduce Verification-First (VF), a strategy that prompts models to verify a provided candidate answer, even a trivial or random one, before generating a solution. This approach triggers a "reverse reasoning" process complementary to standard forward Chain-of-Thought (CoT), which restricts the logical search space of the answer by pruning the LLM's output distribution. We further generalize VF prompting to Iter-VF, a sequential test-time scaling (TTS) method that iteratively cycles the verification-generation process using the model's previous answer. Extensive experiments across various benchmarks and various LLMs confirm that VF prompting with random answer consistently outperforms standard CoT with minimal computational overhead, and Iter-VF outperforms existing TTS strategies. VF is also effective on SOTA thinking models. For example, by using the simple VF prompting, we obtain a new SOTA 94.9% accuracy on GPQA-Diamond with Gemini-3-Pro-Preview where VF reduces its errors by ~30% relatively.

Figures

Figures reproduced from arXiv: 2511.21734 by Quanming Yao, Shiguang Wu.

**Figure 2.** Figure 2: VF prompting with random/trivial answer (Right), comparing with standard CoT prompting (Left) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Illustration of (a) VF prompting with previously generated answer, and iterating such process as (b) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: VF prompting consistently outperforms stan [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Figure 5: Providing different answers to VF for verification. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 6.** Figure 6: Comparison of different TTS methods on MATH500. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

30 extracted references · 15 linked inside Pith

[1]

Lisa Alazraki, Maximilian Mozes, Jon Ander Campos, Tan Yi-Chern, Marek Rei, and Max Bartolo. 2025. No need for explanations: Llms can implicitly learn from mistakes in-context. arXiv preprint arXiv:2502.08550

arXiv 2025
[2]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

Pith/arXiv arXiv 2021
[3]

Theodore Baker, John Gill, and Robert Solovay. 1975. Relativizations of the p=?np question. SIAM Journal on computing, 4(4):431--442

1975
[4]

Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and 1 others. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 17682--17690

2024
[5]

Stephen D Brookfield. 1987. Developing critical thinkers: Challenging adults to explore alternative ways of thinking and acting. Jossey-Bass

1987
[6]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. https://arxiv.org/abs/2107.03374 Evaluating large lang...

Pith/arXiv arXiv 2021
[7]

Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. 2023. Contrastive chain-of-thought prompting. arXiv preprint arXiv:2311.09277

Pith/arXiv arXiv 2023
[8]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

Pith/arXiv arXiv 2021
[9]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

Pith/arXiv arXiv 2024
[10]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

Pith/arXiv arXiv 2021
[11]

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

Pith/arXiv arXiv 2023
[12]

Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. 2025. Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21321

Pith/arXiv arXiv 2025
[13]

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244

Pith/arXiv arXiv 2023
[14]

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. In The Twelfth International Conference on Learning Representations

2023
[15]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534--46594

2023
[16]

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393

Pith/arXiv arXiv 2025
[17]

Jean Piaget. 1976. Piaget’s theory. In Piaget and his school: A reader in developmental psychology, pages 11--23. Springer

1976
[18]

George Polya. 1957. How to solve it

1957
[19]

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

2024
[20]

Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. 2025. Scaling test-time compute without verification or rl is suboptimal. arXiv preprint arXiv:2502.12118

Pith/arXiv arXiv 2025
[21]

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634--8652

2023
[22]

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

Pith/arXiv arXiv 2024
[23]

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

Pith/arXiv arXiv 2022
[24]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[25]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

Pith/arXiv arXiv 2024
[26]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809--11822

2023
[27]

James Xu Zhao, Bryan Hooi, and See-Kiong Ng. 2025. Test-time scaling in reasoning models is not effective for knowledge-intensive tasks yet. arXiv preprint arXiv:2509.06861

arXiv 2025
[28]

Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797

Pith/arXiv arXiv 2023
[29]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[30]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[1] [1]

Lisa Alazraki, Maximilian Mozes, Jon Ander Campos, Tan Yi-Chern, Marek Rei, and Max Bartolo. 2025. No need for explanations: Llms can implicitly learn from mistakes in-context. arXiv preprint arXiv:2502.08550

arXiv 2025

[2] [2]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

Pith/arXiv arXiv 2021

[3] [3]

Theodore Baker, John Gill, and Robert Solovay. 1975. Relativizations of the p=?np question. SIAM Journal on computing, 4(4):431--442

1975

[4] [4]

Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and 1 others. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 17682--17690

2024

[5] [5]

Stephen D Brookfield. 1987. Developing critical thinkers: Challenging adults to explore alternative ways of thinking and acting. Jossey-Bass

1987

[6] [6]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. https://arxiv.org/abs/2107.03374 Evaluating large lang...

Pith/arXiv arXiv 2021

[7] [7]

Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. 2023. Contrastive chain-of-thought prompting. arXiv preprint arXiv:2311.09277

Pith/arXiv arXiv 2023

[8] [8]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

Pith/arXiv arXiv 2021

[9] [9]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

Pith/arXiv arXiv 2024

[10] [10]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

Pith/arXiv arXiv 2021

[11] [11]

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

Pith/arXiv arXiv 2023

[12] [12]

Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. 2025. Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21321

Pith/arXiv arXiv 2025

[13] [13]

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244

Pith/arXiv arXiv 2023

[14] [14]

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. In The Twelfth International Conference on Learning Representations

2023

[15] [15]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534--46594

2023

[16] [16]

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393

Pith/arXiv arXiv 2025

[17] [17]

Jean Piaget. 1976. Piaget’s theory. In Piaget and his school: A reader in developmental psychology, pages 11--23. Springer

1976

[18] [18]

George Polya. 1957. How to solve it

1957

[19] [19]

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

2024

[20] [20]

Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. 2025. Scaling test-time compute without verification or rl is suboptimal. arXiv preprint arXiv:2502.12118

Pith/arXiv arXiv 2025

[21] [21]

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634--8652

2023

[22] [22]

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

Pith/arXiv arXiv 2024

[23] [23]

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

Pith/arXiv arXiv 2022

[24] [24]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022

[25] [25]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

Pith/arXiv arXiv 2024

[26] [26]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809--11822

2023

[27] [27]

James Xu Zhao, Bryan Hooi, and See-Kiong Ng. 2025. Test-time scaling in reasoning models is not effective for knowledge-intensive tasks yet. arXiv preprint arXiv:2509.06861

arXiv 2025

[28] [28]

Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797

Pith/arXiv arXiv 2023

[29] [29]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[30] [30]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...