Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Zero-Shot Verification-guided Chain of Thoughts

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Zero-shot chain-of-thought can be decomposed and self-verified with prompt-only additions, but verifier-guided search does not beat plain self-consistency.

desk verdict A solid, honest zero-shot prompting paper whose COT STEP trick is genuinely useful; the RQ2 verifier numbers are weakened by answer-match labels but the limitation is disclosed. read the letter →

arxiv 2501.13122 v1 pith:L3HI2PM3 submitted 2025-01-21 cs.CL cs.AI

Jishnu Ray Chowdhury , Cornelia Caragea This is my paper · ORCID
classification cs.CLcs.AI
keywords zero-shotpromptingchain-of-thoughtself-verificationreasoningstepdecompositionverifier-guidedsearchself-consistencymathematicalcommonsense
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a language model can verify its own step-by-step reasoning with no fine-tuned verifier, no handcrafted examples, and no extra data. It introduces COT STEP, a zero-shot prompt that appends 'Step 1:' to the standard 'Let's think step by step' opener, forcing the model to emit numbered reasoning steps that can be split automatically. On five reasoning datasets and four openly-licensed LLMs, COT STEP matches plain chain-of-thought accuracy while making the steps parseable. The paper also introduces a zero-shot chain-of-thought verification prompt (COTR-prompt) that classifies whether mathematical reasoning chains are correct, reaching 83.9% accuracy on GSM8K chains for one model. But using the verifier's scores to guide search only helps without self-consistency; with self-consistency, plain majority voting remains the stronger choice.

What carries the argument

The load-bearing mechanism is the COT STEP prompt format: 'Let's think step by step.\n\nStep 1:' appended to the question, which makes the model emit numbered steps that can be split on 'Step <number>:'. The verifier score combines two components: C1, the average token log-probability of the generated step, and C2, the log-probability of the verification answer, mapped to a positive score for 'correct' and a negative score for 'incorrect', then averaged and exponentiated into a unified score. Step-wise greedy search uses that unified score to pick the highest-scoring candidate step at each turn.

What would settle it

Re-annotate a sample of GSM8K chains with step-level correctness labels by human raters, then recompute the COTR verifier's accuracy as a chain classifier on that relabeled set; if the correct-answer-with-wrong-steps rate is non-negligible, the reported 83.9% accuracy for SOLAR on COT STEP chains should fall noticeably.

Watch

Extended reading notes

Core claim

The central finding is that a purely prompt-based pipeline can structure and verify chain-of-thought reasoning in a zero-shot setting. COT STEP is the standard zero-shot COT instruction with '\n\nStep 1:' appended, which induces the model to generate numbered steps separated by 'Step <number>:' and gives an automatic way to split a chain into reasoning steps without exemplars. The verifier prompts, R-prompt (a direct yes/no verification prompt) and COTR-prompt (its chain-of-thought variant), ask the model to judge the last step of a partial chain; COTR-prompt additionally asks it to double-check step by step, and its answers are scored from the token log-probabilities of the step and of the verification response. Evaluated on chain-level correctness classification built from GSM8K and CommonsenseQA, COTR-prompt with COT STEP chains reaches 83.9% accuracy on GSM8K for SOLAR and 75.2% for Phi3, while performance on commonsense questions stays near chance. In search experiments, a step-wise greedy search guided by the verifier improves accuracy over plain COT STEP without self-consistency, but the improvement disappears under self-consistency, and verifier-weighted or filtered majority voting does not beat plain majority voting.

Load-bearing premise

The evaluation assumes a reasoning chain is correct whenever its final answer matches the gold answer, even if intermediate steps are wrong, which the paper itself flags as imperfect in Section 8.

Editorial extensions

If this is right

  • COT STEP can replace zero-shot COT in any pipeline that needs to parse reasoning steps, since it preserves accuracy while adding a clean 'Step N:' delimiter.
  • The COTR-prompt gives a drop-in verifier for mathematical reasoning tasks where no trained verifier or labeled verification data is available.
  • For users already running self-consistency, adding this zero-shot verifier is not worth the extra compute, because majority voting alone matches or beats all three verifier-based selection policies.
  • PS+ and TAB COT should not be assumed to outperform COT on openly-licensed models; the gains reported for those prompts do not generalize consistently.
  • Verifier-guided step-wise greedy search is a useful single-chain baseline, but beam search guided by the same verifier does not improve on it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the answer-based labeling means the reported verifier accuracies are upper bounds on true step-level judgment; a human step-level relabeling could lower them substantially.
  • Beyond the paper, the fact that search gains vanish under self-consistency suggests majority voting already captures the same answer-level signal, so a reranker would need to act only on disagreeing samples to add value.
  • Beyond the paper, COT STEP's numbered structure can likely be combined orthogonally with other zero-shot augmentations, such as re-reading or role-play prompts, without changing the parseable format.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes and evaluates zero-shot prompts for chain-of-thought reasoning: COT STEP, a modification of the standard zero-shot COT prompt that forces numbered step outputs, and two self-verification prompts (R-prompt and COTR-prompt) that score the correctness of the last reasoning step. The authors run three studies: (RQ1) comparison of COT STEP against existing zero-shot prompts on five benchmarks and two LLMs; (RQ2) evaluation of the verifiers on a self-constructed balanced dataset where a reasoning chain is labeled correct iff its final answer matches the gold answer; (RQ3) use of verifier scores to augment self-consistency and to guide step-wise greedy/beam search. Findings: COT STEP matches or slightly exceeds standard COT while giving cleanly separated steps; the COTR-prompt achieves high accuracy on the chain-level correctness classification task for math questions but not commonsense; step-wise greedy search improves over plain COT STEP when self-consistency is not used, but the benefit disappears under self-consistency.

Significance. The paper's main strengths are its honest reporting of negative results (verifier scores do not help self-consistency; beam search does not beat greedy) and its simple, reproducible prompt designs. COT STEP is a practical contribution: it is a drop-in zero-shot prompt that yields structured step outputs without hurting accuracy. The empirical comparison across five prompts, five datasets, and multiple LLMs is useful. However, the evaluation of the verifier's step-level verification capability rests on a final-answer-match proxy, and the comparative claims lack variance estimates. If the verification claim is reframed or re-evaluated with step-level labels, the paper would make a solid empirical contribution to zero-shot reasoning and self-verification.

major comments (3)
  1. [§4.2–4.3, Tables 2 and 7] The automatic ground-truth labels for the correctness classification task are assigned by comparing the extracted final answer to the dataset gold answer (Section 4.3), yet Section 4.2 defines chain-level correctness as 'all intermediate reasoning steps are correct and the final answer matches.' These two notions are not equivalent: a chain can reach the correct final answer through incorrect or unfaithful intermediate steps. As a result, the reported verifier accuracies (e.g., SOLAR + COTR-prompt at 83.90% on (COT STEP, GSM8K)) measure the verifier's ability to predict final-answer match, not its ability to detect step-level errors. The limitation is acknowledged in Section 8, but the conclusion still states that the COTR-prompt is 'useful for verification particularly in mathematical domain' (takeaway 3). To support a step-level verification claim, the authors should either evaluate the verifier on a subset with step-level ground truth (e.g., PRM800K-style annotations or a human-rated sample) or explicitly restate the RQ2 claim as 'classifying whether the final answer is correct.'
  2. [§4.1.1] The verifier's binary prediction is obtained by checking whether the substring 'A)' appears in the generated response. For the COTR-prompt, the maximum generation length is 512 tokens, so the model can produce 'A)' in a context unrelated to the verdict (e.g., 'Option A is incorrect') or can state 'Yes' without emitting 'A)'. This extraction heuristic is a potential source of systematic error that affects both the accuracy numbers in Table 2 and the Filter policy in Section 5.1. The authors should report a manual evaluation of a sample of extractions or use a more robust parsing method (e.g., look at the final answer token or classify the response with a second prompt). Without this, the verifier scores are built on a fragile parse.
  3. [Tables 1–7] All results are point estimates from a single run (or a single set of 20 self-consistency samples), with no error bars, multiple seeds, or significance tests. Several comparative statements in Section 3.1 rely on small differences (e.g., COT STEP at 73.54 vs COT at 73.62 on GSM8K for SOLAR, or COT STEP at 84.91 vs PS+ at 84.90 on GSM8K with self-consistency). These differences are within the noise one would expect from sampling, making the qualitative claims such as 'COT STEP keeps up fairly well' or 'COT STEP is even the best performer in most cases' unsupported. The authors should provide variance estimates or a statistical test (e.g., bootstrap over questions) for at least the headline comparisons, or temper the claims to acknowledge the lack of statistical robustness.
minor comments (6)
  1. [§2.4] Section 2.4 contains a typo: 'Quetion Answering' should be 'Question Answering.'
  2. [Table 2 caption] The caption of Table 2 labels the true negative rate as 'True Negate Rate'; the correct term is 'True Negative Rate.'
  3. [Tables 6, 7, and text] The model name is inconsistently written as 'Llama3' and 'LLama3' across Tables 6, 7, and the text; please standardize (e.g., 'Llama 3').
  4. [§4.1.2] Equation (1) should clarify whether bsC2 is the average log-probability of the entire verification response or of the option token only; the current wording suggests the former, but then the expression log(1 - exp(bsC2)) is not the log-probability of the 'No' answer. Specifying this will make the scoring function interpretable.
  5. [§5.2 and Appendix A] Section 5.2 and Appendix A report the greedy/beam search hyperparameters but not the computational cost (number of LLM calls) compared with the self-consistency baseline; since the conclusion states that self-consistency is a 'better trade-off', a cost comparison would substantiate that claim.
  6. [§5.2, Appendix A] The search hyperparameters (k=5, beam width=16, tau=0.5, alpha=0.5, max steps=18) are fixed without sensitivity analysis; a brief note on their effect (or a citation to prior work justifying the values) would make the zero-shot claim more robust.

Circularity Check

0 steps flagged · score 0.0 of 10

The paper's claims are empirically evaluated against held-out benchmarks; the disclosed answer-match labeling limitation is a validity concern, not a circular derivation.

full rationale

We find no load-bearing circularity. The central comparative claims (RQ1 and RQ3) are measured directly by final-answer accuracy on held-out benchmark questions, and no fitted parameter is renamed as a prediction: the verifier score in Eq. 2 is a fixed average of two log-probability components with no tunable weights, and the Appendix ablation shows the C1 component contributes. COT STEP is a formatting variant of zero-shot COT, and its performance is established empirically rather than derived from its own definition. The RQ2 correctness-classification evaluation uses final-answer match as a proxy for chain correctness (Section 4.3), which the authors explicitly acknowledge as imperfect in Section 8, citing Lanham et al. This is a label-validity limitation: the verifier does not receive the gold answer, and the self-verification setup is the object of study rather than a hidden assumption. The R-prompt is adapted from external prior work (Ling et al., 2023), and there is no self-citation chain invoked to justify the paper's central choices. No equation is equivalent to its input by construction, and no fitted quantity is presented as a prediction. Accordingly, no circular step meets the evidentiary bar, and the appropriate finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its empirical claims rest on three domain assumptions (answer-match labels, parseable 'Step N:' formatting, log-probability informativeness) and six hand-selected search hyperparameters. No constants are fitted to the test sets; the score averaging in Eq. 2 has no tuned weights.

free parameters (6)
  • k (candidate steps per turn in SG) = 5
    Hand-set; no sensitivity analysis. Affects the SG search results in Tables 4, 6, and 9.
  • Beam width for SB = 16
    Chosen to stay close to the 20-sample self-consistency budget (Appendix A).
  • Initial beam-search temperature tau and decay alpha = tau = 0.5, alpha = 0.5
    Set following Xie et al. (2023), not tuned here (Appendix A).
  • Maximum reasoning steps for search = 18
    Hand-set cap for SG and SB (Appendix A).
  • Self-consistency sampling temperature = 0.8
    Standard for SC sampling (Section 3); no analysis of sensitivity.
  • R-prompt max generation length / COTR-prompt max length = 2 / 512 tokens
    Heuristics to force a choice in R-prompt and allow reasoning in COTR-prompt (Section 4.1.1).
assumptions (3)
  • domain assumption A chain whose final answer matches the gold answer is wholly correct; a chain whose answer does not match is incorrect.
    Used to build all correctness-classification datasets (Section 4.3); explicitly flagged as imperfect in Section 8, citing Lanham et al. (2023).
  • domain assumption LLMs will reliably format COT STEP replies with 'Step N:' delimiters that can be parsed.
    The step decomposition (Section 2.3) and all verifier step-level scoring depend on this; the examples in Appendix C show it works for two models, but no parse-success rate is reported.
  • domain assumption Average token log-probability during COT generation and verification responses is a meaningful correctness signal.
    The verifier score sf (Eq. 2) is built from these log-probabilities; the paper does not calibrate or evaluate this assumption directly beyond the resulting accuracy numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-Shot Verification-guided Chain of Thoughts." pith.science (2026). https://pith.science/paper/L3HI2PM3

@misc{pith2026250113122,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot Verification-guided Chain of Thoughts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L3HI2PM3}},
  note         = {Machine review of arXiv:2501.13122}
}
read the original abstract

Previous works have demonstrated the effectiveness of Chain-of-Thought (COT) prompts and verifiers in guiding Large Language Models (LLMs) through the space of reasoning. However, most such studies either use a fine-tuned verifier or rely on manually handcrafted few-shot examples. In contrast, in this paper, we focus on LLM-based self-verification of self-generated reasoning steps via COT prompts in a completely zero-shot regime. To explore this setting, we design a new zero-shot prompt, which we call COT STEP, to aid zero-shot decomposition of reasoning steps and design two new zero-shot prompts for LLM-based verifiers. We evaluate the verifiers' ability to classify the correctness of reasoning chains and explore different ways to use verifier scores in guiding reasoning for various mathematical and commonsense reasoning tasks with different LLMs.

Figures

Figures reproduced from arXiv: 2501.13122 by the authors.

Figure 1
Figure 1. The R-prompt template used for LLMs to verify/evaluate the last (potentially partial) reasoning step in a given generated reasoning chain. The blue color represents placeholders for the template that is to be filled with the input data-specific information (the question, and generated reasoning step with the last step number). erwise competitive in CSQA where COT-style rea￾soning may not be as crucial given that it … view at source ↗
Figure 2
Figure 2. Example generations by Phi 3 Mini Instruct based on COT STEP prompt on GSM8K samples. Bold is [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Example generations by Phi 3 Mini Instruct based on COT prompt on GSM8K samples. Bold is applied [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoning

    cs.CL 2025-08 reject novelty 5.0 of 10

    ASCoT claims later reasoning errors are more harmful than early ones and uses a position-weighted verifier to prune and correct CoT steps, but its key evidence is internally inconsistent.

  2. Large Language Models for Next-Generation Wireless Network Management: A Survey and Tutorial

    cs.NI 2025-09 conditional novelty 4.0 of 10

    A survey and tutorial that organizes LLM-enabled wireless network optimization into formulation, solution, and verification stages, with case studies drawn from the authors' own prior papers.

Reference graph

Works this paper leans on

59 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    u rgen M \

    Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwa \'s niewski, J \"u rgen M \"u ller, Lukas Gianinazzi, et al. 2024. Topologies of reasoning: Demystifying chains, trees, and graphs of thoughts. arXiv preprint arXiv:2401.14295

    arXiv 2024

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  6. [6]

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712

    arXiv 2023

  7. [7]

    Wei-Lin Chen, Cheng-Kuang Wu, Yun-Nung Chen, and Hsin-Hsi Chen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.968 Self- ICL : Zero-shot in-context learning with self-generated demonstrations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15651--15662, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.968 2023

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 59 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. 2023. Strategic reasoning with language models. arXiv preprint arXiv:2305.19165

    2023 arXiv

  3. [11]

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

  4. [12]

    Zitian Gao, Boye Niu, Xuzheng He, Haotian Xu, Hongzhang Liu, Aiwei Liu, Xuming Hu, and Lijie Wen. 2024. Interpretable contrastive monte carlo tree search reasoning. arXiv preprint arXiv:2410.01707

    2024 arXiv

  5. [13]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. https://doi.org/10.1162/tacl_a_00370 Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies . Transactions of the Association for Computational Lingu...

    2021 doi

  6. [14]

    Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, and Zhiting Hu. 2024. https://openreview.net/forum?id=b0y6fbSUG0 LLM reasoners: New evaluation, library, and analysis of step-by-step reasoning...

    2024

  7. [15]

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.507 Reasoning with language model is planning with world model . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

    2023 doi

  8. [16]

    Ruixin Hong, Hongming Zhang, Xinyu Pang, Dong Yu, and Changshui Zhang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.52 A closer look at the self-verification abilities of large language models in logical reasoning . In Proceedings of the 2024 Conference of the North Ameri...

    2024 doi

  9. [17]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  10. [18]

    Feihu Jin, Yifan Liu, and Ying Tan. 2024. Zero-shot chain-of-thought reasoning guided by evolutionary algorithms in large language models. arXiv preprint arXiv:2402.05376

    2024 arXiv

  11. [19]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  12. [20]

    Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.1022 GRACE : Discriminator-guided chain-of-thought reasoning . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1...

    2023 doi

  13. [21]

    Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. 2024. https://doi.org/10.18653/v1/...

    2024 doi

  14. [22]

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf Large language models are zero-shot reasoners . In Advances in Neural Inf...

    2022

  15. [23]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024. https://doi.org/10.18653/v1/2024.naacl-long.228 Better zero-shot reasoning with role-play prompting . In Proceedings of the 2024 Conference of the North American C...

    2024 doi

  16. [24]

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702

    2023 arXiv

  17. [25]

    Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562

    2016 arXiv

  18. [26]

    Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023. https://doi.org/10.18653/v1/2023.acl-long.291 Making language models better reasoners with step-aware verifier . In Proceedings of the 61st Annual Meeting of the Association for Compu...

    2023 doi

  19. [27]

    Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, and Semih Yavuz. 2024. Improving llm reasoning through scaling inference computation with collaborative verification. arXiv preprint arXiv:2410.05318

    2024 arXiv

  20. [28]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

    2024

  21. [29]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. https://doi.org/10.18653/v1/P17-1015 Program induction by rationale generation: Learning to solve and explain algebraic word problems . In Proceedings of the 55th Annual Meeting of the Association for Computational ...

    2017 doi

  22. [30]

    Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/72393bd47a35f5b3bee4c609e7bba733-Paper-Conference.pdf Deductive verification of chain-of-thought reasoning . In Advances i...

    2023

  23. [31]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://proceed...

    2023

  24. [32]

    Ning Miao, Yee Whye Teh, and Tom Rainforth. 2024. https://openreview.net/forum?id=pTHfApDakA Selfcheck: Using LLM s to zero-shot check their own step-by-step reasoning . In The Twelfth International Conference on Learning Representations

    2024

  25. [33]

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022. https://openreview.net/forum?id=HBlx2idbkbq Show your work: Scratchpads for int...

    2022

  26. [34]

    Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2024. https://aclanthology.org/2024.eacl-long.67 REFINER : Reasoning feedback on intermediate representations . In Proceedings of the 18th Conference of the European...

    2024

  27. [35]

    Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. 2024. Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling. arXiv preprint arXiv:2410.16033

    2024 arXiv

  28. [36]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf Reflexion: language agents with verbal reinforcement learning . In Advances...

    2023

  29. [37]

    Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2024. On the self-verification limitations of large language models on reasoning and planning tasks. arXiv preprint arXiv:2402.08115

    2024 arXiv

  30. [38]

    Linzhuang Sun, Hao Liang, and Wentao Zhang. 2024. Beats: Optimizing llm mathematical capabilities with backverify and adaptive disambiguate based efficient tree search. arXiv preprint arXiv:2409.17972

    2024 arXiv

  31. [39]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  32. [40]

    Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. 2023 a . https://doi.org/10.18653/v1/2023.findings-acl.216 Better zero-shot reasoning with self-adaptive prompting . In Findings of the Association for Computational Linguistics: ACL 2023, pages 3493--3514, T...

    2023 doi

  33. [41]

    Xingchen Wan, Ruoxi Sun, Hootan Nakhost, Hanjun Dai, Julian Eisenschlos, Sercan Arik, and Tomas Pfister. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.461 Universal self-adaptive prompting . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  34. [42]

    Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2024. https://openreview.net/forum?id=C4OpREezgj Alphazero-like tree-search can guide large language model decoding and training . In Forty-first International Conference on Machin...

    2024

  35. [43]

    Chaojie Wang, Yanchen Deng, Zhiyi Lv, Shuicheng Yan, and An Bo. 2024 a . Q*: Improving multi-step reasoning for llms with deliberative planning. arXiv preprint arXiv:2406.14283

    2024 arXiv

  36. [44]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023 a . https://doi.org/10.18653/v1/2023.acl-long.147 Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models . In Proceedings of the 61st Annua...

    2023 doi

  37. [45]

    Xiyao Wang, Linfeng Song, Ye Tian, Dian Yu, Baolin Peng, Haitao Mi, Furong Huang, and Dong Yu. 2024 b . Towards self-improvement of llms via mcts: Leveraging stepwise knowledge with curriculum preference learning. arXiv preprint arXiv:2410.06508

    2024 arXiv

  38. [46]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 b . https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Confer...

    2023

  39. [47]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022 a . https://openreview.net/forum?id=yzkSU...

    2022

  40. [48]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022 b . https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reason...

    2022

  41. [49]

    Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.167 Large language models are better reasoners with self-verification . In Findings of the Association for Computational L...

    2023 doi

  42. [50]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  43. [51]

    Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/81fde95c4dc79188a69ce5b24d63010b-Paper-Conference.pdf Self-evaluation guided beam search for reasoning . In Advances...

    2023

  44. [52]

    Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian-Guang Lou, and Shuai Ma. 2024. https://aclanthology.org/2024.emnlp-main.871 Re-reading improves reasoning in large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural...

    2024

  45. [53]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/271db9922b8d1f4dd7aaef84ed5ac703-Paper-Conference.pdf Tree of thoughts: Deliberate problem solving with large lan...

    2023

  46. [54]

    Xiaosong Yuan, Chen Shen, Shaotian Yan, Xiao Feng Zhang, Liang Xie, Wenxiao Wang, Renchu Guan, Ying Wang, and Jieping Ye. 2024. https://openreview.net/forum?id=31xWlIdxTm Instance-adaptive zero-shot chain-of-thought prompting . In The Thirty-eighth Annual Conference on Neural ...

    2024

  47. [55]

    Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2024. https://openreview.net/forum?id=UkFEs3ciz8 Progressive-hint prompting improves reasoning in large language models . In AI for Math Workshop @ ICML 2024

    2024

  48. [56]

    Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. 2023 a . Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921

    2023 arXiv

  49. [57]

    Jin Peng Zhou, Charles E Staats, Wenda Li, Christian Szegedy, Kilian Q Weinberger, and Yuhuai Wu. 2023 b . Don't trust: Verify--grounding llm quantitative reasoning with autoformalization. In The Twelfth International Conference on Learning Representations

    2023

  50. [58]

    Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V Le, Ed H Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. 2024. Self-discover: Large language models self-compose reasoning structures. arXiv preprint arXiv:2402.03620

    2024 arXiv

  51. [59]

    Jin Ziqi and Wei Lu. 2023. https://doi.org/10.18653/v1/2023.findings-acl.651 Tab- C o T : Zero-shot tabular chain of thought . In Findings of the Association for Computational Linguistics: ACL 2023, pages 10259--10277, Toronto, Canada. Association for Computational Linguistics

    2023 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.