Pith. sign in

REVIEW 3 major objections 5 minor 36 references

Are Retrials All You Need? Enhancing Large Language Model Reasoning Without Verbalized Feedback

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that "retrials without feedback"—simply re-attempting a problem after a wrong answer—let plain chain-of-thought prompting outperform costly tree-search and self-reflection frameworks.

desk verdict A cost-efficiency comparison that mostly re-discovers best-of-n sampling; the HotpotQA results are contradicted by the paper's own limitations and need fixing, but the deterministic-verifier findings are worth a serious look. read the letter →

arxiv 2504.12951 v1 pith:OYE5E5AC submitted 2025-04-17 cs.CL cs.AIcs.LG

Nearchos Potamitis , Akhil Arora This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords largelanguagemodelsretrialswithoutfeedbackchain-of-thoughtpromptingcost-efficientreasoningiterativerefinementTree-of-ThoughtsReflexionbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that much of the expensive machinery in iterative reasoning frameworks is unnecessary: an LLM that simply retries a problem from scratch whenever it produces a wrong answer captures most of the benefit at a fraction of the cost. Under a fixed retrial budget, plain Input-Output prompting and Chain-of-Thought prompting outperform Tree-of-Thoughts and Reflexion on Game of 24, HumanEval, and HotpotQA, with both GPT-4o-mini and LLaMA-3.3-70B as base models. The headline result is 94% success on Game of 24 using Chain-of-Thought with retrials, a level the authors say earlier methods reach only after spending hundreds of dollars on GPT-4. The paper reframes evaluation of reasoning frameworks around cost-quality curves rather than single accuracy numbers.

What carries the argument

The machinery is the retrial loop itself: a deterministic verifier checks each proposed output, and any incorrect answer triggers a fresh, context-free retry until the budget is exhausted. This replaces verbalized self-reflection (as in Reflexion) and explicit tree search (as in Tree-of-Thoughts) with repeated independent sampling. The flat per-attempt cost is the load-bearing property: refinement methods grow their context window on every iteration, so their cost climbs with each correction, while a retry costs no more than the original attempt. Temperature becomes a search parameter, since higher temperatures generate more diverse fresh attempts.

What would settle it

Take the same retrial protocol on HotpotQA but trigger retries with the model's own confidence or a self-consistency vote instead of an oracle check; if accuracy collapses or cost explodes, the low-cost gains do not extend to non-verifiable tasks. Alternatively, on Game of 24, give an untriggered baseline the same number of unconditional retries; if it matches the triggered version, the "identify incorrect answer" step is not what drives the result.

Watch

Extended reading notes

Core claim

The central claim is that performance under a retrial budget inverts the usual ranking of reasoning methods. Rather than asking the model to evaluate its own thoughts or generate verbal feedback, the retrial mechanism uses a deterministic check to detect a wrong answer, discards that attempt, and starts over with a fresh prompt. Because the context window is not allowed to grow between attempts, each retry costs roughly the same as the first, and simpler methods inherit the model's native reasoning ability without paying for elaborate scaffolding. Across all three benchmarks and both base models, Chain-of-Thought with retrials is the most cost-efficient strategy, reaching 94% success on Game of 24.

Load-bearing premise

The load-bearing premise is that a cheap, reliable way to recognize a wrong answer exists during the solving process itself; the paper does not explain how retrials are triggered on HotpotQA, where the correct answer is hidden until evaluation.

Editorial extensions

If this is right

  • Any verifiable task can be wrapped in retrials without redesigning the underlying prompts, so the mechanism is a drop-in efficiency upgrade for existing strategies.
  • Cost-quality curves should replace single accuracy numbers when comparing reasoning methods, because the ranking can flip once budget is held fixed.
  • On Game of 24 with GPT-4o-mini, raising temperature improves the retrial success rate, and Chain-of-Thought reaches full success at roughly half its allocated budget at temperature 1.0.
  • The advantage of simple retrial methods is strongest when the base model is already a strong reasoner, so model choice and task choice shape the cost-efficiency ranking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: retrials here act like randomized restarts; because each attempt is independent, the method's power may come from sampling diversity rather than from guided improvement, which a direct comparison against unconditional re-sampling at the same budget would test.
  • Editorial inference: the results imply self-reflection earns its cost mainly when no deterministic verifier exists or when the budget is too small for many retries; on verifiable tasks, accumulated feedback may be actively counterproductive for cost efficiency.
  • Editorial inference: a natural extension is to combine retrials with majority voting over attempts, which may recover the sample efficiency Tree-of-Thoughts shows in the paper's retrial-count plots while keeping the flat cost curve.
  • Editorial inference: for tasks with hidden answers, the method needs an approximate verifier such as self-consistency or a proxy test; whether the same cost advantage survives that substitution is an open empirical question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces "retrials without feedback," a simple iterative mechanism in which an LLM reasoning method retries a problem whenever an incorrect answer is identified, until a computational budget is exhausted. The authors compare IO prompting, Chain-of-Thought, Tree-of-Thoughts, and Reflexion on Game of 24, HumanEval, and HotpotQA using GPT-4o-mini and LLaMA-3.3-70B, reporting success rate or exact match against monetary cost. The central claims are that simpler retrial-based methods, especially CoT, are more cost-efficient than sophisticated frameworks such as ToT and Reflexion, and that higher temperature improves retrial performance, as shown in the reported figures. The paper also states a future direction of extending retrials to tasks without deterministic verifiers.

Significance. If the empirical findings hold, the paper makes a useful practical point: under a retry budget, the added complexity of tree search and verbal self-reflection may not justify its cost. The study is straightforward and the price table makes cost calculations transparent. However, the novelty of the mechanism is limited, since retry-until-success with a deterministic verifier is closely related to repeated sampling with a verifier (e.g., best-of-k sampling), and no comparison to such natural baselines is included. The manuscript also explicitly acknowledges in its Limitations section that the method cannot be applied to tasks without direct answer verification, which is a significant scope restriction. Nevertheless, the cost-efficiency comparison for tasks with deterministic verifiers is a valuable practical contribution, provided the reported experiments are internally consistent.

major comments (3)
  1. [§3.3 vs. Limitations] The HotpotQA results cannot be produced by the stated retrial mechanism. Section 3.3 says "Unsolved samples are re-tried in subsequent trials," which requires knowing, during the solving process, whether a sample is solved. For HotpotQA, however, the Limitations paragraph states that "the correct answer is hidden and only used for evaluation, making direct verification impossible during the solving process." Section 3.1 also describes evaluation by exact match "given an oracle answer." These statements are inconsistent: to know which samples are unsolved, the pipeline must either consult the oracle labels during search (evaluation leakage) or use an unspecified proxy such as an LLM verifier or self-consistency check, which is never described in the paper or Appendix A. Consequently, the HotpotQA panels in Figures 1, 4, 5, and 6 cannot be interpreted as testing "retrials without feedback," and the claim in Section 4 that CoT "consistently outperforms" alternatives "across all benchmarks" rests on an invalid or underspecified experimental condition for one of the three tasks. This issue is load-bearing and should be fixed, for example by removing HotpotQA from the main claims or by explicitly introducing and validating an internal verifier for it.
  2. [§4] The headline cost comparison is not apples-to-apples. Section 4 states that CoT achieved a 94% success rate on Game of 24 and that methods such as RAFA "would need multiple hundreds of dollars (≃600$) to achieve on GPT-4." The reported 94% result is obtained with GPT-4o-mini, while the ≃600$ cost refers to RAFA on GPT-4. Since the two base models differ in price and capability, this comparison conflates method cost with model cost. A controlled comparison on the same base model is needed before claiming that retrials with CoT make the RAFA-style cost unnecessary.
  3. [Figures 1-6 and §3.3] The paper reports no statistical error bars or repeated-run variance, yet Section 4 makes strong comparative claims such as "consistently outperforms" across all benchmarks. The underlying samples are stochastic (temperature sampling), and the evaluation sets are small (100 samples for Game of 24 and HotpotQA), so the reported differences may not be reliable. In addition, the "indicated budget" is never stated numerically anywhere in the paper; this prevents the reader from reproducing the cost curves or verifying that methods were compared under identical budgets. The authors should report budget values and, where feasible, confidence intervals or multiple seeds.
minor comments (5)
  1. [Abstract and title] The phrase "without feedback" is potentially misleading because the retrial mechanism still relies on an external deterministic verifier to decide whether an answer is correct; this is a form of feedback, even if it is not verbalized. The authors should clarify this distinction early in the paper.
  2. [§2, Related Work] Given that retrials with a verifier are closely related to repeated sampling, the paper should discuss and ideally compare against self-consistency (CoT-SC) and best-of-k sampling as baselines. The current comparison with ToT and Reflexion only shows that complex frameworks are not needed; it does not establish that retrials offer an advantage over simpler sampling-based baselines.
  3. [Appendix C] There are several typographical errors: "were were" and "thhe" appear in the first paragraph, and "UsingLlama" appears in the caption of Figure 4. These should be corrected.
  4. [Table 1 and cost calculations] The price table lists "current model prices" without stating the date or snapshot; since API prices change over time, the cost figures should include the exact price as of the experiment period to ensure reproducibility.
  5. [Reproducibility] No code or data release is mentioned. Given the simplicity of the method, providing code with the exact prompts and budgets would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; the empirical comparison is self-contained. The HotpotQA retrial trigger is an experimental-validity flaw, not a circular step.

full rationale

Circularity analysis: this is an empirical study with no fitted equations and no parameter-derived predictions, so the standard circularity patterns do not arise. The core comparison—IO, CoT, ToT, and Reflexion under a retrial budget—is a direct measurement of success rate and dollar cost from API execution on Game of 24, HumanEval, and HotpotQA; the reported quantities are read off the runs rather than derived from the inputs. The only self-citation, Potamitis et al. (2024), appears in a list of prior prompting frameworks in the Introduction and is not load-bearing for any conclusion. The retrial mechanism does reduce to repeated sampling without feedback, which is a novelty and positioning concern rather than a circularity concern. One validity issue must be flagged: in section 3.3 the paper says 'Unsolved samples are re-tried in subsequent trials,' while the Limitations section states that for HotpotQA 'the correct answer is hidden and only used for evaluation, making direct verification impossible during the solving process.' No proxy verifier is described, so the HotpotQA retrials are either oracle-dependent (evaluation leakage) or undefined. This is an experimental-design flaw that threatens the validity of the HotpotQA comparison, but it is not a circular derivation of the paper's main claims. Therefore the circularity score is 2, reflecting only a minor non-load-bearing self-citation; no circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on experimental comparisons rather than derivations. The key assumptions are the existence of deterministic verifiers, the fairness of dollar-budget comparisons, and the applicability of retrials to tasks without verifiers. Temperature and budget are hand-selected experimental parameters that affect the reported curves.

free parameters (2)
  • temperature = 0.2, 0.5, 1.0 (per experiment)
    Temperature is a hyperparameter varied in the temperature analysis; the main results do not state which temperature was used, but the success rate is strongly affected.
  • budget (max cost per benchmark)
    The cost budget determines when retrials stop; figures show curves but no explicit budget values are reported in the text.
assumptions (4)
  • domain assumption Game of 24 and HumanEval have deterministic verifiers that can identify correct answers during the retrial loop.
    Stated in Section 3.3 and repeated in Limitations; central to the retrial mechanism.
  • domain assumption HotpotQA can be used with the retrial mechanism even though the correct answer is hidden and cannot be verified during solving.
    This is contradicted by the Limitations section, making the experimental setup unclear.
  • domain assumption Dollar cost is a valid measure of computational complexity for comparing reasoning frameworks.
    All comparisons in the paper are cost-quality curves; the choice of API prices determines the results.
  • domain assumption Running each method under the same total dollar budget is a fair comparison even though methods spend tokens differently per attempt.
    Assumed throughout Section 3.3 and 4; not tested against alternative budget metrics like number of tokens or attempts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Retrials All You Need? Enhancing Large Language Model Reasoning Without Verbalized Feedback." pith.science (2026). https://pith.science/paper/OYE5E5AC

@misc{pith2026250412951,
  author       = {Pith},
  title        = {Pith review of: Are Retrials All You Need? Enhancing Large Language Model Reasoning Without Verbalized Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OYE5E5AC}},
  note         = {Machine review of arXiv:2504.12951}
}
read the original abstract

Recent advancements in large language models (LLMs) have catalyzed the development of general-purpose autonomous agents, demonstrating remarkable performance in complex reasoning tasks across various domains. This surge has spurred the evolution of a plethora of prompt-based reasoning frameworks. A recent focus has been on iterative reasoning strategies that refine outputs through self-evaluation and verbalized feedback. However, these strategies require additional computational complexity to enable models to recognize and correct their mistakes, leading to a significant increase in their cost. In this work, we introduce the concept of ``retrials without feedback'', an embarrassingly simple yet powerful mechanism for enhancing reasoning frameworks by allowing LLMs to retry problem-solving attempts upon identifying incorrect answers. Unlike conventional iterative refinement methods, our method does not require explicit self-reflection or verbalized feedback, simplifying the refinement process. Our findings indicate that simpler retrial-based approaches often outperform more sophisticated reasoning frameworks, suggesting that the benefits of complex methods may not always justify their computational costs. By challenging the prevailing assumption that more intricate reasoning strategies inherently lead to better performance, our work offers new insights into how simpler, more efficient approaches can achieve optimal results. So, are retrials all you need?

Figures

Figures reproduced from arXiv: 2504.12951 by the authors.

Figure 1
Figure 1. Comparing the cost-quality trade-off of IO, CoT, ToT, and Reflexion using [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparing the cost-quality trade-off of CoT and ToT across different temperature levels using [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparing the cost-quality trade-off of CoT [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparing the cost-quality trade-off of IO, CoT, ToT, and Reflexion using [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparing the sample-quality trade-off of IO, CoT, ToT, and Reflexion using [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparing the sample-quality trade-off of IO, CoT, ToT, and Reflexion using [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 20 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    arXiv 2024

  4. [4]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. https://arxiv.org/abs/2305.10403 Palm 2 technical report . arXiv preprint arXiv:2305.10403

    arXiv 2023

  5. [5]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In AAAI, pages 17682--17690

    work page 2024

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS, volume 33, pages 1877--1901

    work page 2020

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168

    arXiv 2021

  8. [8]

    Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. 2022. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In NeurIPS: Datasets and Benchmarks Track

    work page 2022

Show all 36 references
  1. [9]

    Aaron Grattafiori et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. In EMNLP

    2023

  3. [11]

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. https://openreview.net/forum?id=_nGgzQjzaRy Decomposed prompting: A modular approach for solving complex tasks . In ICLR

    2023

  4. [12]

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In NeurIPS, pages 22199--22213

    2022

  5. [13]

    Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. 2024. Reason for future, act for now: A principled architecture for autonomous LLM agents. In ICML

    2024

  6. [14]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651

    2023 arXiv

  7. [15]

    Maxwell I. Nye, Anders Johan Andreassen, Guy Gur - Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. https://arxiv.org/abs/2112.00114 Show your work: Scratchpads for intermedi...

    2021 arXiv

  8. [16]

    OpenAI and Hurst et al. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

    2024 arXiv

  9. [17]

    Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. https://arxiv.org/abs/2304.01904 Refiner: Reasoning feedback on intermediate representations . arXiv preprint arXiv:2304.01904

    2023 arXiv

  10. [18]

    Nearchos Potamitis, Lars Klein, Roland Aydin, Caglar Gulcehre, Robert West, and Akhil Arora. 2024. https://arxiv.org/abs/2405.06691 Fleet of agents: Coordinated problem solving with large language models . Preprint, arXiv:2405.06691

    2024 arXiv

  11. [19]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In NeurIPS

    2023

  12. [20]

    Bilgehan Sel, Ahmad Tawaha, Vanshaj Khattar, Ruoxi Jia, and Ming Jin. 2024. Algorithm of thoughts: Enhancing exploration of ideas in large language models. In ICML

    2024

  13. [21]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In NeurIPS, pages 8634--8652

    2023

  14. [22]

    Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2023. Adaplanner: Adaptive planning from feedback with language models. In NeurIPS, volume 36, pages 58202--58245

    2023

  15. [23]

    Mirac Suzgun and Adam Tauman Kalai. 2024. https://arxiv.org/abs/2401.12954 Meta-prompting: Enhancing language models with task-agnostic scaffolding . arXiv preprint arXiv:2401.12954

    2024 arXiv

  16. [24]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971

    2023 arXiv

  17. [25]

    Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al

    Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  18. [26]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  19. [27]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In ICLR

    2023

  20. [28]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, pages 24824--24837

    2022

  21. [29]

    Robert West, Joelle Pineau, and Doina Precup. 2009. Wikispeedia: An online game for inferring semantic distances between concepts. In IJCAI, page 1598–1603

    2009

  22. [30]

    Gonzalez, and Bin Cui

    Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E. Gonzalez, and Bin Cui. 2024. https://openreview.net/forum?id=ANO1i9JPtb Buffer of thoughts: Thought-augmented reasoning with large language models . In NeurIPS

    2024

  23. [31]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. In NeurIPS, pages 20744--20757

    2022

  24. [32]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. NeurIPS, 36

    2024

  25. [33]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models . In ICLR

    2023

  26. [34]

    Yang Zhilin, Qi Peng, Zhang Saizheng, Bengio Yoshua, Cohen William, Salakhutdinov Ruslan, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/d18-1259 Hotpotqa: A dataset for diverse, explainable multi-hop question answering . Proceedings of the 2018 Conference on Em...

    2018 doi

  27. [35]

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2024. Language agent tree search unifies reasoning, acting, and planning in language models. In ICML

    2024

  28. [36]

    Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. https://openreview.net/forum?id=WZH7099tgfM Least-to-most prompting enables complex reasoning in large language mode...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.