{"id":"a73085d3-c482-4c35-92c4-48fff845412b","arxiv_id":"2504.12951","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Retrying failed attempts with fresh samples, called retrials without feedback, is a cheap and effective way to boost LLM reasoning, often outperforming complex methods like tree-of-thoughts.","lead":"Large language models can reason more effectively if they simply retry a problem when the first answer is wrong, without any self-criticism or feedback. The paper reports that simple chain-of-thought prompting with retries often beats fancier reasoning frameworks under the same budget.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HotpotQA results cannot be produced by the stated retrial mechanism: the paper's own Limitations section says answers are hidden and only used for evaluation, yet §3.3 retries 'unsolved' samples, which requires knowing the answer.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: for HotpotQA, the retrial mechanism cannot be implemented as stated because no verifier is available during inference. This is not a peripheral issue. The paper's central claim is that simple retrial-based methods outperform complex frameworks 'across all benchmarks,' and HotpotQA is one of only three benchmarks. If the retrial signal there is the oracle label, the reported HotpotQA curves are invalid evidence; if it is an unspecified proxy, the method is not the one defined in the abstract. Either way, the current manuscript does not support the HotpotQA portion of the claim. I considered the other potential concerns: the '$600' comparison is apples-to-oranges (GPT-4 for prior work vs GPT-4o-mini for this work) and should be softened, but it is a rhetorical cost comparison rather than the core evidence, which comes from same-model baselines in Figs. 1 and 4. The missing self-consistency baseline is also notable, but on verifiable tasks retrial-with-stopping dominates majority voting under a fixed sample budget, so it would not overturn the central claim if added. The HotpotQA problem is more fundamental because it is an internal contradiction explicitly acknowledged in the Limitations section, and the experiments do not respect that limitation. The rest of the paper—Game of 24 and HumanEval with deterministic verifiers—is plausible and honestly scoped in the Limitations paragraph, so outright rejection is not warranted. A conditional acceptance requiring clarification of the HotpotQA stopping rule, or removal of those results, is the appropriate outcome. This does not change the reader's verdict, so I recommend UNCHANGED.","tokens_in":7990,"tokens_out":4873,"duration_ms":48651,"concrete_test":"Ask the authors for the HotpotQA retry script/logs, or reconstruct the loop from the released code (if any). The decisive check: identify the exact condition that decides 'unsolved' and triggers the next trial for HotpotQA. If the oracle exact-match answer is used in that condition, rerun the benchmark with a verifier-free protocol (e.g., a fixed number of K attempts per question, no early stopping) and report the cost-quality curves; if they change materially, the HotpotQA-based conclusions should be removed or re-scoped. If an LLM-as-judge proxy was used, its accuracy on a held-out subset must be reported, since imperfect verification changes both success rate and cost.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The mechanism's only stopping/retry signal is 'incorrect answer identified' (Abstract; §3.3). For Game of 24 and HumanEval such a deterministic verifier exists. For HotpotQA the paper's own Limitations paragraph states 'the correct answer is hidden and only used for evaluation, making direct verification impossible during the solving process.' Yet §3.1 describes evaluating HotpotQA by exact match 'given an oracle answer,' and §3.3 says unsolved samples are re-tried until solved. These statements are inconsistent: to know which samples are 'unsolved' the pipeline must either consult the oracle labels during search (evaluation leakage) or use some unspecified proxy (an LLM verifier, self-consistency check, etc.) that is never described. No such proxy is defined in Appendix A, and no code is released. Consequently the HotpotQA panels in Figs. 1/4-6 cannot be interpreted as testing 'retrials without feedback,' and the summary claim that CoT 'consistently outperforms' alternatives 'across all benchmarks' rests on an invalid or unspecified experimental condition for one of the three tasks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces \"retrials without feedback,\" a simple iterative mechanism in which an LLM reasoning method retries a problem whenever an incorrect answer is identified, until a computational budget is exhausted. The authors compare IO prompting, Chain-of-Thought, Tree-of-Thoughts, and Reflexion on Game of 24, HumanEval, and HotpotQA using GPT-4o-mini and LLaMA-3.3-70B, reporting success rate or exact match against monetary cost. The central claims are that simpler retrial-based methods, especially CoT, are more cost-efficient than sophisticated frameworks such as ToT and Reflexion, and that higher temperature improves retrial performance, as shown in the reported figures. The paper also states a future direction of extending retrials to tasks without deterministic verifiers.","tokens_in":8239,"tokens_out":3859,"duration_ms":40778,"significance":"If the empirical findings hold, the paper makes a useful practical point: under a retry budget, the added complexity of tree search and verbal self-reflection may not justify its cost. The study is straightforward and the price table makes cost calculations transparent. However, the novelty of the mechanism is limited, since retry-until-success with a deterministic verifier is closely related to repeated sampling with a verifier (e.g., best-of-k sampling), and no comparison to such natural baselines is included. The manuscript also explicitly acknowledges in its Limitations section that the method cannot be applied to tasks without direct answer verification, which is a significant scope restriction. Nevertheless, the cost-efficiency comparison for tasks with deterministic verifiers is a valuable practical contribution, provided the reported experiments are internally consistent.","major_comments":[{"comment":"The HotpotQA results cannot be produced by the stated retrial mechanism. Section 3.3 says \"Unsolved samples are re-tried in subsequent trials,\" which requires knowing, during the solving process, whether a sample is solved. For HotpotQA, however, the Limitations paragraph states that \"the correct answer is hidden and only used for evaluation, making direct verification impossible during the solving process.\" Section 3.1 also describes evaluation by exact match \"given an oracle answer.\" These statements are inconsistent: to know which samples are unsolved, the pipeline must either consult the oracle labels during search (evaluation leakage) or use an unspecified proxy such as an LLM verifier or self-consistency check, which is never described in the paper or Appendix A. Consequently, the HotpotQA panels in Figures 1, 4, 5, and 6 cannot be interpreted as testing \"retrials without feedback,\" and the claim in Section 4 that CoT \"consistently outperforms\" alternatives \"across all benchmarks\" rests on an invalid or underspecified experimental condition for one of the three tasks. This issue is load-bearing and should be fixed, for example by removing HotpotQA from the main claims or by explicitly introducing and validating an internal verifier for it.","section":"§3.3 vs. Limitations"},{"comment":"The headline cost comparison is not apples-to-apples. Section 4 states that CoT achieved a 94% success rate on Game of 24 and that methods such as RAFA \"would need multiple hundreds of dollars (≃600$) to achieve on GPT-4.\" The reported 94% result is obtained with GPT-4o-mini, while the ≃600$ cost refers to RAFA on GPT-4. Since the two base models differ in price and capability, this comparison conflates method cost with model cost. A controlled comparison on the same base model is needed before claiming that retrials with CoT make the RAFA-style cost unnecessary.","section":"§4"},{"comment":"The paper reports no statistical error bars or repeated-run variance, yet Section 4 makes strong comparative claims such as \"consistently outperforms\" across all benchmarks. The underlying samples are stochastic (temperature sampling), and the evaluation sets are small (100 samples for Game of 24 and HotpotQA), so the reported differences may not be reliable. In addition, the \"indicated budget\" is never stated numerically anywhere in the paper; this prevents the reader from reproducing the cost curves or verifying that methods were compared under identical budgets. The authors should report budget values and, where feasible, confidence intervals or multiple seeds.","section":"Figures 1-6 and §3.3"}],"minor_comments":[{"comment":"The phrase \"without feedback\" is potentially misleading because the retrial mechanism still relies on an external deterministic verifier to decide whether an answer is correct; this is a form of feedback, even if it is not verbalized. The authors should clarify this distinction early in the paper.","section":"Abstract and title"},{"comment":"Given that retrials with a verifier are closely related to repeated sampling, the paper should discuss and ideally compare against self-consistency (CoT-SC) and best-of-k sampling as baselines. The current comparison with ToT and Reflexion only shows that complex frameworks are not needed; it does not establish that retrials offer an advantage over simpler sampling-based baselines.","section":"§2, Related Work"},{"comment":"There are several typographical errors: \"were were\" and \"thhe\" appear in the first paragraph, and \"UsingLlama\" appears in the caption of Figure 4. These should be corrected.","section":"Appendix C"},{"comment":"The price table lists \"current model prices\" without stating the date or snapshot; since API prices change over time, the cost figures should include the exact price as of the experiment period to ensure reproducibility.","section":"Table 1 and cost calculations"},{"comment":"No code or data release is mentioned. Given the simplicity of the method, providing code with the exact prompts and budgets would substantially improve reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The HotpotQA inconsistency is the main technical obstruction: the paper's own Limitations section admits that direct verification is impossible for that benchmark, yet the experimental protocol in §3.3 requires exactly that. This makes one of the three benchmark claims uninterpretable. The novelty is modest—the retrial idea is essentially repeated sampling with a verifier—but the cost-efficiency comparison is still valuable for practitioners if the reported experiments are made internally consistent. I recommend major revision rather than rejection because the central insight for tasks with deterministic verifiers (e.g., Game of 24 and HumanEval) is plausible and the issues appear fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the punchline: the paper's 'retrials without feedback' is rejection sampling/best-of-n with early stopping, which is not new. What is worth reading is the cost-quality comparison: under the same retrial budget, simple CoT and IO beat ToT and Reflexion on Game of 24 and HumanEval. That is plausible and useful for practitioners.\n\nThe paper does several things right. It picks two base models, uses API prices to estimate real dollar costs, and plots quality against cost and trials. The result that CoT with retries reaches 94% on Game of 24, while RAFA/ToT reportedly needed hundreds of dollars on GPT-4, is striking even though the comparison uses different models and pricing. The message that elaborate self-refinement may not be worth the extra tokens is worth airing.\n\nNow the soft spots. The big one is HotpotQA. The retry mechanism needs a deterministic verifier. The paper's own Limitations section says for HotpotQA 'the correct answer is hidden and only used for evaluation, making direct verification impossible during the solving process.' But §3.3 says 'unsolved samples are re-tried in subsequent trials.' To know which samples are unsolved, the pipeline must either peek at the oracle labels or use an unspecified proxy. No proxy is described, no code is released. So the HotpotQA panels in Figs. 1/4-6 cannot be interpreted as testing the stated mechanism. The summary claim that CoT 'consistently outperforms' across all benchmarks rests on an invalid condition for one of three tasks. This is not a minor blemish; it needs to be fixed, either by dropping HotpotQA or by specifying and implementing a verifier.\n\nThe second soft spot is novelty. Retrying without feedback is equivalent to best-of-n sampling, and the paper cites self-consistency but never runs it as a baseline. That omission makes it hard to conclude that retrials are better than a stronger existing method.\n\nThird, the evaluation has no error bars, the budget values are not stated numerically, and the RAFA cost comparison uses GPT-4 vs GPT-4o-mini, so the 'multiple hundreds of dollars' contrast is apples-to-oranges.\n\nFourth, no code release.\n\nNet: the paper as-is should not be accepted as a strong claim across all benchmarks. But the core finding on deterministic-verifier tasks is likely true, and the cost lens is a useful corrective to the field's enthusiasm for complex reasoning frameworks. This deserves peer review with major revisions: a serious editor should send it out, not desk reject, because the empirical question is real and the HotpotQA problem is fixable.\n\nWho it's for: practitioners choosing a cheap reasoning deployment, and researchers working on iterative feedback who need to know that simple retries erode much of the advantage of verbalized feedback. I would not cite it in its current form, but I'd bring it to a reading group to discuss the evaluation pitfall.","headline":"A cost-efficiency comparison that mostly re-discovers best-of-n sampling; the HotpotQA results are contradicted by the paper's own limitations and need fixing, but the deterministic-verifier findings are worth a serious look.","tokens_in":8723,"tokens_out":4522,"would_cite":false,"duration_ms":41088,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that \"retrials without feedback\"—simply re-attempting a problem after a wrong answer—let plain chain-of-thought prompting outperform costly tree-search and self-reflection frameworks.","keywords":["large language models","retrials without feedback","chain-of-thought prompting","cost-efficient reasoning","iterative refinement","Tree-of-Thoughts","Reflexion","reasoning benchmarks"],"falsifier":"Take the same retrial protocol on HotpotQA but trigger retries with the model's own confidence or a self-consistency vote instead of an oracle check; if accuracy collapses or cost explodes, the low-cost gains do not extend to non-verifiable tasks. Alternatively, on Game of 24, give an untriggered baseline the same number of unconditional retries; if it matches the triggered version, the \"identify incorrect answer\" step is not what drives the result.","tokens_in":7819,"feed_emoji":"🔁","tokens_out":8420,"duration_ms":77876,"temperature":0.7,"pith_summary":"This paper argues that much of the expensive machinery in iterative reasoning frameworks is unnecessary: an LLM that simply retries a problem from scratch whenever it produces a wrong answer captures most of the benefit at a fraction of the cost. Under a fixed retrial budget, plain Input-Output prompting and Chain-of-Thought prompting outperform Tree-of-Thoughts and Reflexion on Game of 24, HumanEval, and HotpotQA, with both GPT-4o-mini and LLaMA-3.3-70B as base models. The headline result is 94% success on Game of 24 using Chain-of-Thought with retrials, a level the authors say earlier methods reach only after spending hundreds of dollars on GPT-4. The paper reframes evaluation of reasoning frameworks around cost-quality curves rather than single accuracy numbers.","feed_headline":"Retrials alone beat elaborate LLM reasoning frameworks at lower cost","feed_subtitle":"Under a fixed budget, chain-of-thought with simple retries outperforms tree-of-thoughts and Reflexion on tested tasks.","key_machinery":"The machinery is the retrial loop itself: a deterministic verifier checks each proposed output, and any incorrect answer triggers a fresh, context-free retry until the budget is exhausted. This replaces verbalized self-reflection (as in Reflexion) and explicit tree search (as in Tree-of-Thoughts) with repeated independent sampling. The flat per-attempt cost is the load-bearing property: refinement methods grow their context window on every iteration, so their cost climbs with each correction, while a retry costs no more than the original attempt. Temperature becomes a search parameter, since higher temperatures generate more diverse fresh attempts.","core_discovery":"The central claim is that performance under a retrial budget inverts the usual ranking of reasoning methods. Rather than asking the model to evaluate its own thoughts or generate verbal feedback, the retrial mechanism uses a deterministic check to detect a wrong answer, discards that attempt, and starts over with a fresh prompt. Because the context window is not allowed to grow between attempts, each retry costs roughly the same as the first, and simpler methods inherit the model's native reasoning ability without paying for elaborate scaffolding. Across all three benchmarks and both base models, Chain-of-Thought with retrials is the most cost-efficient strategy, reaching 94% success on Game of 24.","pith_inferences":["Editorial inference: retrials here act like randomized restarts; because each attempt is independent, the method's power may come from sampling diversity rather than from guided improvement, which a direct comparison against unconditional re-sampling at the same budget would test.","Editorial inference: the results imply self-reflection earns its cost mainly when no deterministic verifier exists or when the budget is too small for many retries; on verifiable tasks, accumulated feedback may be actively counterproductive for cost efficiency.","Editorial inference: a natural extension is to combine retrials with majority voting over attempts, which may recover the sample efficiency Tree-of-Thoughts shows in the paper's retrial-count plots while keeping the flat cost curve.","Editorial inference: for tasks with hidden answers, the method needs an approximate verifier such as self-consistency or a proxy test; whether the same cost advantage survives that substitution is an open empirical question."],"forward_implications":["Any verifiable task can be wrapped in retrials without redesigning the underlying prompts, so the mechanism is a drop-in efficiency upgrade for existing strategies.","Cost-quality curves should replace single accuracy numbers when comparing reasoning methods, because the ranking can flip once budget is held fixed.","On Game of 24 with GPT-4o-mini, raising temperature improves the retrial success rate, and Chain-of-Thought reaches full success at roughly half its allocated budget at temperature 1.0.","The advantage of simple retrial methods is strongest when the base model is already a strong reasoner, so model choice and task choice shape the cost-efficiency ranking."],"supporting_citations":[{"why":"Introduces Chain-of-Thought prompting, the simple strategy that the paper's retrial mechanism turns into the most cost-efficient method.","marker":"Wei et al., 2022"},{"why":"Introduces Tree-of-Thoughts, one of the complex baselines outperformed, and supplies the Game of 24 test split used for the headline result.","marker":"Yao et al., 2024"},{"why":"Introduces Reflexion, the verbalized-feedback refinement baseline that retrials improve upon without the feedback.","marker":"Shinn et al., 2023"},{"why":"Provides the RAFA system cited for the roughly $600 cost that prior refinement methods need on Game of 24.","marker":"Liu et al., 2024"},{"why":"Documents LLaMA-3.3-70B, one of the two base models used to show the retrial advantage is not model-specific.","marker":"Grattafiori et al., 2024"},{"why":"Introduces HotpotQA, one of the three benchmarks, and the case where the retrial trigger is empirically unclear.","marker":"Zhilin et al., 2018"},{"why":"Documents GPT-4o-mini, the base model for the main cost-quality figures.","marker":"OpenAI and et al., 2024"}],"fun_headline_variants":["Retrials beat intricate reasoning at lower cost","Simple retries outperform tree-of-thoughts","Retrials without feedback top complex reasoning","Why retrials alone outshine Reflexion and ToT","Retrial-only LLM reasoning rivals complex methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a cheap, reliable way to recognize a wrong answer exists during the solving process itself; the paper does not explain how retrials are triggered on HotpotQA, where the correct answer is hidden until evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Retrials beat intricate reasoning at lower cost","Simple retries outperform tree-of-thoughts","Retrials without feedback top complex reasoning","Why retrials alone outshine Reflexion and ToT","Retrial-only LLM reasoning rivals complex methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1138,"prompt_tokens":892,"completion_tokens":246,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":173}},"tokens_in":508,"tokens_out":246,"duration_ms":2815,"temperature":1.0,"reasoning_tokens":173,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:17:36.626963+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same retrial protocol on HotpotQA but trigger retries with the model's own confidence or a self-consistency vote instead of an oracle check; if accuracy collapses or cost explodes, the low-cost gains do not extend to non-verifiable tasks. Alternatively, on Game of 24, give an untriggered baseline the same number of unconditional retries; if it matches the triggered version, the \"identify incorrect answer\" step is not what drives the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Tree-of-Thoughts, one of the complex baselines outperformed, and supplies the Game of 24 test split used for the headline result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Reflexion, the verbalized-feedback refinement baseline that retrials improve upon without the feedback."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RAFA system cited for the roughly $600 cost that prior refinement methods need on Game of 24."}],"review_version":1}