{"id":"152ee090-2b58-44ef-9fb8-2446096dd91f","arxiv_id":"2501.13122","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"COT STEP, a zero-shot prompt that forces numbered reasoning steps, enables competitive accuracy and step-level self-verification, but verifier-guided search does not beat self-consistency.","lead":"A new zero-shot prompt, COT STEP, makes large language models number their reasoning steps, enabling automatic step decomposition without examples. Testing on five benchmarks and four open models, the authors find self-verification helps a greedy search but does not beat plain self-consistency.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2 verifier accuracy is computed against answer-match proxy labels; if unfaithful chains are common, the 83.9% COTR-prompt claim is not evidence of step-level verification.","rationale":"The paper's headline 'COT STEP offers an elegant zero-shot strategy to decompose reasoning steps virtually without any accuracy loss compared to COT' is an RQ1 claim supported by Table 1, and it does not rely on the verifier labels. The reader's strongest_claim, however, bundles in the COTR verifier result of 83.9% on GSM8K chain-level classification, and that result is exactly where the proxy-label assumption is load-bearing. I considered other candidates: the lack of error bars on Table 1, the fragile 'A)' answer-extraction heuristic, and the inconsistency in the SG result on SOLAR StrategyQA. Error bars would help quantify 'virtually no loss,' but the reported differences are small and that claim is not the central scientific risk; the label issue is a semantic mismatch between the claimed capability (step verification) and the measured target (answer correctness). The search experiments in RQ3 are evaluated on final task accuracy, so they are not confounded by the proxy, and they honestly report that verifier gains vanish under self-consistency. Therefore the concern, while real and load-bearing for the verification-capability claim, does not change the conditional verdict; it reinforces it. I agree with the reader's weakest_assumption, and the proposed human-labeling test would settle whether the proxy materially inflates the reported verification accuracy.","tokens_in":21401,"tokens_out":7156,"duration_ms":72215,"concrete_test":"Take 200-300 chains from the GSM8K/COT STEP/SOLAR pool used in Table 2. Have two annotators label every step correct/incorrect and every chain faithful or unfaithful. Recompute COTR-prompt chain-level accuracy and F1 against the human step-verified chain labels, preserving the same 1:1 positive/negative balance. If accuracy falls materially below 83.9 (for example, by more than about 5 points), the reported verifier accuracy is substantially an artifact of the answer-match proxy; if the number is stable, the proxy is validated for this setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the automatic correctness label used for the RQ2 verifier. In Section 4.3, a reasoning chain is labeled correct if and only if its extracted final answer matches the dataset gold answer. But Section 4.2 defines chain-level correctness as all intermediate reasoning steps being correct and the final answer matching. The label therefore cannot distinguish a chain whose final answer is correct because every step is correct from one that reaches the right answer through an incorrect or unfaithful intermediate step. Section 8 acknowledges this and cites Lanham et al. (2023). Since the positive class in the balanced 1,000/1,000 datasets is defined by answer match, a verifier that mostly checks the final answer can score well without actually verifying intermediate steps. Table 2's headline number (SOLAR + COTR-prompt on GSM8K: 83.90 accuracy, 85.26 F1) is thus not strong evidence for the conclusion that the zero-shot COTR-prompt is useful for step-level verification in the mathematical domain. This does not undermine the COT STEP comparative accuracy claim (Table 1, RQ1) or the RQ3 search results, which are evaluated by final task accuracy, but it does weaken the verification-capability claim. Because the authors disclose the limitation, this is a correctness-risk issue rather than an omission.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes and evaluates zero-shot prompts for chain-of-thought reasoning: COT STEP, a modification of the standard zero-shot COT prompt that forces numbered step outputs, and two self-verification prompts (R-prompt and COTR-prompt) that score the correctness of the last reasoning step. The authors run three studies: (RQ1) comparison of COT STEP against existing zero-shot prompts on five benchmarks and two LLMs; (RQ2) evaluation of the verifiers on a self-constructed balanced dataset where a reasoning chain is labeled correct iff its final answer matches the gold answer; (RQ3) use of verifier scores to augment self-consistency and to guide step-wise greedy/beam search. Findings: COT STEP matches or slightly exceeds standard COT while giving cleanly separated steps; the COTR-prompt achieves high accuracy on the chain-level correctness classification task for math questions but not commonsense; step-wise greedy search improves over plain COT STEP when self-consistency is not used, but the benefit disappears under self-consistency.","tokens_in":21689,"tokens_out":8498,"duration_ms":82124,"significance":"The paper's main strengths are its honest reporting of negative results (verifier scores do not help self-consistency; beam search does not beat greedy) and its simple, reproducible prompt designs. COT STEP is a practical contribution: it is a drop-in zero-shot prompt that yields structured step outputs without hurting accuracy. The empirical comparison across five prompts, five datasets, and multiple LLMs is useful. However, the evaluation of the verifier's step-level verification capability rests on a final-answer-match proxy, and the comparative claims lack variance estimates. If the verification claim is reframed or re-evaluated with step-level labels, the paper would make a solid empirical contribution to zero-shot reasoning and self-verification.","major_comments":[{"comment":"The automatic ground-truth labels for the correctness classification task are assigned by comparing the extracted final answer to the dataset gold answer (Section 4.3), yet Section 4.2 defines chain-level correctness as 'all intermediate reasoning steps are correct and the final answer matches.' These two notions are not equivalent: a chain can reach the correct final answer through incorrect or unfaithful intermediate steps. As a result, the reported verifier accuracies (e.g., SOLAR + COTR-prompt at 83.90% on (COT STEP, GSM8K)) measure the verifier's ability to predict final-answer match, not its ability to detect step-level errors. The limitation is acknowledged in Section 8, but the conclusion still states that the COTR-prompt is 'useful for verification particularly in mathematical domain' (takeaway 3). To support a step-level verification claim, the authors should either evaluate the verifier on a subset with step-level ground truth (e.g., PRM800K-style annotations or a human-rated sample) or explicitly restate the RQ2 claim as 'classifying whether the final answer is correct.'","section":"§4.2–4.3, Tables 2 and 7"},{"comment":"The verifier's binary prediction is obtained by checking whether the substring 'A)' appears in the generated response. For the COTR-prompt, the maximum generation length is 512 tokens, so the model can produce 'A)' in a context unrelated to the verdict (e.g., 'Option A is incorrect') or can state 'Yes' without emitting 'A)'. This extraction heuristic is a potential source of systematic error that affects both the accuracy numbers in Table 2 and the Filter policy in Section 5.1. The authors should report a manual evaluation of a sample of extractions or use a more robust parsing method (e.g., look at the final answer token or classify the response with a second prompt). Without this, the verifier scores are built on a fragile parse.","section":"§4.1.1"},{"comment":"All results are point estimates from a single run (or a single set of 20 self-consistency samples), with no error bars, multiple seeds, or significance tests. Several comparative statements in Section 3.1 rely on small differences (e.g., COT STEP at 73.54 vs COT at 73.62 on GSM8K for SOLAR, or COT STEP at 84.91 vs PS+ at 84.90 on GSM8K with self-consistency). These differences are within the noise one would expect from sampling, making the qualitative claims such as 'COT STEP keeps up fairly well' or 'COT STEP is even the best performer in most cases' unsupported. The authors should provide variance estimates or a statistical test (e.g., bootstrap over questions) for at least the headline comparisons, or temper the claims to acknowledge the lack of statistical robustness.","section":"Tables 1–7"}],"minor_comments":[{"comment":"Section 2.4 contains a typo: 'Quetion Answering' should be 'Question Answering.'","section":"§2.4"},{"comment":"The caption of Table 2 labels the true negative rate as 'True Negate Rate'; the correct term is 'True Negative Rate.'","section":"Table 2 caption"},{"comment":"The model name is inconsistently written as 'Llama3' and 'LLama3' across Tables 6, 7, and the text; please standardize (e.g., 'Llama 3').","section":"Tables 6, 7, and text"},{"comment":"Equation (1) should clarify whether bsC2 is the average log-probability of the entire verification response or of the option token only; the current wording suggests the former, but then the expression log(1 - exp(bsC2)) is not the log-probability of the 'No' answer. Specifying this will make the scoring function interpretable.","section":"§4.1.2"},{"comment":"Section 5.2 and Appendix A report the greedy/beam search hyperparameters but not the computational cost (number of LLM calls) compared with the self-consistency baseline; since the conclusion states that self-consistency is a 'better trade-off', a cost comparison would substantiate that claim.","section":"§5.2 and Appendix A"},{"comment":"The search hyperparameters (k=5, beam width=16, tau=0.5, alpha=0.5, max steps=18) are fixed without sensitivity analysis; a brief note on their effect (or a citation to prior work justifying the values) would make the zero-shot claim more robust.","section":"§5.2, Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The RQ2 evaluation gap is the main blocker. The paper is otherwise methodologically sound in its stated scope, but the abstract and conclusion overstate the verification result. For a journal venue, the authors should add either a human-annotated or step-level-ground-truth evaluation subset, and report multiple seeds for at least the key comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a competent empirical paper with one genuinely useful, minimal contribution (COT STEP) and a set of honestly reported negative results. The verifier evaluation (RQ2) is weaker than the headline numbers suggest because the ground-truth labels are answer-match proxies, but the authors disclose that in Section 8, and it doesn't undermine the RQ1 or RQ3 conclusions.\n\nWhat's actually new: COT STEP is a near-trivial change to zero-shot COT — adding 'Step 1:' after 'Let's think step by step' — but that small change buys automatic step parsing without hurting accuracy. That's a practical win for anyone who wants structured chains without hand-crafted exemplars. The two zero-shot verification prompts (R-prompt and COTR-prompt) are straightforward adaptations of Ling et al. (2023), and the systematic comparison across SOLAR, Phi3, plus Mistral and Llama3 in the appendix is useful. The paper also does a good job reporting negative results: verifier-guided search helps without self-consistency but the benefit largely vanishes once you add self-consistency, and beam search doesn't beat greedy. That is honest and useful for the community.\n\nSoft spots: The RQ2 correctness-classification dataset labels a chain correct if and only if the final answer matches the gold answer. The stress-test concern lands: this can't distinguish a chain with all correct steps from one that reaches the right answer via an unfaithful intermediate step. So the 83.9% COTR-prompt accuracy on GSM8K is not strong evidence of step-level verification. The authors acknowledge the issue and cite Lanham et al. (2023), which is good, but it remains a correctness risk for that specific claim. There are no error bars or significance tests, and no code or data release; the 'A)' answer-extraction heuristic is fragile but minor. These are all addressable.\n\nWho this is for: practitioners who want a cheap structured zero-shot prompt, and researchers working on LLM self-verification. It deserves a serious referee. A revision should add a small human-annotated step-level verification set, significance testing, and code release. I'd send it to review, not desk-reject.","headline":"A solid, honest zero-shot prompting paper whose COT STEP trick is genuinely useful; the RQ2 verifier numbers are weakened by answer-match labels but the limitation is disclosed.","tokens_in":22242,"tokens_out":2764,"would_cite":true,"duration_ms":25077,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot chain-of-thought can be decomposed and self-verified with prompt-only additions, but verifier-guided search does not beat plain self-consistency.","keywords":["zero-shot prompting","chain-of-thought","self-verification","reasoning step decomposition","verifier-guided search","self-consistency","mathematical reasoning","commonsense reasoning"],"falsifier":"Re-annotate a sample of GSM8K chains with step-level correctness labels by human raters, then recompute the COTR verifier's accuracy as a chain classifier on that relabeled set; if the correct-answer-with-wrong-steps rate is non-negligible, the reported 83.9% accuracy for SOLAR on COT STEP chains should fall noticeably.","tokens_in":21190,"feed_emoji":"🧠","tokens_out":10630,"duration_ms":95694,"temperature":0.7,"pith_summary":"This paper tries to show that a language model can verify its own step-by-step reasoning with no fine-tuned verifier, no handcrafted examples, and no extra data. It introduces COT STEP, a zero-shot prompt that appends 'Step 1:' to the standard 'Let's think step by step' opener, forcing the model to emit numbered reasoning steps that can be split automatically. On five reasoning datasets and four openly-licensed LLMs, COT STEP matches plain chain-of-thought accuracy while making the steps parseable. The paper also introduces a zero-shot chain-of-thought verification prompt (COTR-prompt) that classifies whether mathematical reasoning chains are correct, reaching 83.9% accuracy on GSM8K chains for one model. But using the verifier's scores to guide search only helps without self-consistency; with self-consistency, plain majority voting remains the stronger choice.","feed_headline":"Numbered-step prompting matches chain-of-thought at no accuracy cost","feed_subtitle":"A prompt-only verifier checks math chains with no training labels, but majority voting still wins under self-consistency.","key_machinery":"The load-bearing mechanism is the COT STEP prompt format: 'Let's think step by step.\\n\\nStep 1:' appended to the question, which makes the model emit numbered steps that can be split on 'Step <number>:'. The verifier score combines two components: C1, the average token log-probability of the generated step, and C2, the log-probability of the verification answer, mapped to a positive score for 'correct' and a negative score for 'incorrect', then averaged and exponentiated into a unified score. Step-wise greedy search uses that unified score to pick the highest-scoring candidate step at each turn.","core_discovery":"The central finding is that a purely prompt-based pipeline can structure and verify chain-of-thought reasoning in a zero-shot setting. COT STEP is the standard zero-shot COT instruction with '\\n\\nStep 1:' appended, which induces the model to generate numbered steps separated by 'Step <number>:' and gives an automatic way to split a chain into reasoning steps without exemplars. The verifier prompts, R-prompt (a direct yes/no verification prompt) and COTR-prompt (its chain-of-thought variant), ask the model to judge the last step of a partial chain; COTR-prompt additionally asks it to double-check step by step, and its answers are scored from the token log-probabilities of the step and of the verification response. Evaluated on chain-level correctness classification built from GSM8K and CommonsenseQA, COTR-prompt with COT STEP chains reaches 83.9% accuracy on GSM8K for SOLAR and 75.2% for Phi3, while performance on commonsense questions stays near chance. In search experiments, a step-wise greedy search guided by the verifier improves accuracy over plain COT STEP without self-consistency, but the improvement disappears under self-consistency, and verifier-weighted or filtered majority voting does not beat plain majority voting.","pith_inferences":["Beyond the paper, the answer-based labeling means the reported verifier accuracies are upper bounds on true step-level judgment; a human step-level relabeling could lower them substantially.","Beyond the paper, the fact that search gains vanish under self-consistency suggests majority voting already captures the same answer-level signal, so a reranker would need to act only on disagreeing samples to add value.","Beyond the paper, COT STEP's numbered structure can likely be combined orthogonally with other zero-shot augmentations, such as re-reading or role-play prompts, without changing the parseable format."],"forward_implications":["COT STEP can replace zero-shot COT in any pipeline that needs to parse reasoning steps, since it preserves accuracy while adding a clean 'Step N:' delimiter.","The COTR-prompt gives a drop-in verifier for mathematical reasoning tasks where no trained verifier or labeled verification data is available.","For users already running self-consistency, adding this zero-shot verifier is not worth the extra compute, because majority voting alone matches or beats all three verifier-based selection policies.","PS+ and TAB COT should not be assumed to outperform COT on openly-licensed models; the gains reported for those prompts do not generalize consistently.","Verifier-guided step-wise greedy search is a useful single-chain baseline, but beam search guided by the same verifier does not improve on it."],"supporting_citations":[{"why":"Supplies the zero-shot COT baseline and prompt template that COT STEP modifies.","marker":"(Kojima et al., 2022)"},{"why":"Supplies the GSM8K dataset and the trained-verifier paradigm the zero-shot verifier is compared against.","marker":"(Cobbe et al., 2021)"},{"why":"Supplies the self-consistency majority-voting baseline that verifier-score policies fail to beat.","marker":"(Wang et al., 2023b)"},{"why":"Supplies the one-shot verification prompt from which the R-prompt is simplified and the COTR variant is built.","marker":"(Ling et al., 2023)"},{"why":"Supplies the idea of combining generation probability with verification probability and the beam-search recipe.","marker":"(Xie et al., 2023)"},{"why":"Supplies the step-wise greedy search approach that the paper's SG adapts.","marker":"(Khalifa et al., 2023)"},{"why":"Documents that a correct final answer can accompany unfaithful reasoning, the limitation the paper acknowledges for its automatic labels.","marker":"(Lanham et al., 2023)"},{"why":"Supplies the CommonsenseQA dataset used to build the commonsense correctness-classification datasets.","marker":"(Talmor et al., 2019)"}],"fun_headline_variants":["Zero-shot verifier guides chain-of-thought without training","Prompt-only verifier checks reasoning steps, but voting still wins","COT STEP beats plain prompting, but not self-consistency","Self-verification matches chain-of-thought in zero-shot math","Verifier-guided CoT: no labels, but majority vote rules"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes a reasoning chain is correct whenever its final answer matches the gold answer, even if intermediate steps are wrong, which the paper itself flags as imperfect in Section 8.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot verifier guides chain-of-thought without training","Prompt-only verifier checks reasoning steps, but voting still wins","COT STEP beats plain prompting, but not self-consistency","Self-verification matches chain-of-thought in zero-shot math","Verifier-guided CoT: no labels, but majority vote rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1385,"prompt_tokens":948,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":349}},"tokens_in":564,"tokens_out":437,"duration_ms":4136,"temperature":1.0,"reasoning_tokens":349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:46:15.753732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a sample of GSM8K chains with step-level correctness labels by human raters, then recompute the COTR verifier's accuracy as a chain classifier on that relabeled set; if the correct-answer-with-wrong-steps rate is non-negligible, the reported 83.9% accuracy for SOLAR on COT STEP chains should fall noticeably.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the one-shot verification prompt from which the R-prompt is simplified and the COTR variant is built."}],"review_version":1}