{"id":"520e8f64-2c66-46c7-be61-4e6528564f67","arxiv_id":"2608.05643","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Sampling multiple reasoning paths, refining each with self-critique and self-correction, then majority voting improves math reasoning accuracy over width-only or verifier-based test-time scaling on several open-weight LLMs.","lead":"This paper tests a simple idea: instead of only sampling many answers from a language model, also have the model criticize and fix each answer several times, then take a majority vote. On five math benchmarks the approach often beats greedy decoding, majority voting, and verifier-based selection, especially for small models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim lacks an equal-compute comparison: the reported gains over Maj@8 and RM@8 may reflect extra TFLOPs from D=4 refinement rather than depth itself, so 'refining over resampling' is not established.","rationale":"The reader's weakest assumption focuses on whether the same base model can critique and correct reliably enough for majority voting to suppress regressions. That is a meaningful mechanistic risk, but I think the more load-bearing concern is comparative validity: the experiments fix N=8 for all sampling methods and spend far more compute on Ours, so the central conclusion that refinement beats additional sampling is not yet tested. The manuscript is transparent about this in Section 5.3, which is a point in its favor, but transparency does not convert an unsupported comparative claim into a supported one. My proposed check is direct and feasible: scale majority voting to the same total TFLOPs as Ours and see whether the accuracy advantage survives. If it does not, the title claim 'Refining Over Resampling' fails even if individual self-correction dynamics are as described. Secondary issues reinforce the same caution: Table 1 already contradicts the abstract's 'consistently improves' wording (for example, Qwen2.5-Math-7B loses to RM@8 on AIME24 and LLaMA-3.1-8B loses on AIME25), and the absence of seeds, code, or error bars makes single-question differences on AIME/AMC hard to interpret. I credit the paper for including an ablation of the critic, refinement diagnostics, and a compute-normalized analysis, and I am not alleging any fabrication. The concern is that the central causal claim about depth versus width is underdetermined by the current experimental design, which is precisely why the paper should be accepted only with an equal-compute resampling comparison or with the claim restricted to equal-sample-count improvements.","tokens_in":19380,"tokens_out":10065,"duration_ms":102704,"concrete_test":"Reproduce the headline comparisons with an equal-compute majority-voting baseline. Using the paper's own FLOP accounting (Section 3.5), compute the per-rollout inference cost of Maj@8 for each model and benchmark and the total cost C_ours of Ours (N=8, D=4) from Figure 5 and Table 4. Choose N' = floor((C_ours - C_Maj@8) / per_rollout_cost) + 8, sample N' independent rollouts at the same temperature tau=0.7, and aggregate with the same answer extractor and plurality vote. If Maj@N' reaches or exceeds Ours on the average of the five benchmarks, especially Qwen2.5-1.5B on MATH500, the central 'refining over resampling' claim is unsupported; if Ours still wins at matched compute, the concern is settled.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that refining sampled trajectories is more effective than resampling or verifier-based selection at test time. The experiments, however, compare all methods at a fixed sample count N=8, not at a fixed compute budget. Ours runs each of the 8 rollouts through D=4 cycles of continuation, critique, and correction, so its total inference cost is much larger: Table 4 reports ΔTFLOPs over Maj@8 of roughly 728 for Qwen2.5-1.5B, about 2,184-2,624 for the 7B/8B models, and about 2,034 over RM@8 for Qwen2.5-Math-7B. The paper itself states in Section 5.3 that the compute-normalized gain eta 'does not constitute a direct equal-budget comparison in which majority voting is scaled to the same compute.' Consequently, the observed accuracy advantage could in principle be obtained by spending the same additional compute on more independent rollouts, leaving the load-bearing premise that depth, rather than extra forward passes, drives the gains untested. Establishing that refinement is more effective than resampling requires an equal-compute resampling baseline.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free, verifier-free test-time reasoning framework that combines breadth (N independently sampled rollouts) with depth (D rounds of self-critique and self-correction applied to each rollout) and then aggregates the refined rollouts by majority voting. The method is evaluated on AIME24, AIME25, AMC, OlympiadBench, and MATH500 with four open-weight models, reporting accuracy gains over greedy decoding, majority voting, verifier-based best-of-N, beam search, and lookahead decoding, together with TFLOP accounting, compute-normalized gain metrics, ablations of the critic stage, and quantitative diagnostics of refinement dynamics. The paper also includes a limitation section acknowledging the higher inference cost and the dependence on the base model's self-critique ability.","tokens_in":19616,"tokens_out":4747,"duration_ms":48049,"significance":"If the central claim is established, the work would be a useful and timely contribution: it offers a simple, model-agnostic, verifier-free test-time scaling strategy that requires no additional training and no reward model, and it provides a rare transparent discussion of the compute trade-off and of the noisy, sometimes harmful, nature of per-rollout self-correction. The paper includes several strengths: an explicit ablation of the self-critique stage (Table 2), a breadth-depth allocation study (Table 3), compute accounting and compute-normalized gains (Table 4, Figures 5-6), and honest diagnostics showing that per-rollout correction benefit can be negative (Figure 4). However, the main scientific claim that refinement is more effective than resampling is not yet established, because the experiments do not include an equal-compute resampling baseline, the abstract overstates the consistency of the gains relative to Table 1, and the headline numbers lack any measure of statistical uncertainty. These issues are load-bearing for the central claim and require revision.","major_comments":[{"comment":"The central claim that refining sampled trajectories is more effective than resampling requires an equal-compute comparison, but all main results compare methods at a fixed sample count N=8. Under the paper's own TFLOP accounting, Ours uses roughly 728 additional TFLOPs over Maj@8 for Qwen2.5-1.5B and about 2,184-2,624 additional TFLOPs for the 7B/8B models (Table 4). Section 5.3 correctly states that the compute-normalized gain eta 'does not constitute a direct equal-budget comparison in which majority voting is scaled to the same compute.' Without a Maj@K baseline scaled to the same TFLOP budget as D=4 refinement, the observed accuracy advantage could be due to the extra forward passes rather than to the depth of refinement. Please add an equal-compute majority-voting baseline (e.g., Maj@K with K chosen so that total TFLOPs match Ours) and report whether the advantage survives.","section":"§5.3, Table 4"},{"comment":"The abstract's claim that the method 'consistently improves over greedy decoding, majority voting, verifier-based best-of-N, beam search, and lookahead decoding' is contradicted by Table 1. For example, on AIME24 with Qwen2.5-Math-7B, Ours (10.0) is below RM@8 (13.33); on AIME25 with LLaMA-3.1-8B, Ours (6.7) is below Greedy, Maj@8, and RM@8 (all 10.0); and on AIME25 with Ministral-8B, Ours (3.33) ties RM@8 (3.33). Several other cells are ties with Maj@8. The Table 4 W/T/L counts (e.g., 4/1/0 for Qwen2.5-Math-7B vs RM@8, and 2/1/2 for LLaMA-3.1-8B vs RM@8) are more accurate. Please soften the abstract claim to 'generally improves' or 'improves on most benchmark-model pairs' and report the W/T/L summary in the abstract or at least in the results section.","section":"Abstract; Table 1"},{"comment":"The reported accuracies lack error bars, multiple seeds, or significance tests. AIME24 and AIME25 have only 30 questions each, so a 3.33-percentage-point difference is a single question, and several of the claimed gains are exactly one question. For instance, the Qwen2.5-1.5B AIME24 gain from 3.33 (Maj@8/RM@8) to 6.67 (Ours) is one question, and the LLaMA-3.1-8B AIME25 loss from 10.0 to 6.7 is one question. Without multiple seeds or bootstrap confidence intervals, the headline improvements on MATH500 (e.g., 29.6 to 58.0 for Qwen2.5-1.5B) could be partly due to sampling noise. Please report at least three seeds with mean and standard deviation for the main results, or provide bootstrap CIs over questions.","section":"§4.1, Table 1"},{"comment":"Figure 4 shows that the per-rollout net correction benefit B(d) is negative at several depths for some models (e.g., the Qwen2.5-Math-7B curve dips below zero at early depths), and the paper's own limitation section acknowledges that refinement 'can introduce rollout-level regressions.' The argument that majority voting suppresses these regressions is plausible but is not directly tested. The paper reports vote-level recovery and regression rates in Figure 3 but does not provide a statistical test of whether the vote-level net benefit (recovery minus regression) is significantly positive across questions, nor does it quantify how sensitive the final accuracy is to the per-rollout flip rates. Please add a paired bootstrap or sign test comparing depth-0 vote accuracy with terminal vote accuracy per question, and report the per-question distribution of vote-level net benefit.","section":"§5.2, Figure 4"},{"comment":"The method is entirely defined by the role-conditioned prompts used for the generator, critic, and corrector, yet the paper never provides these prompt templates. Appendix B shows only a single qualitative trace for one problem and one model. Without the exact prompts, the experiments are not reproducible and the claimed generality across models cannot be independently verified. Please include the full prompt templates (with the exact role instructions, output format constraints, and any special tokens) in an appendix or supplementary material.","section":"§3.3, Appendix B"}],"minor_comments":[{"comment":"There are several typographical issues in Table 1: 'Ours10.0' is missing a space in the Qwen2.5-Math-7B row, '0.06.67' in the Qwen2.5-1.5B Greedy row should be '0.0 6.67', and the Ministral-8B MATH cell '65.78' appears to be a rendering of '65.78' but should be checked against the value 65.78 in the text.","section":"Table 1"},{"comment":"The y-axis label of Figure 6 reads 'CNG2[acc.pp/103TFLOPs]' which appears to be a typo for 'CNG' (compute-normalized gain) and the units should be formatted consistently with Table 4 and Appendix A.7.","section":"Figure 6"},{"comment":"The perturbation model in Eq. (2) uses constants lambda and h from Liu et al. (2024) without checking whether they apply to the models evaluated here. The paper should state clearly that this equation is an illustrative first-order model, not an empirical characterization of the tested models, and that the i.i.d. perturbation assumption may not hold if errors are correlated across samples.","section":"Equation (2)"},{"comment":"The clustering threshold tau_sim=0.85 and the choice of all-MiniLM-L6-v2 are free parameters of the diversity-saturation analysis in Figure 1. The appendix notes that answer-based clustering is the primary grouping, but the redundancy curves in Figure 1 depend on the sentence-embedding clustering; a sensitivity check over tau_sim (e.g., 0.75, 0.85, 0.95) would strengthen the motivation.","section":"Appendix A.6"},{"comment":"Algorithm 1 says the critic stage is 'parallel / batched' but the pseudocode does not explicitly show how the generation, critique, and correction calls are batched; clarifying the batch structure would help implementation and compute accounting.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: worth a referee, but not on its current claims. The paper tests a natural combination — sample N rollouts, push each through D rounds of self-critique and correction, then majority vote — and reports gains on five math benchmarks across four open-weight models. As far as I know, that combination is genuinely new; prior work either refines a single trajectory or votes over unrefined samples. The paper also does real diagnostic work: compute accounting, compute-normalized gains, recovery/regression curves, a critic ablation, and a qualitative trace. Credit where due: the study is substantial, and the authors are transparent in places — they admit per-rollout correction benefit can be negative and that their compute-normalized metric is not an equal-budget comparison.\n\nThe soft spots are real. First, the title claim 'refining over resampling' is not established. All main comparisons fix N=8 for every method, so the refined method spends far more TFLOPs (roughly 728 extra for the 1.5B model, 2,000+ for the 8B models per Table 4) without an equal-compute resampling baseline. The gains could come from extra forward passes, not from depth per se. The paper admits this in Section 5.3, which makes the abstract's 'consistently improves' and the concluding 'more effective' overreach. Second, 'consistently' is contradicted by their own Table 1: clear losses (Qwen2.5-Math-7B on AIME24, LLaMA-3.1-8B on AIME25 and OlyBench) and W/T/L counts in Table 4 showing 1-2 losses per baseline. Third, no code, data, or seeds, and no error bars; with AIME/AMC sized at tens of questions, several headline deltas are one or two questions. Fourth, N=8, D=4 is presented as a stable trade-off, but Table 3 shows non-monotonic accuracy in both N and D, with no validation protocol.\n\nNone of this is fatal to the core idea — a verifier-free, training-free refinement-plus-vote recipe is plausible and worth testing properly. But the paper currently overstates what it shows. I'd send it to review, not desk-reject, with a request for an equal-compute baseline, error bars or seeds, and a toned-down central claim.\n\nRecommendation: engage with it as a promising empirical study that needs revision.","headline":"Genuinely tested breadth-depth refinement, but the title claim is unsupported without an equal-compute baseline.","tokens_in":20201,"tokens_out":4422,"would_cite":false,"duration_ms":42204,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM math reasoning improves when sampled answers are critiqued and repaired before majority voting, rather than when more samples or a verifier are used.","keywords":["test-time scaling","self-correction","LLM reasoning","majority voting","verifier-free","mathematical reasoning","breadth-depth refinement","self-critique"],"falsifier":"Run the method on AIME24 with Qwen2.5-1.5B at $N=8$, $D=4$, replacing the self-critique prompt with one that always affirms the current trace: if the refined majority-vote accuracy does not fall below unrefined Maj@8, then the claimed benefit is not caused by the critique-and-correct signal, and the loop's value would need another explanation.","tokens_in":19145,"feed_emoji":"🧮","tokens_out":9756,"duration_ms":93428,"temperature":0.7,"pith_summary":"The paper's central claim is that extra inference compute for LLM reasoning produces more accuracy when spent on repairing the solutions the model already sampled than on simply sampling more solutions or relying on an external reward model. It proposes a training-free, verifier-free breadth–depth refinement method: sample $N$ independent reasoning rollouts, run $D$ rounds of self-critique and self-correction on each rollout using the same base model in three roles — generator, critic, corrector — and aggregate the terminal answers by majority voting. Across AIME24, AIME25, AMC, OlympiadBench, and MATH500, with four open-weight models, the method consistently beats greedy decoding, majority voting, verifier-based best-of-$N$, beam search, and lookahead decoding. The largest reported gain is on the smallest model: Qwen2.5-1.5B rises from 29.6% to 58.0% on MATH500. The paper argues this succeeds because depth repairs local reasoning errors before aggregation, while voting suppresses the noisy, sometimes harmful, flips that occur in individual corrections.","feed_headline":"Self-critique lifts LLM math reasoning beyond resampling","feed_subtitle":"Qwen2.5-1.5B jumps from 29.6% to 58.0% on MATH500 when eight sampled answers are refined by the same model before voting.","key_machinery":"The central machinery is a three-stage refinement cycle applied for $D$ depths to each of $N$ independent rollouts: reasoning continuation (the generator extends or rewrites the previous trace), self-critique (the critic names logical, arithmetic, or structural errors or explicitly confirms the trace), and self-correction (the corrector revises the trace conditioned on both). The same base model is role-conditioned by prompts, so no external verifier, process reward model, learned stop policy, or additional training is introduced. The cycle's work is to give each trajectory repeated opportunities to repair local errors; the final parameter-free plurality vote then aggregates the refined terminal answers, relying on cross-rollout agreement to suppress the noisy per-rollout corrections documented in Figure 4.","core_discovery":"On the paper's own terms, the discovery is that refinement — not more resampling and not verifier selection — is the more effective use of test-time compute for mathematical reasoning. Given a problem $x$, the method draws $N$ initial rollouts from the model's generator policy, then for each of $D$ depths produces a reasoning continuation, a natural-language self-critique, and a corrected trace, all with the same base model and no external reward model; terminal traces are mapped to normalized answers and combined by plurality vote. The paper motivates the design with two failure modes: diversity saturation, where extra samples keep revisiting existing reasoning clusters, and a per-trace hallucination floor, where every i.i.d. rollout is exposed to the same chance of perturbing the intended answer away from correct. Its own diagnostics show that per-rollout correction is noisy and can have negative net benefit, so the final claim is that majority voting over refined rollouts retains the shared repairs and suppresses isolated regressions. The paper expects this to matter most for weaker models, where latent reasoning ability exists but is not reliably expressed in a single first-pass answer.","pith_inferences":["Beyond the paper's claims, one can test whether the framework's benefit is really from the critique stage by replacing the critic prompt with an always-confirm prompt; if refined voting accuracy survives that change, the gain would trace to something other than self-critique quality.","Beyond the paper's claims, the same repair-before-aggregate recipe should transfer to code generation or formal proof tasks, where a symbolic checker can play the role the plurality vote plays here; this is a natural next benchmark.","Beyond the paper's claims, the compute-normalized results suggest an untested scaling hypothesis: for a fixed budget, there is an optimal allocation between $N$ and $D$, and the paper's own Table 3 already shows accuracy is not monotonic in either coordinate.","Beyond the paper's claims, the method's dependence on majority voting means it will likely underperform on single-solution tasks or when the model's self-critique is systematically overconfident; a testable variant would weigh refined rollouts by confidence instead of applying a flat vote."],"forward_implications":["Because the method needs no reward model or extra training, it can be applied to any open-weight model at inference time and should transfer across domains where answers are extractable and voteable.","With $N=8$ and $D=4$ fixed, the reported gains are consistent across five benchmarks and across all compared baselines; weaker models gain most, which suggests refinement recovers latent reasoning ability.","The terminal rollout diversity remains high (0.80–0.97 at depth 4), so the voting step is meaningful rather than a rubber stamp of convergent rollouts.","Compute-normalized gains are positive against every baseline, with the largest per-TFLOP gains at small model scale; equal-sample comparisons may still understate the method's compute cost.","The explicit self-critique stage adds accuracy beyond pure correction in most settings (Table 2), meaning the natural-language critique is information-bearing and not just extra tokens."],"supporting_citations":[{"why":"Supplies the majority-voting and self-consistency baseline that the method extends and must beat.","marker":"Wang et al., 2022"},{"why":"Supplies the verifier-based selection framework that RM@8 instantiates and that the paper argues is calibration-dependent.","marker":"Cobbe et al., 2021"},{"why":"Provides the iterative self-refine feedback-and-revision loop that the paper adapts into role-conditioned critic and corrector stages.","marker":"Madaan et al., 2023a"},{"why":"Supplies the warning that naive self-correction can degrade reasoning, which motivates structured depth plus majority voting.","marker":"Huang et al., 2024"},{"why":"Supplies the latent-accuracy and hallucination model behind Eq. (2) and the claim that second-pass correction reduces the hallucination rate.","marker":"Liu et al., 2024"},{"why":"SCoRe trains self-correction via reinforcement learning, serving as the trained-method contrast that the paper's training-free claim is measured against.","marker":"Kumar et al., 2024"},{"why":"ReVISE learns a stop-or-refine policy, another learned-method contrast that supports the paper's claim of being model-agnostic and verifier-free.","marker":"Lee et al., 2025"},{"why":"Provides Qwen2.5-Math-RM-32B, the reward model used to instantiate the verifier-based best-of-$N$ baseline.","marker":"Yang et al., 2024c"},{"why":"Supplies the transformer scaling-law FLOP approximation used for the paper's compute accounting and compute-normalized gains.","marker":"Kaplan et al., 2020"},{"why":"Supplies the scaling-law compute model that the paper relies on when estimating inference TFLOPs.","marker":"Hoffmann et al., 2022"}],"fun_headline_variants":["Self-critique beats resampling for LLM math reasoning","Refinement over resampling: test-time self-correction pays off","LLM reasoning improved by self-refining sampled answers","No verifier needed: self-corrected rollouts win on math","Breadth and depth: self-critique lifts LLM math scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the same base model, prompted as critic and corrector, produces enough net-valid corrections that majority voting over refined rollouts beats majority voting over the original rollouts, even though the paper's own Figure 4 shows per-rollout correction can have negative net benefit and its Limitations section concedes that over-correction can introduce regressions.","fun_headline_variants_meta":{"raw":{"variants":["Self-critique beats resampling for LLM math reasoning","Refinement over resampling: test-time self-correction pays off","LLM reasoning improved by self-refining sampled answers","No verifier needed: self-corrected rollouts win on math","Breadth and depth: self-critique lifts LLM math scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1929,"prompt_tokens":1035,"completion_tokens":894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":804}},"tokens_in":651,"tokens_out":894,"duration_ms":7703,"temperature":1.0,"reasoning_tokens":804,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:59:27.850035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the method on AIME24 with Qwen2.5-1.5B at $N=8$, $D=4$, replacing the self-critique prompt with one that always affirms the current trace: if the refined majority-vote accuracy does not fall below unrefined Maj@8, then the claimed benefit is not caused by the critique-and-correct signal, and the loop's value would need another explanation.","supporting_citations":[],"review_version":1}