{"id":"5b52c173-88e2-4594-b54f-987553f58a67","arxiv_id":"2412.17397","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding a self-correction RL stage before MCTS-DPO improves arithmetic reasoning accuracy by 2.00 to 4.94 points on GSM8K and MATH.","lead":"A two-stage LLM training method, first teaching a model to self-correct on self-generated attempts, then using that self-correcting model as the verifier inside MCTS-guided preference learning, yields 1 to 5 percentage point accuracy gains on GSM8K and MATH arithmetic benchmarks. The gains are modest and the paper's claim of 'intrinsic' self-correction without external feedback is contradicted by its own training objective using ground-truth answer rewards.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage I's 'intrinsic self-correction' uses oracle answers in Eq. (1); without an ablation that removes y*, the claimed mechanism behind the accuracy gains is unsupported.","rationale":"The reader's weakest assumption is the right one. The paper's own text contains an explicit contradiction: it promises intrinsic self-correction without external feedback, then defines the Stage I objective with an oracle answer checker. I agree with the reader that this is load-bearing because the novelty and the interpretation of the accuracy gains rest on the mechanism, not merely on the numbers. The most direct way to settle it is an ablation that removes y* from the Stage I reward. I do not think this requires changing the conditional verdict: the claim is plausible but unverified, and the missing ablation should be a condition of acceptance. No ad hominem intended; the issue is the argument.","tokens_in":8629,"tokens_out":5171,"duration_ms":50338,"concrete_test":"Run the full two-stage pipeline twice on MATH with OpenMath2-Llama3.1-8B, keeping Stage II and all hyperparameters fixed, but in Stage I replace the oracle reward r_hat(y,y*) in Eq. (1) with a self-generated reward that never sees y* (e.g., agreement probability between two independent sampled solutions, or a frozen self-consistency score). If the final accuracy falls from the reported 71.34 to at or below the 68.06 Base+MCTS-DPO row, the oracle reward is the active ingredient and the 'intrinsic self-correction' mechanism is not supported; if the gain persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Stage I is described as relying 'entirely on self-generated data' and 'without any external feedback,' but Eq. (1) maximizes sum_i r_hat(y_i, y*), where r_hat is an oracle answer checker comparing outputs with the ground-truth y*. That is external, answer-level supervision. Stage II then uses the Stage I policy as an 'Enhanced-Self-Verify' reward (Eqs. 5-6): the term pi_theta1(A | prompteval, x, st) is added to R(st) as a correctness confidence. For this to be a valid component of the claimed mechanism, pi_theta1 must have learned genuinely self-generated correction and must transfer to scoring partial reasoning states. Neither is demonstrated: the Stage I objective trains the model to maximize oracle-graded correctness of whole responses, not to score intermediate steps, and the verifier prompt/eval format is never specified or operationalized. Consequently, the Table 2 gains (e.g., 71.34 vs 68.06 for Base+MCTS-DPO on OpenMath2-MATH) could be produced entirely by the oracle-reward REINFORCE stage, which is standard answer-checker RL, with the 'intrinsic' label doing no work. Table 3 does not include a condition that removes y* from the Stage I reward, so it cannot distinguish the proposed mechanism from this confound. No code, hyperparameters, or error bars are provided, which makes the mechanism uncheckable from the text.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage procedure for improving step-level reasoning in LLMs on arithmetic benchmarks. In Stage I, a policy pi_theta1 is trained with a REINFORCE-style objective plus a KL penalty to produce corrected responses across multiple attempts, using an oracle answer checker r_hat(y,y*) as reward. In Stage II, the authors run step-level MCTS-DPO in which the reward at a state combines outcome correctness with a self-evaluation term C(s_t) = pi_theta2(A | prompt_eval, x, s_t), and an 'Enhanced-Self-Verify' term C_hat(s_t) = pi_theta1(A | prompt_eval, x, s_t). Results are reported on GSM8K and MATH for four base models, with the full method claimed to outperform both the base model and the Base + MCTS-DPO baseline; an ablation on MATH compares configurations of policy/reference/reward models.","tokens_in":9014,"tokens_out":5880,"duration_ms":56653,"significance":"If the claimed mechanism were validated, the paper would address a real bottleneck in MCTS-boosted reasoning, namely the reliance on accurate step-level verifiers, by reusing a self-trained policy as an intrinsic verifier. The reported numerical gains are consistent in direction across four models and two datasets, and the MATH ablation is a useful first attempt at isolating the contribution of each stage. However, the central 'intrinsic self-correction' claim is currently entangled with oracle answer-level supervision, the Enhanced-Self-Verify term is not operationalized, and the experimental section omits code, hyperparameters, and error bars. The significance is therefore conditional on resolving these issues.","major_comments":[{"comment":"Eq. (1) maximizes E[sum_{i=1}^{l+1} r_hat(y_i, y*)] using an oracle answer checker that compares outputs with the ground-truth y*. The abstract and the Stage I text say the model is trained 'relying entirely on self-generated data' and 'without any external feedback.' This is internally inconsistent: y* is external, answer-level supervision. To support the 'intrinsic' claim, the paper must distinguish between self-generated rollouts (the inputs are self-generated) and the oracle-graded reward, and it must add an ablation in which Stage I is trained with a self-evaluation-only reward or with y* removed. As written, the Table 2 gains could be produced entirely by standard answer-checker RL, with the 'intrinsic' label doing no work.","section":"Stage I (Eq. (1))"},{"comment":"The reward R(s_correct_t) is defined as R(s_t) + C_hat(s_t), where C_hat(s_t) = pi_theta1(A | prompt_eval, x, s_t), but prompt_eval is never specified and no training objective is given that makes pi_theta1 output a token-level confidence A for a partial reasoning state s_t. Stage I optimizes whole-response correctness over l+1 attempts, not step-level verifier scores. The authors need to specify the exact prompt, the mapping from pi_theta1 logits to A, and any validation of this confidence against step-level correctness. Without this, the mechanism behind the Table 3 gains cannot be distinguished from Stage I answer-checker RL alone.","section":"Enhanced-Self-Verify (Eqs. (5)-(6))"},{"comment":"The self-evaluation term C(s_t) = pi_theta2(A | prompt_eval, x, s_t) uses the same policy being trained by MCTS-DPO. This self-referential reward can reward high confidence rather than true correctness. Since final accuracy is measured on held-out test sets, the evaluation itself is not circular, but the training signal may be. Please report whether pi_theta2's confidence score A correlates with step-level correctness on a validation set; otherwise the self-evaluation component could be a confidence-seeking objective rather than a verifier.","section":"Eq. (4) and Stage II reward"},{"comment":"Table 3 does not contain the conditions needed to attribute the improvement to the proposed mechanism. The design varies policy/reference/reward model identity among base, ISC, and SPL, but there is no row that suppresses the oracle reward in Stage I and no row that isolates the Enhanced-Self-Verify term from the plain self-evaluation term. Moreover, the ISC/ISC/SPL row underperforms the base/base/base row (65.34 vs. 68.06 on OpenMath2-MATH and 47.22 vs. 45.56 on dart-math-MATH), a result that is not discussed and is hard to reconcile with a monotone benefit of Stage I. The absence of hyperparameters (c_puct, lambda, KL coefficient, number of rollouts), random seeds, and code makes the reported gains (e.g., 71.34 vs. 68.06 and 48.06 vs. 45.56) uncheckable and prevents assessment of statistical significance.","section":"Experiments and Ablation (Table 3)"}],"minor_comments":[{"comment":"The relative gain for dart-math-mistral-7b-uniform on MATH is reported as +4.94% in the abstract but +4.64% in the main text (48.06 vs. 43.42); please harmonize.","section":"Abstract and Main Results"},{"comment":"Eq. (2) writes s*_{t+1} = argmax_{st} [...], but the maximization should be over child nodes/actions, not over st; please correct the notation.","section":"Eq. (2)"},{"comment":"The captions of Figures 1 and 2 introduce terms such as 'self-correctness-policy' and 'outer-loop-policy' without connecting them explicitly to pi_theta1 and pi_theta2; please define all acronyms and box labels.","section":"Figures 1 and 2"},{"comment":"The column labeled 'Reference model' is not defined in the text; clarify whether it is the KL reference model in REINFORCE/DPO or a different quantity.","section":"Table 3"},{"comment":"The Discussion refers to 'self-supervised learning' for Stage I, while Stage I is described methodologically as a REINFORCE RL objective; use consistent terminology.","section":"Discussion"},{"comment":"Numerical values for lambda in Eq. (2), the KL penalty coefficient in Stage I, and the number of MCTS rollouts are not reported; please provide these or state that they were tuned on a validation split.","section":"Methodology"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution relative to the MCTS-DPO framework of Xie et al. (2024) is incremental, and the claimed novelty depends on a term (Eq. (6)) that is not operationalized. The main experimental result might be reproducible with standard answer-checker RL, which would change the contribution substantially. If the authors can provide the missing ablations, implementation details, and a validation of the self-verifier, the paper could become suitable for publication; without these, the central claim is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper is a real but incremental extension of MCTS-DPO, and the 'intrinsic self-correction' framing doesn't survive contact with Eq. (1). The two-stage combination—train a self-correction policy with REINFORCE against an answer checker, then use it as the verifier inside step-level MCTS-DPO—is new as a package, and the reported gains (2.0–4.9 points on GSM8K and MATH) are plausible for that kind of recipe. Table 3 has a useful signal: swapping the reward model changes accuracy more than swapping the policy model. That is worth keeping.\n\nThe soft spots are not minor. The method section says Stage I uses 'self-generated data without any external feedback,' but Eq. (1) maximizes oracle reward r̂(y, y*), which compares against ground-truth answers. That's external supervision. The authors may mean 'no separate trained reward model,' but that's not what they wrote, and the distinction matters because the paper's novelty rests on the 'intrinsic' bit. The Enhanced-Self-Verify term in Eqs. (5–6) is equally under-specified: prompt_eval and the confidence token A are never defined, so a reader cannot tell what the Stage I policy is actually scoring. No code, hyperparameters, or error bars are provided, and Table 3 lacks a condition that removes y* from the Stage I reward. Without that ablation, the accuracy gains could come entirely from standard answer-checker RL, with the 'intrinsic' label doing no work.\n\nThat said, the core method is coherent, the benchmarks are standard, and the rankings look believable. The gaps are fixable in principle: run the no-oracle ablation, spell out the verifier prompt, and release hyperparameters. I'd send this to peer review rather than desk reject—it's a legitimate incremental step with an addressable flaw in the central claim. I wouldn't cite it until the mechanism is pinned down, but people working on MCTS-DPO or self-correction RL should read it.","headline":"A plausible incremental extension of MCTS-DPO that overclaims its 'intrinsic' self-correction, with fixable but load-bearing gaps in the mechanism.","tokens_in":9474,"tokens_out":3494,"would_cite":false,"duration_ms":31582,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage self-correction loop is claimed to improve LLM arithmetic accuracy by up to 4.94 percentage points on GSM8K and MATH.","keywords":["large language models","self-correction","reinforcement learning","Monte Carlo Tree Search","direct preference optimization","mathematical reasoning","GSM8K","MATH"],"falsifier":"Retrain Stage I with the reward function changed from the oracle answer-checker $\\hat{r}(y, y^*)$ to a purely internal signal such as self-consistency across sampled rollouts, and keep Stage II fixed; if the reported GSM8K and MATH gains disappear, the mechanism named 'intrinsic self-correction' is not the source of the improvement.","tokens_in":8379,"feed_emoji":"🧮","tokens_out":8483,"duration_ms":67326,"temperature":0.7,"pith_summary":"The paper argues that an LLM's arithmetic reasoning improves reliably when self-correction is learned before step-level preference learning, rather than only at the end. The proposed two-stage procedure first uses REINFORCE to train a policy that, given its own earlier attempts at a problem, produces a corrected answer; this is the 'intrinsic self-correction' stage. The second stage plugs that self-correcting policy into Monte Carlo Tree Search boosted Direct Preference Optimization, using it as a step-level verifier that flags and repairs misleading intermediate reasoning. The reported result is that this combination beats both the base models and the MCTS-DPO-only baseline on GSM8K and MATH, with accuracy gains between 2.00 and 4.94 percentage points across four base models. The authors' intended takeaway is that the two abilities are synergistic: a better self-corrector makes the tree-search preference learner a better judge, and the gains would not be available from either stage alone.","feed_headline":"Self-correct-first training lifts LLM math accuracy by up to 5 points","feed_subtitle":"Stage one teaches the model to fix its own mistakes; stage two uses the fixed policy as a step-level judge.","key_machinery":"The load-bearing machinery is a two-loop training setup. Stage I trains a policy $\\pi_{\\theta_1}$ under a REINFORCE objective with a KL penalty, maximizing the summed oracle reward over multiple attempts (Eq. 1), where the policy receives the problem $x$, its earlier attempts $\\hat{y}_{1:l}$, and an instruction to improve. Stage II is the step-level iterative preference learning: an MCTS tree grows by selection (PUCT), expansion, an 'enhanced-self-verify' step, and backup; the reward at a state is $R(s_t) = O(s_t) + C(s_t)$ with outcome correctness $O$ and self-evaluation $C$, and the Stage-I policy is inserted as an additional correction signal $\\hat{C}(s_t) = \\pi_{\\theta_1}(A \\mid \\text{prompt}_{\\text{eval}}, x, s_t)$, so the corrected state reward becomes $R(s_t^{\\text{scorrect}}) = R(s_t) + \\hat{C}(s_t)$. That insertion—the Stage-I policy acting as an extra verifier inside MCTS—is the specific mechanism the paper claims is responsible for the improvement.","core_discovery":"On the paper's own terms, the central discovery is that an 'intrinsic self-correction' stage—training the model to improve its own responses using only its own outputs as data—can be composed with the existing step-level MCTS-DPO method to yield consistent accuracy gains on two arithmetic benchmarks. The new ingredient is not a new search algorithm but a new ordering: the self-correcting policy is trained first (Stage I), then reused in Stage II as the verification model in the MCTS expansion and roll-out reward. The paper reports concrete numbers: on GSM8K, accuracy rises to 86.76% from an 84.76% baseline for Llama-3.1-8B-Instruct and to 38.06% from 35.78% for Mistral-7B-Instruct-v0.1; on MATH, accuracy rises to 71.34% from 67.16% for OpenMath2-Llama3.1-8B and to 48.06% from 43.42% for dart-math-mistral-7b-uniform. The ablation on MATH reports that the best configuration uses the Stage II policy (SPL-Model) as both policy and reward model, indicating the two stages work together rather than independently.","pith_inferences":["The paper's Eq. (1) uses an oracle reward $\\hat{r}(y,y^*)$ that compares against ground truth $y^*$, despite the abstract claiming no external feedback; this leaves open whether the 'intrinsic' stage would work without ground-truth answers, and testing self-consistency alone as the reward would separate true self-correction from supervised answer-matching.","A natural extension not tested in the paper is to use the Stage-I policy's correction signal as a reward in a policy-gradient pass rather than only inside MCTS, which would show whether the benefit is specific to tree-search integration.","If the corrected-step reward is what helps, one expects the gap between Ours and Base Model + MCTS-DPO to widen as problems get harder; checking accuracy per MATH difficulty band would make the mechanism testable.","The method should transfer to code generation or tool-use benchmarks where a unit test or interpreter supplies the oracle correctness signal, without changing the two-stage structure."],"forward_implications":["If the two-stage claim holds, any step-level MCTS-DPO training pipeline can benefit by first running an intrinsic self-correction loop on self-generated rollouts, at no extra label cost beyond the final answer checker.","The gains are consistent across two different base-model families and two dataset difficulties, so the procedure is not tailored to a single architecture.","Stage-I-only self-correction already improves over the base model in all four configurations, which suggests the self-correction policy transfers some verifier ability before any tree search is used.","Ablations on MATH indicate the full combination dominates either stage alone, so the method's value is in the composition rather than in a single component.","Because the method only assumes an oracle correctness check at the end of a rollout, it should extend to other tasks where final answers can be checked objectively."],"supporting_citations":[{"why":"Supplies the step-level MCTS-DPO baseline and the iterative preference learning loop that Stage II builds on.","marker":"Xie et al. 2024"},{"why":"Supplies the self-correction RL formulation and the multi-turn MDP that Stage I's intrinsic data generation is modeled on.","marker":"Kumar et al. 2024"},{"why":"Provides the REINFORCE-style policy-gradient update with KL penalty used to train the Stage I policies.","marker":"Ahmadian et al. 2024"},{"why":"Provides the oracle reward and answer-checker idea used in Eq. (1) to score corrected attempts against ground truth.","marker":"Uesato et al. 2022"},{"why":"Supports the step-level over outcome-level verification premise that motivates Stage II's step-wise preference learning.","marker":"Lightman et al. 2023"},{"why":"Defines the GSM8K benchmark used for evaluation.","marker":"Cobbe et al. 2021"},{"why":"Defines the MATH benchmark used for evaluation.","marker":"Hendrycks et al. 2021"},{"why":"Provides the Llama-3.1-8B-Instruct base model on which the GSM8K experiments run.","marker":"Dubey et al. 2024"},{"why":"Provides the dart-math-mistral-7b-uniform base model used on MATH.","marker":"Tong et al. 2024"},{"why":"Provides the OpenMath2-Llama3.1-8B base model used on MATH.","marker":"Toshniwal et al. 2024b"}],"fun_headline_variants":["Self-correct stage first boosts LLM math scores by up to 5 points","Train self-correction before MCTS-DPO for higher math accuracy","Order matters: self-correct first, then MCTS-DPO lifts accuracy","Two-step training: self-correction then MCTS boosts arithmetic reasoning","Self-correct pre-training improves MCTS reasoning gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument collapses if Stage I is assumed to be a purely self-supervised 'intrinsic' correction loop while Eq. (1) still rewards success against ground-truth answers; if those rewards are doing the work, the data is not self-generated in the sense claimed, and the improved verifier in Stage II may simply be a better answer-matching model rather than a genuine self-corrector.","fun_headline_variants_meta":{"raw":{"variants":["Self-correct stage first boosts LLM math scores by up to 5 points","Train self-correction before MCTS-DPO for higher math accuracy","Order matters: self-correct first, then MCTS-DPO lifts accuracy","Two-step training: self-correction then MCTS boosts arithmetic reasoning","Self-correct pre-training improves MCTS reasoning gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000663,"raw_usage":{"total_tokens":3081,"prompt_tokens":1050,"completion_tokens":2031,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":1938}},"tokens_in":666,"tokens_out":2031,"duration_ms":14604,"temperature":1.0,"reasoning_tokens":1938,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:27:03.060769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain Stage I with the reward function changed from the oracle answer-checker $\\hat{r}(y, y^*)$ to a purely internal signal such as self-consistency across sampled rollouts, and keep Stage II fixed; if the reported GSM8K and MATH gains disappear, the mechanism named 'intrinsic self-correction' is not the source of the improvement.","supporting_citations":[],"review_version":1}