{"id":"3132f540-4dfd-40d7-af30-5f44b02cb0ae","arxiv_id":"2608.11573","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Self-Fix Step-DPO trains LLMs first on step-level preferences and then on explicit self-correction, yielding small accuracy gains over prior step-level methods.","lead":"This paper introduces a two-stage training method that teaches smaller language models to notice and fix their own math reasoning mistakes. The method improves accuracy on math benchmarks compared with earlier step-by-step reasoning training, but gains are modest and no code is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SFS-DPO vs Step-DPO comparison is confounded by extra training data and epochs in Stage 2; gains may not be due to the self-correction formulation.","rationale":"Table 2 is the central evidence for the paper's main claim. The reader's concern about transition-phrase circularity affects the behavioral metrics (self-correction rate, Error Recall) but not directly the accuracy numbers. The more load-bearing issue is that SFS-DPO receives 8.4K additional preference pairs and extra epochs beyond Step-DPO, so the headline comparison lacks a matched-data control. The Table 4 ablations vary only the initialization stage and cannot isolate the effect of the self-correction objective. A matched-data control would settle whether the gains come from the self-correction formulation or simply from more preference data on the same problems. Since this is a required experiment for the central claim, the reader's conditional verdict remains appropriate, though the specific concern differs.","tokens_in":20665,"tokens_out":6269,"duration_ms":62467,"concrete_test":"Run a matched-data control: initialize with the Stage-1 Step-DPO model, then train on the same 8,416 samples using ordinary step-preference pairs (correct next step vs. incorrect next step) without error-detection transition phrases or teacher rationales, keeping all hyperparameters and epoch counts identical. If this control matches or exceeds SFS-DPO accuracy on MATH/GSM8K, the self-correction formulation is not responsible for the gains; if SFS-DPO remains significantly better, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 gives SFS-DPO and SFS-DPO-R the 10K Step-DPO initialization set plus an additional 8,416 self-correction preference pairs and four extra epochs, while the Step-DPO baseline is trained only on the 10K set. Table 2 therefore conflates the self-correction objective with additional training data and compute. Table 4 ablates the initialization stage (No init., Joint training, RL) but keeps the second stage fixed; it never trains a control on the same 8,416 samples without the self-correction signals. Without this control, the central claim that 'explicitly modeling step-level self-correction yields stronger gains' is not established; the observed gains could come from additional preference optimization on derived pairs from the same questions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SFS-DPO, a two-stage reinforcement learning framework for step-level self-correction in LLM math reasoning. Stage 1 applies step-level preference optimization (Step-DPO-style) to strengthen step-wise reasoning; Stage 2 trains the model to prefer a self-corrected continuation that explicitly detects and fixes an erroneous step over a continuation that leaves the error unaddressed. A teacher-assisted variant, SFS-DPO-R, augments the correction with GPT-4o-generated rationales. The paper evaluates on MATH, GSM8K, GK2023, and OCW across seven 7B-14B backbones, reports gains over Step-DPO and over SFT-based self-correction baselines, and analyzes self-correction frequency, error recall, and qualitative behavior. The central claim is that explicitly modeling step-level self-correction yields stronger gains than prior step-wise preference learning.","tokens_in":20786,"tokens_out":4655,"duration_ms":46891,"significance":"If the central comparison were clean, the paper would make a useful contribution: it connects step-level preference learning with explicit self-correction training, uses substantially less data than several prior self-correction methods, and evaluates across a broad set of modern open-weight LLMs. The loss formulation is a standard DPO variant, which makes the method easy to implement, and the multi-backbone evaluation is a genuine strength. However, the headline comparison against Step-DPO is confounded by additional training data and epochs, and the behavioral metrics are partly circular because the same transition phrases used to construct training targets are used to detect self-correction in evaluation. The core accuracy results are not invalidated, but the evidence for the self-correction mechanism as the cause of the gains is not yet established.","major_comments":[{"comment":"The headline comparison against Step-DPO is confounded by training data and compute. Step-DPO is trained only on the 10K step-preference set, while SFS-DPO and SFS-DPO-R receive the same 10K set plus 8,416 self-correction preference pairs and four additional epochs in the second stage. Table 4 varies the initialization strategy but keeps the second stage fixed, so it does not provide the necessary control: a model trained on the same 8,416 derived pairs with an ordinary next-step preference objective (or with the same epochs) but without the self-correction formulation. Without this control, the observed gains in Table 2 could be due to more preference optimization on question-derived pairs rather than to explicit step-level self-correction. Please add this matched-data control or explicitly re-scope the claim as a systems comparison.","section":"Section 4.1 and Table 2"},{"comment":"The behavioral metrics are partly circular. The training data for Stage 2 are built by appending one of the transition phrases in Figure 6 to the correct step to form the chosen continuation, and the same phrase list is then used to automatically identify self-correction in generated responses. Thus higher self-correction rate and Error Recall may partly reflect imitation of the training template rather than genuine error localization. This affects the interpretation of Section 5.3 and Table 3, where lower self-correction rates are interpreted as 'selective' correction. Please either use a held-out set of detection signals or validate the behavioral metrics against human annotations of true error detection.","section":"Section 4.1 and Figure 6"},{"comment":"The out-of-domain claims are made without uncertainty quantification. Appendix A states that significance tests are omitted for GK2023 and OCW due to small sample sizes, but Section 4.3 asserts 'consistent improvements' and substantial OOD gains. With 385 and 272 test items, differences of a few percentage points correspond to small counts; please provide confidence intervals, exact McNemar p-values (even if underpowered), or a clear statement that OOD differences are within noise. This is needed to support the generalization claim.","section":"Section 4.3 and Appendix A"}],"minor_comments":[{"comment":"The claim that SFS-DPO and SFS-DPO-R 'consistently maintain or improve accuracy across all backbones and datasets' is not literally true for Qwen2-7B-SFT on GK2023, where SFS-DPO scores 46.0 versus a 46.2 baseline.","section":"Section 4.3, Table 2"},{"comment":"The text asserts a positive correlation between self-correction rate and task accuracy, but no correlation coefficient or statistical test is reported; please quantify the relationship.","section":"Section 5.3, Figure 2"},{"comment":"The SFS-DPO-R qualitative trace contains conflicting reasoning, including a statement that '20 hours' is the correct total work time and then a correction that the total should be 40 hours; the example should be edited or replaced to clearly support the claim of targeted correction.","section":"Table 5"},{"comment":"Error Recall is defined only as 'the fraction of incorrect reasoning steps that the model flags via a self-correction signal'; the manuscript should specify how incorrect steps are annotated in evaluation and whether the annotation is step-level gold data or automatically derived.","section":"Section 4.1"},{"comment":"The one-sided McNemar's tests are applied per setting without multiple-comparison correction across the 14 in-domain settings in Table 2; at minimum, report raw p-values and the number of comparisons.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope and the method is plausible, but the central empirical claim is not yet isolated from the extra-data/extra-epoch confound. A matched-data control in the style of Table 4 would probably be sufficient to resolve this. The behavioral metrics should be reframed as template-proxy measures unless validated against held-out detection signals. I would not require new large-scale experiments, but the writing should be adjusted so that claims track the actual evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nTwo things to know about arXiv:2608.11573. First, the proposed SFS-DPO / SFS-DPO-R recipe is genuinely new: a Step-DPO initialization followed by a preference objective that contrasts a self-corrected continuation against an unaddressed erroneous continuation, with a teacher-rationale variant. That specific construction is not in the cited prior work. Second, the paper's central comparison is confounded: SFS-DPO gets 8,416 extra training pairs and four extra epochs in Stage 2, while the Step-DPO baseline appears to get only the 10K initialization set. The paper never runs a control on the same 8,416 samples without the self-correction signal, so the observed gains over Step-DPO can't be attributed cleanly to the self-correction formulation.\n\nWhat the paper does well: the experiments are broad (seven backbones, in-domain plus OOD), the ablations in Table 4 suggest the two-stage design matters, and the authors are transparent about using the same transition phrases for training and evaluation. They also report McNemar's tests on the main benchmarks and explain why OOD tests are underpowered. The qualitative examples are helpful, even though a couple show SFS-DPO-R still going wrong.\n\nSoft spots, in order of severity. The data/compute confound is the big one; it needs a same-data control before the central claim is credible. The behavioral metrics (self-correction rate, Error Recall) are partly circular, since the model is trained to emit the same phrases that later count as self-correction. The effect sizes are small—often under one point—and many differences are not statistically significant. And there's no code or data release, which makes reproducibility a real question.\n\nThe paper is not a waste of time. If you work on self-correction or step-level preference optimization, the idea is worth engaging with. My recommendation: send it to peer review, but tell the authors to add a control that trains on the same 8,416 samples with a standard step-preference objective, and to either de-couple the behavioral metrics from the training phrases or present them as what they are—proxy measures. That would turn a conditional paper into a solid one.","headline":"A novel two-stage self-correction recipe whose headline comparison against Step-DPO is confounded by extra data and compute, yet the idea is worth sending to a serious referee.","tokens_in":21336,"tokens_out":3086,"would_cite":false,"duration_ms":30823,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training LLMs to detect and repair their own erroneous reasoning steps, in two reinforcement-learning stages, produces larger and more robust accuracy gains on math benchmarks than step-level preference learning alone.","keywords":["self-correction","step-level reasoning","preference optimization","direct preference optimization","mathematical reasoning","large language models","reinforcement learning","error detection"],"falsifier":"Run the trained models on a held-out set where every transition phrase from the training list is paraphrased or replaced, and ask human annotators to mark whether the flagged step truly contains an error and whether its position matches the actual first error; if self-correction rate and Error Recall fall to near-chance when the memorized phrases are unavailable, the paper's selective-correction claim is contradicted.","tokens_in":20496,"feed_emoji":"🧮","tokens_out":9677,"duration_ms":97367,"temperature":0.7,"pith_summary":"This paper claims that a language model becomes a more reliable mathematical reasoner when it is explicitly trained to detect and repair its own erroneous reasoning steps, rather than only to prefer better next steps. The proposed two-stage framework first applies step-level preference optimization to strengthen step-wise reasoning, then trains the model to prefer a continuation that flags and fixes an incorrect step over a continuation that leaves the error in place. A teacher-assisted variant adds a rationale explaining each detected error, and the reported evaluations on seven 7B-14B LLMs across four math benchmarks show consistent gains over step-preference and self-correction baselines, including out-of-domain problems. The paper's behavioral analysis argues that what matters is not how often a model corrects itself but how selectively and accurately it does so.","feed_headline":"Explicit step self-correction beats preference-only training in LLMs","feed_subtitle":"A two-stage reinforcement-learning recipe trains 7B-14B models to detect and fix wrong math steps.","key_machinery":"The load-bearing object is the step-wise self-correction preference pair. Formally, for a trajectory prefix ending at an incorrect step $s_k^-$, the chosen continuation $c_k^+$ is built as an error-detection signal $d_{k-1}$ followed by the corrected step $s_k^+$ (plus, in SFS-DPO-R, a teacher rationale $r_{k-1}$); the rejected continuation is $s_{k+1}^-$, the step the model would produce if the error were left unaddressed. A direct preference optimization (DPO) loss $L_{\\mathrm{SC}}$ pushes the policy toward the corrected continuation and away from the error-continuing continuation, while the first-stage loss $L_{\\mathrm{Pre}}$ aligns next-step preferences on correct prefixes. The mechanism converts a coarse answer-level signal into a localized repair-level signal, and the paper argues that selectivity, or knowing when not to correct, is part of what the objective teaches.","core_discovery":"On the paper's own terms, the central discovery is that step-level self-correction should be trained as an explicit preference: given an incorrect step $s_k^-$ following a correct prefix, the model is taught to prefer the continuation $c_k^+ = \\{d_{k-1}, s_k^+\\}$ that first emits an error-detection signal and then a corrected step, over the continuation $s_{k+1}^-$ that would follow if the error were ignored. This objective is applied after an initialization stage of step-level preference optimization, which the experiments show is necessary: removing it, or training correction jointly with reasoning, degrades performance on MATH and GSM8K. With the rationale-augmented variant, each correction is preceded by a teacher-generated explanation of why the previous step is wrong. The paper reports that this recipe outperforms the step-preference baseline in 11/14 settings (SFS-DPO) and 12/14 settings (SFS-DPO-R), yields consistent out-of-domain gains, and produces fewer but more effective self-corrections than SFT-based self-correction baselines.","pith_inferences":["My inference, beyond the paper: because the detection signal and the evaluation metric share the same transition phrase list, the paper's self-correction rate and Error Recall likely overstate genuine error localization; a paraphrase-invariant probe would separate phrase recall from real detection.","My inference, beyond the paper: the two-stage structure, step-preference first and repair-preference second, is a transferable recipe for any domain where errors localize to discrete steps, such as code debugging or multi-hop fact-checking; the paper only tests mathematics.","My inference, beyond the paper: since teacher rationales appear only in training data, the teacher-assisted variant is a form of rationale distillation; a natural extension is to test whether students that generate their own explanations at inference time retain the gains without teacher dependence."],"forward_implications":["If the central claim holds, step-level self-correction training is a more direct route to accuracy gains than step-level preference learning alone: the proposed methods beat the step-preference baseline in 11/14 and 12/14 in-domain settings, with average MATH gains of 1.11% and 1.36% over seven backbones.","The gains transfer out of domain: on GK2023 and OCW, the methods maintain or improve accuracy on every backbone, with Qwen2.5-Math-7B-Instruct improving by 10.9 points on GK2023 and 8.8 points on OCW.","Initialization order matters: removing the step-level RL initialization, training correction jointly with reasoning, or replacing it with standard RL initialization all degrade MATH and GSM8K performance, so the sequential two-stage design is load-bearing.","More self-correction is not better: despite lower self-correction rates and lower Error Recall than prior SFT-based self-correction baselines, the proposed models reach higher accuracy, implying selective correction rather than frequent correction is the operative mechanism.","Explanatory rationales add value: the teacher-assisted variant's error explanations consistently improve over the teacher-free variant, particularly on in-domain accuracy, suggesting that the 'why' behind an error is useful supervision."],"supporting_citations":[{"why":"Supplies the step-level preference dataset and DPO-style objective used in the initialization stage, and serves as the main step-preference baseline.","marker":"Lai et al., 2024"},{"why":"Motivates on-policy RL self-correction by demonstrating distribution shift and behavior collapse in supervised correction training, and provides the SCoRe baseline.","marker":"Kumar et al., 2024"},{"why":"Provides the transition-phrase list used both to construct the chosen self-corrected step and to detect self-correction behavior in evaluation, and serves as a baseline source.","marker":"Pan et al., 2025"},{"why":"Supplies the S2R self-verification and self-correction baseline and the Error Recall metric used in behavioral evaluation.","marker":"Ma et al., 2025"},{"why":"Provides the SuperCorrect teacher-student self-correction baseline that uses template-based supervised initialization.","marker":"Yang et al., 2025b"},{"why":"Supplies the GSM8K benchmark used for in-domain evaluation and part of the training-data source.","marker":"Cobbe et al., 2021"},{"why":"Supplies the MATH benchmark used for in-domain evaluation and part of the training-data source.","marker":"Hendrycks et al., 2021"},{"why":"Provides the strong teacher model that generates explanatory rationales for the teacher-assisted variant.","marker":"Hurst et al., 2024"}],"fun_headline_variants":["Explicit step self-correction beats preference-only training","Two-stage RL: step preference, then explicit self-fix","Step-level RL trains LLMs to verify and correct steps","Explicit correction after step-preference outdoes joint training","Reinforce step reasoning, then teach self-correction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model emitting one of the training signal phrases (such as 'The previous step is incorrect') has genuinely detected and localized an error; if the model only memorizes the phrases without finding real mistakes, the paper's self-correction metrics would overstate its correction ability.","fun_headline_variants_meta":{"raw":{"variants":["Explicit step self-correction beats preference-only training","Two-stage RL: step preference, then explicit self-fix","Step-level RL trains LLMs to verify and correct steps","Explicit correction after step-preference outdoes joint training","Reinforce step reasoning, then teach self-correction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00143,"raw_usage":{"total_tokens":5758,"prompt_tokens":925,"completion_tokens":4833,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":4752}},"tokens_in":541,"tokens_out":4833,"duration_ms":33693,"temperature":1.0,"reasoning_tokens":4752,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:34:31.322006+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained models on a held-out set where every transition phrase from the training list is paraphrased or replaced, and ask human annotators to mark whether the flagged step truly contains an error and whether its position matches the actual first error; if self-correction rate and Error Recall fall to near-chance when the memorized phrases are unavailable, the paper's selective-correction claim is contradicted.","supporting_citations":[],"review_version":1}